Why observability
Monitoring answers the questions you planned for; observability answers the ones you didn't, and why that distinction is worth a new vocabulary.
A production incident rarely announces itself as the failure mode you dashboarded for. The database connection pool dashboard is green, CPU is nowhere near its limit, and the error rate alert hasn’t fired, yet checkout is taking eleven seconds for a slice of customers and nobody can say why. Every piece of monitoring that was built in advance is telling the truth and none of it is useful, because the question on the table — “why is this specific slice of traffic slow” — was never anticipated as a question worth a dashboard.
That gap between what monitoring tells you and what you actually need to know in the moment is where observability starts. It isn’t a rebrand of monitoring or a fancier dashboard; it’s a different relationship to the unknown.
Monitoring answers the questions you already asked
Monitoring, in the sense engineers used the term for the two decades before “observability” entered common use, means watching a predetermined set of signals against predetermined thresholds. CPU above 90%, alert. Error rate above 5%, alert. Disk nearly full, alert. Each of these exists because someone, at some point, decided in advance that this particular number crossing this particular line meant something was wrong.
That’s not a weakness; it’s the correct tool for problems you’ve seen before. A memory leak looks like memory climbing over hours, every time, in every service that has one, and a dashboard that plots memory over time catches it reliably. The entire discipline of SRE runbooks exists because a large fraction of production problems repeat in recognizable shapes, and monitoring is built to recognize shapes you already know.
The failure mode is specific: monitoring can only answer questions it was configured in advance to answer. If the eleven-second checkout delay turns out to be caused by a specific combination of a slow third-party payment provider, a retry loop that wasn’t supposed to retry on that particular error code, and a connection pool that’s sized fine on average but starves under that retry pattern, no dashboard was built for that combination, because nobody could have predicted it. You can’t alert on a shape you’ve never seen.
Observability answers the questions you haven’t asked yet
Observability, borrowed from control theory, where it describes how well a system’s internal state can be inferred from its external outputs, became a software term for a specific capability: the ability to ask a new question about a running system without shipping new code to answer it. Not “is CPU high,” which was decided in advance, but “show me every request in the last hour where checkout took over five seconds, broken down by which payment provider handled it, and show me the logs for just those requests.” That query was never anticipated. It was constructed on the spot, in response to the specific incident in front of you, and it was answerable because the system already emits data with enough structure and enough cardinality to support an arbitrary slice.
That requirement — arbitrary slicing after the fact — is what separates an observable system from a merely monitored one. It needs correlated identifiers (a trace ID that shows up in a span, a metric exemplar, and a log line, so you can pivot between all three for the same request), and it needs attributes with enough detail that filtering by payment.provider = "stripe" or retry.count > 2 is possible without having predicted that filter would matter. The OpenTelemetry observability primer frames this as the difference between a system whose internal state can only be inferred from the external outputs it happens to emit, and one engineered to expose that state on demand.
Known-unknowns and unknown-unknowns
A useful way to sharpen the line between monitoring and observability borrows a phrase from risk analysis: known-unknowns versus unknown-unknowns. A known-unknown is a question you can name in advance even if you don’t know the answer yet — “will disk usage cross 80% this week” is a known-unknown, and a threshold alert answers it fine. An unknown-unknown is a question you can’t name until the moment it happens — “will a specific customer’s retry loop interact badly with a specific payment provider’s rate limiter in a way that only shows up for orders placed between certain currency and certain shipping zone” is not a question anyone writes an alert for, because nobody thinks of it until it’s already happening to a real customer.
Monitoring is built entirely around known-unknowns: name the question, set the threshold, wait for it to fire. Observability is the capability to investigate an unknown-unknown after the fact, using data that was captured broadly enough and with enough structure that an arbitrary new question can be asked of it without having been anticipated. This is why observability tooling emphasizes high-cardinality, high-dimensionality data — many distinct attribute values, many attributes per event — where traditional monitoring tooling optimizes for low-cardinality aggregates that are cheap to store and threshold against. A metric like “requests per second, labeled by status code” is cheap and useful for known-unknowns. A trace carrying the customer ID, the payment provider, the shipping zone, and forty other attributes on every span is expensive by comparison, and it’s exactly that expense that buys the ability to filter by a combination nobody predicted.
What this looks like in the demo system
The six services this tutorial builds make the distinction concrete rather than hypothetical. Suppose the order service’s p99 latency climbs after a deploy. A monitoring dashboard showing order service latency over time will show the climb, and it will show roughly when the deploy happened, which is useful and exactly what monitoring is for. It will not show whether the climb is isolated to orders that call the payment service’s card-charging path versus orders that use a stored payment method, whether it correlates with inventory lookups for a specific warehouse, or whether it only affects orders that also triggered the Kafka-based shipping notification. Those are unknown-unknowns until the incident makes them matter, and answering any of them requires being able to slice traces and logs by attributes nobody wrote a dashboard panel for in advance.
This is also why the demo system’s protocol diversity matters for the argument, not just for coverage. A REST call from a browser to the order service, a gRPC call from order to inventory, and an asynchronous Kafka event from order to shipping each fail in different ways and leave different evidence behind. An observable system correlates all three kinds of evidence back to the single customer-facing request that triggered them, using the same trace ID whether the hop was synchronous or not. A merely monitored system has a separate dashboard for HTTP latency, a separate one for gRPC error rates, and no dashboard at all for “did the Kafka consumer that was supposed to react to this order actually run,” because nobody thought to build one until a customer complained that their shipping confirmation email never arrived.
The five signals, briefly
OpenTelemetry’s answer to “what has to be true for a system to be observable” is five kinds of telemetry, each covering a different angle on the same running request, introduced here and given a full chapter each later in the tutorial.
A trace is the shape of one request as it moves through a distributed system, a tree of timed spans connected by parent-child relationships. It answers “where did this specific request’s time go.” A metric is an aggregate measurement over many requests: a counter, a gauge, a histogram. It answers “how is the system behaving overall, right now, compared to five minutes ago.” A log is a discrete event with a message and structured fields, correlated to the trace that was active when it was written, answering “what did the code itself say was happening.” Baggage carries key-value context across service boundaries alongside the trace, so a customer tier or feature flag set at the edge of the system is still readable three services downstream without re-deriving it. A profile samples where CPU time or memory allocation is actually going inside a process, down to the function, answering a question none of the other four signals can: not just that a span took 400ms, but which line of code consumed those 400ms.
No single signal is sufficient alone. A trace tells you which span was slow; it doesn’t tell you if that slowness is new or has been the baseline for a week, which is a metrics question. A metric tells you error rate spiked at 14:03; it doesn’t tell you which specific requests failed or why, which is a trace-and-log question. The five signals solve adjacent problems, and an observable system is one where all five are available and, critically, correlated to each other through shared identifiers rather than living in five separate tools with five separate logins and no way to jump between them.
The cost of opacity
The cost of not having this is easiest to see as time, not money, though it becomes money quickly enough. An incident where the five signals are correlated looks like: see the latency spike in a metric, click through to a slow trace from that time window, see which span is the bottleneck, click through to the logs for that span, read the exception message, fix it. Fifteen minutes, start to finish, much of it waiting for a deploy rather than waiting to understand the problem.
An incident without that correlation looks like: notice latency is up from a dashboard that happens to be open. Guess which service is likely responsible based on recent deploys. SSH into that service’s hosts, or open its separate logging tool, and grep for errors around the right timestamp, adjusting for whatever clock skew exists between the metrics system and the log timestamps. Find a promising error, but have no way to confirm it’s actually related to the slow requests rather than an unrelated error that happens to share a timestamp. Repeat for the next service that seems plausible. The same incident now takes two hours, most of which is not spent thinking about the problem but spent manually doing the correlation work that instrumented correlation would have done automatically.
Multiply that gap by the number of incidents a team handles in a year, and by the number of engineers paged for each one who could have self-served an answer instead of escalating, and the cost of opacity stops being an abstraction. It shows up as mean time to resolution, as on-call burnout, and as the quieter cost of engineers who stop trusting their own tools and start reaching for print statements and a redeploy instead, because that at least feels like it’s under their control. An observable system doesn’t eliminate incidents. It changes what the two hours is spent on.
Why OpenTelemetry
Observability as a property doesn’t require any particular tool. You could build it with a proprietary agent, a vendor-specific API, and a vendor-specific backend, and plenty of teams did exactly that for most of the last decade, each major monitoring vendor shipping its own SDK with its own instrumentation story. The cost of that approach shows up later: switching vendors means re-instrumenting every service, and supporting multiple backends for different parts of an organization means maintaining multiple competing instrumentation libraries side by side.
OpenTelemetry exists to remove that cost by standardizing the parts that don’t need to be proprietary: the data model for a span, a metric, and a log; the API a developer codes against; the wire protocol, OTLP, that carries the data; and a growing set of instrumentation libraries that emit it automatically for common frameworks. What stays open for competition is the backend — where the data ends up and how it’s queried and visualized — which is exactly the part where real product differentiation exists. Instrument once against the OpenTelemetry API, and the exporter configuration, not the instrumentation, decides whether that data lands in a self-hosted stack, a SaaS vendor, or both at once during a migration.
This tutorial sends that data to the Grafana LGTM stack: Loki for logs, Grafana for the UI, Tempo for traces, and Mimir for metrics, with Pyroscope added for profiles. That choice is pragmatic, not doctrinal. The stack is open source, runs as a single container for local development, and already wires the correlation links between its four stores that the “click a trace, land on its logs” workflow depends on. Nothing about the instrumentation chapters that follow is specific to Grafana; point the same OTLP exporter at a different backend and the spans, metrics, and logs look identical on the wire. Chapter 2 opens up that wire format and how the pieces that produce it (the API, the SDK, and the Collector) fit together.