OpenTelemetry architecture
API versus SDK versus instrumentation, Resource and Context and Propagation, and the Collector's role between a process and its backend.
OpenTelemetry is not one piece of software. It’s a specification plus a layered set of components that implement it, and the layering is the part worth understanding before any instrumentation chapter asks you to add a dependency and call a method. Four terms do most of the work: API, SDK, Resource, and Collector. Get the relationship between them straight once, and every later chapter’s configuration — which dependency to add, why a span shows up with no attributes until a processor is registered, why disabling the SDK doesn’t mean removing instrumented code — stops being arbitrary and starts being a direct consequence of this layering.
API versus SDK: the interface and the implementation
The OpenTelemetry components page draws a hard line between the API and the SDK, and the line exists for a reason that matters well beyond OpenTelemetry’s own code: it’s what lets a library author add tracing to their code without forcing every consumer of that library to adopt a specific tracing backend.
The API defines the types and operations a developer writes against: start a span, record an attribute, add an event, get the current context. Critically, the API ships with a no-op default implementation. Call tracer.startSpan() against the bare API with no SDK configured, and it returns a real object that satisfies the type signature, does nothing, and costs almost nothing to call. This matters because it means a library — say, a database driver, or an HTTP client — can instrument itself against the OpenTelemetry API unconditionally, shipped to every consumer, without imposing a tracing backend, a performance cost, or even a visible dependency on anyone who isn’t actively using OpenTelemetry. The instrumentation is inert until something downstream turns it on.
The SDK is what turns it on. It’s the concrete implementation of the API’s interfaces: the part that actually builds a span, assigns it a context, runs it through configured samplers and processors, and eventually exports it. An application wires up exactly one SDK, once, usually near its entry point, and from that point forward every API call anywhere in the dependency tree — in the application’s own code, in a library that instrumented itself against the API, in an auto-instrumentation agent — routes through that same SDK and produces real telemetry instead of a no-op.
This two-layer design is why “add tracing to a Python library” and “configure tracing for a Python application” are different tasks with different audiences. A library author only ever touches the API. An application author configures the SDK once and gets the combined output of every API call made anywhere underneath it, including calls made by code they didn’t write and may not know is instrumented.
Instrumentation is the third piece, sitting on top of both. It’s the code, in a library or in an auto-instrumentation agent, that actually calls the API at the right moments: wrapping an HTTP handler to start a span per request, wrapping a database driver to record the query. OpenTelemetry ships two flavors. Code-based instrumentation is a library someone imports and calls explicitly, no different from any other dependency. Zero-code instrumentation attaches at process startup — a Java agent that rewrites bytecode as classes load, or a Python process launched with opentelemetry-instrument that monkey-patches known libraries before the application’s own code runs — and produces spans for common frameworks without a single line of application code calling the API directly. Every language chapter in this tutorial starts with zero-code instrumentation because it’s the fastest way to see real telemetry, then layers code-based, manual spans on top for the business logic a generic HTTP or database instrumentation can’t see: a payment authorization, an inventory reservation, anything that’s domain logic rather than a framework boundary.
Resource: who is talking
A span, a metric, and a log all carry a Resource: a fixed set of attributes describing the entity producing the telemetry, attached once per SDK instance rather than repeated on every individual span. service.name, service.version, deployment.environment, and the host or container identity are typical Resource attributes. The Resource concept page frames this as answering “who emitted this,” as distinct from an individual span’s attributes, which answer “what happened during this specific operation.”
The practical consequence shows up the first time two services in this tutorial both get instrumented and their spans land in the same Tempo instance. Without a service.name Resource attribute distinguishing order-quarkus from payment-python, every span in that shared backend looks like it came from the same anonymous source, and no per-service filtering, dashboard, or service graph is possible. Every SDK configuration in this tutorial sets service.name as close to first as any configuration gets, because everything downstream — correlation, dashboards, the service graph in Chapter 22 — depends on Resource attributes being populated correctly from the start.
Context and Propagation: carrying state across a boundary
Context is the mechanism that carries state — which span is currently active, what baggage is attached — through a single process as execution moves from function to function, often across asynchronous boundaries like a thread pool or an event loop callback. Every span created without an explicit parent picks up the currently active span from Context automatically, which is what makes a chain of nested function calls produce a nested chain of spans without each function having to explicitly pass a parent reference to the next.
Propagation is what happens when that state needs to cross a process boundary: an HTTP call from one service to another, a gRPC call, a message placed on a Kafka topic. The context propagation concept page describes this as injecting the current context into the outgoing request’s headers on one side, and extracting it from the incoming request’s headers on the other, using a propagator that both sides agree on. The default propagator in this tutorial is the W3C Trace Context standard, which defines two headers: traceparent, carrying the trace ID, the parent span ID, and sampling flags, and tracestate, carrying vendor-specific extensions. Baggage, introduced in Chapter 12, propagates the same way, via its own baggage header, alongside traceparent.
This is also precisely where a distributed trace can silently fall apart. If a service makes an outbound call using a client library that was never instrumented, or publishes to Kafka without attaching the trace context to the message’s headers, the receiving side has nothing to extract, starts a brand new trace with no parent, and the two halves of what was really one customer request show up in Tempo as two unrelated traces with no link between them. Chapter 13 reproduces this exact failure by publishing to Kafka with propagation turned off, so it’s visible once before the fix makes it invisible again.
The Collector: a vendor-neutral relay
An SDK could export directly to a backend, and in the smallest possible deployment it does. Every language chapter in this tutorial, though, routes through the OpenTelemetry Collector, a standalone process that sits between every instrumented service and the LGTM stack, and understanding why clarifies what the Collector is actually for.
The Collector’s job, in its own documentation’s words, is to receive, process, and export telemetry data in a vendor-agnostic way. Receiving means it exposes the same OTLP endpoints an SDK would otherwise need a backend to expose directly, decoupling “where does my application send data” from “where does that data ultimately live.” Processing means telemetry passes through a configurable pipeline of processors before export: a memory limiter that protects the Collector itself from an unexpected flood of telemetry, a batcher that groups spans together to reduce the number of network calls to the backend, a resource processor that can backfill or correct attributes centrally instead of in every single service. Exporting means the same received data can fan out to one or many backends, which is the detail that actually matters operationally: switching backends, or sending the same data to two of them during a migration, is a Collector configuration change, not a redeployment of every instrumented service.
This tutorial’s Collector runs embedded inside the grafana/otel-lgtm all-in-one container, configured with exactly the floor this tutorial’s Collector configuration file demonstrates: a memory limiter first, a batch processor, a resource processor, and three exporters, one per signal type, feeding Tempo, Mimir, and Loki respectively. Later, Chapter 19 pulls the Collector out as a dedicated subject, and Chapter 20 layers a tail-sampling processor on top of this same pipeline, making a sampling decision only after an entire trace has been collected, which is only possible because the Collector, not each individual SDK, is the point where the whole trace can be seen at once.
OTLP and the two ports every chapter uses
The wire format connecting all of this is OTLP, the OpenTelemetry Protocol, and it comes in two interchangeable transport flavors that show up by port number throughout this tutorial. The OTLP specification defines OTLP/gRPC, using Protocol Buffers over HTTP/2, with a default port of 4317, and OTLP/HTTP, using the same Protocol Buffer message definitions (or JSON) over plain HTTP/1.1, with a default port of 4318. Both carry identical semantic content; the choice between them is almost entirely about the runtime and network environment, not about what data can be expressed.
gRPC’s bidirectional streaming over HTTP/2 tends to suit long-lived services with a persistent connection to their Collector and a runtime with mature gRPC support. HTTP/1.1 tends to suit environments where a strict corporate proxy, a serverless runtime with short-lived connections, or a language with weaker gRPC tooling makes plain HTTP simpler to reason about and debug — a curl command can poke an OTLP/HTTP endpoint directly in a way that’s considerably more awkward against gRPC. This tutorial’s shared infrastructure, defined in stack/compose.yaml, exposes both ports, 4317 and 4318, on the same Collector, and each language’s exporter configuration picks whichever transport that language’s OpenTelemetry SDK defaults to or the chapter explicitly sets, without the choice affecting anything else downstream.
Following one span from creation to storage
The layers above are easier to hold together as a single sequence than as five separate definitions, so trace one span through all of them. A request arrives at the order service’s REST endpoint. Zero-code instrumentation, attached at process start, wraps the HTTP framework’s request handler and calls the OpenTelemetry API to start a span, naming it after the route and method. Because an SDK was configured at startup, that API call isn’t a no-op: it reaches the SDK, which checks the configured sampler to decide whether this trace should be recorded at all, attaches the Resource attributes set once at startup (service.name: order-quarkus, deployment.environment: local), and places the new span into Context as the currently active span for this request’s execution.
Inside that handler, the order service calls inventory over gRPC. A gRPC client instrumentation library, itself written against the bare API, asks Context for the currently active span, starts a new child span for the outbound call, and — this is the propagation step — injects that child span’s trace ID and span ID into the outgoing gRPC metadata as a traceparent value before the call leaves the process. On the inventory side, a matching piece of server-side gRPC instrumentation extracts that traceparent from the incoming metadata, and starts its own span as a child of it rather than starting a fresh, disconnected trace. Two processes, two separate SDK instances, one continuous trace, entirely because both sides agreed on the same propagator and both calls went through instrumentation that knew to use it.
Each span, as it finishes, flows through the SDK’s configured processors — typically a batching processor that holds spans briefly to send them in groups rather than one network call per span — and out through the OTLP exporter toward the Collector, over gRPC port 4317 or HTTP port 4318 depending on configuration. The Collector applies its own processor chain, the same shape regardless of which service or which language sent the span, and exports the batch onward to Tempo. By the time a span is visible in Grafana, it has crossed an API call with no inherent backend, an SDK that gave that call meaning, a Resource that identified its source, a propagator that kept it linked to its siblings across a process boundary, and a Collector that decoupled where it was produced from where it’s stored. None of those five steps is visible in the Grafana UI; all five are why the trace in front of you is coherent instead of a pile of disconnected fragments.
One more piece of this pipeline deserves a name before it reappears throughout the signal chapters: semantic conventions. When the gRPC instrumentation above names its span and attaches attributes like rpc.system and rpc.service, it isn’t inventing those names. It’s following the semantic conventions that OpenTelemetry defines for common operation types, specifically so that a dashboard built against gRPC spans from one language works unmodified against gRPC spans from another. The API and SDK define the mechanism for attaching an attribute to a span; semantic conventions define what that attribute should be called and what its value should mean, and auto-instrumentation libraries across every language in this tutorial are built to follow them without being told to.
With this layering in place, Chapter 3 turns to the destination end of that OTLP connection: the LGTM stack itself, what each of its stores actually holds, and the ports and endpoints that make the correlation between them work.