The LGTM stack
Loki, Grafana, Tempo, and Mimir, plus Pyroscope and the embedded OTel Collector: what each store holds, the all-in-one image, and the ports every later chapter uses.
The backend this tutorial runs against is four storage engines and one UI, shipped as a single container image, with a fifth storage engine for profiles running alongside it. None of the five is interchangeable with another; each one is built around a different shape of data, and knowing which store holds which signal makes every later chapter’s configuration — ports, datasource wiring, what a Grafana panel is actually querying — legible instead of memorized.
Four stores, four shapes of data
Tempo stores traces: the write pattern is a steady stream of spans, grouped into traces by trace ID, and the read pattern is almost always “look up this exact trace ID” or “find traces matching these attribute filters within this time window.” Tempo is not built as a general-purpose database with secondary indexes on every field; trace volume is too high for that to be affordable, so it leans on the trace ID as a highly selective key and treats broader search as a comparatively more expensive operation layered on top.
Mimir stores metrics, speaking the Prometheus remote-write protocol, which means anything that already knows how to push to Prometheus can push to Mimir with no code changes. The write pattern is small numeric samples arriving continuously — a counter incrementing, a gauge updating, a histogram bucket accumulating — identified by a metric name plus a set of labels. The read pattern is aggregation over time ranges: rate of requests over the last five minutes, p99 latency over the last hour, broken down by service label. Mimir’s entire design optimizes for that aggregation pattern at a scale no single Prometheus server could hold.
Loki stores logs, and its defining design choice is indexing only the labels attached to a log stream — service.name, deployment.environment — rather than the full text of every log line, the way a traditional log search product like Elasticsearch does. That keeps Loki’s index small and cheap, at the cost of making full-text search within a stream more like grep across a known set of files than an inverted-index lookup. For this tutorial’s purposes, the label that matters most is trace_id, attached to every log line emitted while a given trace’s span was active, because that label is what makes “jump from a trace to its logs” possible at all.
Pyroscope, the fourth store, holds continuous profiles: compact, timestamped samples of where a process was spending CPU time or allocating memory, aggregated into the flame graphs profiling tools have used for decades. It’s a separate container in this tutorial’s compose file rather than bundled into the all-in-one image, reflecting that continuous profiling is architecturally distinct from the other three signals — and that OpenTelemetry’s own profiling signal, as the profiles concept page notes, is still stabilizing in the specification. Pyroscope already speaks an OTLP-compatible profiling format, which is why it fits into the same correlation story as the other three stores despite running outside their shared container.
Grafana itself stores nothing telemetry-related; it’s the query and visualization layer in front of all four stores, and its job in this stack is less “draw a chart” and more “let one engineer, looking at one trace, click through to that trace’s logs, then to the metrics for the service that emitted it, then to a CPU flame graph for the exact time window that span covered,” without manually copying a trace ID between four separate tools.
The all-in-one image
Running four separate backend processes for a tutorial would mean four separate sets of configuration, four separate health checks, and four separate things that can fail to start cleanly before a reader has written a line of instrumented code. The grafana/otel-lgtm image collapses that into one: Grafana, Tempo, Mimir, and Loki, plus an embedded OpenTelemetry Collector, all running as processes inside a single container, pinned in this tutorial’s stack/compose.yaml at tag 0.35.0.
Because the Collector lives inside that same container, its exporters in stack/otelcol/config.yaml point at localhost rather than at separate compose service names — http://localhost:3200 for Tempo, http://localhost:9090/api/v1/write for the Mimir-compatible remote-write endpoint, http://localhost:3100/otlp for Loki. That configuration file stays minimal: a memory_limiter first, because an unbounded flood of telemetry from a runaway service should degrade the Collector gracefully rather than let it get OOM-killed and take observability itself offline at the exact moment it’s needed most; a batch processor to amortize export calls; a resource processor that can backfill attributes centrally; and three pipelines, one per signal, each named for its destination (otlphttp/tempo, prometheusremotewrite, otlphttp/loki). Every later chapter that touches Collector configuration, including the tail-sampling variant in Chapter 20, builds on top of exactly this floor rather than replacing it.
Two environment variables in the compose file matter enough to call out directly: GF_AUTH_ANONYMOUS_ENABLED and GF_AUTH_DISABLE_LOGIN_FORM, both set to let Grafana open straight to its dashboards with an anonymous admin session rather than demanding a login on a local, throwaway development stack. That’s a deliberate convenience for a tutorial environment and not a pattern to carry into anything internet-facing.
Wiring the correlation links
The all-in-one image ships its own default datasource provisioning, and this tutorial overrides it with stack/grafana/datasources.yaml, mounted into the container’s provisioning directory with a filename (zz-tutorial-datasources.yaml) chosen so it sorts, and therefore loads, after the image’s built-in defaults — Grafana provisions every file it finds in that directory and the last one processed wins on a UID collision, so without that naming trick the image’s own datasource definitions would silently overwrite the ones this tutorial actually needs.
That file is what makes the cross-signal correlation demos in later chapters work at all, and it’s worth reading once end to end because every link in it is a concrete answer to “how does clicking here get me there.” The Tempo datasource’s tracesToLogsV2 field tells Grafana that from any trace, a Loki query filtered by service.name and the trace’s own ID, within a window five minutes before and after the span, will surface that trace’s logs. Its tracesToMetrics field does the same for jumping from a trace to a request-rate or error-rate query scoped to the same service and time window. Its tracesToProfiles field points at the Pyroscope datasource, naming process_cpu:cpu:nanoseconds:cpu:nanoseconds as the profile type to fetch for the span’s service and time range — this is the field that makes “click a slow span, see its flame graph” possible. In the other direction, the Loki datasource’s derivedFields entry recognizes trace_id as structured metadata on a log line (not a pattern matched out of the message text, since OTLP delivers it as a structured field) and turns it into a clickable link back into Tempo. The Prometheus datasource’s exemplarTraceIdDestinations closes the loop a third way, linking a latency histogram’s exemplar points — individual sampled data points attached to an aggregate metric — directly to the trace that produced that one sample.
None of this is automatic the way the signal data itself is automatic once OTLP is flowing. A Collector can receive and store traces, metrics, and logs with no datasource correlation configured at all, and Grafana will still display each of them individually. The correlation links exist because this specific file defines them, which is also why a production Grafana deployment that skips this provisioning step gets four working datasources and none of the “click a trace, land on its logs” workflow this tutorial relies on from the first signal chapter onward.
One investigation, four stores
The correlation wiring above is easiest to appreciate as a single walk through Grafana rather than as a list of configuration fields, so follow one hypothetical investigation end to end. A dashboard panel backed by Mimir shows order service p99 latency climbing past its usual baseline over the last ten minutes; that panel is a Prometheus-style query against aggregated histogram data, and clicking one of its exemplar points, enabled by the Prometheus datasource’s exemplarTraceIdDestinations field, drops straight into the single trace that produced that one latency sample, now rendered from Tempo.
Inside that trace, one span clearly dominates the total duration: a call out to the payment service. From that span, the tracesToLogsV2 wiring runs a Loki query scoped to the payment service and a narrow window around the span’s own start and end time, surfacing the exact log lines the payment service wrote while handling this specific request, not a generic tail of its logs around that time. One of those lines is a warning about a slow upstream card network response, which narrows the question from “why is checkout slow” to “why is this particular downstream dependency slow for this particular request.” If the payment service happens to be CPU-bound rather than waiting on the network, the same span’s tracesToProfiles link opens a Pyroscope flame graph for that service over that exact time range, and the function consuming the most CPU during the slow window is visible directly, no separate profiling session or redeploy with a profiler attached required.
Four queries, across four different storage engines, each one with its own data model and its own strengths, chained together by nothing more exotic than a shared trace ID and a handful of datasource fields that know how to turn that ID into the next query. None of the four stores needed to know anything about the other three to make this work; the correlation lives entirely in Grafana’s datasource configuration, which is exactly why stack/grafana/datasources.yaml is worth reading in full rather than treating as boilerplate.
Why these four stores and not one
A reasonable question, especially for a reader used to a single general-purpose database, is why traces, metrics, and logs need three separate specialized stores rather than one database with three tables. The answer is that each signal’s access pattern is different enough that a store optimized for one of them is a poor fit for the other two. A store built for Loki’s append-heavy, label-indexed log streams would struggle with Mimir’s requirement to aggregate millions of time-series samples into a single number over an arbitrary window in well under a second. A store built for that kind of metric aggregation would be a poor fit for Tempo’s requirement to retrieve one specific trace, by ID, out of a volume of spans that can run into the billions per day at real production scale, with full-attribute search as a secondary, more expensive path. Specialization costs operational complexity — four processes instead of one — and buys performance and cost efficiency that a single undifferentiated store can’t match at the volumes observability systems actually produce. The all-in-one image exists precisely to hide that operational complexity for a local tutorial environment, without pretending the four stores underneath it are actually one thing.
Kafka, Postgres, and the one thing with no GUI
Two more services round out the shared infrastructure in stack/compose.yaml, neither one a signal store but both relied on by later chapters: a single Postgres instance holding the schema every language’s services share, and a single-broker Kafka cluster running in KRaft mode, meaning no separate ZooKeeper process, used for the asynchronous order.placed event that shipping and notification consume.
Kafka gets no GUI in this stack. There is no bundled Kafka UI container in this compose file; the tool for looking inside a topic, listing consumer groups, or producing a test message by hand is kcat, a command-line client started on demand under the tools compose profile (docker compose -f stack/compose.yaml --profile tools up -d kcat) and then driven with docker compose exec kcat kcat ... against the broker’s internal address. That’s a choice for this tutorial rather than an oversight: Grafana, wired up with the correlation links above, is the one visual surface this stack asks you to learn, and every other piece of infrastructure — Kafka included — is inspected from the terminal. Readers expecting a point-and-click Kafka browser alongside Grafana won’t find one here; the CLI habit is the one worth building, since production Kafka clusters rarely ship a GUI either.
The ports this tutorial keeps coming back to
A short list is worth having in one place, since every later chapter assumes it rather than restating it: 3000 is Grafana’s UI, 4317 and 4318 are the Collector’s OTLP gRPC and HTTP ingestion ports respectively, 3200 is Tempo, 3100 is Loki, 9090 is the Mimir-compatible Prometheus remote-write and query endpoint, and 4040 is Pyroscope, running as its own container rather than inside the all-in-one image. Kafka listens on 9092 for host-side clients and 9094 for clients running inside the compose network, a split that matters the first time a chapter has you run kcat from inside a container rather than from the host. With this map in place, and the correlation wiring explained, the setup chapter that follows is where these pieces actually get started on a real machine.