Chapter 29 — Instrumented Thread Pool and the LGTM Stack
Thread-safety and correct propagation are different guarantees — an instrumented thread pool proves both against a real Grafana LGTM observability backend.
The full source for this example lives in
examples/concurrency/29-observing-concurrency/— clone the repo,cdin, and./demo.sh.
Chapter reference: Doc 29 — Observing Concurrency
What it shows
A ch15-style thread pool is extended with a trace per request: each task captures the submitting thread’s active OpenTelemetry context — which still includes the request’s root span — and hands it across the pool’s thread hop as an explicit parent, instead of relying on the SDK’s own thread-safety (which says nothing about propagation). A queue-depth up/down counter makes saturation observable directly rather than inferred from latency. This is the project’s first multi-container demo: a docker compose stack brings up the instrumented service alongside grafana/otel-lgtm, and the verifier queries Tempo and Mimir’s own HTTP APIs directly — asserting a discovered trace has exactly one root span and the right number of correctly parented child spans, and that a queue-depth metric actually reached the backend — rather than trusting the service’s own exit code.
Running it
cd examples/concurrency/29-observing-concurrency
./demo.sh # build + bring up the stack + run the workload + verify
./demo.sh --clean # tear down and remove the images this demo built