Observability with the LGTM stack

Telemetry is configuration. The services already export traces, metrics and logs when OTEL_SDK_DISABLED is false; what remains is giving every pod the same endpoint and a service name, and giving the humans a dashboard. This chapter finishes the umbrella chart with both: environment variables generated by the library chart, and a Grafana dashboard that travels in the Helm release. Then it follows a single dispatch through Kafka in Tempo.

The code is in examples/26-observability/. Run ./demo.sh offline there; the full run needs the LGTM stack from scripts/platform/setup-lgtm.sh. The example is the complete umbrella, namespace hfd-26, release platform.

Figure 26.1 — From the chart's environment variables to Grafana

Download .excalidraw source Edit on excalidraw.com

Telemetry from the library

Each service chart includes the same helper from pc-lib, so the umbrella controls telemetry for every service from one value.

{{- define "pc-lib.otelEnv" -}}
{{- $global := default (dict) .Values.global -}}
{{- $endpoint := default (default "" $global.otlpEndpoint) .Values.otel.endpoint -}}
{{- $svc := default (include "pc-lib.fullname" .) .Values.otel.serviceName -}}
- name: OTEL_SDK_DISABLED
  value: {{ if $endpoint }}"false"{{ else }}"true"{{ end }}

The helper resolves an endpoint from the subchart’s own otel.endpoint, falling back to global.otlpEndpoint. default (default "" ...) keeps a missing global from failing the render. The SDK is disabled whenever the endpoint is empty, so the default install runs with no collector. The remaining lines emit OTEL_EXPORTER_OTLP_ENDPOINT only when set, OTEL_SERVICE_NAME from the fullname (platform-shipping, platform-notification), and OTEL_RESOURCE_ATTRIBUTES built from the data-product domain, global.environment and .Chart.AppVersion. The value comes from values-dev.yaml:

global:
  environment: dev
  otlpEndpoint: http://otel-collector.observability.svc.cluster.local:4318

The endpoint is the base URL without a path: the Python exporters in services/common/pcobs/otel.py append /v1/traces, /v1/metrics and /v1/logs themselves. The service names follow the release name, so two releases in one cluster appear as separate services in Tempo. The OpenTelemetry SDK environment variables define the variable names.

Cross-Kafka tracing

Tracing across Kafka needs the trace context to travel with the message, because Kafka records have headers but no ambient context. Two small functions in services/common/pcobs/propagation.py do it. inject_headers() serializes the active span context into a W3C traceparent header, and the shared producer in pcobs/kafka.py passes it as the headers= argument of send_and_wait. On the other side extract_context(message.headers) rebuilds the context, and the notification consumer starts its span process shipment.dispatched inside it. Without the extract step the consumer would start a disconnected trace, and Tempo would show two unrelated traces for one dispatch.

Nothing in the chart switches this on beyond the endpoint. The chart’s job is the environment: the same OTEL_EXPORTER_OTLP_ENDPOINT in both services, so both reach the same Collector, and service names that distinguish them. A Collector in the middle is the design choice from the platform setup: the services speak only OTLP, and the Collector decides which backend gets traces (Tempo), logs (Loki) and metrics (Mimir). Changing a backend means editing the Collector, not any chart.

The dashboard in the chart

A dashboard that lives only in Grafana’s database disappears with the cluster. A dashboard shipped as a ConfigMap is versioned with the chart that produces the telemetry. Grafana’s chart runs a sidecar that watches for ConfigMaps with a label and loads their JSON. scripts/platform/setup-lgtm.sh configures it with label grafana_dashboard and value 1, and a grafana_folder annotation that names the Grafana folder.

The umbrella gets three files in this example. templates/dashboard.yaml:

metadata:
  name: {{ .Release.Name }}-dashboard
  namespace: {{ default .Release.Namespace .Values.dashboards.namespace }}
  labels:
    {{ .Values.dashboards.label }}: {{ .Values.dashboards.labelValue | quote }}
  annotations:
    grafana_folder: {{ .Values.dashboards.folder | quote }}
data:
  shipping-dashboard.json: |-
    {{- tpl (.Files.Get "files/shipping-dashboard.json") . | nindent 4 }}

The label key and value come from values so a different Grafana configuration needs no template edit. .Files.Get reads the JSON from the chart’s files/ directory, and tpl renders it as a template so queries can use {{ .Release.Name }}: the metric filter becomes job=~".*/platform-.*" for release platform. nindent 4 indents the JSON under the block scalar. The dashboard has three panels: request rate from Mimir, a Tempo table whose query is the TraceQL filter shown below, and a Loki log stream. Legend text such as {{ "{{" }}job{{ "}}" }} is escaped so Helm does not consume Grafana’s own braces.

Values dashboards.enabled, label, labelValue, folder and namespace have defaults in values.yaml and entries in values.schema.json. tests/dashboard_test.yaml asserts the label, the folder annotation, the release name in the queries, the namespace override and that enabled: false renders nothing.

The namespace trap. setup-lgtm.sh installs Grafana without sidecar.dashboards.searchNamespace, and this chapter upgrades the shared Grafana with searchNamespace=ALL. The Grafana chart’s dashboard sidecar watches only Grafana’s own namespace unless sidecar.dashboards.searchNamespace is set. The ConfigMap lands in hfd-26, Grafana runs in observability, and nothing appears: the dashboard search returned [] while the ConfigMap existed, and the sidecar container had no NAMESPACE variable. After the upgrade it carried NAMESPACE=ALL and the dashboard showed up in the Shipping folder. There are two fixes. Set sidecar.dashboards.searchNamespace=ALL on Grafana, which demo.sh does with helm upgrade grafana grafana/grafana --version 8.5.0 -n observability --reuse-values --set sidecar.dashboards.searchNamespace=ALL. Or set dashboards.namespace: observability in the umbrella values, which writes the ConfigMap into Grafana’s namespace from the hfd-26 release. The second keeps Grafana untouched but couples the chart to a namespace name.

The dashboard template, its JSON file, the values block, the schema entry and the unit test are the only differences between this umbrella and the one from chapter 16.

Querying the other two signals

The same service names work in Loki and Mimir. In Grafana’s Explore view, pick the Loki datasource and run {service_name="platform-notification"} to see the consumer’s log lines, which the OTLP log handler sends through the Collector. Pick Mimir (a Prometheus-compatible datasource) and run sum by (job) (rate(http_server_duration_milliseconds_count[5m])) for request rate. That metric name comes from the FastAPI instrumentation’s HTTP server duration histogram as the Collector forwards it, and it exists in this lab’s Mimir. The series carry no service_name label: the Collector’s Prometheus-style export folds service.namespace and service.name into job, so the series read job="shipping/platform-shipping" and job="notification/platform-notification". The dashboard therefore filters on job. If your instrumentation version emits the newer semantic-convention name, the panel is empty until you change the query in files/shipping-dashboard.json. Loki is different: its stream labels include service_name, which is why the log query above uses it. Because the dashboard is a chart file, that fix is a chart change, reviewed and released like any other.

Build, run, observe

[host]$ cd examples/26-observability && ./demo.sh offline

Offline mode lints, runs the 20 unit tests, renders values-dev.yaml through kubeconform, shows the OTEL_* variables per service, checks that the default render keeps OTEL_SDK_DISABLED true, and parses the dashboard JSON out of the ConfigMap. The full run, ./demo.sh, installs the umbrella, creates and dispatches a shipment, and queries Tempo through Grafana’s datasource proxy:

{ resource.service.name = "platform-shipping" && span.http.target =~ ".*dispatch.*" }

The attribute is span.http.target, and the dispatch filter drops the /healthz spans the probes generate. demo.sh dispatches with the bearer token (an unauthenticated dispatch returns 401 and leaves a shipping-only trace) and asserts that the trace holds both services. The reference run returned trace 3ff8ba3e5495645659fcce9edffcd6f5, root POST /api/shipments/{shipment_id}/dispatch, and the trace listed these spans (abridged; repeated SELECT spans removed):

platform-notification | process shipment.dispatched
platform-shipping | POST /api/shipments/{shipment_id}/dispatch http send
platform-shipping | POST /api/shipments/{shipment_id}/dispatch
platform-shipping | SELECT
platform-shipping | UPDATE

One trace holds spans from both services, which proves the context crossed Kafka. Open Grafana at http://127.0.0.1:30300 (user admin, password admin) (published NodePort 30300) and look for “Shipping platform (platform)” in the Shipping folder.

Cross-check

Compare the pod’s environment with the chart’s output:

[host]$ kubectl -n hfd-26 exec deploy/platform-shipping -- env
[host]$ helm get manifest platform -n hfd-26

The OTEL_* values in the live pod match the rendered Deployment, and the dashboard ConfigMap in the manifest is the one the sidecar loaded.

What you learned

One library helper turns telemetry on for every service from global.otlpEndpoint, and leaves it off when the value is empty.
A dashboard ships as a ConfigMap with the sidecar label; the sidecar must be allowed to see its namespace.
TraceQL on span.http.target finds the dispatch trace, which spans shipping and notification.

This ends the delivery chapters. The appendices cover OpenShift Local, the Helm 3 to Helm 4 migration, and a command cheat sheet.