Part 10 3 chapters

Observability

From /proc, /sys, and cgroup stats through the USE method and eBPF tools to OpenTelemetry in all three languages, flowing into the local LGTM stack — with Cockpit and Performance Co-Pilot alongside.

Chapter 36

/proc, /sys, and the agent: a USE-method snapshot read straight from the kernel, no tools required

sysagent v0 is a USE-method metrics collector that reads /proc and /sys/fs/cgroup directly — no root, no shelled-out tools — computing CPU utilization from a two-read /proc/stat delta spanning --interval-ms, run-queue and load from /proc/loadavg, memory from /proc/meminfo, per-disk I/O from /proc/diskstats, network counters from /proc/net/dev, and cgroup PSI (falling back to system-wide /proc/pressure) gathered concurrently with the sampling sleep, exposed identically by a one-shot sample --json and a serve --port /metrics HTTP endpoint, with byte-for-byte identical field names across C++, Go, and Rust.

50 minutesRead →
Chapter 37

The USE method in 60 seconds: a checklist that runs the real tools, not a shortcut around them

loadmix is Brendan Gregg's Utilization/Saturation/Errors checklist as a runnable program — a saturation generator that pegs one resource at a time, and an analyze mode that shells out to uptime, vmstat, mpstat, iostat, sar, and pidstat exactly as a human would, parses each tool's plain-text table by column name from its own header row, runs the five interval samplers concurrently so a 60-second checklist costs 60 seconds and not five minutes, and names the resource whose Saturation signal fired hardest.

50 minutesRead →
Chapter 38

OpenTelemetry to LGTM: instrument a service, then prove the spans and metrics actually arrived

lsp-otel is a traced request/response server that exports OpenTelemetry traces and metrics over OTLP/HTTP to the LGTM stack from Chapter 3 — a parent request span wrapping work and respond children, a requests_total counter and a request_duration histogram — and the verify step proves it by querying Tempo and Prometheus through Grafana rather than trusting the program's own word, across C++, Go, and Rust each tagged with its own service.name.

50 minutesRead →