What in this tutorial is verified on real Fedora 44 hardware, what is in flight, and what still needs validation.
This document tracks the gap between what the tutorial claims and
what has been verified end-to-end on real hardware. It is the single
authoritative list of things to check before any part is declared
done.
How to use this document
New content making a verifiable claim → add a row as unverified.
You actually test a claim on Fedora 44 → promote to
verified (Fedora 44), noting which platform.
Actively debugging → in flight with a note on what’s blocking.
Deliberately not testing this iteration → out of scope with the
reason.
Default state for new claims is unverified. Promotion to
verified requires a real test run by a human, recorded here. An AI
assistant must not self-promote.
Conventions
verified (Fedora 44) — run end-to-end on Fedora 44 with the exact
commands shown. This is the canonical primary-platform marker.
in flight — actively being worked on.
unverified — taken from sources/docs, not yet run here.
out of scope — deliberately not verified this iteration.
Status (superseded — current truth is the snapshot below). The
real-hardware pass is done: the corpus was built and run end-to-end on
the KVM lab (Fedora 44, kernel 7.1.3), and every chapter with a runnable
demo is now verified. See the “Status snapshot — VM smoke campaign”
section immediately below for the authoritative current state (including the
genuinely-broken examples that were fixed and the one hardware skip). The
detailed per-row entries and changelog further down are the historical
authoring ledger — kept as a record of how each claim was tracked from
unverified to verified, not a live to-do list.
Original r1.0 note (historical): “This iteration delivers the lab that
verification will run against, so nothing is verified yet — there was no
Fedora 44 target available while authoring.” That is no longer the case.
Status snapshot — VM smoke campaign (2026-07-11, through r65)
The full corpus was built and run end-to-end on the KVM lab
(ebpf-target / ebpf-peer, Fedora 44, kernel 7.1.3, BTF present).
All chapters with a runnable demo are now verified; the deck build
(101/201) and the diagram accuracy pass are also done. The open items are
now closed (r64–r65):
Ex
Final state
Resolution
41-sudoadd
verified (r64)
Three real bugs, not offsets. (1) Verifier rejected the program: bpf_probe_write_user size is ARG_CONST_SIZE, so the map-derived length’s 0 lower bound failed as “R3 invalid zero-sized read” → clamp to 1..=64. (2) comm-only targeting smashed the loader’s shared-library ELF reads and bricked sudo → retarget to a /etc/sudoers header signature. (3) sudo lseeks back to 0 and re-reads for the actual parse → content-match every offset-0 read, not just the first. Verified: victim → uid=0 while attached, denied on detach, disk untouched.
53-bpf-token
verified (privileged half) (r65)
bpffs accepts + reads back delegate_*; a bogus axis is rejected (kernel enforces, not ignores); BPF_TOKEN_CREATE in the kernel ABI. Aya loader token API confirmed still emerging against aya 0.14.0 (EbpfLoader has no token_path) — illustrative/loader_with_token.rs is intended-shape, not shipped.
60-offload
out of scope — hardware skip
NIC hardware offload; no offload-capable NIC in the lab. Documented skip; no action.
Bonus fix (r64): 39-pidhide was mislabeled verified but only ever
confirmed attach, not the hide. On a live /proc (200+ entries returned in
one getdents64) the for _ in 0..64 record-walk scanned only the first
slice and hid nothing → raised to 0..512. Now behaviorally verified (PID
vanishes from ls /proc, reappears on detach).
Corpus-wide correction (r64):bpf_probe_write_user does not taint the
kernel or emit a dmesg/journal warning on kernel 7.1.3 (verified tainted=0,
no boot-journal notice while actively writing user memory). Chapters 39 and 41
previously led their detection sections with that taint signal; both now rely
on loaded-program enumeration instead and flag the taint as absent.
Everything else that has a runnable demo is verified (Fedora 44, kernel
7.1.3), including 20-javagc (uprobe the G1 collector from libjvm
.symtab), 47-pg-probe (eu-unstrip symbol merge), 49/55/57 (C
references, guest-side), and 51-userspace-ebpf (host rbpf run,
interpreter=42 jit=42 — no VM needed).
To resume after reboot:sg libvirt -c "bash scripts/lab/lab-up.sh"
(both guests snapshot to lab-ready; revert-vm.sh is the undo), bring up
the stack with examples/03-observability-stack/demo.sh, then tackle 41
then 53. lab-down.sh when done.
A. Pinned tool versions
The versions this tutorial is written against. Every later claim
implicitly assumes these.
Status
Tool
Version
Where pinned
Notes
unverified
Fedora (host + guest)
44
Ch 1–2
Released ~Apr 2026; BTF-enabled stock kernel
unverified
Rust toolchain
1.96.0
Ch 4, rust-toolchain.toml
Beta at authoring (late May 2026); goes stable ~2026-06-05. Stable today is 1.95.0 — use it if 1.96.0 stable isn’t out yet
unverified
Rust nightly
latest + rust-src
Ch 4
For the BPF target via build-std
unverified
aya (user space)
0.13.x
Ch 4, 6 hello/Cargo.toml
crates.io current line at authoring
unverified
aya-ebpf (kernel)
0.1.x
Ch 4, 6 hello-ebpf/Cargo.toml
crates.io current line at authoring
unverified
aya-log / aya-log-ebpf
0.13.x / 0.1.x
Ch 6
kernel log forwarding
unverified
aya-build
0.1.x
Ch 6 hello/build.rs
build-time BPF compile; template approach may differ
unverified
bpf-linker
latest
Ch 4
cargo install; LLVM fallback from Fedora llvm/llvm-devel
unverified
cargo-generate
latest
Ch 4
scaffolds from aya-template
unverified
Podman
5.x
Ch 1, 3
rootless
unverified
podman-compose
latest
Ch 1, 3
Fedora package
unverified
grafana/otel-lgtm
0.28.0
Ch 3 compose.yaml
bundles Grafana+Tempo+Mimir+Loki+Prometheus+Pyroscope+OTel Collector (+OBI); current tag at authoring
unverified
Python
3.14
Ch 3 client/Containerfile
UBI ubi9/python-314; clients + FastAPI target
unverified
FastAPI
current
Ch 16 (Python target)
Python 3.14 app target, containerized
unverified
Java
25 (LTS)
Ch 16 (Java target)
Quarkus runtime
unverified
Quarkus
3.33 (LTS)
Ch 16 (Java target)
containerized, UBI + multi-stage
unverified
crun
1.27.1
Ch 16, container-observation chapters
Fedora default OCI runtime; eBPF + SELinux
unverified
opentelemetry (Rust)
0.27.x
Ch 6 hello/Cargo.toml
exporter API moves between minors
unverified
opentelemetry (Python SDK)
1.30.0
Ch 3 client/requirements.txt
OTLP/HTTP exporter
unverified
libvirt / qemu-kvm / virt-install
Fedora 44 packages
Ch 1–2
@virtualization group
unverified
bpftool / bpftrace / bcc-tools / perf
Fedora 44 packages
Ch 2, 6
Fedora repos only (tooling policy)
B. Foundations — per-chapter claims (Ch 0–6, all r1.0)
Status
Claim
Chapter
unverified
virt-host-validate qemu passes hardware virtualization on a VT-x/AMD-V laptop
Ch 1
unverified
Rootless podman run ubi9/ubi-minimal echo OK works without subscription
Ch 1
unverified
provision-vm.sh ebpf-target boots a Fedora 44 guest via cloud-init + virt-install
Ch 2
unverified
The pinned Fedora Cloud Base image filename resolves at the mirror
Ch 2
unverified
The guest exposes /sys/kernel/btf/vmlinux (CO-RE works)
Ch 2
unverified
cloud-init installs bpftool/bpftrace/bcc-tools/perf from Fedora repos in the guest
Ch 2
unverified
deploy-to-target.sh copies a binary to the guest and runs it under sudo
Ch 2
unverified
Second guest ebpf-peer is reachable from ebpf-target on the default network
Ch 2
unverified
grafana/otel-lgtm comes up healthy under rootless Podman with the compose file
Ch 3
unverified
The Python 3.14 client exports OTLP and ebpf_events_total appears in Grafana
Ch 3
unverified
host.containers.internal:4318 reaches the host stack from a rootless container
Verified:(historical ledger entry — superseded by the status snapshot
at the top; these examples are now verified on Fedora 44 / kernel 7.1.3.)
At authoring no Fedora 44 target was available; all code was written to
current conventions but unrun.
Known risks to check first on real hardware: (1) the Aya
build.rs/aya-build wiring in Chapter 6 vs. whatever the current
aya-template generates; (2) the exact Fedora 44 Cloud Base image
filename in provision-vm.sh; (3) the opentelemetry 0.27 exporter
builder names; (4) whether aya-ebpf now provides a panic handler,
making ours redundant; (5) Rust 1.96.0 stable availability vs. the
beta timing.
Learned: authored against verified-current facts (Rust 1.95
stable / 1.96 beta; aya 0.13.x; Fedora 44; otel-lgtm 0.28.0; Python
3.14) but the lab to verify against is itself part of this delivery,
so r1.1 is explicitly a verification pass.
r2.0 — Chapter 7: kprobe + unlink
Shipped:_docs/07-kprobe-unlink.md; examples/07-kprobe-unlink/
(unlinksnoop workspace — kprobe on do_unlinkat, RingBuf events,
OTLP reporting, demo.sh); reconciliation Section C rows for Ch 7.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) the filename read assumes
struct filename begins with its name pointer — likely needs CO-RE
field access if the layout differs; (2) ctx.arg::<*const u8>(1)
argument indexing on the target kernel; (3) RingBuf::reserve/
submit and user-side RingBuf::next API names in aya 0.13.x.
Note: Chapter 8 (fentry + unlink) will revisit the same target to
contrast kprobe fragility with fentry’s BTF-typed argument access —
that contrast is the pedagogical payoff, so r3.0 should be drafted to
build directly on this chapter’s code.
r3.0 — Chapter 8: fentry + unlink + CI build-on-push
Shipped:_docs/08-fentry-unlink.md; examples/08-fentry-unlink/
(fentrysnoop — fentry+fexit on do_unlinkat, HashMap entry→exit
correlation, return-value capture, RingBuf events, OTLP with
ok/fail label, demo.sh); reconciliation Section C rows for Ch 8;
.github/workflows/pages.yml updated to build on every push (any
branch) and on PRs, deploying only from main.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) fexit return-value access
ctx.arg::<i64>(2); (2) Btf::from_sys_fs() + FEntry/FExitload(fn, &btf) + attach() API names in aya 0.13.x; (3) the
struct filename layout assumption (shared with Ch 7); (4) whether
the target kernel permits fentry/fexit (BTF + not locked down).
CI note: the deploy job is now guarded
if: github.event_name == 'push' && github.ref == 'refs/heads/main',
so feature-branch pushes and PRs build (validate) without deploying.
r4.0 — Chapters 9–10: opensnoop + sigsnoop
Shipped:_docs/09-opensnoop.md, _docs/10-sigsnoop.md;
examples/09-opensnoop/ (enter+exit openat tracepoints, user-memory
filename read, ok/err result) and examples/10-sigsnoop/ (single
kill tracepoint, signal name mapping); reconciliation Section C rows
for Ch 9–10.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) tracepoint field offsets vs. the
kernel’s format files (sys_enter_openat, sys_enter_kill);
(2) TracePointContext::read_at::<T>(offset) API in aya 0.13.x;
(3) bpf_probe_read_user_str_bytes (user vs kernel reader) for the
openat filename; (4) high openat event volume — future chapters add
in-kernel filtering.
Teaching arc: Ch 7 (kprobe) → Ch 8 (fentry/fexit, return values)
→ Ch 9 (stable tracepoints + user-memory reads + enter/exit) → Ch 10
(minimal single tracepoint). The four together cover the main
attach mechanisms before process-lifecycle tracing (execsnoop/
exitsnoop) in r5.0.
r5.0 — Chapters 11–12: execsnoop + exitsnoop
Shipped:_docs/11-execsnoop.md, _docs/12-exitsnoop.md;
examples/11-execsnoop/ (execve tracepoint, bounded argv read into
fixed slots, event written into the ring slot) and
examples/12-exitsnoop/ (exit_group tracepoint, exit-code decode);
reconciliation Section C rows for Ch 11–12. Closes the “Tracing the
kernel” part.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) the argv bounded loop passing the
verifier — highest risk in the whole iteration set; (2)
bpf_probe_read_user single-value signature; (3) writing an ~800B
event into the ring slot via reserve; (4) execve/exit_group offsets;
(5) the exit-code decode (& 0xff on the raw exit_group arg, NOT the
>> 8 used for task_struct->exit_code).
Correctness note: chose the exit_group tracepoint over
sched:sched_process_exit specifically to avoid a task_struct
CO-RE read, keeping Ch 12 robust; documented that signal-deaths won’t
appear (they don’t call exit_group).
Shipped:_docs/13-bashreadline.md, _docs/14-uprobe-rust.md;
examples/13-bashreadline/ (uretprobe on bash readline) and
examples/14-uprobe-rust/ (uprobe on a #[no_mangle] extern "C"
function in a bundled target-app); reconciliation Section C rows for
Ch 13–14. First chapters of Part “User-space & language probing”.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) UProbe/#[uprobe]/#[uretprobe]
ProbeContext::arg/RetProbeContext::ret API in aya 0.13.x;
(2) the attach(Some(sym), offset, target, pid) signature;
(3) where readline resolves on Fedora 44 bash (binary vs
libreadline); (4) whether #[inline(never)] + #[no_mangle] survives
release+LTO so compute stays attachable (else build target-app
without LTO).
Note: these introduce the user-space side; the new memory rule is
“uprobe reads belong to the traced process → user probe helpers”.
Remaining Part-3 chapters (USDT, sslsniff, funclatency, runtimes) build
on this attach model.
r6.1 — site: two-level Part/Chapter navigation (no new chapters)
Shipped: a parts collection (_parts/*.md, one per Part) +
_layouts/part_index.html; the homepage now shows one Part card
(with chapter count) instead of a flat chapter grid; each Part page
lists its chapters as cards; chapter breadcrumbs link Home → Part →
Chapter; Part pages have prev/next-part navigation. Config gains the
parts collection + default part_index layout.
Rationale: a flat 60+-chapter card grid would be unusable; the
two-level hierarchy keeps the homepage scannable as content grows.
Verified: nothing — unverified until the site is built. Static
checks pass (all 10 _parts parse; every doc part matches a
part_name; Liquid clean).
To check on build: the parts collection renders at
/parts/<slug>/; Part cards show correct chapter counts; empty parts
show “Coming soon”; future chapters MUST set part: to the exact
part_name string in the matching _parts file or they won’t group.
r6.2 — docs: neutral source-provenance wording
Shipped: removed country-specific naming from the source-provenance
policy across CONTRIBUTING.md, PRD.md, README.md, _docs/00-outline.md,
_plans/prd-reconciliation.md, onboarding/STARTING-WITH-CLAUDE.md.
Replaced with a neutral norm: insight from anywhere; ship original,
clearly-licensed code; no line-for-line copying/porting.
Verified: N/A (wording change). No country references remain in
the repo.
r7.0 — Chapter 15: BTF-assisted uprobe
Shipped:_docs/15-btf-uprobe.md; examples/15-btf-uprobe/
(target-app passing a *const Order, a uprobe reading the whole
struct via bpf_probe_read_user, BTF inspection via pahole -J +
bpftool btf dump); reconciliation Section C rows for Ch 15.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) bpf_probe_read_user::<T> reading
a whole struct (vs str/bytes variants) in aya 0.13.x; (2)
attachability of process_order under release+LTO; (3) debug = true
leaving DWARF that pahole -J can convert; (4) Order#[repr(C)]
offsets matching the BTF dump.
Scope note: framed user-space CO-RE honestly — kernel CO-RE is
turnkey, user-space relocation is newer; the robust path taught is the
shared/BTF-generated #[repr(C)] mirror + bpf_probe_read_user. Full
relocation deferred to the CO-RE deep-dive (Ch 56).
r7.1 — conventions: container policy, version pins, first diagram
Shipped: expanded CONTRIBUTING container policy (everything
user-space in Podman except the privileged Aya loader on the VM;
multi-stage UBI Containerfiles mandatory; crun 1.27.1 + container
observation + SELinux; Java 25/Quarkus 3.33 + Python 3.14/FastAPI
target pins; Excalidraw diagram workflow). PRD + reconciliation
version table updated with Java/Quarkus/FastAPI/crun. Rewrote the
Ch 3 client Containerfile as multi-stage UBI (builder venv → minimal
runtime). Added the first real diagram — assets/diagrams/lab-topology
(.svg + .excalidraw) — embedded in Chapter 2.
Verified: N/A for policy/diagram; the multi-stage Containerfile is
unverified (not built here).
To check on build: the multi-stage build resolves
ubi9/python-314 (builder) + ubi9/python-314-minimal (runtime) and
the venv copy runs; the SVG renders via the excalidraw include at
/assets/diagrams/lab-topology.svg.
Captured as durable project requirements (carried forward to all
future chapters): the container/loader split, multi-stage UBI, the
language-target pins, crun coverage, and the Excalidraw workflow.
Shipped:_docs/16-container-targets.md;
examples/16-container-targets/ — FastAPI (Python 3.14) and Quarkus
(Java 25 + Quarkus 3.33) targets as multi-stage UBI containers,
contrace (cgroup-scoped openat tracer via
bpf_get_current_cgroup_id() + config Array), compose.yaml, and a
demo that runs targets on the VM and scopes observation to one
container. Added podman/crun/dwarves to the target VM cloud-init.
Reconciliation Section C rows for Ch 16.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) cgroup-id resolution
(rootless/rootful + cgroup-manager variance) — demo falls back to
unscoped; (2) ubi9/openjdk-25 tag availability + Quarkus 3.33 build
(Containerfile carries a fallback); (3) bpf_get_current_cgroup_id /
Array::set API; (4) re-provisioning the VM so podman/crun are
present.
Model refinement: observed app-target containers run on the
target VM (not the host), because eBPF attaches to that kernel —
documented in the chapter and the lab topology applies. Host still
runs the stack + load driver.
r9.0 — Chapters 17–18: sslsniff + funclatency
Shipped:_docs/17-sslsniff.md, _docs/18-funclatency.md;
examples/17-sslsniff/ (uprobes on libssl SSL_write/SSL_read,
entry/return correlation for reads, bounded plaintext capture, ethics
note) and examples/18-funclatency/ (uprobe+uretprobe timing with
bpf_ktime_get_ns, per-call ring events → OTLP f64_histogram,
console log2 histogram, bundled target-app). Reconciliation Section C
rows for Ch 17–18.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) bpf_probe_read_user_buf with a
dynamic min(len,CAP) length passing the verifier (sslsniff); (2)
OpenSSL 3 symbol names in libssl.so.3; (3) bpf_ktime_get_ns +
uprobe/uretprobe-on-same-symbol + entry/exit HashMap; (4)
f64_histogram in opentelemetry 0.27; (5) slow_op/SSL symbols
attachable under release+LTO.
Note: funclatency ships the per-call-event approach (simple, OTLP
histogram); the in-kernel-histogram optimization is documented as the
high-call-rate alternative. This closes the core techniques of Part
“User-space & language probing”; remaining Part-3 chapters (goroutine
states, javagc) apply them to specific runtimes.
r10.0 — Chapters 19–20: goroutine states + javagc (closes Part 3)
Shipped:_docs/19-goroutine-states.md, _docs/20-javagc.md;
examples/19-goroutine-states/ (Go target + uprobe on
runtime.casgstatus reading the Go-ABI RCX register, uprobe-only) and
examples/20-javagc/ (HotSpot USDT gc__begin/gc__end timed via
uprobes at resolved offsets, OTLP GC-pause histogram, Java target).
Reconciliation Section C rows for Ch 19–20.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) Go register ABI read (RCX) +
pt_regs.rcx field name; (2) USDT offset resolution and the
readelf-Location→file-offset assumption (javagc — most experimental);
(3) Go symbol presence; (4) JDK hotspot USDT availability; (5) new host
toolchains needed: golang (Ch 19) and a JDK (Ch 20), both Fedora
repos.
Two hazards documented as first-class lessons: never uretprobe Go
(moving stacks corrupt return trampolines); non-C languages need
manual register mapping (ctx.arg assumes the C ABI). bpftrace’s native
USDT + reg() are the cross-check references.
Milestone: Parts 0–3 complete (Foundations, Tracing the kernel,
User-space & language probing) — 21 chapters, 16 examples. Next:
Part 4 Performance & resources (r11+).
r11.0 — Chapters 21–22: runqlat + hardirqs (opens Part 4)
Shipped:_docs/21-runqlat.md, _docs/22-hardirqs.md;
examples/21-runqlat/ (sched_wakeup/_new + sched_switch, in-kernel
log2 histogram in an Array, OTLP observable-gauge percentiles) and
examples/22-hardirqs/ (irq_handler_entry/exit keyed by CPU, per-IRQ
HashMap totals, OTLP observable-gauge per vector). Reconciliation
Section C rows for Ch 21–22.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) sched + irq tracepoint field
offsets; (2) Array::get_ptr_mut / HashMap accumulation in-kernel;
(3) the observable-gauge callback API in opentelemetry 0.27 (first
use of registered-once observable gauges — both chapters); (4)
TASK_RUNNING==0 and nested-IRQ simplification.
Technique milestone: both chapters use in-kernel aggregation
(the hot-path technique flagged in Ch 18) rather than per-event ring
buffers — runqlat via a log2 Array histogram, hardirqs via a per-IRQ
HashMap. This is the Performance-part idiom.
r12.0 — Chapter 23: profile (sampling CPU profiler)
Shipped:_docs/23-profile.md; examples/23-profile/ — a
perf_event program sampling at 99 Hz on every CPU, capturing kernel
user stacks via bpf_get_stackid into a StackTrace map, a count
map keyed by (pid,comm,kstack,ustack), kernel symbolization via
kernel_symbols(), and folded flame-graph output. Reconciliation
Section C rows for Ch 23.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) the PerfEvent::attach signature
/ SamplePolicy / PerfEventScope and online_cpus() in aya 0.13.x
(first perf_event program); (2) get_stackid (ebpf) and
StackTraceMap::get().frames() (user) — first stack-walking example;
(3) user-stack capture depending on frame pointers/unwind info.
New ground: first perf_event program type, first stack walking,
first sampling (fixed-cost) tool. User-frame symbolization left as hex
(wire in blazesym); Pyroscope push noted as the continuous-profiling
extension (otel-lgtm bundles Pyroscope).
r13.0 — Chapters 24–25: memleak + biopattern
Shipped:_docs/24-memleak.md, _docs/25-biopattern.md;
examples/24-memleak/ (malloc/calloc+free pairing, alloc-site stacks
via bpf_get_stackid reused from Ch 23, pid-scoped, bundled leaker.c)
and examples/25-biopattern/ (block_rq_issue tracepoint, per-device
sequential/random classification by sector arithmetic, OTLP ratio
gauge). Reconciliation Section C rows for Ch 24–25.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) glibc malloc/calloc/free uprobe
attach + user-stack capture needing frame pointers (memleak); (2)
block_rq_issue field offsets vs. the format file (biopattern —
layout has drifted across kernels); (3) dev_t decode; (4) u64_gauge
and observable-gauge APIs in otel 0.27.
Continuity: memleak deliberately reuses the Ch 23 stack-walking
primitive (StackTrace + get_stackid), reinforcing it as a building
block. Both remain consistent with the aggregate-in-kernel idiom.
r14.0 — Chapter 26: energy monitoring (closes Part 4)
Shipped:_docs/26-energy.md; examples/26-energy/ — a
sched_switch tracepoint crediting per-task on-CPU time
(ONCPU/USAGE HashMaps), with user-space energy attribution by
CPU-time share × system power (RAPL when present, flat-TDP model when
not — the VM reality). Exports estimated_power_watts{comm} +
system_power_watts. Reconciliation Section C rows for Ch 26.
Verified: nothing — unverified pending a real Fedora 44 run.
Known risks to check first: (1) sched_switch offsets; (2) RAPL
absence on KVM guests (fallback model exercised); (3) observable-gauge
API in otel 0.27; (4) the model is an estimate by construction.
Scope note: chapter is explicit that RAPL/vPMU are usually NOT
exposed in VMs, so absolute watts are modeled on the lab VM while the
attribution (shares) stays correct; bare metal gives real RAPL. This
mirrors Kepler’s cloud accommodation. Hardware-counter accuracy upgrade
(PERF_EVENT_ARRAY + bpf_perf_event_read_value) documented as the
extension, not shipped, to avoid an uncertain API in the core example.
MILESTONE: Part 4 (Performance & resources) complete — Ch 21–26
(runqlat, hardirqs, profile, memleak, biopattern, energy), 27 chapters
/ 22 examples total. Parts 0–4 done. Next: Part 5 Networking (r15+),
which needs the two-VM peer build-out.
r14.1 — diagrams pass (Tier 1–3)
Shipped: 19 new Excalidraw+SVG diagram pairs under
assets/diagrams/ (plus a spec-based generate.py), embedded one or
two per chapter across Ch 3,4,5,6,8,9,11,13,15,16,17,19(×2),20,21,23,
24,25,26. Covers the foundational/reusable concepts (lifecycle,
RingBuf path, data path, workspace build, entry/exit correlation,
user-vs-kernel memory reads), the language-probing mechanics
(probing-surfaces menu, struct/BTF, container observation, TLS
boundary, goroutine state machine, Go-vs-C ABI, USDT-as-uprobe), and
the performance pipelines (runqlat timeline, profiler, memleak, bio
seq/random, energy attribution). README catalogue updated.
Verified: SVGs are well-formed XML; Excalidraw files parse; all
chapter front matter still parses; includes use the existing
excalidraw.html partial with alt text + figure captions.
Not verified (rendering): exact visual layout/overflow in a real
Jekyll build (no local Jekyll); .svg are clean themed exports rather
than the hand-drawn Excalidraw aesthetic — .excalidraw sources are
included so they can be refined/re-exported.
Deferred by design: networking diagrams (packet path / hook
points, two-VM topology, TCP lifecycle, XDP-vs-tc) ship with the
Part 5 chapters (r15+).
Shipped:_docs/27-tcpconnlat.md, _docs/28-tcpstates.md;
examples/27-tcpconnlat/ (kprobes on the TCP connect path, sock* key,
sock-field offset reads) and examples/28-tcpstates/
(sock:inet_sock_set_state tracepoint, full state machine). Two-VM
build-out: cloud-init net tools (nmap-ncat/socat/iproute/iputils/curl/
tcpdump), scripts/lab/lab-ips.sh helper; peer is
provision-vm.sh ebpf-peer. Networking diagrams authored:
net-hooks, tcp-handshake (Ch 27), tcp-states (Ch 28).
Verified: nothing — unverified pending real Fedora 44 + two VMs.
Known risks to check first: (1) sock_common offsets (Ch 27 —
CO-RE removes this in Ch 56); (2) inet_sock_set_state tracepoint
offsets + port byte order (Ch 28); (3) kprobe attach to TCP symbols;
(4) two guests routing to each other.
Pedagogical contrast (deliberate): Ch 27 kprobe + struct-offset
(powerful, fragile) vs. Ch 28 stable tracepoint (less reach, durable)
— stated explicitly as the kernel-tracing through-line.
Networking diagrams shipped with the chapters (per the r14.1 plan
to fold net diagrams into r15+).
Chapters 29–30 — HTTP L7, sockops (r16.0)
Status
Claim
Chapter
unverified
examples/29-http-l7 builds
Ch 29
unverified
socket_filter program + SkBuffContext load/load_bytes in aya 0.13.x
Ch 29
unverified
AF_PACKET raw socket setup (libc) + SocketFilter::attach(fd)
Ch 29
unverified
Eth→IPv4(no-options)→TCP parse + data-offset math reaches payload
Ch 29
unverified
HTTP method/HTTP/ detection + first-line capture; cleartext only
Ch 29
unverified
ebpf_http_lines_total{method} in Grafana
Ch 29
unverified
examples/30-sockops builds
Ch 30
unverified
sock_ops program + SockOps::attach(cgroup) in aya 0.13.x
established op constants (ACTIVE=4/PASSIVE=5); port byte-order convention
Ch 30
unverified
requires cgroup-v2 at /sys/fs/cgroup; ebpf_sock_established_total{dir} in Grafana
Ch 30
r16.0 — Chapters 29–30: HTTP L7 + sockops
Shipped:_docs/29-http-l7.md, _docs/30-sockops.md;
examples/29-http-l7/ (socket_filter parsing Eth/IPv4/TCP → HTTP line,
AF_PACKET raw socket) and examples/30-sockops/ (sock_ops on the
cgroup-v2 root, established callbacks, 4-tuple from the context).
Diagrams l7-socketfilter (Ch 29) and sockops-cb (Ch 30).
Verified: nothing — unverified pending real Fedora 44 + two VMs.
Known risks to check first: (1) socket_filter + SkBuffContext API
and AF_PACKET setup (Ch 29 — first packet-content program); (2)
sock_ops attach + SockOpsContext accessors + op constants (Ch 30 —
first cgroup-attached callback program); (3) byte-order conventions;
(4) IHL==5 / cleartext-only simplifications.
Two new program types introduced (socket_filter, sock_ops). L7
taught both ways: socket filter (wire/cleartext) vs. syscall+uprobe
(buffer/encrypted, ties back to Ch 17). sock_ops framed as the
observe-and-act, cgroup-scoped, callback model.
#3 diagrams (fixed): generator now draws nodes before edges so
arrowheads render on top of boxes (root cause of “arrows go behind
boxes / stop short”); arrowheads enlarged; edge labels get a white halo.
All 24 edge-bearing SVGs reordered; goroutine-states redesigned with
short adjacent arrows instead of one long cross-diagram arrow.
assets/diagrams/generate.py synced to the fixed generator.
#4 VM/networking setup (clarified): the two-VM lab already lives in
Foundations (Ch 2); added an explicit “What the networking part needs”
subsection (peer reachability / interface / cgroup-v2 table + resource
sizing) and a back-reference from Ch 27 to Ch 2.
#2 code-explanation depth (model established): Ch 9 (opensnoop)
rewritten with a full “How the code works” walkthrough — both maps and
why each type, the entry stash and exit pair/emit with per-call
explanation, and the user-space attach + ring-drain + decode. This is
the depth/style to roll through the remaining chapters in subsequent
passes (pending author confirmation).
Verified: nothing new — diagrams are visual-only; code remains
unverified.
r16.2 — code-depth pass, Part 1 (Tracing the kernel)
Brought Part 1 to the Ch 9 walkthrough standard (“explain what Rust+Aya
are doing, as if the reader were writing it, with a BCC side-by-side”).
Deepened: Ch 10 (sigsnoop) — full kernel handler explained + the
previously-absent user-side attach/drain/format walkthrough; Ch 12
(exitsnoop) — real handler instead of a stub + concrete user-side
decode (kept the strong “encoding gotcha” section).
r16.3 — code-depth pass, Part 2 (User-space & language probing)
Reviewed Ch 13–20 against the Ch 9 standard. Already at depth (left
as-is): Ch 14 (Rust uprobe / no_mangle), Ch 15 (struct read + BTF),
Ch 16 (cgroup-id scoping handler + inode resolution), Ch 17 (sslsniff
entry/return library probes), Ch 18 (entry/return timing + in-kernel-
vs-userspace histogram trade-off), Ch 19 (Go ABI pt_regs/RCX read,
no-uretprobe-on-Go), Ch 20 (USDT-as-uprobe at offset). These already
show real handlers, the Aya mechanics, and register/BTF cross-checks.
Deepened: Ch 13 (bashreadline) — replaced the stub uretprobe body
with the full reserve/read-user-string/submit handler + explanation.
Docs-only; code remains unverified.
r16.4 — diagram + wording fixes; Part 3 review
Diagram fix:container-observe (Ch 16) rebuilt — the “container”
is now a framing band (top-left label) so no centered text sits
behind the nested app/libjvm boxes. Audited all diagrams for
node-contains-node label-hiding; only this one was affected
(lab-topology’s labels are top-left, so it was a false positive).
Wording: removed all “roadmap” closers and the “(iteration)
roadmap” links from every chapter; removed “honest”/”an honest”
framing (6 spots) — reworded, e.g. Ch 15 “An honest scope note” → “Scope
note”, Ch 26 “The honest part” → “The hard part”. Repaired the two
sentences left dangling by the removals (Ch 0, Ch 7).
Nav: dropped the public “Roadmap” menu item; the “Tutorial” item
already points to the outline (more useful to readers).
Ch 17 (sslsniff): added a FIPS section — FIPS mode changes the
cipher/provider, not where plaintext sits, so the SSL_read/SSL_write
uprobes capture identically; with the kTLS caveat as the real
boundary-mover.
Part 3 (Ch 21–26) review: already at the Ch 9 depth standard
(in-kernel Array log2 histogram + get_ptr_mut; per-key/per-CPU
HashMap aggregation with race + frame-pointer notes; get_stackid
stack walking; OTLP observable-gauge percentile callbacks; RAPL +
VM fallback). No rewrites needed.
r16.5 — code-depth pass, Part 4 (Networking) — rollout complete
Brought Ch 27–30 to the Ch 9 walkthrough standard (these were the
tersest chapters). Each now shows the real handler(s) and a user-side
walkthrough with per-call explanation:
Ch 27 (tcpconnlat): full two-kprobe handlers (sock* key, struct-field
reads), user-side dual KProbe attach + ring drain + OTLP histogram.
Ch 29 (http-l7): completed the parse→capture→submit, plus the
distinctive AF_PACKET raw-socket open + SocketFilter::attach(fd).
Ch 30 (sockops): concrete ctx.op() handler with reserve/submit,
plus cgroup-v2 SockOps::attach(File) user side.
Code-depth rollout now complete across Parts 0–4: Part 1 (r16.2)
deepened Ch 10/12; Part 2 (r16.3) deepened Ch 13; Parts 2/3 otherwise
already at depth; Part 4 (r16.5) deepened all four. Docs-only; code
remains unverified.
Diagram fix:tcp-handshake (Fig 27.2) redrawn — both kprobe boxes
now sit above the client and each dashed connector lands on its event
(connect→client, rcv_state_process→the SYN-ACK arrow); the previous
dangling diagonal is gone.
Ch 20 (javagc) expanded per request:
New jvm-observable diagram (Excalidraw+SVG): the HotSpot USDT probe
surface across GC / memory pools / JIT / threads / monitors /
allocation+classes, each reachable as a uprobe at its offset.
Collectors: noted the JDK/OpenJDK-UBI ships G1 (default), ZGC, and
Shenandoah; all fire gc__begin/gc__end, but ZGC/Shenandoah are mostly
concurrent so the signal shifts to concurrent-cycle/allocation.
“Many JVMs on one node”: one probe per distinct libjvm.so path
(per container overlay), attribute by cgroup/PID, label by
container+collector (ties to Ch 16).
“Why GC monitoring matters”: stop-the-world pauses as tail latency,
time-in-GC as saturation, what to alert on, and the out-of-process
advantage (no JVM flags / agent / verbose:gc parsing).
Part numbering fixed: homepage cards render Part {order}, so
Networking (order 4) is Part 4. Corrected stale “Part 5” refs in
Ch 26 and Ch 27 (description + prose). (“Part 6” in Ch 26 = sched_ext,
order 6 — correct, left.)
Ch 27 environment readiness: added an up-front “Before you start —
this part needs two VMs” block (stack up / target running / peer
provisioned, with the exact checks) so the reader confirms the
environment before running; trimmed the now-redundant mid-chapter
peer-provisioning into a short topology note.
Explicit run guidance in every chapter: appended a one-line run
hint to each program chapter’s code-location line (“./demo.sh there
builds, deploys, and runs it; its README.md covers …”) — 25 chapters
(06–26, 28–30) plus tailored hints for Ch 03/14/20; Ch 27 carries the
fuller block.
Getting-Started: added a reader-facing “Running any chapter’s
example” section (the universal cd examples/NN → ./demo.sh
pattern, README per example, demo.sh self-docs, and the
stack/target/peer assumptions).
Docs-only. No test runner exists (test-all-examples.sh was stale in
notes); examples are exercised via each demo.sh + cargo build.
r17.0 — Part 4 (Networking) continues: tc + first XDP — UNVERIFIED
New chapters and examples (all unverified — not yet run on Fedora 44):
Ch 31 (tc-classify) — #[classifier] on clsact egress; counts
packets/bytes per L4 proto in-kernel HashMaps; drops traffic to
BLOCK_PORT with TC_ACT_SHOT. New diagram tc-clsact. Risks to
confirm: Aya tc API (qdisc_add_clsact, SchedClassifier,
TcAttachType::Egress), network-types 0.0.7 field/LEN names,
TcContext::load/len, user-space HashMap::iter()/get deltas,
that TC_ACT_SHOT drops on egress.
Ch 32 (xdp-drop) — #[xdp] ingress; raw data/data_end with a
ptr_at bounds check; counts per proto, drops ICMP with XDP_DROP.
New diagram xdp-path. Risks: virtio-net native XDP vs the
SKB_MODE fallback, XdpContext::data/data_end, verifier acceptance
of ptr_at, that ping to the target stops while attached.
Both reinforce the in-kernel-aggregation lesson (no per-packet ring on
a data path) and introduce verdicts (acting, not just observing).
New shared dep introduced: network-types = "0.0.7" (header parsing).
Fixed Fig 31.1 (tc-clsact): the two hook boxes were covering the
band’s top-left “clsact qdisc (eth0)” label; boxes now start well below
the label so it’s legible.
Added a “Reading the command blocks” callout to Ch 2 defining the
prompt prefixes ([host]$/[vm]$/[peer]$) and the <…> placeholder
convention, with how to get a shell and find <iface>.
Standardized the new networking chapters on [vm]$/[peer]$ (were
[target]$); rewrote the Ch 31 and Ch 32 cross-checks to discover the
interface first (ip -br link, concrete enp1s0), show option meanings
(tc -s, ip link show, bpftool net show) and expected output, and
define <target-ip> with a concrete example. No remaining bare
command-block placeholders.
r17.2 — document the lab credential model (docs only)
Made the lab credential model explicit in the Ch 2 conventions callout:
user fedora, SSH-key-only login (account password locked via cloud-init
lock_passwd: true), and passwordless sudo (NOPASSWD:ALL) — so there is
no sudo password to list, and every [vm]$ sudo … runs without a prompt.
Noted how to set a console password if ever needed.
Ch 33 (xdp-capture) — #[xdp] read-only tap: parse Eth/IPv4/TCP via
ptr_at, count all packets in SEEN, and ship a FlowRecord over a
RingBuf only for TCP SYN/FIN/RST (in-kernel filter keeps volume low);
always XDP_PASS. New diagram xdp-capture. Confirm: RingBuf in #[xdp],
network-types field names, TCP flags byte at offset 13, output matches
tcpdump 'tcp[tcpflags] & (…) != 0'.
Ch 34 (xdp-lb) — #[xdp] UDP port load balancer: ptr_at_mut to
rewrite udp.dest round-robin across an Array of backends (cursor via
get_ptr_mut), zero the optional IPv4 UDP checksum, XDP_PASS; per-backend
HITS. New diagram xdp-lb. Confirm: ptr_at_mut writes pass the
verifier, zeroed UDP checksum accepted end-to-end, Arrayget/get_ptr_mut/set, native-vs-SKB attach, even split. Documents the
production gaps (return path / DSR / conntrack, incremental checksum,
XDP_REDIRECT to remote backends) as forward work.
New diagrams: 28 → 30. Both reuse the ptr_at discipline from Ch 32; Ch 34
introduces packet mutation and map-driven backend selection.
Regenerated lab-topology (Fig 2.1): the previous hand-authored SVG had
only its and no drawing body, so it rendered blank. Now generated
with the diagram tool (host + Podman containers, target VM, peer VM, with
scp/OTLP/traffic arrows).
Replaced the ASCII-art lifecycle “picture” in Ch 5 with a real diagram,
ebpf-runtime-loop (Fig 5.2): the runtime loop (hook fires → program
writes map → loader reads → Grafana), complementing Fig 5.1’s build→attach
pipeline.
Confirmed the Ch 2 T=$(./vm-ip.sh …) && P=$(./vm-ip.sh …) && echo …
command is correct: vm-ip.sh prints only the IP (errors to stderr, exits
non-zero), and the line runs in scripts/lab/ as set by the preceding cd.
Ch 35 (xdp-test) — a BPF_PROG_TEST_RUN harness: load a small XDP
filter (drop ICMP / pass else + PKTS counter), build synthetic Eth/IPv4
packets in user space, run each through the program, assert the verdict,
and check a map side-effect; exits non-zero on failure (CI-ready). New
diagram xdp-test-run. KEY risk: the test-run binding — the example issues
the bpf() command via a libc::syscall wrapper with a hand-laid bpf_attr
test struct + prog.fd(); confirm against the installed Aya/kernel (or use
Aya’s own test_run if present).
Ch 36 (tcx) — the Ch 31 classifier reattached via tcx (kernel 6.6+):
no qdisc_add_clsact, attach(&iface, TcAttachType::Ingress) returns a
bpf_link that auto-detaches on drop; cross-check shows tcx/ingress in
bpftool net show with an empty tc filter show and no clsact qdisc. New
diagram tcx-chain. Risk: exact Aya tcx attach API + link lifetime.
Networking part (Ch 27–36) is now complete. New deps unchanged
(network-types already in; Ch 35 adds libc to that one loader). Diagrams
31 → 33.
r20.0 — Part 5 (Security & LSM) opens: BPF LSM + signal programs — UNVERIFIED
Ch 37 (lsm-confine) — first BPF LSM program: #[lsm(hook="socket_connect")]
denies connect() for processes in a confined cgroup (return -EPERM),
loaded against kernel BTF; teaches the allow/deny model, prior-verdict
respect, and the bpf LSM enablement requirement. New diagram lsm-decide,
new lab helper scripts/lab/enable-bpf-lsm.sh. Risks: Aya LSM API
(Lsm::load(hook,&btf)/attach), LsmContext arg indexing (ret at 3),
cgroup-id == dir inode.
Ch 38 (signal-kill) — signal program: tracepoint on sys_enter_execve
matches a forbidden filename prefix and calls bpf_send_signal(SIGKILL)
to kill the process pre-exec; emits KillEvents over a RingBuf. New diagram
signal-kill. Framed LAB-ONLY. Risks: bpf_send_signal from a tp, execve
filename offset 16, bounded starts_with.
Pulled Ch 38 (bpf_send_signal) into r20 alongside Ch 37 (plan had it in
r21); iteration plan reconciled. Diagrams 33 → 35. Part 5 card now shows
2 chapters.
Ch 39 (pidhide) — LAB-ONLY rootkit technique: two tracepoints on
getdents64 (enter captures the user buffer, exit walks linux_dirent64 and
splices out the target /proc/ by extending the previous record's
d_reclen via `bpf_probe_write_user`). Heavy detection section (kernel taint,
bpftool prog show, /proc vs real task list). New diagram `pidhide`. Risks:
getdents64 enter/exit offsets (dirent @24, ret @16), linux_dirent64 offsets
(d_reclen @16, d_name @19), bpf_probe_write_user permission, bounded walk.
Ch 40 (lsm-fileprotect) — defensive counterpart: #[lsm(hook=
"inode_permission")] denies MAY_WRITE on one protected inode (read-only
even for root), reading inode->i_ino via bpf_probe_read_kernel at a
hard-coded I_INO_OFFSET (flagged version-specific; CO-RE in Part 9 fixes
it). Fail-open on read error. New diagram lsm-file-protect. Risks: arg
indexing (inode@0/mask@1/ret@2), MAY_WRITE value, i_ino offset.
Diagrams 35 → 37. The hard-coded i_ino offset is the canonical CO-RE
motivation; forward-ref to Ch 56 reinforced.
r21.1 — fix blank Ch 39 body (rendering bug)
Root cause: _includes/excalidraw.html emitted {{ include.alt }} as raw,
unescaped HTML fallback text inside
Fix: {{ include.alt | escape }} in both the aria-label attribute and the
fallback text — protects every chapter from <…> in alt forever. Also
tidied Ch 39’s alt (/proc/<pid> → /proc/PID). Ch 39 was the only
chapter with a tag-like alt. No content change to the chapter body.
Ch 41 (sudoadd) — LAB-ONLY: intercept sudo’s read() (comm==”sudo”),
overwrite the returned buffer with an injected NOPASSWD sudoers line via
bpf_probe_write_user; on-disk file untouched. enter_read stashes buf/count,
exit_read rewrites; payload (<user> ALL=(ALL:ALL) NOPASSWD:ALL #) from a
map, write bounded to 64B. Detection section + LSM-can-block-the-primitive.
New diagram sudo-escalate. Risks: read enter/exit offsets (buf @24,
count @32, ret @16), comm match, payload/padding vs real sudoers layout.
Ch 42 (secsensor) — defensive capstone: three tracepoints (execve,
ptrace, setuid) emit a uniform SecEvent over one RingBuf; user side
classifies type+severity → ebpf_sec_events_total{type,severity}. Frames
observe-vs-shield (telemetry broad/cheap; enforce narrow via LSM). New
diagram security-sensor. Risks: tracepoint names (esp. setuid vs
setreuid/setresuid), one RingBuf shared by three programs.
Security & LSM part complete (Ch 37–42): confine, react, hide, protect,
escalate, sense. Pulled Ch 42 into r22 (plan had it as r23); plan
reconciled, the stray r23 row folded in. Diagrams 37 → 39.
Ch 43 (scx-simple) — sched_ext/struct_ops model taught via the real
scx_simple callbacks (shown in C, faithful to upstream): select_cpu (idle
fast path → SCX_DSQ_LOCAL), enqueue (→ SCX_DSQ_GLOBAL), init/exit,
SCX_OPS_DEFINE; DSQs; the safety watchdog (evict → fall back to default).
HONEST framing callout: sched_ext today = BPF-in-C + Rust-userspace
(scx/scx_utils); Aya kernel-side struct_ops is emerging, so we run the
shipping scx_simple (Fedora scx-scheds) and OBSERVE it with an Aya
sched_switch probe (per-CPU ctx-switch counter → ebpf_ctxsw_total{cpu}).
New diagram scx-simple. Example 43-scx-simple = Aya observer (scx-watch)
reference/scx_simple.bpf.c (read-only) + runbook demo. Requirements:
kernel ≥ 6.12, CONFIG_SCHED_CLASS_EXT. New part “Schedulers (sched_ext)”.
NOTE: first chapter to use C for the (reference) BPF side — deliberate and
documented, because that’s the current reality of sched_ext tooling.
r24.0 — Part 7 (Schedulers): a realistic policy (scx_nest) — UNVERIFIED
Ch 44 (scx-nest) — the nest policy: concentrate work on a small set of
warm, high-frequency cores; primary nest + reserve, promote under load,
demote when idle. Shows simplified nest_select_cpu (C, ref) using
scx_bpf_pick_idle_cpu over cpumasks + bpf_cpumask_* kfuncs (kfunc forward-
ref to Part 8). Same honest pattern: run the real scx_nest (scx-scheds) and
OBSERVE it with an Aya per-CPU busy probe (sched_switch, attribute interval
to busy if prev_pid != 0 → ebpf_cpu_busy_ns_total{cpu}; live busy% bar), so
the nest is visible as a few hot cores. New diagram scx-nest. Example
44-scx-nest = cpu-busy probe + reference/scx_nest.bpf.c (simplified) +
moderate-load runbook. Risks: scx_nest availability, sched_switch prev_pid
offset (24), idle==prev_pid 0.
Schedulers part now has its two-chapter tour (minimal + realistic). Diagrams
40 → 41.
Ch 45 (nginx-probe) — uprobes on a containerized nginx’s request
functions: req_start (ngx_http_process_request) stamps STARTS keyed by the
request pointer (ctx.arg(0), never dereferenced); req_done
(ngx_http_finalize_request) computes elapsed → log2(us) HIST. Teaches the
two real-target frictions: SYMBOLS (nm/objdump; distro nginx stripped →
debuginfo/build-with-symbols) and CONTAINER NAMESPACES (attach via
/proc//root/usr/sbin/nginx to reach the in-container inode).
Example = Aya uprobe latency tool + multi-stage UBI Containerfile building
nginx from source with --with-debug -g (keeps .symtab) + containerized
runbook (podman build/run on target, drive load, resolve worker, attach).
New diagram `nginx-uprobe`. Risks: nginx symbol presence, UProbe::attach
signature (fn/offset/target/pid), /proc//root resolution, ctx.arg(0)
ABI. Reuses the funclatency (Ch18) histogram, keyed by request object.
Diagrams 41 → 42. First real third-party app target.
Ch 5 “shape of every chapter” rewritten to match the real chapter structure
(concept+diagram → How the code works → Build/deploy/observe → Cross-check →
What you learned → Verification status), instead of the old 6-step “recipe”
that listed three separate crate sections (they’re one “How the code works”
section) and omitted the diagram and recap. Noted -common is per-chapter
(when a shared record exists), and that the book never self-promotes.
Fixed the outline tables in Ch 0 and PRD: Security & LSM grew to 6 chapters
(37–42), so everything after shifted +1 — Schedulers 43–44, Application
targets 45–46, Advanced kernel surface 47–56, Operating eBPF 57–61. PRD
Security row description updated to its 6 chapters.
De-numbered the CO-RE forward-refs (were “Chapter 56”/”Ch 56” in Ch 7, 8,
15, 27) to “Part 9” so they can’t drift as earlier parts grow.
r25.2 — docs: make the Grafana side of “observe” real (and referenced)
Ch 3: replaced the ASCII data-path block under “How Rust eBPF programs
report in” with a new diagram reports-in (Fig 3.2) showing the opensnoop
example fanning out to TWO faces of output — a live terminal table and
ebpf_* metrics pushed via OTLP to otel-lgtm/Grafana. Added a “Where your
output shows up” subsection that fixes the terminal-vs-Grafana distinction
and gives a concrete Grafana viewing recipe (127.0.0.1:3000 → Explore →
Prometheus → query ebpf_). Strengthened “Leave it running” to state every
later chapter feeds this stack, and updated the closing checklist.
Disambiguated chapter “Build, deploy, observe” steps so “watch X” is clearly
the terminal and the ebpf_* metric is clearly Grafana: Ch 44 (per-CPU
busy % in terminal; ebpf_cpu_busy_ns_total{cpu} rate() in Grafana) and
Ch 43 (ebpf_ctxsw_total{cpu}). Ch 45 already named its Grafana metric.
CONVENTION going forward: every program chapter’s “Build, deploy, observe”
names both the terminal live view and the specific ebpf_* metric to chart
in Grafana. Older chapters (6–42) still describe terminal output well but
mostly don’t name the Grafana panel — a focused “observe pass” can retrofit
them; tracked here as outstanding.
r25.3 — observe pass: every chapter now names its Grafana view
Renamed 5 stray metrics to the ebpf_ prefix so ALL series are discoverable
under ebpf_: ebpf_function_latency_ms (18), ebpf_runqueue_latency_us (21),
ebpf_memleak_outstanding_bytes (24), ebpf_bio_sequential_ratio (25),
ebpf_estimated_power_watts + ebpf_system_power_watts (26).
Added an “In Grafana” line to the “Build, deploy, observe” section of every
program chapter (06–45): the exact PromQL to graph, shaped per metric type —
counters → rate(); gauges (21,24,25,26) → raw; the funclatency histogram (18)
→ histogram_quantile/heatmap; shared ebpf_events_total chapters → filter by
the chapter’s ebpf-* service + sum by(program). 35 (BPF_PROG_TEST_RUN) is
noted terminal-only. 43/44/45 already had theirs (r25.2 / r25.0).
Enhanced the auto-provisioned dashboard (ebpf-overview.json, already mounted
into otel-lgtm) into the universal front door: added a “Metric (explorer)”
query variable over metrics(ebpf_.*) feeding a rate panel (counters) and a
raw panel (gauges), alongside the existing events + logs panels. Referenced
from Ch3 and the example README. (No manual import — it loads on stack up.)
The earlier “watch X — where?” gap is now closed end to end: terminal view
for the live glance, named ebpf_* metric/query for Grafana, one dashboard
that browses them all.
r26.0 — capstone: tying the three signals together (Ch 46) — UNVERIFIED
New Ch 46 (three-signals), a short capstone in the Application targets
part. Core lesson: Java (JIT) and Python (interpreted) have no app symbols to
uprobe, so instrument at the kernel SOCKET layer — the OBI insight. kprobes
on tcp_recvmsg/tcp_sendmsg keyed by struct sock* (as u64), bpf_get_current_comm
to tag the service, ring buffer of Req{dur_ns,comm}. Loader mints ONE trace_id
per request and emits three correlated signals: span (Tempo), log (Loki), RED
metrics (Prometheus) — finally exercising the trace/log/metric correlation the
Ch3 datasources provisioned. Honest seams flagged: loader-minted trace_id is
intra-service (not distributed W3C propagation); exemplar (metric→trace) hop
may be missing in the Rust metrics SDK; recv/send pair is a simplification of
real L7 parsing. Positions OBI (OpenTelemetry eBPF Instrumentation, donated
Beyla) as the production tool, with its architecture diagram + docs link.
Example 46-three-signals: httpwatch-{ebpf,common,loader} + Java (plain JDK
HttpServer) and Python (FastAPI) containerized services + demo. Two new
diagrams: three-signals, obi-arch. New metrics: ebpf_http_server_requests_total,
ebpf_http_server_duration_ms.
RENUMBER (Security stayed 6, now capstone inserted): postgres 46→47;
Application targets 45→47; Advanced kernel surface 47–56 → 48–57; Operating
eBPF 57–61 → 58–62. Updated Ch0 + PRD tables, part blurb, Ch45 next-pointer,
plan rows. CO-RE refs already de-numbered to “Part 9” so unaffected.
HIGHEST-uncertainty code in the book (3-signal OTel SDK wiring); marked
unverified with the specific API surfaces to confirm.
A LARGER capstone is planned for the very end of the book to reinforce this.
Ch 47 (pg-probe) — a multi-process database target. ONE uprobe on the
postgres binary covers every backend (attach pid=None); the single-threaded
backend pid is a clean per-connection key. uprobe/uretprobe on
exec_simple_query → per-query latency + SQL text (bpf_probe_read_user_str_bytes
of arg0, 128B bounded, PII note); uprobe/uretprobe on ProcSleep → lock-wait
time (contention invisible from outside). Introduces USDT (query__start /
lock__wait__start, .note.stapsdt) as the stable DB-observability API, with an
honest note that Aya USDT attach is still maturing so we uprobe the wrapped
functions. Cross-check against pg_stat_statements + pg_stat_activity (the DB’s
own books). New diagram pg-probe. Example = pg-probe-{ebpf,common,loader} +
demo running stock postgres image + a lock-contention scenario. New metrics
ebpf_pg_query_duration_ms, ebpf_pg_lock_wait_ms. Symbol/debuginfo caveat
documented (stock images stripped; need debug symbols or –enable-dtrace).
Application targets part (45 nginx, 46 capstone, 47 postgres) now complete.
Next: Part 8 Advanced kernel surface (48+), opening with detach/pinning.
Ch 48 (pin-demo) — the lifecycle foundation. BPF programs/maps/links are
refcounted by fds; loader exit frees them. Pinning to bpffs (/sys/fs/bpf)
creates a named reference that keeps them alive. Map pinned via
HashMap::pinned (LIBBPF_PIN_BY_NAME) + EbpfLoader::map_pin_path; link pinned
via take_link → FdLink::try_from → FdLink::pin; read back elsewhere via
MapData::from_pin. pinctl load/read/detach demonstrates: load+pin+exit, the
program keeps counting execs, a fresh process reads the pinned map, rm the
pin detaches. Cross-check with bpftool prog/link/map show + map dump pinned.
New diagram pinning. New metric ebpf_pinned_execs_total. Decoupling the
eBPF lifecycle from the loader → forward-ref to Part 9 (zero-downtime).
Plan Phase 9 rows resynced to current numbering (Advanced = 48–57). Next:
Ch 49 syscall programs.
Ch 49 (syscall-prog) — BPF_PROG_TYPE_SYSCALL: attaches to nothing, run via
BPF_PROG_RUN, calls bpf() itself via bpf_sys_bpf (sleepable; bpf_sys_bpf/
bpf_sys_close/bpf_btf_find_by_name_kind). Teaches loader programs (“strace of
libbpf” replayed in-kernel) and light skeletons (bpftool gen skeleton -L
embeds a generated syscall/loader program + data → no libelf/libbpf at load;
path to signed loading). HONEST frame: Aya IS the user-space loader, so this
is mechanism-to-recognize, not a tool to wield; aya-ebpf syscall-program
authoring is experimental. Concept chapter: new diagram syscall-prog +
illustrative/loader_program.rs (read-only sketch, like sched_ext reference)
demo using bpftool gen skeleton -L on a real built object. No ebpf_ metric
(inspection chapter, like Ch35) — no In-Grafana line, by design.
Next: Ch 50 user ring buffer (user space → BPF direction).
r30.0 — Advanced surface: user ring buffer (Ch 50) — UNVERIFIED (experimental)
Ch 50 (user-rb) — BPF_MAP_TYPE_USER_RINGBUF (kernel 6.1): the ring buffer
reversed — user space produces (reserve/submit), a BPF program consumes via
bpf_user_ringbuf_drain(&map, callback, ctx, flags); each sample arrives as a
bpf_dynptr (forward-ref to the dynptr chapter later in this part). When to use
vs a plain map: stream of variable-length messages vs a few settings. Drain
happens when the program runs (here a getpid tracepoint the loader triggers).
HONEST frontier: Aya knows the map type but producer + drain/dynptr wrappers
are settling → reference/user_ringbuf.bpf.c canonical, user-rb-ebpf an Aya
sketch (UserRingBuf::drain), flagged experimental. Example round-trips 1000
samples → AGG count/sum (expect 1000 / 500500). New diagram user-ringbuf.
New metric ebpf_userrb_messages_total (+ value_sum_total). In-Grafana line
present. Renumber note: plan row split (50 done, 51 userspace eBPF next).
Ch 51 (userspace-ebpf) — eBPF is a portable bytecode ISA; user-space
runtimes run the same instructions with no kernel/root. Covers uBPF (C,
original; OVS/DPDK), rbpf (Rust port: interpreter + x86-64 JIT + assembler),
and bpftime (user-space uprobes/USDT, ~10x faster, shares maps with kernel,
runs libbpf/Aya programs). Example is HOST-ONLY (no lab VM, no root — the
point): rbpf assembles a tiny program (ldxb/call/exit), registers a Rust
helper (key 1 = double), runs interpreter + JIT, asserts they agree (->42),
and –disasm shows it’s ordinary eBPF. Notes rbpf’s toy verifier vs kernel’s.
New diagram userspace-ebpf. No ebpf_ metric / no In-Grafana line, by design
(nothing touches the kernel or OTLP). Single-crate example (no -ebpf/-common).
Plan Phase 9 resynced: 52–57 now mapped one chapter per row (kfuncs, bpf
token, bpf wq, struct_ops general, dynptr/arena, bpf iters).
USER FEEDBACK (repeated): advanced-part chapters had grown sparse on context/
explanation. CORRECTED here and going forward: restore the full depth standard
(full handler walked the way the verifier reads it, the WHY behind each choice,
flags/mechanics in prose, stability caveats, real cross-check) even when Aya
code must be flagged. Ch 52 = 1711 words vs ~1050-1120 for the sparse ones.
Ch 52 (kfunc-task) — kfuncs: the modern typed alternative to the frozen
helper UAPI. WHY helpers stopped scaling (permanent ABI commitment; can’t take
typed kernel pointers). kfuncs = BTF-registered kernel fns, typed pointer args,
per-fn flags, modular per-subsystem registration, NO ABI stability (pin
kernels). Flags explained: KF_ACQUIRE/KF_RELEASE (ref tracking), KF_RET_NULL
(forced null-check), KF_TRUSTED_ARGS, + KF_SLEEPABLE/KF_RCU/KF_DESTRUCTIVE/
KF_ITER_*. Worked example: bpf_task_from_pid (KF_ACQUIRE|KF_RET_NULL) +
bpf_task_release (KF_RELEASE), walked per-line as the verifier sees it; the
verifier REJECTS the load if the release is dropped on any path (README invites
deleting it to watch). Declared as extern fns resolved via BTF; task_struct from
aya-tool vmlinux (placeholder shipped; CO-RE → Part 9). New diagram kfuncs
(helpers vs kfuncs + acquire/release band). New metric ebpf_task_lookups_total
{result}. In-Grafana line present. UNVERIFIED bits: kfunc presence in kernel
BTF, aya kfunc-declaration + vmlinux/BTF ergonomics, verifier-rejects-on-leak.
Next: Ch 53 bpf token (delegating BPF into unprivileged containers) — at depth.
Ch 53 (bpf-token) — kernel 6.9 (Andrii Nakryiko). The capability problem:
BPF needs init-ns CAP_BPF (+ CAP_PERFMON/CAP_NET_ADMIN); BPF isn’t namespace-
scoped so there’s no namespaced CAP_BPF → pre-6.9 was all-or-nothing for
containers. Token model: privileged runtime mounts a userns-bound bpffs with
four delegation axes (delegate_cmds/maps/progs/attachs, bitmasks); trusted
unprivileged container opens it, derives a token via BPF_TOKEN_CREATE, passes
token_fd to bpf(PROG_LOAD/MAP_CREATE/…); kernel checks the token not init-ns
caps. Bounded to owning userns; trust explicit (CAP_BPF still checked in the
container’s own userns at create). Runtime wiring: LXD/Incus
security.delegate_bpf.*, systemd. Aya angle: token is a property of HOW you
load (loader threads token_fd through load/map/link) — EMERGING in Aya;
libbpf does it via bpf_token_path; program unchanged. Example: demo mounts a
delegated bpffs and prints the policy (runnable privileged half) +
illustrative/loader_with_token.rs (Aya sketch, flagged). New diagram bpf-token.
Control-plane feature → no ebpf_ metric / no In-Grafana line, by design.
Kept at FULL DEPTH (the restored standard): ~ richer prose, problem framing,
security reasoning, runtime integration, cross-check.
Next: Ch 54 BPF timers & workqueues (deferred work inside the kernel).
Ch 54 (bpf-timer) — deferred work for run-to-completion programs.
Shared model: timer/wq lives in a MAP VALUE (struct bpf_timer/bpf_wq field),
init → set_callback → start, callback (map,key,value) runs after the program
returns; map must have a user ref (fd/pin) or -EPERM; map lifecycle owns the
timer; pending timer keeps the prog loaded. TIMERS (5.15): softirq, non-
sleepable, self-rescheduling → in-kernel periodic aggregation with ZERO
user-space wakeups (the worked example: count/arm/tick computing per-second
rate, walked per-line). WORKQUEUES (6.10, Tissoires, kfunc-based): “bpf_timer
in process context”, SLEEPABLE callback → fast-path/slow-path pattern; choose
timer for cheap periodic, wq when deferred work must sleep. Aya angle: helper/
kfunc calls are fine; the hard part is callback-as-subprogram → C reference
canonical (reference/timer.bpf.c), aya-ebpf a flagged sketch; USER side is
real Aya (holds map open, arms once via execve, reads rate, observable gauge).
New diagram bpf-timer. New metric ebpf_timer_events_per_sec. In-Grafana line
present. Kept at FULL DEPTH.
Next: Ch 55 struct_ops (general) — implementing kernel policy interfaces in
BPF, the mechanism beneath Part 6’s schedulers.
Ch 55 (structops-cc) — generalizes Part 6’s sched_ext. The inversion:
normal BPF reacts at a kernel-owned hook; struct_ops has the kernel define an
INTERFACE (struct of fn ptrs / vtable) and BPF supply the whole impl, which
the kernel calls at the interface’s call sites — BPF as a pluggable kernel
module. Mechanism: BPF_MAP_TYPE_STRUCT_OPS map holds the struct (SEC
.struct_ops.link), SEC(“struct_ops/") programs fill slots, BTF
reflection matches members by name+kind+size, register via a link installs;
verifier permits writing the interface's required fields. Worked example: a
minimal Reno-style tcp_congestion_ops (ssthresh/cong_avoid/undo_cwnd) walked
as the TCP stack runs it; tcp_slow_start/tcp_cong_avoid_ai as exposed building
blocks; select via sysctl/setsockopt. Generalization: same machinery = sched_ext,
HID-BPF, bpf Qdisc, FUSE. Aya struct_ops authoring EMERGING → reference/cc.bpf.c
canonical + registered via bpftool struct_ops register (real, no Aya needed) +
illustrative/cc_aya.rs sketch. New diagram structops. Control-plane → no ebpf_
metric / no In-Grafana (observe via sysctl tcp_available_congestion_control,
bpftool struct_ops show, ss -ti). FULL DEPTH maintained.
Next: Ch 56 dynptr & bpf arena (variable-length data + shared large memory).
Ch 56 (dynptr-arena) — escaping fixed-size memory. DYNPTR (5.19, Koong): a
verifier-tracked HANDLE to a variable-length region; bpf_dynptr_data(off,len)
→ bounds-checked slice (len const), read/write copy, slice/slice_rdwr for
skb/xdp (non-contiguous, may return NULL → null-check). Abstracts over backing
(local/ringbuf/skb/xdp/user-ringbuf) — the thing Ch50’s callback received;
slice invalidated when dynptr is. Worked: ring-buffer dynptr emitting
variable-length records (exactly len bytes). ARENA (6.9, Starovoitov):
BPF_MAP_TYPE_ARENA sparse shared region (≤4GB), bpf_arena_alloc_pages +
__arena pointers → build real lists/trees/hashes with normal pointer ops;
user mmaps zero-copy bidirectional; replaces faking pointers with array-map
indices; use cases KV accelerator / in-kernel structures / BPF heap. Both
emerging in Aya → reference/{dynptr_ringbuf,arena_list}.bpf.c canonical;
dynptr-ebpf aya rendering (fixed Record + logical len, since aya dynptr reserve
emerging); REAL Aya loader reads variable-length records → ebpf_dynptr_records_total
(In-Grafana). Arena compiled+loaded via clang -D__BPF_FEATURE_ADDR_SPACE_CAST
bpftool. Two diagrams dynptr + bpf-arena. FULL DEPTH.
Next: Ch 57 BPF iterators — closes the Advanced part.
Ch 57 (bpf-iter) — the last inversion: kernel walks a SET (tasks/sockets/
map elements) and calls the program ONCE PER ELEMENT; you read fields directly
and emit via seq_file → cat. Push filtering into the kernel vs parsing /proc.
Two entities: iterator PROGRAM TYPE (SEC(“iter/"), ctx = bpf_iter__X
with meta{seq,seq_num} + element ptr; seq_num==0 header, NULL element summary;
BPF_SEQ_PRINTF; pin via `bpftool iter pin`, read via cat/read(); map-elem iter
takes `map MAP`) and OPEN-CODED iterators (bpf_for/bpf_repeat + bpf_iter_*
KF_ITER kfuncs from Ch52 — bounded loops in any program). Worked: iter/task
process table (tgid/pid/comm). Aya iterator support emerging → reference/
task_iter.bpf.c canonical + bpftool iter pin (real, no Aya) + illustrative/
iter_aya.rs sketch. New diagram bpf-iter. Inspection chapter (output IS the
dump) → no ebpf_ metric / no In-Grafana, stated explicitly. FULL DEPTH.
PART 8 (Advanced kernel surface, Ch 48–57) COMPLETE. Next: Part 9 Operating
eBPF (58–62), opening with the CO-RE deep-dive forward-ref’d throughout.
Reader flagged the conflated “BTF + musl static binary” line. FIXED: CO-RE
gives KERNEL-version portability and is libc-INDEPENDENT (our loader is a
glibc Fedora binary scp’d to target — all a homogeneous Fedora fleet needs).
A musl-linked STATIC loader is a SEPARATE, optional axis for userspace DISTRO
portability (one binary across unlike distros); musl is a build target you opt
into (x86_64-unknown-linux-musl), Alpine’s default — NOT Fedora/UBI (glibc).
Edited both the “How Aya does it” passage and the What-you-learned bullet.
Reader asked for detailed background on glibc vs musl, and on Clang/the eBPF
tooling, in a Foundations chapter. Added to Ch 4 (Rust + Aya toolchain), the
natural home (already owned bpf-linker + kernel tooling). Three additions:
(1) “What’s actually compiling this: LLVM, Clang, and rustc” — eBPF is LLVM-
BPF-backend bytecode; C path = Clang, Rust/Aya path = rustc + bpf-linker (no
Clang); Clang still needed for reference .bpf.c, vmlinux.h, and aya-tool
(bindgen/libclang). (2) “The eBPF tooling landscape” — replaces the brief
VM-tooling note: bpftool/libbpf/BCC/bpftrace/Clang-LLVM/pahole-dwarves/
llvm-objdump/perf, build-on-laptop vs inspect-on-guest, Fedora-repos policy.
(3) “The loader’s libc: glibc vs musl” — loader is an ordinary userspace
binary; glibc (Fedora/UBI default, dynamic, forward-compat) vs musl (Alpine
default; on Fedora a build target x86_64-unknown-linux-musl, static); dynamic
glibc right for our homogeneous Fedora fleet, static musl for cross-distro;
EXPLICITLY separate axis from CO-RE (userspace/distro vs kernel-version).
Lays the foundation the Ch58 r38.1 correction referenced. Intro tweaked (“no
Clang in the loop for the programs you write”), duration 25→35 min,
description quoted, checklist gained two understanding bullets. Ch4 1217→2258 words.
r38.3 — Ch 4 honesty pass: where the kernel side is C, not Rust
Reader correctly noted the “no Clang for the programs you write” line was too
clean — several chapters DO use C in kernel space because Aya cannot author
the feature yet. Audited the repo and added an explicit caveat to Ch 4 listing:
KERNEL-SIDE IS C (loaded via bpftool/scx, user side Rust/bpftool): Ch 43-44
sched_ext (scx_*.bpf.c; aya crate is a companion observer), Ch 55 struct_ops
(cc.bpf.c via bpftool struct_ops register), Ch 57 iterators (task_iter.bpf.c
via bpftool iter pin), Ch 56 arena (arena_list.bpf.c via clang+bpftool).
C-CANONICAL + flagged/approximate aya program built: Ch 50 user-rb, Ch 54
timer, Ch 56 dynptr, Ch 58 CO-RE. Intro softened (“great majority … no Clang
… a handful of advanced chapters are the exception”). Framed as: goal is
all-Rust; 2026 reality is the newest kernel surface speaks C first, so the
toolchain carries both. Ch4 now ~2480 words.
Ch 59 (bpf-lifecycle) — second Part 9 chapter. Default lifetime = loader’s
fds → dies on exit; operating means decoupling. THREE PILLARS: (1) PINNING
(bpffs) decouples lifetime — loader disposable, privilege separation
(privileged pins, unprivileged from_pin; cf Ch53 token); aya
EbpfLoader::map_pin_path / MapData::from_pin / FdLink::pin. (2) LINKS + atomic
BPF_LINK_UPDATE — swap program under a LIVE link, new live before old gone →
zero gap (no dropped packets/events); XDP replace flags / tcx ordered links /
libxdp dispatcher. (3) PINNED MAPS reused across versions → state (counters/
conntrack/LRU) survives upgrade = hot-swap vs cold restart. Orchestrators:
L3AF (LF Networking/Walmart — full lifecycle, chaining, GRACEFUL RESTART of
control plane w/o touching data plane, CO-RE, k8s) and bpfman (CNCF, k8s-native
pin/manage). Example: real Aya — counter program, map_pin_path + FdLink.pin,
exits leaving pins, program keeps counting (bpftool shows pinned link w/ no
owner + climbing map), run 2 reuses pinned map → count CONTINUES. Metric
ebpf_service_events_total (In-Grafana, smooth across handoff). New diagram
bpf-lifecycle (3 pillars). link_update atomic-swap ergonomics flagged evolving.
FULL DEPTH.
Next: Ch 60 hardware/accelerator offload (program runs off the host CPU).
Ch 60 (offload) — run eBPF off the host CPU. XDP THREE MODES: generic(SKB,
emulated, slow), native(DRV, in driver, HOST CPU — Ch32-36), offload(HW, on
the NIC). Aya selects via XdpFlags::HW_MODE/DRV_MODE/SKB_MODE. TRUE HW OFFLOAD:
driver JITs eBPF → NIC machine code; flagship Netronome Agilio NFP (now
Corigine), ~5.5x vs host iptables; verifier stricter (NIC’s helper/map subset).
HONEST CAVEAT: hardware narrow & waning — NFP basically the only lineage;
BlueField-2 does only DRIVER-mode XDP, offload “not supported”; virtio lab NIC
won’t offload → falls back. DPU = practical 2026 answer: BlueField-3 = full ARM
Linux on the NIC; run ORDINARY eBPF on its Linux (switching plane redirects to
ARM cores before host); Aya UNCHANGED, cross-compile aarch64 + deploy (ties to
Ch58 CO-RE). FRONTIER: FPGA synthesis hXDP/eHDL (research, 10-100x); GPU/AI =
eBPF filters near wire + GPU/Tensor inference on telemetry (NVIDIA Morpheus) —
NOT eBPF-on-GPU (that’s research). Example: real Aya XDP_PASS counter, attach
walks HW→DRV→SKB and REPORTS engaged mode (expect DRV/SKB on virtio); metric
ebpf_offload_packets_total (In-Grafana). New diagram offload. FULL DEPTH +
deliberately careful about marketing-vs-deployable.
Next: Ch 61 power & efficiency (measuring/reducing energy cost).
Ch 62 (correlation) — sets up the capstone. We used only METRICS for 60
chapters; otel-lgtm also runs Tempo(traces)+Loki(logs)+Pyroscope(profiles) +
OTel Collector + Grafana. ACCURACY: otel-lgtm uses PROMETHEUS (not Mimir);
MIMIR = production Prometheus-compatible swap (remote-write, PromQL, 1B series,
multi-tenant) — Grafana points at it via Prometheus datasource type, queries
unchanged. Trace = request as parent/child SPANS sharing trace_id; W3C
traceparent header propagates it across services. Grafana CORRELATION wiring:
exemplars (metric→trace, exemplarTraceIdDestinations), Tempo tracesToLogs/
tracesToMetrics, Loki derivedFields (log→trace), Tempo→Pyroscope. eBPF joins
the trace by CAPTURING the id (L7/uprobe — Ch14/29/45) or via OBI (Ch46), or
correlating by time+pid+service (kernel doesn’t natively know trace_id) — the
capstone’s bridge. Example: instrumented FastAPI (Podman/UBI py3.14) emitting
span+metric(app_requests_total)+trace-stamped log on /work; curl with
traceparent; Grafana datasource correlation.yaml (exemplars/tracesToLogs/
derivedFields). New diagram signal-correlation. Backend/app chapter → no ebpf_
metric of its own by design (eBPF correlation = capstone). FULL DEPTH.
Next: Ch 63 CAPSTONE — one curl → Quarkus + FastAPI (Podman) → spans(Tempo) +
metrics(Prometheus) + logs(Loki) + eBPF/Aya kernel view, all on one trace_id.
r43.0 — CAPSTONE: one request, every layer (Ch 63) — UNVERIFIED (full depth)
Ch 63 (capstone) — the grand integration, closing Part 9. Scenario: curl/
Postman + traceparent → FastAPI /checkout → calls Quarkus /inventory (context
propagated) → ONE distributed trace, both services’ spans. Three app signals
on one trace_id: spans→Tempo, app_requests_total+latency(exemplars)→Prometheus,
trace-stamped logs→Loki. FOURTH VIEW: Aya socket observer (sys_enter_read per-
comm counts → ebpf_capstone_syscalls_total{comm}) = kernel-side truth. Joining
eBPF to the trace: (a) L7 traceparent extraction (OBI/uprobe, reference/
l7traceparent.bpf.c canonical-illustrative), or (b) correlate by time+pid+
service (runnable path; demo captures trace_id from response). Read end-to-end
in Grafana: Tempo trace (both services) → span→Loki logs → metric exemplar→
trace → ebpf_capstone* for the window = app intent + app state + kernel reality
in one investigation. Example: fastapi-app (Podman/UBI py3.14, calls Quarkus),
quarkus-app (Podman/UBI openjdk-25 + Quarkus 3.33, auto-instrumented), capstone
Aya observer + loader, reference L7 .bpf.c, compose.yaml, demo orchestrates +
Grafana walkthrough. New diagram capstone. Completes Ch46’s three-signals story
with full LGTM correlation + eBPF. FULL DEPTH. UBI openjdk-25 tag + Quarkus
build flagged unverified.
PART 9 (Operating eBPF, Ch 58–63) COMPLETE. Next: Part 10 (optional field
guide) — bpftrace+python, bpftool+python, BCC tools tour (w/ diagrams); then a
closing retrospective. TODO logged (_plans/todo.md): tooling-coverage setup pass.
r44.0 — Part 10 OPENS: bpftrace from Python (Ch 64) — UNVERIFIED (full depth)
New PART 10 “Field guide” (_parts/field-guide.md, order 10, part_name “Field
guide”) — OPTIONAL tool-focused chapters on the validation CLIs used all book,
each driven from Python.
Ch 64 (bpftrace-python) — bpftrace = awk-for-the-kernel DSL (probes/
predicates/builtins/maps: count/hist/lhist), compiles via LLVM (Ch4). KEY:
-f json → NDJSON, one {“type”,”data”} per line (attached_probes/printf/map/
hist buckets/value/stats). Python subprocess wrapper reads stdout, json.loads,
dispatch by type → live top-N table / histogram. Boundary: bpftrace gathers,
Python decides use (table/alert/OTel). bpftrace = exploration/validation (no
build/deploy); Aya = production (typed/embeddable/one binary) — find signal w/
bpftrace, ship w/ Aya. Example: stdlib-only bpftrace_tool.py + programs/
{syscount,readsize}.bt + demo (runs on VM where bpftrace lives). New diagram
bpftrace-python. Terminal tool → no Grafana panel by design (OTLP hook noted).
FULL DEPTH.
DEFERRED: re-sync Ch0 outline + PRD outline tables once Part 10 complete (new
part + chapters). TODO (_plans/todo.md) tooling-coverage pass still open.
Next: Ch 65 bpftool from Python; Ch 66 BCC tools tour; then retrospective.
r44.1 — Ch 64 expanded with working bpftrace programs (reader request)
Reader: add several working examples (explanations already good). Added 7 more
runnable .bt programs mirroring earlier chapters: opensnoop(Ch9), execsnoop(Ch11),
killsnoop(Ch38), runqlat(Ch21), profile(Ch23), vfsstat, tcpconnect(Ch27) — plus
existing syscount + readsize(Ch24). Three output shapes: streams(printf),
tables(map), histograms(lhist). Upgraded bpftrace_tool.py: –list (with per-file
descriptions), -e/–oneliner inline programs, handles printf streams + map/hist.
Demo now lists + runs three (syscount/execsnoop/runqlat). Chapter gained a
“bundled programs” table (program → shows → output → mirrors). Same
several-working-examples standard to carry into Ch65 (bpftool) + Ch66 (bcc).
r45.0 — Field guide: bpftool from Python (Ch 65) — UNVERIFIED (full depth, 8 examples)
Ch 65 (bpftool-python) — bpftool inspects/manages LOADED objects (our
cross-check all book: prog/map/link show, map dump, struct_ops register, iter
pin). -j → JSON arrays. prog object: id/type/name/tag/run_time_ns/run_cnt/
bytes_memlock/map_ids/pids(5.8+). run_time/run_cnt need sysctl
kernel.bpf_stats_enabled=1 (5.1+, slight perf cost). Wrapper bpftool_tool.py
(stdlib) = 8 working commands: progs (inventory), top (avg ns/run, –enable-stats),
maps, dump <name|id>, links, net (xdp/tc), features (from feature probe), audit
(join prog+link → holders+attachments). 4-line bpftool() helper; audit shows
cross-object correlation. Ops value: read-mostly ground truth (what’s loaded,
what it costs, who owns it), repeatable across a fleet. Demo loads a throwaway
bpftrace probe then runs progs/maps/links/audit/top/features. New diagram
bpftool-python. Terminal tool → no Grafana. FULL DEPTH + several examples.
Next: Ch 66 BCC tools tour (closes field guide); then retrospective + Ch0/PRD
outline re-sync + tooling-coverage pass (_plans/todo.md).
r46.0 — Field guide: BCC tools tour (Ch 66) — UNVERIFIED (full depth, several examples) — CLOSES PART 10
Ch 66 (bcc-tools) — BCC = ~100 ready-made tracers (execsnoop/opensnoop/
biolatency/runqlat/tcpconnect/profile — the cross-check names all book). TWO
GENERATIONS: classic BCC (Python + inline C, RUNTIME-compiled via Clang/LLVM →
needs clang+llvm+kernel headers matching uname -r; /usr/share/bcc/tools on
Fedora, -bpfcc suffix on Debian; “Failed to compile BPF text” gotcha) vs
libbpf-tools (precompiled CO-RE binaries, no runtime deps, BTF relocation — Ch58
story; Fedora libbpf-tools pkg). Suite table grouped (process/sched/block/net/
mem), several mirror our chapters. BCC emits COLUMNAR TEXT (not JSON) → Python
pattern = resolve/run/parse. Example bcc_runner.py: resolver (/usr/share/bcc/
tools→PATH→-bpfcc), run for duration + SIGINT to flush, parsers for execsnoop/
opensnoop/tcpconnect (top-N), capture+print for histogram tools (biolatency/
runqlat/profile). hello_bcc.py = minimal BCC-library inline-C kprobe (BPF(text=)
compiled at runtime) — contrast w/ Aya AOT Rust binary = book in miniature.
Positioning: BCC depth + quick custom (trace/argdist); bpftrace quick DSL;
bpftool inspect loaded; Aya production. New diagram bcc-tools. Terminal → no
Grafana. FULL DEPTH + several examples.
PART 10 (Field guide, Ch 64–66) COMPLETE. Technical body COMPLETE (Ch 0–66).
REMAINING: (1) retrospective final chapter (detailed, across the whole arc;
user wants it as the finale). (2) Ch0 outline + PRD outline-table RE-SYNC (Part
10 + chapters 64–66 + retrospective). (3) tooling-coverage pass (_plans/todo.md;
note bcc-tools needs clang+kernel-devel — add to that audit). (4) skill
chapter-template terminal-view+Grafana-metric requirement still pending.
Reader nits addressed: (1) field-guide docs now use EXPLICIT example paths
(examples/NN-name/file) matching READMEs — Ch64/65/66. (2) Top-level README:
replaced ASCII lab-model with assets/diagrams/lab-topology.svg image + refreshed
prose; fixed Mimir->Prometheus (otel-lgtm uses Prometheus; Mimir = prod swap);
Status updated to “technical body complete Ch0-66, field guide + capstone,
retrospective pending, all unverified”. (3) Ch65 “eight working commands”
converted from bullets to a TABLE (command|what|built-on); scanned all chapters
— no other >=6-consecutive-bold-bullet sections remain. (4) Ch66 bcc_runner.py
expanded: 8 parsers now (execsnoop/opensnoop/statsnoop/tcpconnect/tcpaccept/
tcplife/killsnoop/syscount) + capture list (biolatency/runqlat/profile/biotop/
cachestat/…) + –list; chapter gained a mode table. (6) Ch66 links upstream
github.com/iovisor/bcc.
NEW DIAGRAMS (4, our house-style versions of the classic Brendan Gregg layouts;
curated subsets, .excalidraw source shipped so they are editable; credited to
bcc/bpftrace projects):
bpf-architecture (Ch5) — user-space bytecode->load->verifier->BPF<->maps/events
(item 8, like the iovisor user/kernel workflow figure).
linux-events-bpf (Ch5) — event landscape: kprobes/uprobes/tracepoints/PMCs/
software events across the stack + kernel-version availability (item 9).
bpftrace-tools-map (Ch64) — bpftrace tools mapped to the layer each observes
(item 7).
bcc-tools-map (Ch66) — bcc tools mapped to the layer each observes (item 5).
Ch5 (foundations) expanded 1449->2000 words: new “Anatomy: how bytecode becomes
a running probe” (bpf-architecture) + “The wider event landscape”
(linux-events-bpf) sections (item 8 “a bit more on eBPF”).
Note: layered maps are CURATED subsets (teaching), not full ~100-tool reproductions.
Peripheral note text was #888 (light/unreadable). Re-emitted all 4 diagrams with
DARK labels (#1d1d1d, size 13) + amber bold titles. Fixes light text on
bpftrace-tools-map (Fig 64.2), bcc-tools-map (Fig 66.2), linux-events-bpf (5.4).
Fig 5.4 (linux-events-bpf): arrows were scattered/random-looking. Redesigned so
each tracepoint subsystem label sits next to its target layer with a clean short
arrow that lands on the right box: syscalls:->syscall, sock:->sockets,
sched/signal/timer/workqueue:->scheduler, kmem/vmscan/writeback:->VM, ext4/jbd2:->FS,
block/scsi:->block, net/skb:->net, irq:->drivers. Dynamic-tracing span arrow +
kprobes(4.1)/uprobes(4.3)/tracepoints(4.7)/PMCs+sw-events(4.9) version callouts +
explicit “Fedora 44 modern 6.x kernel” banner.
Fig 5.3 (bpf-architecture): added dashed vertical USER|KERNEL divider; kept the
numbered steps; caption now spells out the circled steps.
Ch5 prose: “Anatomy” section now walks the full pipeline as 7 numbered steps
(compile->load->verify->JIT->attach->run->consume), tied to the diagram’s circled
numbers and the divider, ending with how the SAME kernel path is driven from
Python by the Part 10 tools. Landscape section now states Fedora 44 runs the 6.x
kernel with all listed capabilities. “Where bpftool and bpftrace fit” bullets ->
TABLE (tool | what it is | what for | try). Ch5 word count grew accordingly.
r47.2 — table cell wrapping fix (reader feedback)
Inline code rule (:not(pre) > code { white-space: nowrap }) made long code in
table cells unbreakable, pushing the whole table past page width so NO column
wrapped (Ch5 bpftool/bpftrace table). Fix in site.css: th,td { vertical-align:
top; overflow-wrap: anywhere } and :is(th,td) code { white-space: normal } so
cell content + inline code wrap; applies to every table site-wide. Also trimmed
the long bpftrace one-liner in that table cell.
r47.3 — table fix correction (reader feedback: r47.2 made it worse)
r47.2 used overflow-wrap:anywhere on th,td, which shrinks each column min-content
to one character; with width:100% + a code-heavy column, the prose columns
collapsed to 1 char/line. Corrected: keep cell sizing default; allow wrapping on
INLINE CODE only via :is(th,td) code { white-space: normal; overflow-wrap:
break-word }. break-word preserves min-content (longest token) so columns size
normally and only a too-long token breaks when it would overflow.
r47.4 — table code-wrap: fix CSS source-order/specificity (r47.3 still squeezed)
Root cause finally pinned: the cell code-wrap rule sat BEFORE :not(pre) > code
{ white-space: nowrap }; equal specificity -> later (nowrap) won, so cell code
stayed nowrap, forced the Try column wide, squeezed the prose column. Fix: moved
the override to AFTER the Code section and raised specificity with a leading
=> .
anywhere keeps code min-content small (only the code column yields), so the long
command wraps and prose keeps its width. (No overflow-wrap on th,td, so prose
columns never collapse.)
r47.2-47.4 fixed code WRAPPABILITY but the auto table-layout algorithm still
preferred showing the Try code on one line and shrank the flexible prose column.
Deterministic fix: table { table-layout: fixed } (with width:100%) so columns
share width equally and content wraps within its share; combined with the
table-cell code-wrap override (white-space:normal; overflow-wrap:anywhere) the
long command now wraps inside its fixed column. Applies to all tables (equal
columns); acceptable for the book’s reference tables (<=4 cols).
r48.0 — RETROSPECTIVE (Ch 67) — book narrative COMPLETE
New Part 11 “Retrospective” (_parts/retrospective.md, order 11). Ch 67 = the
detailed finale the reader asked for, “across the whole thing”: how the parts
built on each other (Part 0 foundations → observe → network/secure → frontier →
operate → field guide); what held constant (the four-move loop; Rust earns its
place at the verifier; maps the only channel; lab model; observability-by-default
three-signal spine; explore-with-tools/ship-with-Aya); how your mental model of
the kernel changed (spectator → participant); the honest unverified state (reader
promotes by running on Fedora 44 — by design); and where eBPF/Aya go next
(all-Rust kernel side maturing; observe→control via sched_ext/cpufreq_ext/LSM;
OBI zero-code; bpfman/L3AF operating layers; DPU/offload). New diagram book-arc.
Reflection chapter → no Aya example/Grafana panel by design; avoids “honest” in
prose per convention. ~ closing word + thanks.
BOOK NARRATIVE COMPLETE: Chapters 0–67, Parts 0–11.
REMAINING CLEANUPS (post-finale): (1) Ch0 outline + PRD outline-table re-sync to
reflect Parts 10–11 + chapters 64–67. (2) tooling-coverage setup pass
(_plans/todo.md; incl. turbostat, clang+kernel-devel for classic bcc). (3) skill
chapter-template: require naming terminal-view + Grafana metric in observe.
r49.0 — post-finale cleanups (the three logged TODOs) — DONE
Outline re-sync: Ch0 grouping table + PRD §5 table now reflect Operating eBPF
58–63 (added signal correlation + capstone), the optional Field guide 64–66,
and the Retrospective 67.
Tooling-coverage pass: audited every CLI referenced across chapters vs the Ch2
cloud-init. Missing-and-referenced added to the VM seed: kernel-tools (turbostat,
Ch61 — confirmed Fedora package), libbpf-tools (Ch66), jq (Ch62/65/66), openssl
(Ch14/62/63). Added a “What the lab’s tooling is for” TABLE to Ch2 (tool →
package/source → where (VM vs laptop) → value), covering kernel-side tools
(cloud-init) and host Rust chain (rustup, Ch4). Tools referenced=0 (ethtool,
xdp-tools, powerstat, cpupower) intentionally NOT added.
Skill chapter-template: ebpf-aya-tutorial skill updated so “Build, deploy,
observe” MUST name BOTH the terminal live-view AND the specific Grafana ebpf_*
query (or state “no Grafana panel by design”); added as depth-standard item 5.
Shipped as ebpf-aya-tutorial-skill-updated.zip.
Book COMPLETE (Ch 0–67, Parts 0–11) + all backlog cleared.
r50.0 — ADDENDUM: HE observability (Ch 68, new Part 12 “Addenda”)
Capstone addendum requested by the reader. New Part 12 “Addenda” (order 12);
Ch 68 “Capstone addendum: observing a homomorphic-encryption workload”.
Placement: kept as a standalone Addenda part rather than inserting after Ch 63,
which would renumber 64–67 and break every cross-reference + figure number; the
chapter is written as a direct extension of the Ch 63 capstone instead.
Content covers (per the reader’s asks): what HE is (compute on ciphertext;
partial/somewhat/FHE; bootstrapping; LWE/RLWE; TFHE-rs = Zama pure-Rust) and
WHY you want metrics without observing data (operands are ciphertext = impossible
to read; operator is the threat = forbidden to read; so measure behavior/latency,
never values — eBPF times symbols + samples stacks, not operands). Ties explicitly
to the capstone (same uprobe/OTel machinery, inverted goal).
Example examples/68-he-observability/: he-workload (TFHE-rs; he_keygen/encrypt/
compute/decrypt as #[no_mangle][inline(never)] extern "C" boundaries),
he-observer-ebpf (1 uprobe he_enter on all 4 symbols + 1 uretprobe per symbol;
START HashMap keyed by pid_tgid + EVENTS RingBuf), he-observer loader (attach
pairs, drain ring, f64_histogram ebpf_he_op_latency_seconds{op}, OTLP), he-common
(Sample{op,dur_ns}, no operands), demo.sh (workload starts 4s late so the probe
attaches first and catches one-shot keygen). New diagram he-observability.
Observe section names BOTH the terminal live-view AND the Grafana metric
(ebpf_he_op_latency_seconds) per the r49 skill rule. License note for TFHE-rs.
All unverified. Book now: Ch 0–68, Parts 0–12.
r50.1 — HE addendum enhancements (reader-requested)
Second op: added he_add (homomorphic addition, op id 4) alongside he_compute
(multiply). Wired through workload (boundary fn + main loop encrypt→add→compute→
decrypt), ebpf (he_add_ret uretprobe), loader (SYMBOLS/RET_PROGRAMS 4→5, op_name
4=>”add”). Diagram re-emitted with he_add chip + {op} set incl add. Chapter notes
the add-vs-compute cost gap on one histogram.
Profiling: new subsection “Where the time goes: profiling he_compute to the NTT”
— eBPF on-CPU stack sampling (Ch23) attributes compute’s time to NTT/polynomial
routines; viewable as a Pyroscope flamegraph panel (otel-lgtm bundles Pyroscope);
stays data-blind (addresses, not operands); symbol/frame-pointer caveat noted.
Added examples/68-he-observability/profile.sh (bcc profile -f folded stacks +
flamegraph/Pyroscope guidance). Cross-check adds he_add funclatency contrast.
All unverified.
Reader-requested appendix on Performance Co-Pilot × eBPF. Ch 69 covers: what PCP
is (pmcd + PMDAs; self-describing metrics — PMNS/PMID/units/semantics/instance
domains; live + pmlogger archives); how it’s used (pminfo/pmrep/pmstat/pmchart,
pmlogger replay, pmie alerts, pmproxy/pmseries); what it’s useful for (always-on
low-overhead recording = flight recorder, retrospective root-cause, fleet);
how it’s extended (PMDAs in C/Python/Perl; pmdaopenmetrics scrapes Prom/OpenMetrics);
and the eBPF intersection — pmdabpf (BPF CO-RE ELF modules → bpf.* metrics; bpf.conf,
modules dir, ./Install), pmdabpftrace (bpftrace scripts as metrics + histograms),
legacy pmdabcc, bridging the book’s Aya ebpf_* via pmdaopenmetrics, and grafana-pcp
in the same Grafana. Plus a when-to-use-which vs the book’s push/OTLP model.
Example examples/69-pcp-ebpf/: demo.sh (install pcp+pcp-pmda-bpf+bpftrace+grafana-pcp,
enable runqlat/biolatency, ./Install, pminfo/pmrep, pmlogger record+replay, optional
PROM_URL openmetrics bridge), openmetrics-ebpf.url.example, README. New diagram pcp-ebpf.
Facts grounded via search (pmdabpf manpage + Fedora pcp-pmda-bpf pkg + RHEL eBPF/PCP
blog + bpftrace PMDA README). All unverified. Book now: Ch 0–69, Parts 0–12.