Sanitizers and Dynamic Analysis
ThreadSanitizer and AddressSanitizer, Valgrind's Helgrind and DRD, what each tool catches and misses, and wiring them into CI.
What we’ll cover
- Cover ThreadSanitizer’s compile-time instrumentation approach to detecting data races.
- Cover AddressSanitizer’s approach to memory errors and where it overlaps with and differs from TSan.
- Cover Valgrind’s Helgrind and DRD as binary-instrumentation alternatives, with their own trade-offs.
- Give a clear picture of what each tool catches, what it misses, and the runtime cost of running it.
- Wire at least one sanitizer into a CI job that fails the build on a detected issue.
The previous chapter read one race and one deadlock off a single gdb session and a single TSan report — useful for the bug already in front of you, useless for the one nobody has triggered yet. This chapter is about the tools built to run continuously, against every commit, so that a race or a memory error gets caught before it ever needs a debugger at all. Two families do this work, for fundamentally different reasons: compile-time instrumentation (ThreadSanitizer, AddressSanitizer, UBSan) rebuilds the binary with extra checks baked into the generated code, cheap enough to run on every pull request; binary instrumentation (Valgrind’s Helgrind, DRD) re-translates an unmodified binary under a software CPU, slower by an order of magnitude but requiring no rebuild at all, which matters the day you need to check a binary you cannot recompile.
ThreadSanitizer
ThreadSanitizer instruments every memory access at compile time (-fsanitize=thread) with inline bookkeeping that maintains the same per-thread vector clock the previous chapter’s report relied on, so it catches exactly the class of bug that clock is built to track: any two unsynchronized accesses to the same location, at least one a write, for which no happens-before edge connects them, observed on the schedule the instrumented binary actually ran. That net is wider than a naive “same variable, two threads” check — it catches a shared container accessed through two mutexes that were each meant to protect it but used inconsistently across call sites, atomics marked memory_order_relaxed where the surrounding code assumed a stronger guarantee than relaxed provides, and accesses to file descriptors and other OS resources TSan models via its own syscall interception, not only plain memory.
What it misses falls into three categories. First, coverage: a race in a code path the instrumented run never executes, or in a precompiled third-party .so not itself rebuilt with -fsanitize=thread, produces no report regardless of how real the race is, because TSan can only reason about instructions it instrumented. Second, scale: TSan tracks a bounded number of threads per process (on the order of a few thousand, implementation-dependent), and a program that exceeds it can silently lose coverage on the overflow rather than failing loudly. Third, and subtlest, TSan’s own instrumentation changes timing — a race that depends on a specific narrow interleaving can become more or less likely to manifest once every memory access carries extra bookkeeping overhead, which cuts both ways: TSan sometimes surfaces races a plain run would almost never hit, and can just as easily mask one whose window only existed at un-instrumented speed. The cost for all this, well-documented across projects that run it continuously, lands around 5-15x wall-clock time versus an uninstrumented build — far from free, but cheap enough to run as a standard CI configuration rather than reserving it for an occasional manual pass.
AddressSanitizer
AddressSanitizer instruments memory accesses for safety rather than for synchronization, using shadow memory: one shadow byte tracks every eight bytes of application memory, and specific shadow values flag specific states — a redzone placed around every heap allocation to catch an access a few bytes past the end, and a freed block’s bytes repainted with a poison value the instant it’s released, so any further access through a dangling pointer trips immediately rather than silently reading or corrupting memory the allocator has already reused for something else. This is what makes ASan catch heap-buffer-overflow, stack-buffer-overflow, global-buffer-overflow, heap-use-after-free, use-after-scope, and double-free, each with the allocation and (where relevant) deallocation call stacks attached alongside the faulting access — stack-use-after-return specifically needs an opt-in runtime flag (ASAN_OPTIONS=detect_stack_use_after_return=1) because checking it costs enough that it isn’t default-on. ASan and TSan overlap in that both are compile-time instrumentation from the same sanitizer family, but they cannot usually be linked into the same binary — each wants to own the allocator and certain runtime hooks in ways that conflict — so a project that wants both runs two separate instrumented builds rather than one combined one. A detached-thread scenario is a clean example of where ASan’s coverage matters specifically in concurrent code: a thread that frees a buffer while a second, detached worker still holds a raw pointer into it produces a heap-use-after-free the instant that worker touches the freed memory, a bug born from concurrent lifetime management even though ASan itself has no model of threads at all — it only sees one thread do an invalid access to memory another thread already freed. The overhead is close to native by comparison, commonly cited around 2x, because the inline shadow-memory check costs a handful of instructions per access rather than a software-emulated CPU.
UBSan, usually built alongside ASan in the same compilation (-fsanitize=address,undefined), checks a different, narrower set of properties entirely — signed integer overflow, invalid enum values, misaligned pointer access, null-pointer arithmetic — none of which overlap with either TSan’s race detection or ASan’s heap/stack safety, so its absence from a given report is not evidence it isn’t working, only evidence that particular class of bug didn’t happen to be present.
Valgrind Helgrind and DRD
Valgrind takes the opposite approach entirely: it needs no rebuild and no compiler cooperation, because it operates on the compiled binary directly, re-translating every basic block through its own intermediate representation (VEX) before executing it under a virtual CPU, inserting checks around memory and synchronization operations as it goes. That is the entire point of reaching for it — a vendored binary with no build system you control, a release artifact you want to check exactly as shipped, or a target platform whose compiler has no sanitizer runtime ported to it at all are all cases where “rebuild with -fsanitize=” is not an available option and binary translation is.
Helgrind detects races through a hybrid lockset-and-happens-before algorithm: it tracks, for every shared memory location, the set of locks consistently held whenever that location is accessed, and flags a location accessed under two different, non-overlapping lock sets as a probable race, which lets it report certain lock-order risks (two code paths that could, in principle, deadlock) even in a run where the problematic interleaving never actually occurred — a useful complement to TSan’s stricter rule that it only reports a conflict actually observed happening. DRD is a second, independent Valgrind thread-checking tool using a closer-to-pure happens-before model, generally faster and with a narrower false-positive surface than Helgrind’s lockset heuristics on some codebases, at the cost of less detailed lock-order diagnostics — the practical choice between them on a given project is often just “whichever one produces a cleaner signal on this codebase’s locking idioms,” determined by trying both. Both run at roughly an order of magnitude slower than an uninstrumented binary (commonly cited in the 10-50x range for Valgrind’s tools generally), the cost of a software-emulated CPU checking every basic block rather than a few extra inlined instructions per access.
The practical division of labor follows directly from these trade-offs: TSan as the default day-to-day tool, since cheap-enough-for-every-CI-run and rebuild-is-trivial together beat Valgrind’s heavier, no-rebuild model for the common case of code you control and can recompile; Valgrind reserved for the case that actually needs its specific property — a binary you cannot rebuild, a platform without sanitizer-runtime support, or confirming a bug exists in the literal artifact about to ship rather than in a parallel instrumented build of it.
CI Integration
The right default is two separate, purpose-built CI jobs rather than one heavyweight one. An ASan+UBSan build (-fsanitize=address,undefined -fno-omit-frame-pointer, the frame-pointer flag kept for readable stacks) costs little enough, at roughly 2x, to run on every pull request as a required, blocking gate alongside the plain test suite — treat any nonzero exit from that build’s test run as a merge blocker, the same discipline Error Handling vs Exception Handling argues for any detected invariant violation. A TSan build, at 5-15x, is still cheap enough to run on every PR for most test suites, but is the first candidate to scope down to just the concurrency-relevant test targets if the full suite’s wall-clock time becomes a bottleneck, rather than dropping it to a less-frequent schedule — a race caught a day late because the gate only runs nightly is a race that already merged into every branch built against it in the meantime. Helgrind or DRD, at 10-50x, is the one tier where a nightly or weekly scheduled job is the right trade-off rather than a cop-out, specifically because Valgrind’s value is catching something the cheaper, rebuild-based tools structurally cannot (no sanitizer runtime available for a given target, or confirming the exact shipped binary) rather than catching the same race faster.
A TSAN_OPTIONS=suppressions=<file> (and the equivalent .supp mechanism for Valgrind) exists for a real but narrow case: a race inside a third-party dependency you cannot fix and cannot rebuild with instrumentation yourself. Treat every entry in that file as a reviewed, justified exception rather than a quiet escape hatch — a suppression file that grows without review is indistinguishable, in practice, from turning the gate off, and the distinction between “we investigated this and it’s a known, accepted library limitation” and “the gate was in our way” is exactly the distinction a reviewer needs to be able to draw from the file’s history.
The CI runner’s own environment deserves a specific check: sanitizer runtimes reserve large shadow-memory regions at process start, and a CI container with an aggressive ulimit -v or a restrictive memory-overcommit setting can reject that reservation before a single test runs, producing a sanitizer initialization failure that reads nothing like a test failure and gets misfiled as “the sanitizer job is flaky” rather than “the runner’s resource limits are too tight for this tool.”
Running in a container
Valgrind’s relationship to container security policy differs from both gdb’s and the compile-time sanitizers’, and in a useful direction: Valgrind does not use the kernel’s ptrace(2) mechanism at all, since it fully re-implements CPU execution in userspace rather than tracing a running process through the kernel’s debugging interface, so a hardened container that has dropped CAP_SYS_PTRACE entirely — exactly the posture that blocks gdb’s -p <pid> attach from the previous chapter — does not block Valgrind from running. What Valgrind does need is the flexibility to map and manage memory regions aggressively as it builds its shadow-memory and translation caches, which can run into trouble specifically under a sandboxed container runtime (one that intercepts and reimplements syscalls, such as gVisor) whose own syscall emulation layer was not built with a second full CPU emulator running on top of it in mind; a plain OCI runtime with standard capabilities is the safer target for a Valgrind-based CI job.
CPU quota and sanitizer overhead compound rather than add: a container capped at a quarter of a core running a TSan build’s 5-15x multiplier, or a Helgrind run’s 10-50x, turns a test that completes in seconds unthrottled into one that can threaten a CI job’s own timeout budget — easy to misdiagnose as a regression in the code under test when the actual cause is sizing the timeout against the tool’s native-speed runtime instead of its instrumented one. The fix is the same one earlier chapters apply to any other cgroup-throttled workload: size the budget against the quota and the instrumentation multiplier together, not either alone.
Standards: C++20 / C++23 / C++26
Sanitizers and Valgrind’s tools operate beneath the C++ standard, instrumenting generated code and intercepting runtime calls rather than consuming any particular language feature, so there is no three-way delta across C++20, C++23, and C++26 in how any of these tools are invoked or what their reports look like — the single block below shows the one piece of standard-adjacent syntax that does apply uniformly across all three, a compiler attribute for suppressing a specific, deliberately unchecked function.
// A deliberately lock-free, intentionally racy counter used only inside
// a benchmark harness that already accounts for the imprecision -- the
// same suppression syntax and semantics hold unchanged across C++20,
// C++23, and C++26, since it's a compiler attribute, not a language
// feature tied to any standard revision.
__attribute__((no_sanitize("thread")))
void benchmark_only_increment(int& counter) {
++counter; // TSan would otherwise (correctly) flag this
}
What does shift across standards is what the tools are being pointed at, not the tools themselves. C++20’s std::atomic and std::jthread, covered in Atomics and the Memory Model and Threads in Modern C++, are exactly what TSan already models precisely through its atomic interceptors and thread-lifecycle tracking — nothing about this chapter’s tools needed to change to support either feature. C++23 adds no library surface specifically relevant to dynamic race or memory-error detection. C++26’s proposed std::hazard_pointer and std::rcu (from P2530 and P2545, experimental, with no shipping compiler implementation as of this writing) raise a different verification question for whichever dynamic-analysis tooling eventually supports them: their safety property — no reader ever observes an object after it’s been reclaimed — is a global invariant over the lifetime of every participating thread, not a pairwise happens-before edge between two specific accesses the way TSan’s model directly targets, so confirming it dynamically will need either new tool support purpose-built for the reclamation scheme or a different verification strategy entirely, a question this book’s tools do not yet have a settled answer to.
What we learned
- Compile-time instrumentation (TSan, ASan, UBSan) rebuilds the binary with inline checks, cheap enough — roughly 2x for ASan, 5-15x for TSan — to run on every CI build; binary instrumentation (Helgrind, DRD) needs no rebuild at all but costs an order of magnitude more, the trade worth making only when recompiling isn’t an option.
- TSan’s vector-clock model and ASan’s shadow-memory redzones catch different bug classes (races versus memory-safety violations) and are usually built as two separate instrumented binaries rather than one, since the two sanitizers’ runtimes conflict when linked together.
- Helgrind’s lockset-and-happens-before hybrid can flag a lock-order risk that hasn’t manifested in any single run, a different signal from TSan’s stricter “only report what this execution actually hit,” and DRD trades some of that diagnostic detail for speed and a narrower false-positive surface.
- A sanitizer gate belongs in CI as a required, blocking job sized against both its own instrumentation overhead and the CI container’s CPU quota together — an under-sized timeout misdiagnosed as flakiness is as damaging to trust in the gate as never running it at all.
Next, observing concurrency turns from catching a defect before or during a test run toward watching a correct, already-shipped system’s concurrent behavior in production, where neither a debugger nor a sanitizer is available at all.