The instinct is to go hunting for a smaller base image or a leaner compiler install. That is the wrong lever. The toolchain is not bloat to be trimmed; it is build-time machinery that has no business being in the runtime image at all. The fix is to leave it behind.

What the 685 MB is doing

A first-pass Containerfile builds and runs in the same image. Install the compiler, copy the source, compile, set an entrypoint. It works, and it produces this:

Every one of those lines is present only because the build happened in the image you shipped. None of it is a dependency of the program. The compiler does not run in production. The Conan cache resolved your dependencies months ago. The .o files were consumed by the linker and never read again.

Multi-stage builds: the whole trick is one line

A multi-stage Containerfile declares more than one FROM. Each FROM begins a fresh image. You name the build image with AS, compile in it, then start a second image for runtime and copy just the finished binary across. Only the final stage is tagged; the build stage is discarded the moment the build finishes.

Here is the real build stage from the tutorial’s demo, lightly trimmed:

# Containerfile.ubi-multistage — build stage
ARG UBI_VERSION=10.2
FROM registry.access.redhat.com/ubi10/ubi:${UBI_VERSION} AS build
RUN dnf install -y --setopt=install_weak_deps=False \
        gcc gcc-c++ cmake ninja-build git \
    && dnf clean all
WORKDIR /src
COPY src/ ./src/
COPY CMakeLists.txt CMakePresets.json ./
RUN cmake --preset release \
 && cmake --build --preset release -j"$(nproc)" \
 && strip --strip-all build/release/demo-svc

Then the runtime stage starts over from a smaller base and copies one file:

# Containerfile.ubi-multistage — runtime stage
FROM registry.access.redhat.com/ubi10/ubi-minimal:${UBI_VERSION} AS runtime
RUN microdnf install -y --setopt=install_weak_deps=0 libstdc++ \
 && microdnf clean all
WORKDIR /app
COPY --from=build /src/build/release/demo-svc /app/demo-svc
USER 1001:1001
EXPOSE 8080
ENTRYPOINT ["/app/demo-svc"]

COPY --from=build /src/build/release/demo-svc /app/demo-svc is the entire mechanism. It reaches into the finished build image and lifts out one file. The compiler, the Conan cache, the object files, and the source tree stay behind in a stage that is never tagged and never pushed. They die with stage one.

Build time barely changes, because you are still compiling the same code. What changes is what ships. On ubi-minimal this gets the demo from 689 MB to about 114 MB. The next decision takes it the rest of the way.

Picking the runtime base is an incident-response decision

Red Hat’s Universal Base Image ships in three runtime tiers, and the gap between them is not about megabytes:

It is tempting to read that table top to bottom and reach for the smallest row. Resist that. The real question is what your on-call playbook does when the service misbehaves at 3 a.m.

ubi-minimal keeps a working bash and microdnf, so podman exec -it ... bash drops you into a live container and you can install ldd or poke around. That convenience costs roughly 85 MB over ubi-micro.

ubi-micro is about 14 MB before your binary lands. There is no shell, no dnf, no ldd, no strace. You cannot exec a bash into it because there is no bash to exec. If your binary needs libssl or libstdc++, you either link them statically or COPY them explicitly from the build stage. The tutorial’s ubi-micro variant goes fully static and lands at 26.4 MB:

# Containerfile.ubi-micro — runtime stage (binary is statically linked)
FROM registry.access.redhat.com/ubi10/ubi-micro:${UBI_VERSION} AS runtime
WORKDIR /app
COPY --from=build /src/build/release-static/demo-svc /app/demo-svc
# ubi-micro has no /etc/passwd lookups to worry about
USER 1001:1001
EXPOSE 8080
ENTRYPOINT ["/app/demo-svc"]

So the decision is: if your playbook starts with “shell into the container,” ship ubi-minimal. If your playbook starts with “attach an ephemeral debug sidecar that shares the PID namespace and carries gdb,” ship ubi-micro. The sidecar posture is where most production deployments land, and it is what makes the 26 MB image safe to operate rather than merely small.

A 26 MB image is opaque, so write down what’s inside it

Shrinking the image removes your ability to interrogate it. On ubi-micro you cannot run dnf list installed, you cannot ldd the binary, you cannot ask the image what it was built against. Six months from now a CVE drops against a range of libstdc++ versions and nobody can tell from the image whether you are affected.

The answer is to write the toolchain identity into the image metadata at build time, where a small image can still carry it cheaply:

LABEL org.opencontainers.image.title="demo-svc" \
      org.opencontainers.image.revision="${GIT_SHA}" \
      tutorial.toolchain="gcc-14" \
      tutorial.linkage="static" \
      tutorial.lto="thin"

podman inspect <image> | jq '.[0].Config.Labels' reads them back at incident time without running anything. The standard org.opencontainers.image.* keys cover provenance; a project-specific prefix records the things that bite C++ specifically: which libc and libstdc++ the binary expects, which micro-architecture it was tuned for, whether LTO or PGO is in play. The image is too small to hold the toolchain, so it holds the toolchain’s name tag instead.

The trap that makes this a C++ problem

A Go binary carries its runtime inside itself; an Uberjar carries the JVM’s class libraries. C++ does not work that way. A typical C++ binary depends on a specific libstdc++.so.6 and a specific libc.so.6, and those dependencies are implicit right up until they are fatal.

The fatal version looks like this. Build the service on one distro’s glibc and run it on an older one, and the dynamic linker goes looking at startup for a symbol version the runtime’s libc.so.6 does not carry:

/app/demo-svc: /lib64/libc.so.6: version `GLIBC_2.41' not found (required by /app/demo-svc)

The tutorial reproduces this on demand by building on Fedora 42 (glibc 2.41) and running on ubi-micro (glibc 2.39). The subtle part: compiling with -static-libstdc++ does not save you, because that statically links the C++ runtime while glibc stays dynamic, so a newer glibc symbol baked in at build time is still resolved against whatever libc.so.6 the runtime ships. Two fixes work. Link glibc statically too (-static), which is what the 26.4 MB ubi-micro variant does and why it has no libc to mismatch. Or, more directly, use the same UBI release for both stages so the build glibc and the runtime glibc are identical by construction. UBI-10-on-UBI-10 sidesteps the whole class of failure without any static linking at all.

That is the same shape as the micro-architecture pitfall, where a binary built with -march=native on an AVX-512 host crashes with SIGILL on an older runtime CPU. In both cases the build environment and the runtime environment disagree, and the image strategy is what makes that disagreement visible or invisible.

The takeaway

Image size for a C++ service is not a packaging afterthought. It is a decision about which dynamic libraries travel with the binary, which ones you assume at the runtime end, and how you will diagnose the thing once it is small enough to be opaque. Multi-stage builds leave the toolchain behind. The UBI tier you pick encodes your incident-response posture. Labels buy back the introspection the small image gave up. Get those three right and the same 4 MB binary that was shipping in 689 MB ships in 26.

Source project

This post is mined from §4, “Container Strategy: UBI, ubi-micro, multi-stage” of Optimizing Modern C++ with Containers, which walks through the full argument, the layer-caching ordering rules, and the production diagnostic recipe for reading a small image.

The code samples are drawn from the tutorial’s runnable demo-01-image-strategy, which builds the same service four ways and prints the verified sizes: Containerfile.single-stage-naive (689 MB), Containerfile.ubi-multistage (114 MB), Containerfile.ubi-micro (26.4 MB), and Containerfile.ubi-micro-glibc-mismatch (the one engineered to fail).

Full project: site · repo.

The Quarkus data-mesh reference architecture runs all three answers side by side over the same order-triage domain, with the same business logic underneath, so the only thing that varies is the coordination shape. That makes it a clean place to compare them. This post walks the three, then gets to the distinction that matters in an incident: not whether the sequence is written as code or declared as data, but whether any single artifact knows the sequence at all.

One domain, three coordinators

The three legs coordinate the same work. Two of them are “orchestration” and still disagree sharply, which is the first clue that the usual two-way framing is too coarse.

• Kafka choreography. Four services react to each other through topics with no coordinator. order-service publishes order.placed and has never heard of the others. payment-service subscribes to order.placed and emits payment.captured; shipping-service subscribes to that and emits shipment.dispatched; notification-service also subscribes to order.placed directly, a fourth reaction running in parallel with the payment chain.
• A Camel route. One route in ai-rules-service sequences the steps imperatively, top to bottom, in a single file.
• A Quarkus Flow workflow. The same steps in the same service, declared as a task-graph document the engine interprets, structurally the same shape as a CNCF Serverless Workflow definition.

Who knows the sequence

Start with the choreography chain, because its answer is the surprising one: no file in the repo contains the sequence. No artifact holds the string “order.placed, then payment.captured, then shipment.dispatched” as a single thing. PaymentProcessor.process knows it consumes one topic and produces another, and nothing more. ShipmentProcessor.process knows the mirror image. Each method is a complete description of what it does. That these methods chain into a three-hop saga is a fact about the running system, not a fact recorded anywhere in it. Reconstructing “what happens when an order is placed” means reading four files in four Maven modules and joining them in your head by topic name. The system has that knowledge; no single part of it does.

The Camel route is the opposite extreme. The route is the sequence, and it is short enough to read in one sitting:

// Camel orchestration — OrderTriageRoute.java
// The route is the sequence: read top to bottom, nothing elided.
from("direct:triage")
    .routeId("triage-order")
    .unmarshal().json(JsonLibrary.Jackson, OrderCreate.class)
    .bean(triageService, "classify")
    .bean(triageService, "decide")
    .marshal().json(JsonLibrary.Jackson);

A reviewer who reads those lines has read the entire business process for that endpoint, in order. The Flow workflow occupies the same slot but answers “what is the sequence” with a document instead of a trace:

// Quarkus Flow orchestration — OrderTriageWorkflow.java
// Same two steps, declared as a task graph the engine interprets.
return FlowWorkflowBuilder.workflow("order-triage")
    .tasks(
        FlowDSL.function("classify", triageService::classify),
        FlowDSL.function("decide", triageService::decide))
    .build();

Both orchestration legs beat choreography on “where do I even look,” but they beat it in different senses. The Camel route’s sequence is knowable by reading code. The Flow workflow’s sequence is knowable by reading data: you can inspect, diff, or version the descriptor without executing it. That is the distinction the usual declarative-versus-imperative framing points at, and it is real. It is just not the one that hurts most at three in the morning.

Coupling: what changes when you add a step

Measure coupling by asking what has to change, and where, when a step is added.

Add a fifth reaction to order.placed, say an analytics service, and the choreography chain costs the existing services nothing. order-service does not import a client, does not add a call, does not even learn the new service exists. The new service pays the entire wiring cost by subscribing on its own, exactly as notification-service already does:

// Kafka choreography — PaymentProcessor.java
// Reacts to order.placed; knows nothing of shipping or notification downstream.
@Incoming(Topics.ORDER_PLACED_CHANNEL)
@Outgoing(Topics.PAYMENT_CAPTURED_CHANNEL)
public PaymentCaptured process(OrderPlaced orderPlaced) {
    String orderId = orderPlaced.getOrderId();
    PaymentCaptured existing = paymentStore.findByOrderId(orderId);
    if (existing != null) {
        return existing; // idempotent on redelivery, not a compensation
    }
    // ... build, persist, and return PaymentCaptured ...
}

The only shared thing is the schema, the OrderPlaced Avro type. No participant holds a reference to another participant’s class, bean, or address.

Here is the part that cuts against the tidy story. Both orchestration legs hold a direct @Inject TriageService reference and call its methods, so their coupling to the business logic is equally tight, and tighter than choreography’s. Adding a step means editing the one coordinator: in Camel you insert a line into the fluent chain; in Flow you add an entry to the .tasks(...) list. Both are one edit, one file, one diff, one deploy. On this axis Camel and Flow sit almost on top of each other, and both sit far from choreography, the declarative-versus-imperative difference notwithstanding.

Failure handling: what is implemented, not what is possible

This is where comparisons usually go wrong, because the engines can all do far more than any given codebase exercises. So be precise about what this repo implements today.

Choreography’s failure handling is per-hop idempotency, not compensation. PaymentProcessor and ShipmentProcessor both guard against redelivery by checking for an existing record before acting, which is what makes at-least-once Kafka delivery safe to build on. But nothing reacts to a failed payment by un-dispatching a shipment, and nothing refunds a capture when a shipment fails. There is no saga orchestrator and no dead-letter configuration. Kafka’s consumer redelivery is the only safety net wired up.

Camel’s failure handling here is default propagation. The route has no .onException(...) and no custom error handler, so an exception from classify propagates back through the REST binding. What the route does get by default is that any exception stops execution at the point of failure, so decide never runs against a classify that returned garbage without raising.

Flow’s failure handling is, in this repo, the same as Camel’s: unexercised. The descriptor declares two tasks and nothing else, no retry policy and no compensation task. A 120-second timeout exists at the caller boundary, but that is HTTP hygiene, not workflow-level compensation.

The fair comparison, then, is not “which engine handles failure better.” All three lean on idempotency or exception propagation today. The difference is where the fix would go when you need real compensation:

• In choreography, a new consumer that reacts to a failure event, itself published as an event, keeping the pattern decentralized.
• In Camel, .onException() or .doTry()/.doCatch() clauses on the one route.
• In Flow, retry and compensation nodes added to the one workflow document. The Serverless Workflow spec supports them as first-class constructs, which is the one real edge of the declarative shape: compensation becomes expressible as data. “The spec supports it” is not the same as “this workflow uses it,” and keeping those two sentences apart is the difference between reading a demo and believing its marketing.

Debuggability

An incident forces a concrete question: where do you put your eyes first?

For choreography, the answer is “it depends which hop is silent,” and the path is per-service. Check order-service’s logs and the order.placed topic, then payment-service and payment.captured, then shipping-service. There is no single place that reports “did the whole chain finish.” That is the standing cost of loose coupling, paid on every investigation.

The Camel route is one log stream: one request, one thread, one place to read. The Flow workflow is inspectable per instance and per task, because progress is tracked against named nodes rather than only as a log stream. That is a real advantage for long-running or branching workflows. For this particular two-task, caller-blocks-on-await workflow it stays mostly theoretical, a caveat the comparison should name rather than skip.

Operational cost tracks coupling, almost exactly

Running choreography in production means running, monitoring, scaling, and deploying four services plus Kafka plus a schema registry. The payoff is that those failure domains are independent: a slow notification-service never backs up payment-service, because they share nothing but a topic. Both orchestration legs run inside a single service, so they are cheaper to operate and they share fate. If ai-rules-service is down, both endpoints are down together; if the shared Ollama model is slow, both feel it identically, because they call through the same bean.

So the trade is legible. Choreography’s loosest coupling buys the most independent operability at the highest service count. Both orchestration legs buy operational simplicity at the cost of shared fate for everything the coordinator touches.

Reach for which, when

The question worth carrying

The split that survives contact with production is not declarative versus imperative. Camel and Flow land in nearly the same place on coupling, on where the business logic lives, and on how much failure handling exists today. They differ in how you would extend each one: edit a method chain, or edit a task graph. Choreography is the real outlier, because it removes the central artifact entirely, and that single decision ripples through coupling, debuggability, testing, and operational cost in lockstep.

So when you reach for a coordinator, ask the question the three columns answer differently. When this breaks in production, who knows the sequence? If the answer is “no one, by design, and that is the point,” you want choreography. If the answer needs to be “this file,” you want a route. If it needs to be “this document, which we can inspect and version,” you want a workflow. Pick the shape whose answer you can live with during an incident, and the library choice mostly follows from there.

Bonus: Camel’s Saga EIP, and why it changes the state story

Here is a property the three coordinators share: all of them are stateless. The Camel route and the Flow document each describe the sequence, but the running service holds nothing between requests. A request arrives, the sequence executes, the service forgets. That is what lets you run many replicas and scale them to zero without a second thought, and it is the same posture a well-behaved service keeps in any language.

Camel has a first-class way to make a route survive partial failure: the Saga EIP, its implementation of the saga / long-running-action pattern. You mark the route .saga(), and each participating step registers a .compensation() endpoint (and, if it needs one, a .completion() endpoint). If a later step fails, Camel calls the earlier steps’ compensations in reverse order to undo their side effects:

// Camel Saga — OrderSagaRoute.java
// Same top-to-bottom sequence, but each step now declares how to undo itself.
from("direct:place-order")
    .saga()
        .timeout(30, TimeUnit.SECONDS)           // a stuck saga auto-compensates
    .to("direct:capture-payment")
    .to("direct:arrange-shipping");

from("direct:capture-payment")
    .saga()
        .propagation(SagaPropagation.MANDATORY)
        .compensation("direct:refund-payment")   // called in reverse if a later step fails
        .option("paymentId", simple("${body.paymentId}"))
    .bean(paymentService, "capture");

from("direct:refund-payment")                     // the compensating action — must be idempotent
    .bean(paymentService, "refund");              // reads the paymentId saved via option()

That is a real improvement over the default exception propagation in the comparison above. But look at what it costs. A saga is a state machine: it moves through Active, Completing, and Completed on the happy path, or through Compensating and Compensated on failure, with a timeout that auto-compensates a saga that gets stuck. Something has to hold where each in-flight saga currently is.

The default CamelSagaService is InMemorySagaService, and it keeps all of that in the JVM heap. Fine for a demo, and fatal in production: if the process dies between “payment captured” and “shipping arranged,” the saga state dies with it, no compensation ever runs, and you are left with a payment that was captured and will never be refunded. The stateless property we just praised is gone, in the one way that hurts most.

So making a Camel saga correct is, in the end, a state-externalization exercise. You delegate the saga state to a coordinator that persists it: camel-lra plus a MicroProfile LRA coordinator (Narayana) writes the participant list, the compensation URIs, and the saved options to durable storage, and after a restart the coordinator drives the outstanding compensations to completion.

// Non-durable: saga state lives in the JVM heap, lost on crash.
camelContext.addService(new InMemorySagaService());

// Durable: delegate saga state to an LRA coordinator that persists it,
// so the Camel service itself goes back to holding nothing.
LRASagaService saga = new LRASagaService();
saga.setCoordinatorUrl("http://lra-coordinator:8080");
saga.setLocalParticipantUrl("http://ai-rules-service:8080");
camelContext.addService(saga);

The rest of the work follows from treating it as a state machine with external storage: give every step a compensation route, use .option(...) to persist the identifiers those compensations need, make each compensation idempotent because the coordinator may retry it, and set a .timeout(...) so a wedged saga does not sit half-applied forever. Do that and the state lives outside the process again, and the service is stateless once more. It is the same externalize-your-state move that keeps any service replaceable, applied to the one piece of coordination that does need to remember something.

This is why the saga sits in a bonus rather than in the three-way comparison. The route and the workflow answer “who knows the sequence.” The saga answers a second question on top of it, “who knows how far we got, and will that knowledge survive a crash,” and answering it well means putting the progress somewhere durable. The sequence can live in code or in a document. The saga’s state has to live in a store. Keeping those two questions separate is the whole point: add a saga only when you need the second answer, and know that you are trading the stateless posture for a coordinator you now have to operate.

For the mechanics, see Camel’s Saga EIP and the camel-lra component.

Source project. This post is drawn from Chapter 13, orchestration styles and the deeper Appendix comparing the three engines of the Data Mesh Reference Architecture on Quarkus, which runs all three legs green over one shipping/order domain. The code shown here is real: OrderTriageRoute.java, OrderTriageWorkflow.java, and PaymentProcessor.java. Full source is on GitHub.

Three cost problems hide behind that bill, and they are distinct: metric cardinality, log volume, and trace volume. Each has its own failure mode and its own fix, and all three get cheaper to control the earlier you give the collector a say in what leaves the building.

A metric is a family of time series

Start with the part that surprises people most. A metric is not one stream of numbers. It is a family of time series, one for every unique combination of label values it has ever carried. A counter named http_requests_total with labels method, route, and status_code does not produce one series; it produces one per observed combination: GET /cart 200, POST /checkout 500, GET /cart 404, and so on. The metrics backend (Mimir, in the stack this is drawn from) allocates storage, an index entry, and query-time bookkeeping for every one of them, and keeps it until retention ages it out.

That is fine when the label values come from a small, closed set. HTTP method has a handful of values. A templated route such as /cart/{id} has a few dozen. Status code has a few hundred, of which a service exercises maybe twenty. Multiply those together and a well-behaved metric produces a few thousand series, which any backend built in the last decade handles without noticing.

The failure mode is a label whose value set has no ceiling: a raw user ID, a session token, a request ID, an email address, a full URL with its query string still attached, a container ID. Each one multiplies into the series count, and because the set of values grows without bound, the series count grows without bound too. A counter that looked like it tracked requests per route quietly becomes requests per route per user who has ever hit the service, and the backend ends up carrying a series for every person who tried the product once two years ago and never came back. That is a cardinality explosion, and it is the most common reason a metrics backend falls over or a bill spikes.

OpenTelemetry’s own guidance is explicit here: attributes with unbounded value sets belong on spans and logs, which carry per-event detail, not on metric labels, which exist to be aggregated (see the semantic conventions and the metrics specification). The rule of thumb that holds up across every language: before adding a label to a metric, ask whether you could list every value it will ever take. If you cannot, it does not belong on that metric.

Cardinality has a second cost that bills as latency rather than dollars. A query for the error rate of one route has to scan and aggregate every series that matches, so the more series exist, the more work the engine does to collapse them into the single line a dashboard shows. An explosion does not only make storage expensive; it makes every dashboard built on the affected metric slower, sometimes slow enough to time out. That timeout is usually the first symptom an on-call engineer notices, long before anyone opens a billing report.

Label hygiene is a design discipline

Fixing cardinality after the fact is a scramble: finding which series exploded, patching instrumentation in a dozen places, then waiting out a retention window for the damage to roll off storage. Applied up front it is cheap, and it comes down to a few habits.

• Favor templated routes over literal paths. A framework that already knows the route template should report /orders/{orderId}, not /orders/8f21ac. The identifier belongs on the span or the log line, where it tags one event instead of multiplying a whole family of series.
• Bucket instead of enumerate. A wide-ranging numeric value, a payload size or a duration, belongs in a histogram, whose buckets are a bounded dimension. The raw value as a label is not.
• Push high-cardinality identifiers down the signal stack. A user ID, order ID, or trace ID is exactly the right thing to attach to a span attribute or a structured log field. It should never also become a metric label.
• Review new labels like schema changes. A reviewer who asks one question, “what is the set of values this can take, and is it bounded?”, catches almost every cardinality problem before it ships, because the answer is usually obvious once it is said out loud.

Logs and traces fail differently

Logs do not explode in the metrics sense, because a log line does not pre-aggregate into a time series. Loki, the log store in this stack, indexes a small set of labels and keeps the rest of each line as compressed text. The log cost problem is blunter: volume. Every DEBUG line left on in production, every health check logged at INFO, every framework that logs once per database query, scales linearly with traffic, and storage is priced by the gigabyte ingested and the days retained.

Three controls compound here. Log-level discipline keeps DEBUG and TRACE in development and short diagnostic windows, not steady-state production. Deduplication folds a retry loop that would log the same warning on every attempt into one line with a count. Retention tiering keeps recent data hot and moves or drops the rest; that is where the multi-month savings come from, more than trimming what gets logged in the first place.

One cardinality mistake does carry over to logs: a stream label that varies per request shatters a single log stream into thousands of tiny ones and defeats the compression that makes Loki affordable. Keep stream labels bounded (service, environment, level) and let high-cardinality detail live in the line content, searchable with a query-time filter.

Traces carry the highest per-unit cost of the three, because one trace is a tree of spans, each with its own attributes, events, and timing, and Tempo stores and indexes all of it. Capturing every trace is viable at modest traffic and untenable as volume grows, which is what sampling is for: head and tail strategies decide which traces are worth keeping before they reach storage. Sampling sits upstream of everything else here, so a service emitting one hundred percent of its traces is the top candidate for runaway storage regardless of how clean its spans are. On top of sampling, span attribute hygiene matters: a span that attaches a full request body, or a stack trace on every exception, adds bytes to every stored span, multiplied by however many survive sampling.

Control spend at the collector

Everything above is something a service owner sets at the instrumentation layer: which labels to add, which log level to run, which attributes to attach. The collector is the second, and often more practical, place to enforce it, because it applies the decision centrally without redeploying every service that emits telemetry.

A filter processor drops whole signals, spans, or log records that match a condition; high-volume, low-value health-check traces are the usual first target. An attributes processor in delete mode strips a specific attribute as telemetry passes through, which is the practical backstop for cardinality: even if a library attaches a raw user ID to every span by default, the collector removes it before export rather than waiting for every team to patch its code.

processors:
  attributes/drop-user-id:
    actions:
      - key: user.id
        action: delete
  filter/drop-health-checks:
    traces:
      span:
        - 'attributes["http.route"] == "/healthz"'

A transform processor goes further, rewriting a high-cardinality attribute into a bucketed or templated form instead of dropping it outright, keeping some diagnostic value while still bounding the cardinality. For metrics, aggregation (in the collector, or through SDK views it forwards) lets a team decide a metric needs an application-wide total rather than a per-pod breakdown, cutting the series count by the number of replicas without losing the number alerting depends on.

The thread through all of these is central ownership. A platform or observability team sets retention and cost policy once, in the collector configuration, and it applies across every service, every language, and every team, whichever ones were careful about cardinality and which were not. That is the payoff of putting a collector in the data path: a direct SDK-to-backend pipeline has no interception point, so every cost control becomes a per-service, per-language exercise instead of one shared policy.

The habit worth keeping

Cost control in observability is not a single switch. It is the sum of small decisions: a bounded label, a sane log level, a sampling rate matched to traffic, a collector processor that catches what instrumentation missed. None of them is expensive on its own. What is expensive is discovering that all three signals have been growing unbounded at once, usually the same morning a billing alert fires or a backend starts rejecting writes under load.

So carry one question forward. When you add any label, attribute, or log field, ask what values it can take and whether that set has a ceiling. When you read a collector configuration, check whether it forwards everything it receives or actively decides what is worth keeping. A pipeline with no filtering, no attribute processing, and no sampling is not a more complete observability setup. It is a more expensive one, usually without a matching gain in anything a human can use while debugging a live incident.

Source project. This post is drawn from the cardinality and cost chapter of the OpenTelemetry Observability Tutorial, which builds all five signals (traces, metrics, logs, baggage, and profiles) across Spring Boot, Quarkus, and Python on one Grafana LGTM stack. The sampling chapter covers trace volume in depth, and the collector chapter covers the data-path pipeline these cost controls plug into. Full source is on GitHub.

Four layers (edge, gateway, mesh, and the application itself) each see and decide different things, and most of the work happens deeper than teams expect. The decision chain runs layer by layer: what the mesh does with TLS, sticky sessions, and traffic steering; what only the service itself can decide with rule-driven routing; and why east/west is where most routing lives.

L4 versus L7: what each layer can decide on

A Layer-4 load balancer sees connections: IPs, ports, the TCP 5-tuple. It picks a backend and hash-balances, but it can’t read URLs, headers, cookies, methods, or gRPC metadata. A Layer-7 router sees the whole conversation: host, path, headers, cookies, JWT claims, query parameters, the HTTP method, the gRPC service.method pair. Every routing decision below operates on that envelope.

TLS is the one nuance to keep in mind. An L4 balancer can route by SNI without terminating TLS, which is cheap but opaque. An L7 router typically terminates TLS so it can read everything else, which is richer but slightly slower.

The four-layer stack

L7 routing lives in four layers, and each handles a different kind of decision. A request is a chain of independent L7 decisions across all four.

The edge / ingress layer (HAProxy, Nginx, Envoy, Traefik) is where TLS terminates and coarse host/path routing happens. The API gateway (Kong, Apigee, an Istio gateway, Spring Cloud Gateway) handles product-level concerns: authentication, rate limiting, and transformation. The service mesh (Istio with Envoy sidecars, or Linkerd) governs east/west traffic, giving every internal call mTLS, retries, outlier detection, and locality-aware load balancing. When a decision depends on business rules the platform can’t know, routing pushes into the app itself. Each layer can refuse, transform, or branch the request, and each emits a trace span, so the request’s journey is observable end to end.

Content-based routing

HAProxy and Nginx are the workhorses of the edge: mature, fast, and predictable. HAProxy reads the envelope with ACLs (path_beg for URL prefixes, hdr for headers, content-type for the body kind) and use_backend dispatches on them; Nginx uses location blocks and the map directive to turn a header value into an upstream pool. Envoy is the modern sidecar choice because it speaks HTTP/2 and gRPC natively.

But the form most teams reach for today is the mesh’s: the same content-based primitives expressed as Kubernetes CRDs that Istio compiles down to Envoy config across every sidecar.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: orders }
spec:
  hosts: [orders]
  http:
  - match:                                    # gRPC by service.method
    - uri: { prefix: /acme.orders.v1.OrderService/ }
    route: [{ destination: { host: orders, subset: v2 } }]
  - match:                                    # by header
    - headers: { x-internal: { exact: "1" } }
    route: [{ destination: { host: orders, subset: v2 } }]
  - match:                                    # by cookie
    - headers: { cookie: { regex: ".*canary=on.*" } }
    route: [{ destination: { host: orders, subset: v2 } }]
  - route: [{ destination: { host: orders, subset: v1 } }]   # default

Three match rules, by URI prefix (which is how you route on a gRPC service.method, since gRPC paths look like /package.Service/Method), by header, and by cookie, each send to a different subset. The default route at the bottom is critical: without it, unmatched requests have nowhere to go. The CRD lives in Git, gets reviewed like any other change, and Istio reconfigures every sidecar within seconds.

The runnable example backing this post uses a lighter touch at the edge: plain Envoy weighted clusters and a header override, with no CRDs required: examples/22-l7-routing/envoy/envoy.yaml.

TLS termination

TLS at the edge comes in three modes. Terminate-and-decrypt is the common case; it lets the router do everything L7. Terminate-and-re-encrypt is the most secure default in a mesh, since mTLS inside is automatic. Passthrough is for traffic you must not see (client mTLS for regulated flows): route by SNI only, decrypt nothing. cert-manager is the standard answer for public certificates; SPIFFE-style identity and Envoy’s SDS handle service-to-service certs automatically inside the mesh. A handshake is milliseconds, so amortise it with session resumption, TLS 1.3, and HTTP/2 multiplexing.

Sticky sessions

There are three affinity modes, and the default is the one you want. No affinity sends any request to any pod, which keeps load even and assumes client state lives elsewhere (Redis, the JWT, the database). Cookie-based stickiness is the fallback when something stateful must stay on one pod: the balancer sets a cookie and routes subsequent requests back to the same pod. That ties the session to one pod’s lifetime, so a restart loses it. IP-hash needs no cookie but is fragile behind NAT and mobile networks, where many users share an address and load skews.

The real cost is that stickiness fights elastic scaling, fast restarts, and graceful shutdown, which together are the entire cloud-native delivery model. Treat it as a deliberate choice for a known constraint, never a default, and prefer to externalise state (Redis for session data, JWT claims for identity, a pub/sub backplane for fan-out) so the balancer can return to no-affinity.

Intelligent traffic steering

Six familiar release shapes are all the same L7 primitive with different inputs and weights: canary (a small weighted slice to a new version), blue/green (two environments flipped atomically), A/B (a stable cohort mapped to a variant by a user id), header-based dark launches (route opted-in users by a header), geo/latency (EU users to EU pods), and shadow/mirror (copy a percentage of real traffic to a new version without returning its responses). The canonical one is the weighted canary: a VirtualService splits traffic by weight, and a DestinationRule defines the subsets by pod label.

apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: orders }
spec:
  hosts: [orders]
  http:
  - route:
    - destination: { host: orders, subset: v1 }
      weight: 90                              # 90% on the stable version
    - destination: { host: orders, subset: v2 }
      weight: 10                              # 10% on the canary
---
apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: orders }
spec:
  host: orders
  subsets:
  - name: v1
    labels: { version: v1 }
  - name: v2
    labels: { version: v2 }

Ramping the canary means changing the weights in Git (typically 10, 25, 50, 100 percent, with metric gates between steps), and a controller like Flagger or Argo Rollouts flips the weights back in seconds if error rate or latency on v2 crosses a threshold. Header-based steering adds precision: an upstream identity proxy sets a header for opted-in users, and the VirtualService routes those users to v2 on real production data with zero blast radius.

Payload modification is cheap: header inject/strip and URL rewrites are nearly free because the router is already parsing the request line and headers. (This is how the W3C traceparent header propagates, and how a gateway validates a JWT once and injects trusted claims downstream.) Body transformation is expensive: it forces the router to buffer the full payload, which kills streaming and risks memory under load. The rule of thumb: if you can do it in the headers, do it in the headers.

Routing inside the app by business rules

There’s a threshold where routing leaves the network and enters the service. If the decision fits in a few YAML lines and depends only on the envelope, do it at the mesh. If it depends on the payload, requires a decision table that domain experts edit, or involves many intersecting conditions (routing a healthcare message by patient class and encounter type, say), push it into a rule engine inside the app, behind the mesh, rather than a network CRD. The discipline is “data, not code”: the routing logic lives in an externalised ruleset the domain experts own, separate from the dispatcher, and changes without redeploying the service.

Each ecosystem composes this from its own rule engine (usually a Rete forward-chaining engine) feeding a small dispatcher. The ruleset reads as a set of “when … then …” statements that set a list of destinations:

# orders_rules.py  —  edited by domain experts, not developers (durable-rules)
from durable.lang import ruleset, when_all, m

with ruleset("orders"):

    @when_all((m.amount_cents > 50_000) & (m.tier == "VIP"))
    def vip(c):                              # high-value VIP orders
        c.m.where_to = ["kafka:orders.priority", "http:notify-vip"]

    @when_all(m.region == "EU")
    def eu(c):                               # European orders → EU region
        c.m.where_to = ["kafka:orders.eu"]

    @when_all(+m.order_id)                   # catch-all (lowest priority)
    def default(c):
        if not getattr(c.m, "where_to", None):
            c.m.where_to = ["kafka:orders.default"]

The dispatcher enriches the input with whatever context the rules need, evaluates the ruleset, and fans out to each destination over the right transport. The pattern is the classic content-based router from the enterprise-integration literature. The value is that the decision is data a domain expert edits, not code a developer redeploys.

The runnable version of this dispatcher is a FastAPI service that routes VIP orders to a priority topic and lets you change the threshold at runtime with no redeploy: examples/22-l7-routing/python/router-service/main.py.

The cost of L7 hops

Every L7 capability costs time, and it’s a budget you spend before your service even runs. TLS termination is sub-2ms, path/header routing sub-1ms, JWT validation a few ms, a WAF a few more, each mesh sidecar about a millisecond, a rate-limit Redis lookup a couple of ms. Body inspection is the outlier at roughly 5–50ms depending on size.

The working heuristic is to stay under about 10ms of total L7 on hot paths. With four layers (edge, gateway, sidecar out, sidecar in), that leaves little room per layer. The common pitfalls follow from this: too many hops nobody measured until p99 spiked; expensive regex matches; sticky sessions fighting the platform; body inspection blowing the budget; per-request auth lookups cascading under load; and timeout misalignment, which is the gateway-to-mesh-to-service version of a retry storm. The fix is the same one that fixes retry storms: bound retries, align timeouts top-down, and give the deepest layer the strictest budget.

East/west: L7 between services

Most L7 decisions in a microservices system aren’t at the edge; they happen east/west, every time one service calls another. The mesh’s sidecar pair handles that traffic with no app code: mTLS (encryption and identity), HTTP/2 multiplexing, native gRPC, retries with backoff and jitter, outlier detection that ejects misbehaving pods, locality-aware load balancing that prefers same-zone pods, and per-call traces. Same Istio CRDs as the edge, pointed at an internal host:

apiVersion: networking.istio.io/v1
kind: DestinationRule
metadata: { name: payments }
spec:
  host: payments.acme.svc.cluster.local
  trafficPolicy:
    connectionPool:
      http: { http2MaxRequests: 1000, maxRequestsPerConnection: 10 }
    outlierDetection:                              # eject bad pods
      consecutive5xxErrors: 5
      interval: 30s
      baseEjectionTime: 30s
    loadBalancer:
      localityLbSetting: { enabled: true }         # prefer same zone
  subsets:
  - { name: v1, labels: { version: v1 } }
  - { name: v2, labels: { version: v2 } }
---
apiVersion: networking.istio.io/v1
kind: VirtualService
metadata: { name: payments }
spec:
  hosts: [payments.acme.svc.cluster.local]            # internal DNS — east/west
  http:
  - route:
    - { destination: { host: payments, subset: v1 }, weight: 95 }
    - { destination: { host: payments, subset: v2 }, weight: 5 }

That last block is a canary release of a backend service no external client knows exists, which you cannot do through an edge load balancer.

gRPC east/west has a few sharp edges. Route on the gRPC URI prefix for per-method canaries. Prefer least-request load balancing over round-robin, because gRPC connections are long-lived and round-robining connections (rather than requests) can starve backends. Be careful retrying streaming RPCs, since replaying a half-finished stream redoes half the work. And remember gRPC returns its real status in HTTP trailers, so outlier detection has to read the trailer: a 200 with a non-OK gRPC status is a failure, and a check that ignores it would call a failing service healthy.

When to use which layer

Each layer has a job it does well, and pushing work to the wrong one is where complexity comes from. Edge/ingress for the things every client crosses (TLS, host/path). API gateway for product-shaped policy (auth, rate limits, plans). Mesh for east/west and everything internal. In-app for business rules. This split prevents the slow growth of baroque VirtualService configs that encode rules domain experts need to maintain: if it’s a business rule, it belongs in a rule engine inside a service, not a network CRD. The organisational failure to avoid is the cluster that ends up running three ingress controllers because three teams each picked one. Choose one ingress, one gateway, one mesh; manage routing through GitOps; watch certificate rotation; and load-test the failure paths.

Prove it yourself

Apply the canary VirtualService/DestinationRule, drive a few hundred requests with hey, and confirm the split lands near the configured weight: roughly 10% of responses carrying the v2 build marker (a response header or version field), the rest v1. Bump the weight in Git, re-apply, and watch the ratio move within seconds. Then flip a request’s x-internal header and confirm it goes to v2 every time regardless of weight, since header match beats weight.

For the in-app router: post one VIP order over the configured threshold and one ordinary order, and confirm the VIP order landed on orders.priority (and triggered the notify-vip HTTP call) while the ordinary one fell through to orders.default. Then change only the ruleset, not the dispatcher, and confirm the routing changes without a redeploy. Weights that track the config, and a ruleset that can move routing without a code change, are the two halves of L7 routing working. The full runnable version is Envoy config plus a FastAPI dispatcher, with a verify.sh that checks all of it, in examples/22-l7-routing/ (README).

Source project: this post is adapted from Appendix I of Cloud-Native Design Patterns, where it’s part of a larger tutorial covering event-driven architecture, sagas, observability, and the rest of the cloud-native API toolbox.