The final project
The routing, tracking and quote services end to end: all four RPC types, deadlines and retries, rich errors, mTLS and JWT, OTel across the seam, buf-gated, ghz load-tested, Compose to minikube, both stacks.
Every prior chapter in this book picked one topic and held everything else constant. Chapter 19 put RoutingService.GetQuote behind mTLS and left JWT, tracing, and the other three RPC shapes untouched. Chapter 20 put the same GetQuote behind a bearer token and left transport security to the chapter before it. Chapter 21 traced a REST-to-gRPC seam and never asked whether the call behind it was authenticated at all. A chapter teaching one mechanism is easier to verify and easier to read when nothing else is moving, but it leaves one question unanswered: do these mechanisms still work when they are all turned on at once, against a service that actually implements every RPC shape the contract declares, not just the one shape each lean example needed? This chapter answers it. examples/27-complete-project/ runs shipping.routing.v1.RoutingService (GetQuote, unary; NegotiateRoute, bidirectional), shipping.quote.v1.QuoteService (the downstream hop), and shipping.tracking.v1.TrackingService (WatchShipment, server streaming; ReportPositions, client streaming) in one process per stack, behind mTLS, a JWT interceptor with per-method scopes, and OpenTelemetry tracing across every hop, with an always-on Grafana LGTM stack and a pinned ghz load-test gate — on both Quarkus and Python, verified by one verify.sh CI runs on every push.
Nothing here is a new RPC. Every method this chapter calls was already declared in contracts/proto/shipping/{routing,quote,tracking,type}/v1/*.proto from the earliest chapters that introduced each service; git diff main -- contracts/ for this iteration is empty, and buf breaking in CI proves it stays that way. What changes is that several of these methods — NegotiateRoute, WatchShipment, and ReportPositions chief among them — have answered UNIMPLEMENTED in every lean, single-topic example that declared them, because implementing a streaming RPC wasn’t that chapter’s lesson. This chapter is where all three run together, for the first time.
Four RPC shapes, one system, no shortcuts
The Decision-Framework’s first axis — RPC shape — only means something when a system actually has more than one shape to choose between. GetQuote stays unary because pricing a shipment is a single question with a single answer: the caller sends an origin, a destination, and a cargo manifest, and gets back one Money and one Route. NegotiateRoute is bidirectional because a negotiation is inherently a back-and-forth — the caller streams NegotiateRouteRequest proposals and the server streams back one NegotiateRouteResponse per proposal, each carrying a counter route and an accepted flag, with no requirement that the two streams move in lockstep beyond this implementation’s own choice to answer each proposal as it arrives. WatchShipment is server streaming because a shipment’s location is a continuous feed the caller doesn’t know the length of in advance — this example streams the same four deterministic TrackingEvents Chapter 8 established (PENDING at the origin port, CONFIRMED at the same port, IN_TRANSIT and DELIVERED at the destination), in order, over one HTTP/2 stream. ReportPositions is client streaming for the mirror-image reason: a bulk position upload is naturally many messages folded into one acknowledgment, so the server reads the inbound stream to exhaustion and returns a single ReportPositionsResponse carrying accepted_count — the count of reports it saw, not an echo of their contents.
RoutingService.NegotiateRoute’s policy is deterministic and small: a proposal priced at or under $5,000.00 is accepted as-is, and anything over that is countered at 90% of the proposed price, computed with the same decimal, round-half-up arithmetic every quote formula in this book uses so verify.sh can assert on it without floating-point slop. Two test proposals exercise both branches of that policy in one bidirectional call — one priced at $2,500.00, which comes back accepted: true with the counter equal to the proposal; one priced at $9,000.00, which comes back accepted: false with a counter at $8,100.00 — proving the bidi response pair, not just that the stream opened and closed cleanly.
The REST-to-gRPC seam, now carrying a certificate and a token
Chapter 21’s POST /booking/quote called RoutingService.GetQuote over a loopback channel with nothing guarding it. This chapter’s front door calls the identical method over the identical kind of loopback channel, except that channel now presents the mTLS client certificate certgen mints (the Chapter 19 shape: one shared CA, one server identity covering both the quarkus and python in-container names plus localhost, one client identity) and attaches an internal service token as authorization: Bearer <jwt> metadata on every call — the same AuthzInterceptor that rejects an external caller’s missing or wrong-scope token rejects an internal one too, because there is no code path in this example that treats a loopback hop as implicitly trusted. RoutingServiceImpl.getQuote calling downstream into QuoteServiceImpl.getQuote presents that same certificate and the same internal token on its own loopback hop, a second hop behind a second, independent mTLS-and-JWT check rather than a single edge check that the interior of the system takes on faith.
That internal token is scoped broadly enough to cross every hop this example’s own code makes — quote:read for GetQuote on both services, plus the scopes any other internal call might need — but a caller presenting any OTHER token is held to the per-method map the Chapter 20 interceptor introduced: GetQuote needs quote:read, NegotiateRoute needs routes:negotiate, and ListRoutes (still UNIMPLEMENTED, as in every prior chapter) is gated on routes:admin, a scope none of this example’s demo tokens carry. That last mapping is deliberate: ListRoutes is this chapter’s admin-gated RPC specifically because reusing an already-UNIMPLEMENTED method to prove PERMISSION_DENIED keeps the authorization check demonstrably independent of whether a handler exists behind it — the interceptor rejects the call before RoutingServiceImpl/RoutingServiceServicer.listRoutes is ever reached, the identical property Chapter 20 established and this chapter inherits unchanged.
Tracing a seam that now has guards on it
Chapter 21 proved a W3C traceparent crosses a REST-to-gRPC boundary and one downstream gRPC hop with zero manual propagation code, because quarkus-opentelemetry and opentelemetry-instrument’s zero-code instrumentation inject and extract it automatically. Layering mTLS and a JWT interceptor on top of that same seam does not disturb it: traceparent is ordinary gRPC metadata, riding in the same HEADERS frame as the authorization entry AuthzInterceptor reads, and TLS encrypts that frame without inspecting its contents. The interceptor runs after the TLS handshake has already succeeded and reads one metadata key without touching another, so tracing, authentication, and authorization stack as three independent checks on the same wire format rather than three competing mechanisms. BookingResource.quote (and the Python booking_quote handler) logs the trace id it reads from the current span; RoutingServiceImpl.getQuote/RoutingServiceServicer.GetQuote logs the same id a hop later; QuoteServiceImpl.getQuote/QuoteServiceServicer.GetQuote logs it a hop after that. verify.sh’s trace-id-continuity check is the same synchronous docker compose logs | grep assertion Chapter 21 introduced, now run across three logged lines instead of two, because this chapter’s request path is one hop longer than Chapter 21’s.
WatchShipment, ReportPositions, and NegotiateRoute are not called through the REST front door in this example — verify.sh exercises them directly over gRPC, the same way a service mesh sidecar or another internal service would — so they are not part of the three-hop trace Figure 27.2’s numbered sequence draws through the REST path. They are still authenticated, still authorized, and still mTLS-protected by the identical interceptor and server configuration; they simply are not where this chapter’s trace-id-continuity assertion looks. A production system exposing WatchShipment to a browser would put it behind its own instrumented gateway hop, exactly as GetQuote sits behind this one.
Two failure layers before any handler runs, not one
A call against this example’s gRPC port can fail for reasons that have nothing to do with each other, and verify.sh asserts each one with the tool built for it. A grpcurl call presenting no client certificate against a client-auth: REQUIRED server fails at the TLS handshake — the Chapter 19 assert_tls_rejected pattern, which checks the error text names a transport failure (tls: certificate required, EOF, or a dial-level context deadline exceeded) rather than a clean application-level gRPC status, because a handshake failure and a PERMISSION_DENIED are not the same claim and the test has to tell them apart. A -plaintext call against the same server fails the same way, one layer earlier: there is no gRPC exchange to produce a status code at all, because the server never negotiated a TLS session to carry one. Both of these are transport-layer rejections, and both happen before a single byte of AuthzInterceptor’s logic runs.
Only once mTLS succeeds does the JWT layer get a turn: a call with no authorization metadata gets UNAUTHENTICATED, using the exact grpc_expect_code helper and canonical-code normalization Chapter 20 wrote. A call with a well-formed, correctly-signed, unexpired token that simply lacks the scope a given method requires gets PERMISSION_DENIED instead — the authenticated-but-not-entitled case this chapter demonstrates against ListRoutes, reusing Chapter 20’s mint_token-and-header-aware-assertion pattern rather than inventing a new one. Three failure classes, three distinct checks, three distinct status families — a dial-level failure, UNAUTHENTICATED, and PERMISSION_DENIED — stacked in the order a real call actually encounters them, never collapsed into one undifferentiated “rejected” bucket.
Proving the gate actually bites
An integration test that always passes is not proof of anything; it might be asserting on a field the handler doesn’t even set. After verify.sh ran green on both stacks, this example’s authoring process seeded one wrong expected value — changing the asserted GetQuote amount for the standard 1,000 kg booking from 2500.00 to 2500.01 — and re-ran verify.sh to confirm it failed loudly at exactly that assertion, with lib.sh’s assert_eq reporting the expected-versus-actual mismatch rather than the run silently passing. That seeded change was then reverted before this chapter was considered done. The purpose isn’t paranoia for its own sake: a semantic check that can’t be observed to fail is functionally equivalent to no check at all, and this book’s examples have, in earlier chapters, caught real defects (Chapter 21’s missing wget binary, Chapter 21’s synchronous-versus-aio health servicer mismatch) precisely because their checks were built to fail loudly when something was actually wrong.
ghz, buf, and the two gates that never touch this chapter’s logic
buf lint, buf build, and buf breaking run against contracts/ exactly as they have since Chapter 5 — this chapter adds zero proto changes, so the only thing to prove is that it didn’t accidentally make one. The ghz smoke test is new to this example’s shape only in scale, not in kind: Chapter 18’s pinned ghcr.io/bojand/ghz:0.121.0 image, fed contracts/proto/shipping/routing/v1/routing.proto directly over a read-only volume mount (ghz has no server-reflection support, so it needs the .proto file, not just an address), runs a short burst of GetQuote calls against one stack with latency and error-rate bounds loose enough to absorb CI jitter, backed by three retries — CI runners are shared and noisy, and a flaky performance assertion that fails the build on unrelated infrastructure jitter teaches the wrong lesson. The point of running ghz here at all isn’t a performance claim; it’s proof that the mTLS-and-JWT-protected server under load behaves the same as it does under verify.sh’s one-request-at-a-time assertions — that neither mechanism silently degrades or starts timing out once concurrent load shows up.
Three tiers, one contract, two of them run by hand
This chapter’s Compose stack is CI-gated — examples/27-complete-project/verify.sh is auto-discovered by the same examples/*/verify.sh glob every chapter’s harness uses, and runs on every push. Nested one level deeper, examples/27-complete-project/k8s/, istio/, scripts/, and loadclient/ are a separate, run-by-hand tier: a minikube deployment of this chapter’s RoutingService/QuoteService/TrackingService logic, rebuilt as a plaintext, unauthenticated server for this tier only, that reruns Chapter 22’s load-balancing fix — a plain ClusterIP Service pins one gRPC channel’s entire lifetime to a single pod (0 out of 3 other pods ever answer), a headless Service with client-side round_robin fans the identical 30 calls out roughly evenly across all three, and switching the Deployment into an Istio mesh with a ROUND_ROBIN DestinationRule makes the ORIGINAL plain Service fan out evenly too, because the sidecar — not the client — now owns the upstream connections. None of this tier’s scripts is named verify.sh, and it is never auto-discovered by CI; it is recorded, by hand, in expected-output.md, the same convention Chapter 22 established for what CI cannot see. mTLS and JWT are deliberately absent from this tier’s server, for the same reason Chapter 22’s own demo never layered them in: doing so would obscure the one mechanism this tier exists to teach, and Appendix A’s OpenShift Service Mesh tier demonstrates STRICT mesh-wide mTLS at the platform layer instead, where it belongs architecturally. A fourth, lightweight step in this tier fires GetQuote, NegotiateRoute, and WatchShipment against the deployed service and greps the resulting trace id out of kubectl logs on both the RoutingService and QuoteService log lines — proof the propagate-and-log pattern this book has used since Chapter 21 survives a real cluster deployment. That hop runs in-process on the pod (RoutingService calling QuoteService over the loopback channel), so the continuity it demonstrates is the application-level one, not a through-the-sidecar trace — and it needs no Tempo instance inside minikube to show it.
What you learned
- A service can run every RPC shape its contract declares — unary, server streaming, client streaming, and bidirectional — in one process, with mTLS, a JWT interceptor, and OpenTelemetry tracing all active simultaneously, because each mechanism operates on a different layer of the same call (transport, metadata, and span context respectively) and none of them need to know the others exist.
- An internal, loopback gRPC hop gets no exemption from the checks an external caller faces: this example’s own REST-to-Routing and Routing-to-Quote hops present the identical client certificate and bearer token grpcurl does, verified by the identical interceptor.
- Transport failures (a rejected TLS handshake), authentication failures (
UNAUTHENTICATED), and authorization failures (PERMISSION_DENIED) are three different claims, encountered in that order, and a test suite that can’t tell them apart is weaker than one that can. - An integration test that has never been observed to fail hasn’t proven anything; seeding one wrong expected value and confirming the gate turns red — then reverting it — is how this chapter confirmed its own assertions are real.
- Splitting a system into a CI-gated tier and run-by-hand tiers isn’t a compromise forced by this book’s constraints — it is how you keep a fast, deterministic contract gate (Compose, every push) separate from a slower, environment-dependent proof (a real cluster, a real mesh, a real load balancer), without losing either one.
On the wire
Server reflection on the Quarkus stack lists every service this chapter implements, alongside the standard health and reflection services:
grpcurl -cacert .host-out/ca.crt -cert .host-out/client.crt -key .host-out/client.key \
-servername quarkus localhost:9270 list
grpc.health.v1.Health
grpc.reflection.v1.ServerReflection
shipping.quote.v1.QuoteService
shipping.routing.v1.RoutingService
shipping.tracking.v1.TrackingService
A plaintext call against the same port fails at the handshake, never reaching a gRPC status:
grpcurl -plaintext localhost:9270 list
Failed to dial target host "localhost:9270": tls: first record does not look like a TLS handshake
A valid mTLS call with no bearer token gets UNAUTHENTICATED before RoutingServiceImpl ever runs:
grpcurl -cacert .host-out/ca.crt -cert .host-out/client.crt -key .host-out/client.key \
-servername quarkus -d '{"origin":{"unLocode":"USNYC"},"destination":{"unLocode":"NLRTM"},"cargo":[{"weightKg":1000}]}' \
localhost:9270 shipping.routing.v1.RoutingService/GetQuote
ERROR:
Code: Unauthenticated
Message: missing 'authorization: Bearer <jwt>' metadata
And the full flow, mTLS and a valid token both present, over the REST front door:
curl -s -X POST localhost:8270/booking/quote -H 'content-type: application/json' \
-d '{"origin":"USNYC","destination":"NLRTM","cargoKg":1000}'
{
"traceId": "7a1f9c3e4b2d4e6f8091a2b3c4d5e6f7",
"quote": { "currency": "USD", "amount": "2500.00" },
"route": { "id": "rte_USNYC-NLRTM", "legs": [ { "carrier": "MAERSK" } ] }
}
Swap port 8270/9270 for 8271/9271 and the Python stack answers identically, because examples/27-complete-project/verify.sh asserts precisely that, on both stacks, before it prints PASS.
Decision
Run this chapter's design through the same five axes every chapter in this book answers to. RPC shape: each of the four methods this example implements keeps the shape its own communication pattern demands — unary for a single question-and-answer (GetQuote), bidirectional for a two-way exchange that runs as long as the negotiation takes (NegotiateRoute), server streaming for an open-ended feed (WatchShipment), client streaming for a bulk-then-acknowledge upload (ReportPositions) — none of them forced into unary the way a REST-only mindset would default to. Sync vs. stream: the REST front door stays a single synchronous request/response, because a caller asking for one quote wants one answer, not a feed; the two tracking RPCs stay streams, because neither a shipment's location history nor a batch of position reports is naturally one message. Deadline and budget: this chapter inherits Chapter 10's unary deadline-propagation shape for the GetQuote-to-QuoteService hop unchanged and does not re-litigate it here — but a long-lived bidirectional stream like NegotiateRoute needs its own idle-timeout and overall-duration policy, distinct from a single unary deadline, that a production deployment of this example would have to add explicitly; this build does not set one, which is a real gap worth naming rather than quietly leaving implicit.
Failure class: stacking a transport rejection, an authentication rejection, and an authorization rejection as three independently-asserted outcomes, in the order a real call actually encounters them, tells a caller more than collapsing authentication and authorization into one "auth failed" bucket would — a caller debugging a PERMISSION_DENIED needs to know their credentials were accepted and their entitlements weren't, not just that something about the call was rejected. Concurrency model: every handler in this example is non-blocking end to end — Mutiny's Uni/Multi on Quarkus, grpc.aio coroutines on Python — including the loopback calls that attach mTLS credentials and a bearer token on every hop, because a system layering three cross-cutting concerns onto every call cannot afford even one of them to block a thread the runtime needed for other work. Where this chapter is weaker than it could be: a reader taking this into production should treat the long-lived internal service token as a placeholder for a short-lived, rotated, IdP-issued credential, and should give NegotiateRoute an explicit stream-level deadline before trusting it with real negotiations.