Backend I/O concurrency
Quarkus event-loop vs @Blocking worker pool vs @RunOnVirtualThread, Python grpc.aio vs a thread pool vs the GIL, and measuring the difference an execution model makes under concurrent blocking I/O.
Chapter 10 ended its cancellation story with a promise: the two-second delay that chapter cancelled mid-flight was modeled as a Mutiny pipeline or an asyncio coroutine specifically because only a cancellable abstraction lets a server stop work a caller no longer wants. This chapter asks the question that promise deferred — not “can this server stop working when told to,” but “can this server start enough work at once in the first place.” RoutingService.FindOptimalRoute gets its body here, backed by a database lookup and a file read that together take about 400 milliseconds, and the whole chapter is about one decision: which thread runs those 400 milliseconds. Get it right and eight callers finish in the time one would have taken. Get it wrong and a service with plenty of CPU to spare still queues requests behind each other, one at a time, because the thread that was supposed to be accepting the ninth connection spent the whole window waiting on a disk.
The code is in examples/15-backend-io-concurrency/. The run script there (verify.sh) builds and starts both stacks and asserts the concurrency claims this chapter makes; its README.md covers what it does and how to drive it by hand.
The event loop is not slow — it is shared
Every chapter before this one treated “the server” as a single thing that either answers a call or doesn’t. That framing hides the detail this chapter is built on: a Quarkus gRPC server, like most modern async servers, runs incoming calls on a small, fixed pool of event-loop threads — Vert.x’s default is twice the host’s CPU count — kept small by design, because an event loop’s whole value proposition is handling thousands of concurrent connections without a thread per connection. An event-loop thread must never sit still waiting on anything: a call that returns control quickly, even while real work continues elsewhere (a database driver’s own async callback, a completed Uni), lets that same thread immediately pick up the next waiting call. A call that blocks the thread outright — a Thread.sleep, a synchronous JDBC query, a blocking file read — takes that thread completely out of rotation for every other caller waiting on the same loop, for however long the blocking operation takes. With only a handful of event-loop threads serving potentially thousands of connections, one slow blocking call can stall a disproportionate number of requests that have nothing to do with it.
This is a different failure mode from the one Chapter 10 covered. A deadline exceeded because a downstream dependency was slow is a latency problem with one cause and one victim. An event loop blocked by synchronous I/O is a throughput problem whose victims are whoever else happened to be scheduled on the same thread — usually unrelated services, unrelated customers, unrelated endpoints, since the assignment of calls to event-loop threads is arbitrary from the caller’s point of view. The fix is not “make the I/O faster”: a 400-millisecond database round trip is 400 milliseconds no matter which thread waits on it. The fix is choosing which thread that is.
Quarkus: three ways to answer the same question
By default, every method on a @GrpcService class runs on the event loop — Quarkus does not guess whether your code blocks; it assumes it doesn’t, and the burden is on you to say otherwise. io.smallrye.common.annotation.Blocking is that declaration: annotate a method @Blocking and quarkus-grpc dispatches every invocation of it onto the default Vert.x worker pool (a fixed thread pool, sized around twenty by default) before a single line of the method body executes. Inside a @Blocking method, a synchronous database call, a blocking file read, or a Thread.sleep is exactly as safe as it would be in a classic thread-per-request server, because the thread doing the waiting isn’t one of the scarce event-loop threads — it’s one of a pool explicitly provisioned to absorb this. This chapter’s example implements FindOptimalRoute this way by default: the method is annotated @Blocking, and its ~400 millisecond lookup runs wherever quarkus-grpc already put it.
@RunOnVirtualThread answers the same question differently. Instead of handing the call to a shared, fixed-size pool, it starts a fresh platform-independent virtual thread for that one invocation — Project Loom’s contribution to this problem, available because this book pins Java 25. A virtual thread that blocks on ordinary I/O unmounts from its carrier platform thread rather than occupying it, so blocking code inside a @RunOnVirtualThread method costs almost nothing to run: orders of magnitude more of them can be in flight than worker-pool threads, because each one costs kilobytes rather than a platform thread’s megabyte-scale stack. “Unmounts” is the catch — it only happens for operations the JVM has taught virtual threads to unmount around. A synchronized block used to defeat this entirely, pinning the virtual thread to its carrier for the block’s duration, until JEP 491 in Java 24 removed synchronized-caused pinning for good. What’s left, even on Java 25, is native code: a virtual thread that calls into native code (JNI, the Foreign Function & Memory API) and blocks there still pins its carrier, because the JVM has no visibility into what a native frame is doing — a JDBC driver’s native networking layer or a library wrapping a C extension can reintroduce the exact problem virtual threads exist to solve, with the code calling into them looking no different whether it pins or not. The other caveat: libraries that lean on ThreadLocal for pooled, reused objects (a parser, a buffer, a formatter) stop pooling anything once every call gets its own disposable virtual thread — the pool turns into an allocation per call, a correctness-neutral but potentially expensive surprise for code that assumed thread reuse.
This chapter’s example shows @RunOnVirtualThread as a live third mode rather than only describing it, because a single findOptimalRoute method can’t carry two conflicting execution-model annotations at once. RoutingServiceImpl.onVirtualThread reproduces by hand what the annotation would do automatically if it were this method’s only annotation: it starts Thread.ofVirtual().start(...) around the identical lookup and completes a Uni from inside it. If this method’s only job were the virtual-thread model, the idiomatic version would simply read:
@RunOnVirtualThread
@Override
public Uni<FindOptimalRouteResponse> findOptimalRoute(FindOptimalRouteRequest request) {
return Uni.createFrom().item(() -> buildResponse(request));
}
— one annotation, no manual dispatch. The manual version exists here only because this chapter needs to hold three models up for comparison behind one RPC, which the next section explains.
The starving path, kept reachable
A chapter about choosing the right execution model is more convincing with a live wrong answer sitting next to the right one, not just a description of what a wrong answer would look like. RoutingServiceImpl.findOptimalRoute reads an execution-mode flag out of request.origin.name — a field that carries no routing meaning anywhere else in this example, repurposed here rather than adding a field to the frozen contract — and branches on it. The default (empty) value takes the @Blocking path already described. A value of "starving" does something the chapter’s whole argument says never to do: it runs the identical 400-millisecond lookup directly on an event-loop thread, via Context.runOnContext, hopping back onto the event loop from the worker thread @Blocking had already dispatched to. The point isn’t that this is a realistic mistake to type by accident — a method marked @Blocking won’t spontaneously decide to misbehave — it’s that the chapter’s timing assertions need a reachable counter-example to compare against, not an assertion about code nobody can run.
Making that counter-example reproducible took one more decision worth naming. Vert.x’s event-loop pool defaults to twice the host’s CPU count, so on any machine with four or more cores, eight concurrent starving-mode calls would mostly land on eight different event loops and never contend with each other — the demonstration would only work by accident. RoutingServiceImpl fixes this by lazily capturing one io.vertx.core.Context on the first starving call and reusing that same context for every one after it (starvingContext()), so every starving call is pinned to the same underlying thread no matter how many event loops the host actually runs. An earlier version of this example instead set quarkus.vertx.event-loops-pool-size=1 globally, shrinking the whole application to a single event loop — it worked, until a batch of starving calls left that one shared loop backed up long enough to slow down an unrelated batch of correct-model calls run immediately afterward. Pinning only the demo’s own starving path, rather than crippling the server’s entire concurrency budget to make one code path misbehave reliably, is the fix that stuck.
Python: grpc.aio, asyncio.to_thread, and the GIL
Python’s version of this problem has a second axis Java’s doesn’t: the Global Interpreter Lock. grpc.aio runs every RPC handler as a coroutine on a single asyncio event loop — there is exactly one thread executing Python bytecode for all of a grpc.aio server’s concurrent calls, full stop. A handler that calls a blocking function directly, with no await in between, runs that function to completion before the event loop can do anything else: no other coroutine on that loop — not even an unrelated RPC to a different method — gets a turn until it returns. This chapter’s starving mode does exactly that: blocking_lookup(origin_code, dest_code) is called in-line inside the async def FindOptimalRoute handler, synchronously, and the ~400 milliseconds it takes is 400 milliseconds the one event-loop thread spends doing nothing else.
The correct, default mode hands the same function to asyncio.to_thread, which schedules it on the loop’s default thread-pool executor (sized min(32, cpu_count + 4) by Python’s own default) and suspends the coroutine until it completes — the await point is exactly what lets the event loop serve other coroutines while this call’s I/O is in flight somewhere else. This is the same shape as Quarkus’s @Blocking: a small, explicit hop off the single scarce thread and onto a pool sized for blocking work, made at the one call site that needs it rather than as a property of the whole server.
Here is where the GIL matters, and where it’s easy to overstate the danger. time.sleep and a blocking file read both release the GIL for their duration — true of CPython’s I/O primitives for decades, independent of asyncio or threads entirely — so eight concurrent calls to blocking_lookup, each on its own thread-pool thread, actually overlap: each thread blocks on its own I/O while holding no lock that would stop the others from doing the same. The GIL only bottlenecks CPU-bound work — a pure Python loop with no I/O at all — because CPython has no equivalent release point for bytecode execution itself; eight threads running a CPU-bound function compete for the one GIL and serialize on it regardless of thread or core count. asyncio.to_thread is the right tool for this chapter’s I/O-bound lookup because the GIL gets out of the way for the part of the work that’s slow; it would not rescue a CPU-bound workload the same way, because threads don’t buy parallelism CPython’s interpreter loop refuses to grant.
This chapter’s claims stop at the edge of what was actually measured. Python 3.14 introduced an official free-threaded build (PEP 703) that removes the GIL entirely, which would in principle let CPU-bound work scale across threads the way I/O-bound work already does — but that build is a separate interpreter artifact from the standard one, grpcio’s free-threaded wheels are new enough that compatibility is still settling across the ecosystem, and this project’s CI and container image both run the standard, GIL-enabled 3.14 build pinned everywhere else in this book. Nothing here asserts a free-threaded measurement, because none was taken. The narrower claim that does hold: this chapter’s blocking work is I/O-bound by construction (a sleep and a file read, not a computation), which makes its overlap result GIL-agnostic — releasing the GIL during I/O was never what free-threading changes, so the same overlap should hold on either build.
How the code works
FindOptimalRoute’s shape is identical on both stacks: read two UN/LOCODEs and an execution-mode flag out of the request, perform the simulated blocking lookup on whichever thread the mode selects, and return a Route. The lookup itself (PricingCache.lookup in Java, blocking_lookup in Python) is the one piece of real work both modes share, so it’s worth walking before the dispatch logic around it.
The simulated I/O, named explicitly rather than hidden in one sleep. Both implementations do three things in sequence: sleep ~200ms standing in for a database round trip, perform a real blocking read of a small bundled cache file (pricing-cache.txt / pricing_cache.txt, five CSV-like rows — the same five routes Chapter 13’s catalog used, copied rather than shared, since this example has no code dependency on that chapter), then sleep ~200ms standing in for a slow, network-attached store. Two named delays around a real file read, rather than one sleep(0.4), keeps the “database hop” and “storage hop” shapes visible the way a real two-dependency lookup would actually be structured — an obvious seam for a reader to replace each sleep with a real call. A no-match pair falls back to the cache’s first row, a documented simplification: this chapter’s subject is where the blocking work runs, not route-matching logic.
Quarkus’s dispatch: one method, three branches, each choosing a thread. RoutingServiceImpl.findOptimalRoute is annotated @Blocking, so by the time its body runs, quarkus-grpc has already moved execution onto a worker-pool thread — the default branch does nothing further, running buildResponse right there and wrapping it in Uni.createFrom().item(...). The "starving" branch calls onEventLoop, which captures the lazily-cached starvingContext() and schedules the lookup onto it via context.runOnContext(...) — hopping off the worker thread the method was already safely running on and back onto the shared event loop, the one place blocking work isn’t supposed to go. The "virtual" branch calls onVirtualThread, which starts a fresh Thread.ofVirtual() per call and runs the lookup there. All three branches funnel into the same buildResponse helper and the same Uni.createFrom().emitter(...) pattern for the two branches that need to signal completion from a different thread than the one that received the request — emitter.complete(...) on success, emitter.fail(...) if the lookup throws, mirroring the same emitter contract Mutiny uses everywhere a result arrives asynchronously from outside the reactive pipeline itself.
Python’s dispatch: an if around one await. RoutingServiceServicer.FindOptimalRoute is a coroutine; its entire execution-model decision is one branch. The default path is route = await asyncio.to_thread(blocking_lookup, origin_code, dest_code) — the await is what returns control to the event loop while the thread-pool thread blocks on the sleeps and the file read. The starving path is route = blocking_lookup(origin_code, dest_code) with no await, no to_thread, called exactly like any other synchronous Python function — which is the point: nothing about the syntax warns that this line will block the one event loop every other concurrent call on this server depends on. That silence is the real hazard an asyncio codebase defends against by convention and code review, since the language doesn’t make a blocking call look any different from a non-blocking one the way Quarkus’s execution-model annotations at least attempt to.
The other seven RPCs. GetQuote, ListRoutes, NegotiateRoute, RecomputeRoutes, GetOperation, WatchOperation, and CancelOperation all return UNIMPLEMENTED — explicitly overridden on the Quarkus side (the same discipline every prior chapter’s example has followed), left unoverridden on the Python side, where the generated RoutingServiceServicer base class already does this for any method a subclass skips. GetOperation, WatchOperation, and CancelOperation are new to the contract as of Chapter 16’s Operation-lookup RPCs; they stay unimplemented here because this chapter’s subject is backend I/O concurrency on a single unary call, not the long-running-job machinery Chapter 16 builds next.
Decision
Pick an execution model from the shape of the work, not from habit. I/O-bound work — a database query, a file read, a downstream RPC, anything that spends most of its time waiting rather than computing — belongs offloaded off the scarce event-loop thread: @Blocking or @RunOnVirtualThread on Quarkus, asyncio.to_thread or run_in_executor on Python. Between the two Quarkus options, reach for @Blocking first — simpler to reason about, and a fixed worker pool's backpressure (callers wait once every worker thread is busy, rather than the server accepting unbounded work) is often what real load calls for. Reach for @RunOnVirtualThread only when a worker pool's fixed size actually caps the concurrent in-flight blocking calls you need, and audit dependencies first for native-code calls and ThreadLocal-heavy pooling, both of which quietly defeat the model. CPU-bound work — real computation, not I/O wearing a slow mask — is the one case none of this chapter's offloading helps: on the JVM it still needs a thread (worker pool or virtual, either is fine, since there's nothing to unmount), and on Python it runs into the GIL regardless of executor, unless specifically targeting a free-threaded build, which this book's pinned, standard-build CI does not. The failure mode worse than picking the wrong model is not picking one at all: an unannotated Quarkus method that blocks, or a Python coroutine with a bare blocking call and no await, starves every other concurrent caller on the same thread silently, with no error pointing at the cause — exactly the contrast this chapter's starving mode exists to make visible.
Build, run, observe
cd examples/15-backend-io-concurrency && ./verify.sh
verify.sh brings both services up with docker compose, confirms a single correct-model call returns the expected route.id and route.price, fires eight concurrent correct-model calls and asserts every one succeeds with identical values in well under the ~3.2 seconds eight serial 400-millisecond calls would take, then fires eight concurrent starving-mode calls and asserts that batch takes at least twice as long as the correct-model batch did. Expect PASS once every assertion holds for both stacks; the full timing numbers from a real run are in examples/15-backend-io-concurrency/README.md.
What you learned
- An event-loop thread’s scarcity, not its speed, is what blocking I/O threatens: a handful of threads serve every concurrent connection, so one blocked thread stalls every other caller scheduled on it, regardless of how much idle CPU the host has.
- Quarkus offers three execution models for the same blocking call — the event loop a method must never block,
@Blocking’s worker pool, and@RunOnVirtualThread’s per-call virtual thread — and the right choice depends on the work’s shape, not a style preference; the virtual-thread model’s own caveats (pinning on native code,ThreadLocalpooling) are new failure modes, not a universal upgrade. - Python’s
grpc.aioruns every coroutine on one event-loop thread, soasyncio.to_threadis the offload@Blockingcorresponds to; the GIL releases during I/O (letting offloaded I/O-bound work actually overlap) but not during pure computation, a distinction the free-threaded Python 3.14 build changes for CPU-bound work specifically, unmeasured on this book’s standard-build CI image.
Chapter 16 picks up the Operation-lookup RPCs this chapter left UNIMPLEMENTED — RecomputeRoutes returning a server-owned Operation a client polls, watches, or cancels, the async/long-running-job pattern this chapter’s single blocking call was never trying to be.
On the wire
# Correct model: offloaded, returns in about 400ms.
grpcurl -plaintext -d '{"origin":{"unLocode":"USNYC"},"destination":{"unLocode":"NLRTM"}}' \
localhost:9150 shipping.routing.v1.RoutingService/FindOptimalRoute
{
"route": {
"id": "rte_001",
"legs": [{ "carrier": "MAERSK", "origin": {"unLocode":"USNYC"}, "destination": {"unLocode":"NLRTM"} }],
"price": { "currency": "USD", "amount": "1200.00" }
}
}
# Starving model: same request, same ~400ms of work, but it runs on the
# shared event loop / blocks the one coroutine thread instead of offloading.
grpcurl -plaintext -d '{"origin":{"unLocode":"USNYC","name":"starving"},"destination":{"unLocode":"NLRTM"}}' \
localhost:9150 shipping.routing.v1.RoutingService/FindOptimalRoute
A single call to either variant looks identical from a grpcurl prompt — same request, same response, the same roughly 400-millisecond wait. The difference only shows up under concurrency, which is why verify.sh fires eight calls at once rather than timing one. A tool built for that kind of load test, ghz, would run this comparison at a scale verify.sh’s eight calls only sketch: ghz --insecure -d '{"origin":{"unLocode":"USNYC"}}' -c 50 -n 500 localhost:9150 shipping.routing.v1.RoutingService/FindOptimalRoute against the correct-model endpoint, then the same command with "name":"starving" added to the payload, comparing p50/p99 latency and throughput between the two runs. Those ghz numbers are illustrative of the shape this chapter argues for, not numbers this book has measured and pinned — verify.sh’s own loose, relative assertions (< 2.5s for eight concurrent correct calls, correct_total * 2 < starving_total) are the only timing claims this chapter stands behind, chosen to stay true across slower and faster hardware alike rather than to match one machine’s exact benchmark.
Verification status: unverified.
verify.sh ran green against both stacks on the authoring machine (a 22-core
host), including the eight-concurrent-call overlap and starve-vs-scale
assertions, across repeated runs; the highest-risk things a reader’s own run
should confirm are that the timing margins (< 2.5s,
correct_total * 2 < starving_total) hold on meaningfully slower or more
contended hardware, and that Vertx#getOrCreateContext()’s single-context
behavior (what starvingContext() depends on to make the starving path’s
serialization reproducible regardless of host core count) continues to
return the same context on repeated calls across future Quarkus/Vert.x
point releases.