Continuous Batching

Continuous batching is now the default assumption behind every modern LLM serving stack, but “default” hides a chain of decisions: which scheduling model to run, which framework implements it well, and who a request queue serves first when GPU capacity runs short. This entity sits inside the inference optimization stack, directly below the serving-layer lever the theme compares against quantization and sampling — the five articles here carry the full path from mechanism to production deployment to the fairness question the benchmark numbers don’t answer.

The continuous batching reading path: mechanism before deployment

Start with how iteration-level scheduling maximizes GPU throughput — it explains the scheduling mechanism itself: how a server slots a new request into a running batch the moment another one finishes, instead of waiting for the whole group to drain. Read the prerequisites and hard limits of continuous batching next; it names exactly what has to be in place first — KV-cache memory management, PagedAttention — and where the technique stops helping even once those pieces exist.

Once the mechanism is settled, the vLLM, TensorRT-LLM, and SGLang deployment guide turns it into a configuration decision: which engine, which queue depth, which preemption policy for your SLA. For evidence the decision pays off, Stripe’s 73% cost cut and SGLang’s RadixAttention surveys what production deployments are actually getting from it in 2026. Close with who waits longest when continuous batching decides — the scheduling question the throughput numbers never answer.

How continuous batching differs from inference, quantization, and sampling

Continuous batching gets folded into “just optimize inference” more often than any other lever in the stack, which loses a distinction worth keeping straight: inference is the umbrella runtime discipline — the whole autoregressive decoding loop — while continuous batching is one scheduling technique layered on top of it, specific to serving multiple concurrent requests. You can run inference perfectly well, one request at a time, with no batching at all.

It is also not quantization wearing a different name. Quantization changes what the model is — lower-precision weights, applied once before deployment — while continuous batching changes how requests are scheduled onto the GPU, a live decision remade every iteration. The two stack: a quantized model still benefits from a better scheduler, and vice versa. But fixing one when the other is the actual bottleneck wastes an engineering cycle.

The sharpest confusion is with temperature and sampling: both feel like “tuning knobs” set at request time, but continuous batching decides when a request’s tokens get computed, while sampling decides which token gets picked once that computation runs. A batching misconfiguration shows up as latency; a sampling misconfiguration shows up as wrong or repetitive output. Debugging one with the other’s lever burns a cycle without moving the number you’re chasing.

Common questions about continuous batching

Q: Is continuous batching the same thing as dynamic batching or in-flight batching? A: Yes — these are vendor-specific names for the same iteration-level scheduling technique; Hugging Face’s TGI and NVIDIA’s stack favor “in-flight batching,” others say “dynamic batching.” The mechanism explainer covers the technique regardless of which name your framework uses.

Q: Do I need continuous batching if my server only handles one request at a time? A: No — the entire benefit comes from overlapping multiple in-flight requests, and a single-request server has nothing to schedule. The prerequisites read is still worth knowing once your traffic grows past that point.

Q: If I’ve already quantized my model, does adding continuous batching still cut costs? A: Yes — the two compound rather than substitute for each other. Quantization shrinks what fits in memory; batching raises how much of that freed-up capacity actually gets used. Stripe’s deployment is a documented case of exactly that stacking.

Q: Can continuous batching make some requests wait far longer than others? A: Yes — admission into the running batch is a scheduling policy, and without explicit fairness constraints it can systematically favor short requests or high-priority tiers over others. The queueing analysis examines who loses out and why.

Part of the inference optimization stack · closest neighbour: inference. New to this from a software background? Start with the story: Inference Optimization for Developers: What Transfers and What Breaks.