Continuous Batching

What continuous batching is

Every time you send a prompt to an AI model like Claude or ChatGPT, your request joins a queue. The server needs to process multiple requests at once to keep the GPU busy — that’s batching. But how those batches get organized changes how many people the system can serve at once.

With static batching (the older approach), the server collects a group of requests, processes them all together, and waits until every single one finishes before accepting new ones. Imagine a restaurant that seats everyone at once but refuses to clear any table until the last diner finishes dessert. Short meals wait for long ones, and empty seats go unused.

Continuous batching fixes this by treating each forward pass — one computation cycle that produces the next token — as a chance to update who’s in the batch. When one request finishes generating, the system immediately slots a waiting request into that open position without pausing anyone else. A request that needs ten tokens leaves after ten steps, and its slot gets reused right away.

This technique was introduced in the Orca system by Yu et al. at OSDI 2022 under the name “iteration-level scheduling.” The core insight was that since each token generation step is independent, there’s no reason to lock the entire batch until all sequences complete. According to Yu et al., this approach achieved up to 36.9 times the throughput of static batching at comparable latency — a difference that comes from eliminating all the dead time where GPUs waited on the slowest request.

In a modern inference pipeline, continuous batching handles the scheduling side of the problem: deciding when requests enter and leave the GPU. It works in tandem with memory-level optimizations like KV-cache management and PagedAttention, which decide how the model’s attention state gets stored in GPU memory. Scheduling without efficient memory management (or the reverse) leaves performance on the table. That’s why these three components — continuous batching, KV-cache, and PagedAttention — show up together as the foundational building blocks of every production inference stack.

How continuous batching is used in practice

If you’re calling any cloud-hosted AI model today — Claude, ChatGPT, Gemini — continuous batching is already running behind the scenes. The inference engines that serve these models (vLLM, SGLang, TensorRT-LLM, and Hugging Face TGI among them) all implement continuous batching as their default scheduling strategy. You benefit from it without configuring anything.

For teams deploying their own models on-premises or in a private cloud, continuous batching is what allows a single GPU to handle dozens of concurrent users without most of them noticing a delay. When one user’s short two-sentence answer finishes, the freed GPU slot gets filled instantly by the next waiting request. The hardware stays busy generating tokens rather than sitting idle between batches.

During traffic spikes, the difference is stark — short requests cycle through quickly instead of getting stuck behind long-running completions.

If you’re evaluating self-hosted inference options, confirm the serving framework supports continuous batching out of the box. All mainstream engines do, but older or custom setups may still default to static batching — switching that single configuration can multiply your throughput without any hardware changes.

The continuous batching reading path: mechanism before deployment

Start with how iteration-level scheduling maximizes GPU throughput — it explains the scheduling mechanism itself: how a server slots a new request into a running batch the moment another one finishes, instead of waiting for the whole group to drain. Read the prerequisites and hard limits of continuous batching next; it names exactly what has to be in place first — KV-cache memory management, PagedAttention — and where the technique stops helping even once those pieces exist.

Once the mechanism is settled, the vLLM, TensorRT-LLM, and SGLang deployment guide turns it into a configuration decision: which engine, which queue depth, which preemption policy for your SLA. For evidence the decision pays off, Stripe’s 73% cost cut and SGLang’s RadixAttention surveys what production deployments are actually getting from it in 2026. Close with who waits longest when continuous batching decides — the scheduling question the throughput numbers never answer.

How continuous batching differs from inference, quantization, and sampling

Continuous batching gets folded into “just optimize inference” more often than any other lever in the stack, which loses a distinction worth keeping straight: inference is the umbrella runtime discipline — the whole autoregressive decoding loop — while continuous batching is one scheduling technique layered on top of it, specific to serving multiple concurrent requests. You can run inference perfectly well, one request at a time, with no batching at all.

It is also not quantization wearing a different name. Quantization changes what the model is — lower-precision weights, applied once before deployment — while continuous batching changes how requests are scheduled onto the GPU, a live decision remade every iteration. The two stack: a quantized model still benefits from a better scheduler, and vice versa. But fixing one when the other is the actual bottleneck wastes an engineering cycle.

The sharpest confusion is with temperature and sampling: both feel like “tuning knobs” set at request time, but continuous batching decides when a request’s tokens get computed, while sampling decides which token gets picked once that computation runs. A batching misconfiguration shows up as latency; a sampling misconfiguration shows up as wrong or repetitive output. Debugging one with the other’s lever burns a cycle without moving the number you’re chasing.

When continuous batching fits — and when it doesn't

Common questions about continuous batching

Q: Is continuous batching the same thing as dynamic batching or in-flight batching? A: Yes — these are vendor-specific names for the same iteration-level scheduling technique; Hugging Face’s TGI and NVIDIA’s stack favor “in-flight batching,” others say “dynamic batching.” The mechanism explainer covers the technique regardless of which name your framework uses.

Q: Do I need continuous batching if my server only handles one request at a time? A: No — the entire benefit comes from overlapping multiple in-flight requests, and a single-request server has nothing to schedule. The prerequisites read is still worth knowing once your traffic grows past that point.

Q: If I’ve already quantized my model, does adding continuous batching still cut costs? A: Yes — the two compound rather than substitute for each other. Quantization shrinks what fits in memory; batching raises how much of that freed-up capacity actually gets used. Stripe’s deployment is a documented case of exactly that stacking.

Q: Can continuous batching make some requests wait far longer than others? A: Yes — admission into the running batch is a scheduling policy, and without explicit fairness constraints it can systematically favor short requests or high-priority tiers over others. The queueing analysis examines who loses out and why.

Part of the inference optimization stack · closest neighbour: inference. New to this from a software background? Start with the story: Inference Optimization for Developers: What Transfers and What Breaks.

Sources

• Yu et al.: Orca: A Distributed Serving System for Transformer-Based Generative Models - Original paper introducing iteration-level scheduling for LLM inference
• Anyscale Blog: Achieve 23x LLM Inference Throughput with Continuous Batching - Practical throughput benchmarks and implementation guidance