Continuous Batching

Continuous batching is an LLM inference scheduling technique that adds new requests to a running batch at every forward pass, replacing static batching to maximize GPU throughput and reduce waiting time for concurrent users.

What It Is

Every time you send a prompt to an AI model like Claude or ChatGPT, your request joins a queue. The server needs to process multiple requests at once to keep the GPU busy — that’s batching. But how those batches get organized makes a dramatic difference in how many people the system can serve at once.

With static batching (the older approach), the server collects a group of requests, processes them all together, and waits until every single one finishes before accepting new ones. Imagine a restaurant that seats everyone at once but refuses to clear any table until the last diner finishes dessert. Short meals wait for long ones, and empty seats go unused.

Continuous batching fixes this by treating each forward pass — one computation cycle that produces the next token — as a chance to update who’s in the batch. When one request finishes generating, the system immediately slots a waiting request into that open position without pausing anyone else. A request that needs ten tokens leaves after ten steps, and its slot gets reused right away.

This technique was introduced in the Orca system by Yu et al. at OSDI 2022 under the name “iteration-level scheduling.” The core insight was that since each token generation step is independent, there’s no reason to lock the entire batch until all sequences complete. According to Yu et al., this approach achieved up to 36.9 times the throughput of static batching at comparable latency — a difference that comes from eliminating all the dead time where GPUs waited on the slowest request.

In a modern inference pipeline, continuous batching handles the scheduling side of the problem: deciding when requests enter and leave the GPU. It works in tandem with memory-level optimizations like KV-cache management and PagedAttention, which decide how the model’s attention state gets stored in GPU memory. Scheduling without efficient memory management (or the reverse) leaves performance on the table. That’s why these three components — continuous batching, KV-cache, and PagedAttention — show up together as the foundational building blocks of every production inference stack.

How It’s Used in Practice

If you’re calling any cloud-hosted AI model today — Claude, ChatGPT, Gemini — continuous batching is already running behind the scenes. The inference engines that serve these models (vLLM, SGLang, TensorRT-LLM, and Hugging Face TGI among them) all implement continuous batching as their default scheduling strategy. You benefit from it without configuring anything.

For teams deploying their own models on-premises or in a private cloud, continuous batching is what allows a single GPU to handle dozens of concurrent users without most of them noticing a delay. When one user’s short two-sentence answer finishes, the freed GPU slot gets filled instantly by the next waiting request. The hardware stays busy generating tokens rather than sitting idle between batches.

During traffic spikes, the difference is stark — short requests cycle through quickly instead of getting stuck behind long-running completions.

Pro Tip: If you’re evaluating self-hosted inference options, confirm the serving framework supports continuous batching out of the box. All mainstream engines do, but older or custom setups may still default to static batching — switching that single configuration can multiply your throughput without any hardware changes.

When to Use / When Not

Common Misconception

Myth: Continuous batching makes each individual request faster. Reality: It makes the system serve more requests in the same time window. Your individual response might not arrive quicker — what improves is how many users get answers simultaneously. The GPU stays productive, which reduces average wait time across all users rather than speeding up any single response.

One Sentence to Remember

Continuous batching keeps every GPU slot filled by swapping finished requests for waiting ones at every token step — the scheduling backbone that makes high-traffic LLM serving practical alongside memory optimizations like KV-cache and PagedAttention.

FAQ

Q: What is the difference between continuous batching and static batching? A: Static batching waits for all requests in a group to finish before starting new ones. Continuous batching replaces completed requests immediately at each generation step, so the GPU never idles waiting for the slowest request.

Q: Do I need to configure continuous batching myself? A: Usually not. Major inference frameworks like vLLM, SGLang, and TensorRT-LLM enable it by default. If you’re using a cloud AI provider, it’s already active on their servers.

Q: How does continuous batching relate to PagedAttention? A: They solve different bottlenecks in the same pipeline. Continuous batching optimizes request scheduling — when requests enter and leave the batch. PagedAttention optimizes memory allocation — how KV-cache blocks get stored. Together, they form the two core building blocks of modern inference systems.

Sources

• Yu et al.: Orca: A Distributed Serving System for Transformer-Based Generative Models - Original paper introducing iteration-level scheduling for LLM inference
• Anyscale Blog: Achieve 23x LLM Inference Throughput with Continuous Batching - Practical throughput benchmarks and implementation guidance