Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
Rebuilding the inference batch after every token so a finished sequence leaves and a waiting request joins immediately, instead of every request waiting for the slowest one. It is what keeps a serving GPU busy.
In short
Continuous batching rebuilds a model server's batch at every generation step, so a finished request leaves at once and a waiting one takes its slot. It keeps the GPU doing useful work instead of waiting for the longest reply in a fixed batch.
In short
Continuous batching rebuilds a model server's batch at every generation step, so a finished request leaves at once and a waiting one takes its slot. It keeps the GPU doing useful work instead of waiting for the longest reply in a fixed batch.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Continuous Batching across a run of lessons, not one page. These 4 alone are about 305 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
Does batching help a small model on a CPU, and what does it cost in latency? On a quiet 1-vCPU EC2 box, predict_proba on 1,024 rows took 5.24 times as long as on one row, so one row in a big batch was 195 times cheaper. Then I put a micro-batcher into the chapter's FastAPI service and sent it random (Poisson) arrivals from half to 2.5 times the most it answered with no batching (S, 1,098.4 a second): 17 settings, 7 rates, 5 rounds, every setting compared with no batching on the same arrivals. With no batching the service fell behind at S and gave about 1,045 answers a second at most. Batches of up to 32 with no timed wait kept up at every rate, gave 2.62 times as many answers at 2.5 x S, and even at half of S cut the p99 from 5.26 ms to 2.50 ms. Every timed wait of 1 ms or more raised the median at that low rate. All 7,179,151 scores were equal to the bit to single-row scores. Every AWS step is recorded so you can build the box yourself.
Why GPU serving gets expensive and how to fix it: the VRAM ceiling, batching, quantization, distillation, caching, card sharing, bin-packing, parallelism, autoscaling, spot, and horizontal vs vertical scaling.
I built one small model service twice on a rented computer with one processor core: once as first written, and once with three changes the chapter had measured, a warm-up, one time limit with a stand-in answer, and a batcher. Behind both sat a slow helper that freezes on 1 lookup in 100, so the plain service's failure was built in: it waits for every lookup, and no rate passed. The checked service kept its slowest 1 in 100 answers within 25 ms, with 95 in 100 answers live, up to 450 requests a second in the middle round, about $0.0224 per million answers; the helper's 16 slots, not the processor, set that limit. The lesson ends with the whole chapter as a one-page checklist, and every AWS step is recorded so you can build the machine yourself.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with LLM Inference Optimization: Serving More Tokens Per GPUpay once,
yours for life
A language model writes one token at a time, and each new token depends on all the tokens before it. This is called autoregressive generation. To write a 500-token answer, the model runs 500 times in sequence, and there is no way to skip ahead within one reply.
Each of those steps is limited by memory, not by arithmetic. To produce one token, the GPU must read the model's weights from its memory; for a 13-billion-parameter model at 16 bits that is about 26 GB per step. The GPU's math units mostly wait for data. You can see the same imbalance on a laptop: with qwen2.5:3b in Ollama, the median reading speed over 35 calls was 194 tokens a second, and the median writing speed only 30.
Batching uses that imbalance. If the weights must be read anyway, apply them to many requests in the same pass. Serving 16 sequences in one step costs little more memory traffic than serving 1, so total tokens per second rise sharply. The question is how to form the batch, and that is where continuous batching comes in.
Static batching collects a group of requests, runs them together, and holds the whole group until the last one finishes. Reply lengths vary a lot: one request needs a one-line answer, another a 2,000-token essay. The short request finishes early, but its slot on the GPU sits empty until the long one is done. Think of a long restaurant table that is only cleared when every diner has finished, while a queue waits at the door.
Continuous batching rebuilds the batch at every decode step, the step that produces one token for each active request. When a sequence finishes, it leaves immediately and its slot is freed. A waiting request joins on the very next step instead of waiting for the whole batch to end. The GPU always works on as many active sequences as it can hold.
The idea appeared in research as iteration-level scheduling in the Orca system (Yu and colleagues, OSDI 2022) and was made widely used by vLLM. NVIDIA's TensorRT-LLM calls it in-flight batching. A simple illustration with four slots over 20 steps shows static batching near a third of the GPU's capacity and continuous batching near full use; those numbers come from a drawn example, not a benchmark, but the reclaimed idle time is the real source of the gain.
Each request in the batch has its own KV cache, the stored keys and values of every token it has seen so far. That cache lives in GPU memory and grows by one token's worth at each step. To add and remove requests at every step, the server must be able to hand out and take back cache memory quickly.
Older servers reserved one contiguous block of cache memory per request, sized for the longest reply allowed. A short reply used a fraction of its block and the rest could not be lent to anyone. PagedAttention, from the vLLM paper by Kwon and colleagues in 2023, stores the cache in small fixed-size blocks, like memory pages in an operating system, handed out as a sequence grows and returned the moment it finishes.
That is why the two are usually described as a pair. Continuous batching keeps the compute busy; paged memory keeps the memory busy. New requests also need their prompt read (the prefill phase) while other requests are writing. Servers such as vLLM can split a long prefill into chunks so that one long prompt does not stall every reply already in progress.
Batch size trades one user's speed for the whole server's efficiency. As the batch grows, total tokens per second climb, because each weight read serves more sequences. But each user's own streaming speed drifts down, because more sequences share each step. A worked chart for a 13B model on one H100 draws total throughput rising from about 42 to about 1,250 tokens a second between batch sizes 1 and 64, while each user's speed falls from about 42 to about 20. Treat those as illustrative, not measured.
The right setting is a product decision. A chat app caps the batch so text still streams briskly for each person. An overnight summarization job pushes the batch as high as memory allows, because only total throughput matters. A clean method is to pick a minimum per-user tokens per second as your service level, then raise the batch until you are about to break it.
Because a GPU rents at a fixed hourly price, cost per token follows throughput directly. At 3 dollars an hour and 40 tokens a second, the GPU produces 144,000 tokens an hour, so a million tokens cost 3 / 0.144, about 20.83 dollars. At 560 tokens a second it produces 2,016,000 tokens an hour, about 1.49 dollars per million. The throughput figures are assumptions for the example; the arithmetic is what to reuse with your own measured numbers.
It matters when many requests of different lengths arrive at once: a public chat endpoint, an internal assistant with many users, or agents that make many model calls. For one person running a model on a laptop, there is usually one request at a time and nothing to batch.
Its limit is memory. Every active sequence needs KV cache space, so long contexts and many users fill the GPU quickly, and when the cache is full new requests wait. Quantization helps indirectly: smaller weights leave more memory for the cache. Under heavy load each user's streaming speed and time to first token can still get worse, so watch both.
You almost never build this yourself. vLLM, Hugging Face Text Generation Inference, TensorRT-LLM and SGLang all include it. The work left to you is tuning: in vLLM, settings such as --max-model-len and --gpu-memory-utilization decide how much cache room there is, and a load test with your real prompt and reply lengths tells you where per-user speed falls below your target.
vLLM
vLLM schedules requests with continuous batching and stores the KV cache with PagedAttention.
Hugging Face Text Generation Inference
TGI, Hugging Face's serving toolkit, uses continuous batching to serve many generation requests at once.
NVIDIA TensorRT-LLM
TensorRT-LLM implements the same idea under the name in-flight batching.
SGLang
SGLang is an open-source serving runtime that also batches requests continuously during generation.
It is a scheduling method where the server rebuilds the batch after every generated token. Finished requests leave right away and waiting requests join on the next step. This keeps the GPU full instead of idling while the longest reply in a fixed batch finishes.
Static batching runs a fixed group to completion. Dynamic batching waits a short, bounded time to collect requests and then runs them together, which suits models that answer in one pass. Continuous batching changes the batch at every token step, which suits text generation where reply lengths vary widely.
Not by itself. It raises total throughput and cuts waiting in the queue, but each user's streaming speed can drop a little as the batch grows. Servers cap the batch size to keep per-user speed above a target.
Continuous batching adds and removes requests at every step, so it needs a cheap way to give and take back KV cache memory. PagedAttention provides that by storing the cache in small blocks allocated on demand. vLLM combines the two.
Almost never. Serving runtimes such as vLLM, TGI, TensorRT-LLM and SGLang include it. Your job is to set memory limits and batch settings and to load-test with realistic prompt and reply lengths.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.
This page explains the idea. The "LLM Inference Optimization: Serving More Tokens Per GPU" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the LLM Inference Optimization: Serving More Tokens Per GPU lessoncourse 2
Take models from notebook to production, with labs on real models.