free tool · no sign-up · every formula shown
Static and Continuous Batching Visualizer: why serving engines never wait for the slowest answer
A GPU can write many answers at once, but answers end at different times. Static batching waits for the slowest answer in each group before starting the next group. Continuous batching, which vLLM and SGLang use, gives a free place to the next person at once. Watch the idle time on each.
Model sizes, GPU specifications and prices were read from their official sources on 13 November 2026, and every number says where it came from.
Static batching against continuous batching
batching visualizerTwelve people ask questions whose answers have different lengths, and the GPU can work on four at once. Each row is one of those four places, each column is one step, and a hatched square is a step where that place sat idle.
Static: the next four start only when all four have finished
Busy 77% of the time so far, 12 of 12 answers finished.
Continuous: a free place takes the next person at once
Busy 88% of the time so far, 12 of 12 answers finished.
With these answers, static batching needs 34 steps and keeps the GPU busy 77% of the time; continuous batching needs 30 steps at 88%. On average a person gets their whole answer at step 19.4 against 16.8.
The assumptions behind these numbers
- Every step writes one token for every busy place, and each step takes the same time. On a real GPU a step with more people takes a little longer.
- The prompts are left out to keep the picture simple. vLLM also mixes the reading of new prompts into the same steps (chunked prefill).
- The answer lengths are drawn the same way every time, so both rows get exactly the same work: the same twelve answers, the same total number of tokens.
Three things to try
Make every answer the same length
Move the slider to all equal. The two ways become identical, which is why static batching once looked fine.
Make the lengths very different
Move it to a lot. Static batching now spends many steps waiting for one long answer.
Watch the average finish time
Continuous batching helps the average person most, because short answers no longer wait behind long ones.
Questions people ask
What is continuous batching?
It is scheduling at every step instead of every batch. After each step the engine removes finished answers and adds waiting requests in their place, so the GPU never sits idle waiting for the slowest answer in a group. It is also called in-flight or iteration-level batching.
Why does static batching waste the GPU?
A static batch runs until its longest answer is done. Every place whose answer finished early sits idle until then, and the people waiting in the queue wait too.
Does continuous batching make each answer faster?
It makes the GPU busier and the queue shorter. Each step may take a little longer with more people in it, so one person's tokens can arrive slightly slower, while the whole service answers far more people.
The other inference tools: GPU Memory Calculator for LLMs, LLM Latency and Throughput Estimator, LLM Cost per Million Tokens Calculator, KV Cache and PagedAttention Visualizer, Speculative Decoding Simulator. They come from the course Hands-On Inference Engineering, where each sum is worked on paper and then measured on a real GPU. To see a tool used on a real question, try the free exercises with worked answers.