free tool · no sign-up · every formula shown
KV Cache and PagedAttention Visualizer: why vLLM keeps the cache in small pages
Every conversation on a GPU needs its own KV cache, and nobody knows in advance how long an answer will be. Watch the same stream of requests fill two GPUs: one reserves room for the longest answer in one piece, the other takes one small block at a time from anywhere.
Model sizes, GPU specifications and prices were read from their official sources on 13 November 2026, and every number says where it came from.
Why vLLM stores the KV cache in small pages
PagedAttention visualizerThe same requests arrive at two GPUs with the same memory. Each square is one block of 16 tokens, and each colour is one person's conversation. Press play and watch which GPU serves more people.
Reserve the longest answer, in one piece
Hatched squares are reserved for a request but hold nothing yet.
38% holds real tokens, 38% is reserved but empty.
2 running, 8 waiting, 5 finished.
PagedAttention: one block at a time
Blocks can be anywhere. A small table remembers where each conversation's blocks are.
63% holds real tokens.
4 running, 0 waiting, 11 finished.
So far, of the memory handed out to requests, 37% held real tokens the old way and 100% with paging. The paged GPU has finished 11 requests and the other 5, with the same memory.
The assumptions behind these numbers
- This is a teaching model, not vLLM itself: every running request writes one block per step and stops at its own length. It counts every block as full, so paging shows 100%; in real vLLM the last block of each request is partly empty, which is why the paper measured 96.3%.
- Requests need between 2 and 12 blocks by the end of their answer, drawn the same way every time, so both GPUs get exactly the same work.
- The PagedAttention paper (Kwon and others, SOSP 2023) measured that earlier serving systems used only 20.4% to 38.2% of their KV cache memory for real tokens, against 96.3% for vLLM in the same figure.
- When the paged GPU runs out of blocks, the newest running request is paused, its blocks are freed, and it goes back to the front of the queue to be worked out again later. vLLM does the same (preemption by recomputation), though it rebuilds the paused request's cache in one prefill step, while this picture refills it one block per step, which the course covers on the scheduler page.
Three things to try
Make the longest answer longer
The old way reserves more for every request, so fewer fit at once. Paging does not care, because it only takes what it uses.
Step through slowly
Use one step at a time and watch a long request's blocks land in different places on the paged side.
Count the finished requests
At the end, compare how many requests each GPU finished with the same memory.
Questions people ask
What is PagedAttention?
It is the way vLLM stores the KV cache: in small blocks of a fixed number of tokens, placed wherever there is free memory, with a block table that remembers where each conversation's blocks are. It is the same idea as pages in an operating system's virtual memory.
Why does reserving memory up front waste so much?
An answer might be 20 tokens or 2,000, so a system that reserves room for the longest one wastes most of it on short answers. The PagedAttention paper measured that earlier systems used only 20.4% to 38.2% of their KV cache memory for real tokens.
How big is one block in vLLM?
It depends on the hardware and the attention code. 16 tokens is a common size on GPUs. vLLM rounds the cache down to whole blocks when it starts, so the number of tokens it prints is always a whole number of blocks.
The other inference tools: GPU Memory Calculator for LLMs, LLM Latency and Throughput Estimator, LLM Cost per Million Tokens Calculator, Static and Continuous Batching Visualizer, Speculative Decoding Simulator. They come from the course Hands-On Inference Engineering, where each sum is worked on paper and then measured on a real GPU. To see a tool used on a real question, try the free exercises with worked answers.