What Is PagedAttention? How vLLM Stores the KV Cache in Pages
What is PagedAttention, and which inference engines support it?
PagedAttention is how vLLM stores a model's KV cache, the notes it keeps on every earlier token: in small pages of 16 tokens taken from a shared pool as a conversation grows, with a block table listing each conversation's pages in order. Nothing is reserved for a conversation that may never grow, so far more people fit on one GPU. It comes from vLLM's 2023 paper. Hugging Face TGI and TensorRT-LLM also have paged KV caches, and SGLang uses a pool of small slots with RadixAttention on top.
Save your progress and get the AI Engineering cheat sheet PDF
This page explains PagedAttention in plain words, starting from the problem it solves, and then shows what it looked like on a real server: the pages vLLM made, how many people fitted, and what happened when the pages ran out. It answers 14 questions, with numbers we measured on a rented RTX 4090.
14 questions · 4 sections · answers from the course
These answers come from our course Hands-On Inference Engineering: Serve LLMs on GPUs with vLLM and SGLang: 43 lessons ready so far, each one run and measured on a rented NVIDIA GPU. The first lessons are free to read, with no card needed.


The short version
Say the problem first: servers used to reserve memory for the longest possible conversation, and most of it stayed empty. Then the fix: small pages of the same size, handed out as needed, and a block table per conversation, like virtual memory in an operating system. Then the result: on our RTX 4090, 64 conversations ran at once on a server whose log said only one full-length conversation would fit.
If this helped, the full hands-on course goes much deeper.
In the course you start vLLM on a rented RTX 4090, count its pages from /metrics, watch one conversation take pages as it grows, then fill the cache with 160 people and read the preemption counter yourself.
1. What PagedAttention is
PagedAttention is a way of storing memory, not a new kind of attention maths. To see why it matters you need one idea first: the KV cache.
What is the KV cache that PagedAttention stores?
When a language model writes an answer, every new token looks back at every earlier token. To avoid working the earlier tokens out again each time, the model keeps their keys and values, two lists of numbers per token in every layer, in the GPU's memory. Those notes are the KV cache.
It grows with every token and it is large. The room left for it after the model's weights decides how many conversations one GPU can hold at once.
What we measured: For Qwen3-8B the notes take 147,456 bytes, 144 KiB, for every token. One conversation of 32,768 tokens needs about 4.5 GiB.
What is PagedAttention?
PagedAttention is the way vLLM stores each conversation's KV cache: in small pages of a fixed number of tokens, 16 by default, taken from one shared pool as the conversation grows. A block table, one per conversation, records which pages hold its tokens and in what order, so the pages can sit anywhere in memory.
It was introduced in the 2023 paper "Efficient Memory Management for Large Language Model Serving with PagedAttention", the paper that introduced vLLM. The paper compares it to an operating system's virtual memory: "one can think of blocks as pages, tokens as bytes, and requests as processes".
What we measured: On our RTX 4090 with Qwen3-8B, vLLM made 2,076 pages of 2.25 MiB each, room for 33,216 tokens in all.
What problem does PagedAttention solve?
Nobody knows how long a conversation will be. A question of 40 tokens may get an answer of 30 tokens or 3,000. The simple way to store notes is to give each conversation one solid block big enough for the longest it might become, the moment it starts. Almost all of that block is never written, and nobody else can use it.
The PagedAttention paper measured this in the serving systems of the time and found that only 20.4% to 38.2% of the KV cache memory held real token notes. The rest was reserved and empty, or lost in gaps between blocks of different sizes.
What we measured: With a limit of 32,768 tokens on our card, one reserved block is 4.50 GiB and the whole cache is 4.56 GiB, so one conversation fits. A conversation that ended after 193 tokens wrote 0.6% of its block.
Is there a simple way to picture it?
Think of a cloakroom with one long rail of hooks. The wasteful way keeps the whole rail for each guest, just in case, and turns the next guest away. The better way hands out one hook per coat as the coats arrive and writes each hook's number on the guest's ticket. A guest's hooks can be anywhere on the rail; the ticket keeps them together.
In vLLM the rail is the KV cache, a hook is a page of 16 tokens, and the ticket is the block table.
2. How the pages work
Once you know the picture, the details are short: pages are all the same size, handed out one at a time, and listed in order in a block table.

What is a block table?
It is the list, for one conversation, of which pages hold its tokens and in what order. When the model works out attention, it reads that conversation's keys and values page by page in the order the block table gives. As the conversation grows, it takes any free page and adds the page's number to the end of its table.
So a conversation's third page might be page 1,408 and its fourth page 630. Nothing ever has to be moved to keep one conversation's notes together.
Does PagedAttention waste any memory at all?
Very little. Only a conversation's last page can be part empty, because pages are handed out one at a time as each fills. With pages of 16 tokens, a conversation wastes at most 15 tokens of room, never thousands. And because every page is the same size, there are no awkward gaps between blocks of different sizes.
What we measured: A conversation of 340 tokens fills 21 pages and uses 4 of the 16 places on its 22nd page.
What is the block size in vLLM, and should you change it?
It is the number of tokens in one page, 16 by default in vLLM 0.33. You can read it, with the number of pages, from the vllm:cache_config_info counter at the server's /metrics address.
Smaller pages waste less room at the end of each conversation, and larger pages mean fewer entries in each block table. The default suits most models, and the attention backend in use may limit which sizes work, so measure before you change it.
Lesson: Memory and the KV cache in vLLM (settings reference)
How big is one page in bytes?
Multiply the tokens in a page by the KV cache size of one token. One token's size is 2 (a key and a value) times the number of layers, the number of key-value heads, the head size and the bytes per number, all read from the model's config.json.
What we measured: For Qwen3-8B, one token is 2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes, so one page of 16 tokens is 16 × 147,456 bytes = 2.25 MiB.
If this helped, the full hands-on course goes much deeper.
In the course you start vLLM on a rented RTX 4090, count its pages from /metrics, watch one conversation take pages as it grows, then fill the cache with 160 people and read the preemption counter yourself.
3. How many people fit, and what happens when the pages run out
This is the part interviewers and on-call engineers care about most: reading vLLM's start-up log correctly, and knowing what the server does when memory is full.

vLLM's log says "Maximum concurrency 1.01x". Can it serve only one person?
No. That number is the KV cache's size divided by the longest conversation the server allows: how many conversations of the full length would fit at once. It is a worst case. Because pages are handed out only as conversations grow, real conversations, which are much shorter, fit many at a time.
What we measured: With --max-model-len 32768 our log said 1.01x for 33,216 tokens of cache. The same server ran 64 conversations of about 340 tokens at once with the cache at most 66.8% full.
How do you work out how many users one GPU can serve with PagedAttention?
Work it out in pages. Divide the number of pages vLLM made by the pages one of your conversations needs at its longest, which is its tokens divided by 16, rounded up. That is roughly how many can run at once before the server starts pausing people.
What we measured: On our card, 2,076 pages and conversations of about 340 tokens (22 pages each) give about 94 at once before preemption begins.
What happens when vLLM's KV cache is full?
New requests wait in the queue. When a running request needs a new page and none is free, vLLM preempts a request: it frees that request's pages, puts it back in the queue, and works its notes out again when there is room. Nothing fails; the paused person simply waits longer.
A few preemptions at a peak are normal. Preemptions all the time mean you need more KV cache or fewer people per GPU.
What we measured: 160 people at once filled the cache to 100.0%, with 64 preemptions and up to 62 requests waiting. Every answer still came back complete, at 21.9 tokens a second each instead of 43.4 with 64 people.

4. Which engines use it, and what it made possible
PagedAttention started in vLLM, and the idea of a paged KV cache has spread. It also made a second trick cheap: sharing pages between requests.
Which inference engines support PagedAttention?
vLLM introduced PagedAttention and uses it for every model it serves. The idea of storing the KV cache in fixed-size blocks has been adopted widely: Hugging Face Text Generation Inference (TGI) and NVIDIA's TensorRT-LLM both document a paged KV cache.
SGLang takes a related path. It keeps the KV cache in a pool of small slots and builds RadixAttention on top, a tree of earlier prompts that lets new requests reuse any shared beginning. The names differ between engines, so check each engine's own documentation for the exact design in the version you run.
How does PagedAttention make prefix caching possible?
Because pages are the unit of storage, two requests that start with the same tokens can point their block tables at the same pages. vLLM labels each full page with a hash of its tokens and of the page before it, keeps the pages of earlier requests, and reuses them when a new request starts the same way. Only full pages of 16 tokens are shared, and matching stops at the first page that differs.
What we measured: Eight questions behind the same 2,958-token beginning found 87% of their prompt tokens in the cache, and their first word came after about 46 ms instead of 326 ms.
Does PagedAttention make the model compute faster?
Not directly. The attention maths is the same; what changes is how the keys and values are laid out in memory. The gain is capacity: memory that used to be reserved and empty now holds other people's conversations, so the GPU can work on a bigger batch, and a bigger batch writes far more tokens per second in total.
What we measured: On the same RTX 4090 with a small model, one request alone got 359 tokens a second, and 64 at once got 12,087 tokens a second in total.
Where to go next
If this helped, the full hands-on course goes much deeper.
In the course you start vLLM on a rented RTX 4090, count its pages from /metrics, watch one conversation take pages as it grows, then fill the cache with 160 people and read the preemption counter yourself.
Every number on this page comes from Hands-On Inference Engineering: Serve LLMs on GPUs with vLLM and SGLang, where you rent the GPU yourself, run each command and compare your numbers with ours. Start with the free introduction and the free lessons on prefill and decode and the KV cache. If you are also preparing for interviews, the two big courses together and the Inference Engineering interview pack are below.
These questions are a start. The interview asks more.
Get all 120 questions in the Inference Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (141 diagrams in all). A PDF you keep, junior to staff level.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
