Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
A way of storing a language model's KV cache in small fixed-size blocks instead of one big reserved slab, so GPU memory is not wasted and more requests fit at once.
In short
PagedAttention is a way to store a language model's KV cache in small fixed-size blocks that are handed out on demand, like memory pages in an operating system, instead of one large block reserved per request. Less GPU memory sits empty, so one GPU can serve many more requests at once.
In short
PagedAttention is a way to store a language model's KV cache in small fixed-size blocks that are handed out on demand, like memory pages in an operating system, instead of one large block reserved per request. Less GPU memory sits empty, so one GPU can serve many more requests at once.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers PagedAttention across a run of lessons, not one page. These 4 alone are about 257 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
Why is p99 so far from the median, and what sits in the tail? I sent 100,000 real requests, one at a time, to the features chapter's model on a quiet 1-vCPU EC2 box, five times, and repeated it with four single changes. Here the p99 was only 1.08 times the median and the p99.9 1.23 times, but the slowest request of every run was the very first one, 7.7 times the median. My main guess was wrong: no garbage collection ran during any of the 500,000 measured base requests. Slow requests came in bursts in which every step was slower, and the request after a slow one was slow too 41% to 62% of the time. Because of those bursts, pages of 10 calls in a row met the tail 3.3% to 5.4% of the time, not the 9.56% of 1 - 0.99^10. A warm-up removed the slow first request; gc.freeze() and a higher gc threshold could not be told apart from run-to-run noise. Every AWS step is recorded so you can build the box yourself.
How feature stores end train and serve skew, keep training and serving in sync, and let teams reuse features across every model.
A language model never sees letters or words. It sees tokens: small pieces of text, each with a number. This lesson shows how real tokenizers cut text, measured with OpenAI's tiktoken. This course's 1.27 million words come to 1.29 tokens per word. Numbers are cut into pieces of up to three digits. One sentence written in eight languages took 11 tokens in English but up to 100 in other languages with GPT-2's tokenizer, and 12 to 19 with GPT-4o's. That difference is paid in money and in context space.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with LLM Inference Optimization: Serving More Tokens Per GPUpay once,
yours for life
PagedAttention is a memory-management technique for serving large language models. It was introduced with vLLM, an open-source serving system, in a 2023 paper by Kwon and colleagues. The idea is borrowed from operating systems, which have managed memory in small pages for decades.
To see why it matters, you need one fact about how serving works. The thing that fills a GPU while it serves many users is usually not the model. It is a per-request store called the KV cache. PagedAttention changes how that store is laid out in memory. It does not change the model or its answers.
A language model writes one token at a time. A token is a small piece of text. Each new token depends on all the tokens before it, so the model looks back at the whole history on every step.
Inside the model, attention turns each token into two vectors, a key and a value. Once computed, they never change, because the token behind them never changes. So servers compute them once and keep them in GPU memory, also called VRAM. That store is the KV cache. Without it, every step would redo the work of all earlier steps, and generation would get slower and slower as the reply grew.
The cost is memory. The KV cache grows with every token, for every request running at the same time. The lesson gives an illustration: for a model like Llama-2 13B the cache is on the order of 1 megabyte per token. At 4,096 tokens of context, that is about 4 GB per user. On an 80 GB card holding a 13 GB model, four such users already need more memory for their caches than for the model.
So the number of users one GPU can serve is mostly a question of how well you use memory for the KV cache. That is the problem PagedAttention solves.
Older serving systems stored each request's KV cache as one contiguous block. Contiguous means all in one piece, side by side in memory. The server does not know in advance how long a reply will be, so it reserved a block big enough for the maximum possible length.
Most replies are much shorter than the maximum. A short reply left most of its reservation empty, and that empty space could not be lent to any other request. Gaps between blocks of different sizes also went unused. This waste is called fragmentation.
The vLLM paper measured how bad this was. In the existing systems it studied, only 20.4% to 38.2% of the KV cache memory held actual token data. The rest was reserved but empty, or lost to fragmentation. Since memory decides how many requests fit at once, wasted memory means fewer users per GPU and a higher cost per token.
PagedAttention cuts each request's KV cache into small blocks. Each block holds the keys and values for a fixed number of tokens. vLLM's default block size is 16 tokens. The blocks do not need to sit next to each other in memory.
A small lookup table, called a block table, records where each request's blocks really are. The request sees its cache as one ordered list of logical blocks. The table maps each logical block to a physical block somewhere in GPU memory. This is the same trick an operating system uses with virtual memory and page tables.
Based on a figure in the lesson LLM Inference Optimization: Serving More Tokens Per GPU.
Because the blocks are scattered, the attention calculation itself has to be changed to read keys and values through the block table. That is why it is called PagedAttention, and why it needs a custom GPU kernel built into the serving runtime.
PagedAttention matters most together with continuous batching. Batching means running many requests through the model in the same pass. Each step must read all the model weights from memory anyway, so applying them to many requests at once is nearly free extra throughput.
Continuous batching rebuilds the batch on every step. A finished request leaves at once, and a waiting one joins on the next step. That only works if KV cache space can be handed out and taken back cheaply. Paging provides exactly that. In the lesson's words, continuous batching keeps the compute full and paging keeps the memory full.
The vLLM paper reports that this design improved throughput by 2 to 4 times at the same latency, compared with earlier systems such as FasterTransformer and Orca. The gain was larger with longer sequences, larger models and more complex decoding. The lesson puts the two together at the center of a much bigger gap. A tuned server often gets 10 to 20 times the throughput of a naive one on the same hardware.
You almost never write this yourself. vLLM is the project that made continuous batching and PagedAttention mainstream. In its architecture, the scheduler runs continuous batching and the KV block manager runs PagedAttention. Hugging Face's TGI offers similar continuous batching, so the choice between them is mostly about which ecosystem you already use.
PagedAttention does not make the model smaller or each step faster. It does not reduce the size of the KV cache either; it stops memory from being wasted around it. To shrink the cache or the weights you need other tools. Quantization stores numbers in fewer bits. Grouped-query attention is a model design where several attention heads share one set of keys and values.
In practice, two vLLM settings decide how much room the cache gets. max-model-len caps the context length, and gpu-memory-utilization sets how much of the card vLLM may use. Teams tune both so the KV cache never runs out at peak load.
To see where PagedAttention fits next to quantization, speculative decoding and batch-size tuning, with the cost of each lever, read the LLM Inference Optimization lesson. The broader topic is covered in inference optimization and continuous batching.
vLLM
The open-source serving system PagedAttention was built for. Its KV block manager hands out fixed-size blocks, 16 tokens each by default, and its scheduler runs continuous batching on top.
Hugging Face TGI
Hugging Face's Text Generation Inference server, which the lesson describes as offering similar continuous batching and quantization support to vLLM.
NVIDIA TensorRT-LLM
NVIDIA's compiled runtime, which the lesson describes as trading a heavier build step for the last bit of latency and throughput on one model at large scale.
Yes. PagedAttention was introduced together with vLLM in the same 2023 paper, and it is how vLLM stores the KV cache. Its KV block manager hands out fixed-size blocks on demand, while its scheduler runs continuous batching.
They solve different problems. The vLLM paper describes FlashAttention as using tiling and kernel optimizations to cut the peak memory and the memory reads of the attention calculation itself. PagedAttention is about where the KV cache lives: it manages that cache in blocks so less memory is wasted across many requests.
No. It changes how the keys and values are laid out in GPU memory, not what they contain. The attention kernel reads the same numbers through a block table, so the model computes the same thing.
Because it borrows paging from operating systems. Memory is split into small fixed-size pages, here called blocks, and a table maps each request's logical blocks to physical ones. The attention step reads keys and values through that table.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
Continuous Batching
Rebuilding the inference batch after every token so a finished sequence leaves and a waiting request joins immediately, instead of every request waiting for the slowest one. It is what keeps a serving GPU busy.
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
Model Serving
Exposing a trained model behind an API that answers predictions within a latency budget, with batching, autoscaling, and versioning. Where most of the production cost and the request-path complexity lives.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "LLM Inference Optimization: Serving More Tokens Per GPU" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the LLM Inference Optimization: Serving More Tokens Per GPU lessoncourse 2
Take models from notebook to production, with labs on real models.