Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
In short
The KV cache stores the keys and values that attention computed for every earlier token, so the model does not recompute them for each new token. It grows with every token and every concurrent request, which makes it a main limit on memory and on how many users one GPU can serve.
In short
The KV cache stores the keys and values that attention computed for every earlier token, so the model does not recompute them for each new token. It grows with every token and every concurrent request, which makes it a main limit on memory and on how many users one GPU can serve.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers KV Cache across a run of lessons, not one page. These 3 alone are about 156 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Measured on a laptop with qwen2.5:3b over 30 shuffled calls: the first word came back after a median 1.0 s for a 256-token prompt and 36.6 s for about 7,900 tokens. With the laptop's own slowing taken out, writing got about 1.1 tokens a second slower for every 1,000 prompt tokens. A bigger context window made Ollama report more memory: 2.05 GB at 2,048 tokens, 2.79 GB at 32,768.
Measured on a laptop with qwen2.5:3b: a 3,951-token prompt took a median 14.35 s to read, and 0.09 s when it was sent again with only the question after it changed. Changing one word 25%, 50% or 75% of the way in made the second read take 0.78, 0.55 and 0.28 of the first, close to 'everything before the first difference is reused'. The reported token count never dropped. Put what changes last.
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
Inside a transformer, attention is the step where each token gathers information from the tokens before it. In every layer, each token is turned into three vectors: a query, a key and a value. A new token compares its query with the keys of all earlier tokens and then mixes their values. K stands for keys and V for values.
The key and value of a past token never change, because the token itself never changes. So the model computes them once and keeps them. That store is the KV cache. Without it, writing token 100 would redo the work for tokens 1 to 99, token 101 would redo 1 to 100, and generation would get slower and slower as the reply grew.
With the cache, each new step computes a key and value only for the one new token and reuses everything stored for the past. Think of someone writing a report from a stack of pages: they keep notes on each page they have already read instead of rereading the whole stack before every sentence. The notes cost space, and that space is the price of the KV cache.
You can compute the size from the model's shape. For qwen2.5:3b, Ollama reports 36 layers, 16 attention heads, 2 key/value heads, and 2,048 numbers per token. 2,048 divided among 16 heads gives 128 numbers per head.
For one token the model stores a key and a value (x 2), in each of 36 layers (x 36), for each of 2 key/value heads (x 2), each 128 numbers long (x 128). That is 2 x 36 x 2 x 128 = 18,432 numbers. At 16 bits, 2 bytes per number, it is 36,864 bytes, about 36.9 KB per token. For 8,000 tokens that is about 0.29 GB. The general formula is 2 x layers x key/value heads x numbers per head x bytes per number, per token, per request.
The number is small here because 16 query heads share just 2 sets of keys and values. This design is called grouped-query attention, and models such as Llama 3 and Mistral use it. The cache can also be stored in fewer bits. In the q8_0 format, 32 numbers take 34 bytes, which gives 18,432 x 34 / 32 = 19,584 bytes, about 19.6 KB per token, a little over half.
A lab on a laptop (Apple M4, 24 GB) loaded qwen2.5:3b in Ollama with different context window sizes. The context window is the most tokens one request can hold, set in Ollama as num_ctx. Ollama reported 2.046 GB at a 2,048-token window, 2.262 GB at 8,192 and 2.793 GB at 32,768. Above 8,192, each extra 8,192 tokens added 0.177 GB, about 21.6 KB per token. The prompt was one word each time: the memory is reserved when the model loads, whether you use it or not.
That measurement caught a surprise. The 16-bit arithmetic, 36.9 KB per token, was far above what was measured. The reason: Ollama had been installed with Homebrew, and its service file set OLLAMA_KV_CACHE_TYPE to q8_0. The q8_0 arithmetic, 19.6 KB, explains about nine tenths of the measured step. Check your own server: the llama-server command line shows the format after --cache-type-k.
The cache also slows writing slightly. Each new token must read the stored keys and values of every earlier token. Across 30 shuffled calls, with the laptop's own slowdown taken out by a straight-line fit, writing got about 1.1 tokens a second slower for every 1,000 prompt tokens, from a starting speed of about 41. After a 7,900-token prompt that is about 8.6 tokens a second slower, roughly a fifth.
On a server, the model weights are loaded once and stay fixed. The KV cache grows with every token of every active request. For a model like Llama-2 13B the cache is on the order of 1 MB per token at 16 bits, so a few thousand tokens of context times dozens of users can use more GPU memory than the weights themselves. Long before compute runs out, you run out of room to hold everyone's cache.
Older servers reserved one contiguous block per request, sized for the longest possible reply. A short reply left most of that block empty, and nobody else could use it. PagedAttention, introduced with vLLM, stores the cache in small fixed-size blocks handed out on demand, the way an operating system hands out memory pages, and takes them back as soon as a request finishes. Less waste means more requests fit on one card.
The cache can also outlive a request. Prefix caching, called prompt caching by hosted providers, keeps the keys and values of a prompt's start so the next request that begins the same way skips reading it. In the lab, a 3,951-token prompt took a median 14.35 seconds to read, and 0.09 seconds when sent again with only the final question changed. Changing one word 25, 50 or 75 percent of the way in made the second read take 0.78, 0.55 and 0.28 of the first.
Set the context window to what you need, not the maximum the model allows. In the lab, going from 2,048 to 32,768 tokens added about 0.75 GB to a model of about 2 GB, more than a third, and all of it was reserved up front.
Put what changes last. Everything before the first difference between two prompts can be reused, so stable instructions and shared documents belong at the start, and the user's question, a timestamp or a user ID belongs at the end. A date on the first line makes the whole prompt new every time.
Do not time a repeated prompt. Ollama reports the same token count whether it read the prompt or reused it; only the time drops. In one check, a 1,130-token prompt took 2.53 seconds to read the first time and 0.04 seconds the second, with the same count both times. Give each timed prompt a unique first line, or you are measuring the cache.
vLLM
vLLM manages the KV cache with PagedAttention, storing it in fixed-size blocks that are allocated on demand and freed when a request finishes.
Ollama and llama.cpp
Ollama's OLLAMA_KV_CACHE_TYPE setting and llama.cpp's --cache-type-k option choose the storage format of the cache, such as f16 or q8_0.
Hugging Face Transformers
Transformers reuses past keys and values during generation through its cache, controlled by the use_cache setting.
Anthropic and OpenAI prompt caching
Both hosted APIs can reuse the processed start of a prompt; Anthropic uses a cache_control field and OpenAI applies caching automatically on supported models.
It is the stored keys and values that attention computed for every token the model has already processed. Keeping them means each new token only needs its own key and value computed. Without it, every step would redo the work for the whole history.
Per token it is 2 x layers x key/value heads x numbers per head x bytes per number. For qwen2.5:3b that is about 36.9 KB per token at 16 bits and about 19.6 KB at q8_0, so about 0.29 GB or 0.16 GB for 8,000 tokens. Multiply by the number of concurrent requests on a server.
The KV cache lives inside one request so the model can write without recomputing. Prompt caching keeps that cache after the request ends, so a later request with the same start skips reading it. In a laptop lab this cut a 3,951-token read from a median 14.35 seconds to 0.09 seconds.
Servers such as Ollama reserve cache memory for the whole window when the model loads. In the lab, raising num_ctx from 2,048 to 32,768 tokens raised the reported memory from 2.05 GB to 2.79 GB, with a one-word prompt each time.
Storing the cache in 8 bits halves its memory and loses a little precision. The lab that measured the q8_0 cache did not test whether it changed any answers, so check it on your own task before relying on it.
Reusing a Prompt's Start: Read Once, Answer Many Times
Measured on a laptop with qwen2.5:3b: a 3,951-token prompt took a median 14.35 s to read, and 0.09 s when it was sent again with only the question after it changed. Changing one word 25%, 50% or 75% of the way in made the second read take 0.78, 0.55 and 0.28 of the first, close to 'everything before the first difference is reused'. The reported token count never dropped. Put what changes last.
LLM Inference Optimization: Serving More Tokens Per GPU
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
Quantization
Storing model weights in fewer bits, such as FP8 or INT8, to shrink memory and speed up inference. Modern methods lose little quality on most tasks.
Continuous Batching
Rebuilding the inference batch after every token so a finished sequence leaves and a waiting request joins immediately, instead of every request waiting for the slowest one. It is what keeps a serving GPU busy.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "What a Long Prompt Really Costs: Wait, Writing Speed and Memory" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the What a Long Prompt Really Costs: Wait, Writing Speed and Memory lessoncourse 2
Take models from notebook to production, with labs on real models.