What Is LLM Inference? Prefill, Decode and the KV Cache, Measured
What is LLM inference?
LLM inference is the work a language model does to answer a request. First it reads the whole question in one pass, called prefill. Then it writes the answer one token at a time, called decode, keeping notes on every token in the KV cache. On a rented RTX 4090, a small model read a question at 35,796 tokens a second and wrote at 642 tokens a second: reading is fast, writing is slow, and that gap explains most of how inference is tuned.
Save your progress and get the AI Engineering cheat sheet PDF
This guide explains, in plain words, what a language model does every time it answers, and why some answers are fast and others slow. It answers 10 questions with numbers we measured on a rented RTX 4090, and each answer links to the lesson the number comes from.
10 questions · 3 sections · answers from the course
These answers come from our course Hands-On Inference Engineering: Serve LLMs on GPUs with vLLM and SGLang: 43 lessons ready so far, each one run and measured on a rented NVIDIA GPU. The first lessons are free to read, with no card needed.

The short version
Think of two jobs. Reading the question decides how long people wait for the first word. Writing the answer decides how fast the words arrive, and it is limited by how quickly the GPU can read the model's weights from memory. The KV cache, the notes kept for every token, decides how long a conversation can be and how many people one GPU can serve.
1. What LLM inference is
Inference is the everyday part of a language model's life. Training happens once; inference happens every time somebody asks it something.
What is LLM inference?
LLM inference is the work a large language model does to answer a request: it reads the question, then writes the answer one token at a time. A token is a small piece of text, often part of a word. Training is how the model learned; inference is the model being used, and it is what a GPU spends most of its life doing in a real product.
Every request has two halves. In prefill the model reads all the tokens of the question in one pass through its weights. In decode it writes the answer, one pass for every new token, because each token depends on the one before it.
What is the difference between prefill and decode?
Prefill reads the question all at once, so it is fast per token. Decode writes the answer one token at a time, so it is slow per token. On a rented RTX 4090, measured with llama.cpp's benchmark, the small model Qwen3-0.6B read a 256-token prompt at 35,796 tokens a second and wrote at 642.0 tokens a second, about 56 times slower per token.
The two halves feel different to a user. Prefill decides how long they wait before the first word appears. Decode decides how fast the words then arrive.
Why is writing the answer so much slower than reading the question?
Every pass through the model reads all of its weights from the GPU's memory. In prefill one read is shared by every token of the question, so the GPU's arithmetic stays busy. In decode the whole read is paid for a single token, so the GPU spends most of its time waiting for memory.
That makes the memory speed, not the arithmetic, the limit for one person. With Qwen3-8B in 16 bits on our RTX 4090, the measured memory speed of 955 GB a second divided by the 15.15 GB the model reads for each token gives at most 63 tokens a second. We measured 58.2.
2. The KV cache: the model's notes on the conversation
If you remember one idea from LLM inference, make it this one. It decides how long a conversation can be and how many people one GPU can hold.

What is the KV cache?
The KV cache is the set of notes a model keeps on every earlier token of a conversation: a key and a value in every layer, worked out once and kept in GPU memory. Without it, the model would work them out again for every new token it writes.
The notes are not small. For Qwen3-0.6B they take 112 KiB per token, so one conversation of 32,768 tokens needs 3,584 MiB, about six times the 604 MiB of the model's own weights.
How do you work out the KV cache size per token?
Multiply 2 (a key and a value) by the number of layers, the number of key-value heads, the head size and the bytes per number. All of these are in the model's config.json file.
For Qwen3-8B that is 2 × 36 × 8 × 128 × 2 bytes = 147,456 bytes, 144 KiB, for every token. On a 24 GB RTX 4090, vLLM loaded the 16-bit weights in 15.27 GiB and had 4.56 GiB left for the cache, room for 33,216 tokens.
Why do long conversations make a model answer more slowly?
For every new token, attention reads the notes of every earlier token as well as the model's weights. The longer the conversation, the more there is to read.
On our RTX 4090, a 4-bit copy of Qwen3-8B in llama.cpp wrote 164.1 tokens a second with an empty cache and 114.9 tokens a second with 16,384 tokens already in it.
3. Measuring speed, and making it faster
These are the numbers a team watches once a model is in front of real users, and the first thing most of them change.

What is time to first token (TTFT)?
Time to first token is how long a user waits after sending a question before the first word of the answer appears. It is mostly prefill, plus any time spent waiting in the server's queue, so it grows with the length of the question and with how busy the server is.
With Qwen3-8B on our RTX 4090 and vLLM, the median TTFT was 76.0 ms with one person asking and 439.4 ms with 16 people at once. The slowest 1 in 10 of those 16-person answers waited 1,364.4 ms or more.
Why measure p90 and p99 latency instead of the average?
An average mixes fast and slow answers and lands where few people are. In the 16-person run above, the mean wait for the first word was 612.1 ms and the median 439.4 ms, while 5 of the 48 answers waited 1,364.4 ms or more.
The median tells you what a typical person felt. The p90 and p99, the values that 90% and 99% of requests stay under, tell you what the unlucky ones felt, and those are the people who complain.
How does batching make inference faster and cheaper?
Batching puts several people's next tokens into the same pass through the weights, so one read of the weights produces a token for each of them. With llama.cpp, Qwen3-0.6B wrote 586 tokens a second for one person and 4,854 tokens a second in total for 32 people at once.
Servers such as vLLM use continuous batching: between every step they remove people whose answers have finished and add people who are waiting, as long as their KV cache fits. With Qwen3-8B in vLLM, 32 people at once got 20.2 times the total output of one person.
What are the main ways to optimise LLM inference?
Most of them follow from the ideas above. Batch more people together to use the GPU's spare arithmetic. Store the weights or the KV cache in fewer bits, called quantization, so more conversations fit. Reuse the notes of a prompt that many requests share, called prefix caching. Let a small model guess the next few tokens and the big model check them, called speculative decoding.
Each one trades something. Fewer bits can lower the quality of the answers, and a bigger batch makes each person slower. Measure the change on your own model and GPU before you keep it.
Where to go next
Every number on this page comes from Hands-On Inference Engineering: Serve LLMs on GPUs with vLLM and SGLang, where you rent the GPU yourself, run each command and compare your numbers with ours. Start with the free introduction and the free lessons on prefill and decode and the KV cache. If you are also preparing for interviews, the two big courses together and the Inference Engineering interview pack are below.
These questions are a start. The interview asks more.
Get all 120 questions in the Inference Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (141 diagrams in all). A PDF you keep, junior to staff level.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
