Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
In short
Inference optimization is the set of serving techniques that get more tokens from each GPU-second without changing the model: the KV cache, continuous batching, paged memory, quantization, prefix caching and speculative decoding. How well they are applied largely sets the cost per token.
In short
Inference optimization is the set of serving techniques that get more tokens from each GPU-second without changing the model: the KV cache, continuous batching, paged memory, quantization, prefix caching and speculative decoding. How well they are applied largely sets the cost per token.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Inference Optimization across a run of lessons, not one page. These 4 alone are about 265 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Why LLM inference is slow and expensive, and the techniques that fix it: the KV cache, continuous batching, PagedAttention, quantization, and speculative decoding.
Why GPU serving gets expensive and how to fix it: the VRAM ceiling, batching, quantization, distillation, caching, card sharing, bin-packing, parallelism, autoscaling, spot, and horizontal vs vertical scaling.
How a trained model becomes a live API. Covers online vs batch serving, REST vs gRPC, dynamic batching, autoscaling and p99, with a batch simulator.
RAG triples a chat bill through retrieved context. Cut it with a semantic cache, embedding reuse, and a reranker, without serving confidently wrong answers.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with LLM Inference Optimization: Serving More Tokens Per GPUpay once,
yours for life
Every model call has two phases. Prefill is reading: the whole prompt goes through the model in one parallel pass. Decode is writing: the reply comes out one token at a time, each token fed back in before the next. They behave very differently. On a laptop running qwen2.5:3b in Ollama, 35 calls in shuffled order gave a median reading speed of 194 tokens a second and a median writing speed of 30. Even the slowest reading was faster than the fastest writing.
The reason is memory bandwidth, how fast bytes move from memory to the chip that computes. To write one token, the chip must use every weight once, so every weight is fetched from memory for every token. Reading a prompt fetches each weight once and uses it for many tokens. So writing waits on memory, while reading keeps the arithmetic busy.
These two phases map to the two numbers users feel. Time to first token (TTFT) is mostly prefill: on the same laptop, a 256-token prompt showed its first word after a median 1.00 second, and a 7,900-token prompt after 36.58 seconds, more than 99 percent of it reading. Tokens per second after that is set by decode. A chat app cares most about TTFT; a batch job cares only about total throughput. Decide which you are optimizing before changing any setting.
The KV cache is the first lever and every server uses it. It stores the keys and values attention computed for past tokens so they are not recomputed at each step. It turns work that would grow with the square of the reply length into work that grows in a straight line, at the cost of memory that grows with every token of every request.
Quantization stores weights in fewer bits, so each decode step moves fewer bytes. On the laptop, the same Qwen2.5 3B model wrote 11.1 tokens a second at 16 bits (6.18 GB), 18.3 at 8.5 bits (3.29 GB) and 28.4 at about 5 bits (1.93 GB), while reading speed barely changed. On a server GPU, a smaller model also leaves more memory for the KV cache, which means more concurrent users.
The cache itself can be made smaller. Grouped-query attention, chosen when a model is designed, lets several attention heads share one set of keys and values. For qwen2.5:3b, 16 query heads share 2 key/value heads, and one token's cache takes about 36.9 KB at 16 bits, or about 19.6 KB if the cache is stored in the q8_0 format.
Continuous batching applies each weight read to many requests at once and rebuilds the batch at every step, so a finished request leaves immediately and a waiting one joins. Static batching, by contrast, holds a whole group until the longest reply ends, leaving slots idle. Continuous batching is where most of the throughput gain of a tuned server comes from.
Paged KV memory makes that possible. PagedAttention, introduced with vLLM, stores each request's cache in small fixed-size blocks handed out on demand instead of one block reserved for the maximum length. Less memory sits reserved but unused, so more requests fit on the same card.
Speculative decoding attacks the one-token-per-step limit. A small draft model guesses the next few tokens, and the large target model checks all of them in one parallel pass, which costs about the same as producing one token because that pass is limited by memory reads. The longest run of agreed tokens is kept; at the first disagreement the target's own token is used. The output matches what the target model would have produced on its own, and the speedup depends on how often the draft guesses right.
Many requests share the same start: a system prompt, a long document, a set of instructions. Prefix caching keeps the KV cache of that start after a request ends, so the next request that begins the same way skips reading it. On the laptop, a 3,951-token prompt took a median 14.35 seconds to read, and 0.09 seconds when sent again with only the final question changed. Ten questions about the same start cost about 144 seconds of reading without reuse and about 15.2 seconds with it.
Hosted APIs offer the same thing. According to their documentation as read in September 2026, OpenAI applies prompt caching automatically on supported models for prompts of at least 1,024 tokens, and Anthropic turns it on with a cache_control field, charging 1.25 times the input price to write a cached start and 0.1 times to read it on most models. By that pricing, a 10,000-token start reused across 10 questions bills like 21,500 input tokens instead of 100,000, about 22 percent. Prices change, so check the current page.
Only an exactly matching start is reused. In the laptop lab, changing one word 25, 50 or 75 percent of the way into the prompt made the second read take 0.78, 0.55 and 0.28 of the first. The rule is to put stable content first and anything that changes, such as the question, a date or a user ID, last.
Because a GPU rents at a fixed hourly price, cost per token is the hourly price divided by tokens produced per hour. A worked example with assumed throughputs on a 3-dollar-an-hour H100: at 40 tokens a second a million tokens cost about 20.83 dollars; with continuous batching and paged memory at 560 tokens a second, about 1.49; adding 4-bit quantization at 900, about 0.93; adding speculative decoding at 1,600, about 0.52. The throughputs are illustrative; measure your own and redo the division.
None of these levers is free. Quantization can cost some quality: on the laptop, the 8-bit file kept the 16-bit file's top next word on 24 of 24 prompts, while the roughly 5-bit file kept 19 of 24, all changes on near ties. Bigger batches slow each user a little. Speculative decoding needs memory for a second model. A larger context window reserves cache memory whether you use it or not.
In practice you pick a serving runtime that already has these built in, such as vLLM, Hugging Face TGI or TensorRT-LLM, and spend your time tuning and measuring. Measure with medians over several runs, give each timed prompt a unique start so prefix caching does not fake a fast result, and test quality on your own examples after every change that touches the model's numbers.
vLLM
An open-source serving engine with continuous batching, PagedAttention, quantized model support and an OpenAI-compatible API.
Hugging Face Text Generation Inference
Hugging Face's serving toolkit, with continuous batching and support for quantized models.
NVIDIA TensorRT-LLM
NVIDIA's library that compiles models into optimized GPU kernels and uses in-flight batching.
llama.cpp
A C/C++ inference engine, used by Ollama, that runs quantized GGUF models on CPUs and consumer GPUs.
It is the engineering that makes a fixed model produce more tokens per GPU-second: caching keys and values, batching requests continuously, managing cache memory in pages, quantizing weights, reusing shared prompt starts and speculative decoding. The model's weights stay the same; the serving changes.
Time to first token is the wait before anything appears, mostly spent reading the prompt. Tokens per second is how fast the reply streams after that, set by the writing phase. On a laptop test, a 7,900-token prompt took 36.58 seconds to the first word, over 99 percent of it reading.
On a busy server, continuous batching with paged KV memory usually gives the largest throughput gain, because it keeps the GPU full of useful work. For a single user, quantization and prefix caching are often the biggest wins. Measure your own workload before choosing.
No. The large model checks the draft model's guesses and keeps only those it agrees with; at the first disagreement it uses its own token. The result matches what the large model would produce alone, and only the speed changes.
Send less prompt, since reading is most of the wait; reuse a shared start with prefix caching; keep the model loaded so there is no cold start; and use faster hardware. Faster writing does not help, because only one token is written before the first word appears.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
Continuous Batching
Rebuilding the inference batch after every token so a finished sequence leaves and a waiting request joins immediately, instead of every request waiting for the slowest one. It is what keeps a serving GPU busy.
Quantization
Storing model weights in fewer bits, such as FP8 or INT8, to shrink memory and speed up inference. Modern methods lose little quality on most tasks.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "LLM Inference Optimization: Serving More Tokens Per GPU" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the LLM Inference Optimization: Serving More Tokens Per GPU lessoncourse 2
Take models from notebook to production, with labs on real models.