The course map: every lesson, and what to read before it
Each card is one page of the book, grouped by part. An arrow runs from a page to each later page that builds on it, so you can see what to read first before you open anything. Point at a page, or tap it once on a phone, to light up the pages it needs and the pages that need it. Click it, or tap it a second time, to open it.
The same map as a list
Start here
Part 1 · Inference from zero
- What a model is Read first: Introduction: what this course builds, Before you start
- Inside a Hugging Face model, file by file Read first: Before you start, What a model is
- Tokens: how a model reads and writes text Read first: Before you start, What a model is
- How a language model writes, one token at a time Read first: Before you start, What a model is, Tokens: how a model reads and writes text
- Inside a transformer, in plain words Read first: Before you start, What a model is, How a language model writes, one token at a time
- What a GPU is, and why models run on one Read first: Before you start, What a model is, Inside a transformer, in plain words
- Why inference is where the money goes Read first: Before you start, Tokens: how a model reads and writes text, What a GPU is, and why models run on one
- Timing one answer from llama.cpp, step by step Read first: Before you start, Tokens: how a model reads and writes text, How a language model writes, one token at a time
- Serving a model with vLLM Read first: Before you start, Inside a transformer, in plain words, Timing one answer from llama.cpp, step by step
Part 2 · The maths of one request
- Prefill and decode Free Read first: Introduction: what this course builds, Before you start
- Why writing is limited by memory speed Read first: Before you start, Prefill and decode
- When a GPU waits for memory and when it waits for maths Read first: What a GPU is, and why models run on one, Prefill and decode
- The KV cache Free Read first: Before you start, Prefill and decode
- Sizing GPU memory Read first: Before you start, Why writing is limited by memory speed, The KV cache
- Why newer models need a smaller cache Read first: Inside a transformer, in plain words, The KV cache
- Batching: serving many people at once Read first: Prefill and decode, Why writing is limited by memory speed, The KV cache, Sizing GPU memory
- The latency numbers that matter Read first: Prefill and decode, Why writing is limited by memory speed, Batching: serving many people at once
- Kernels and FlashAttention Read first: Prefill and decode, When a GPU waits for memory and when it waits for maths, The KV cache
- Long context: 128K tokens and beyond Coming
Part 3 · vLLM in depth
- How vLLM is built Read first: Serving a model with vLLM, Prefill and decode
- PagedAttention Read first: Serving a model with vLLM, The KV cache, How vLLM is built
- The vLLM scheduler Read first: Serving a model with vLLM, How vLLM is built, PagedAttention
- Chunked prefill Read first: Serving a model with vLLM, Prefill and decode, The vLLM scheduler
- Prefix caching Read first: Serving a model with vLLM, Prefill and decode, PagedAttention
- vLLM memory settings Read first: Serving a model with vLLM, The KV cache, Sizing GPU memory, How vLLM is built
- vLLM serving settings Read first: Serving a model with vLLM, How vLLM is built
- Sampling parameters Read first: Tokens: how a model reads and writes text, Serving a model with vLLM, How vLLM is built
- Calling vLLM well from your code Read first: Serving a model with vLLM, Batching: serving many people at once, How vLLM is built
- Serving many fine-tuned versions on one GPU Read first: Serving a model with vLLM, The KV cache, How vLLM is built, Calling vLLM well from your code
- Start-up time and cold starts Read first: Serving a model with vLLM, Why writing is limited by memory speed, How vLLM is built
- Loading models and chat templates
- Repetition and length controls
- Speed modes
- Running vLLM as a real service
- The scheduler's other controls
Part 4 · Smaller and faster models
Part 5 · SGLang
- Your first SGLang server Coming
- RadixAttention Coming
- SGLang settings that change a number Coming
- Structured output at serving time Coming
- A fair vLLM and SGLang benchmark Coming
- Choosing a serving engine Coming
- TensorRT-LLM and NVIDIA Triton Coming
- llama.cpp, GGUF and Ollama Coming
- Serving embedding and reranker models Coming
- Serving vision and multimodal models Coming
Part 6 · Many GPUs
- Why split a model across GPUs Coming
- Tensor parallelism Coming
- Pipeline parallelism Coming
- More copies instead of bigger splits Coming
- Mixture-of-experts models and expert parallelism Coming
- The links between GPUs Coming
- AMD GPUs and ROCm Coming
Part 7 · Serving at scale with Kubernetes and llm-d
Part 8 · Measure, watch, pay and prove it
- Benchmarking honestly Coming
- Service levels and load tests Coming
- Cost per million tokens Coming
- Watching a model server Coming
- Tracing and profiling a slow request Coming
- Reliability and security for model serving Coming
- Managed endpoints: Bedrock, SageMaker, Vertex AI and NIM Coming
- Host it yourself or use an API Coming
- Capstone: plan a real service Coming
- Capstone: serve it and prove the cost Coming
- Inference engineer interview questions Coming
Part 9 · vLLM settings reference
- Loading a model
- Memory and the KV cache
- The scheduler and batching
- Parallelism
- Quantization
- Speculative decoding
- Structured output, tool calling and reasoning
- LoRA adapters
- Images, audio and video
- The API server and its security
- Logs, metrics and traces
- Compilation, CUDA graphs and kernels
- Distributed and disaggregated serving
- Sampling and request parameters
The words this course teaches
Each word links to the page that explains it properly. In the pages themselves, the first time one of these words appears it is underlined with dots: point at it or tap it to read what it means.
- API server vLLM's front process: it speaks HTTP in OpenAI's format, turns questions into tokens, turns the answer's tokens back into text and streams them out. Taught in "How vLLM is built"
- Arithmetic intensity The number of operations a job does for every byte it reads from memory. It tells you whether the job will wait on memory or on arithmetic. Taught in "When a GPU waits for memory and when it waits for maths"
- Attention The half of each layer where the token being worked on looks back at the earlier tokens and decides which of them matter for what comes next. Taught in "Inside a transformer, in plain words"
- Attention backend The library of GPU programs a serving engine uses for the attention step, such as FlashAttention in vLLM. Taught in "Kernels and FlashAttention"
- Attention head One of several separate looks back that run side by side in a layer, each with its own query. Taught in "Inside a transformer, in plain words"
- Batching Serving several people at once, so one read of the weights writes a token for each of them. Because writing waits on memory, the extra people cost very little. Taught in "Batching: serving many people at once"
- BF16 (bfloat16) The 16-bit number format most models use, 2 bytes a number. It has the range of a 32-bit number but only two to three significant digits. Taught in "What a model is"
- Block table The list, for one conversation, of which pages hold its tokens and in what order: the ticket in the cloakroom picture. Taught in "PagedAttention"
- Chat template The model's own rules for turning a list of messages into one string of tokens, with a few extra marker tokens that show who is speaking. Taught in "Prefix caching"
- Chunked prefill Reading a long question over several steps, in pieces that fit beside the tokens being written for everyone else, so one long prompt does not freeze every other answer. Taught in "Chunked prefill"
- Cold start and warm start A cold start is the first start on a machine where nothing has been prepared, so the server downloads, compiles and records everything. A warm start finds that work saved from an earlier start and is much faster. Taught in "Start-up time and cold starts"
- Compile cache The folder where vLLM keeps the code it compiled, so a later start can load it instead of compiling again. Taught in "Start-up time and cold starts"
- Compute-bound Limited by how fast the GPU can do arithmetic, while the memory waits. Reading a prompt of a few hundred tokens or more is usually compute-bound. Taught in "When a GPU waits for memory and when it waits for maths"
- Concurrency How many people, or requests, are being served at the same moment. Taught in "Batching: serving many people at once"
- Continuous batching Deciding the batch afresh at every step, so finished requests leave and new ones join at once and the batch stays full. Taught in "Batching: serving many people at once"
- Cost per million tokens The price of a GPU hour divided by the tokens written in that hour, times a million. Hosted models are priced the same way, usually higher for output tokens than for input tokens. Taught in "Why inference is where the money goes"
- CPU (processor) The computer's central processing unit, with a few large, flexible cores. It is good at many different tasks, and much slower than a GPU at the arithmetic a model does. Taught in "What a GPU is, and why models run on one"
- CPU offload Keeping part of a model's weights in the computer's ordinary memory instead of on the GPU. It makes room, but the GPU must read those weights across PCIe for every token, so writing becomes much slower. Taught in "vLLM memory settings"
- CUDA NVIDIA's software for running programs on its GPUs. NVIDIA also calls the GPU's small cores CUDA cores. Taught in "What a GPU is, and why models run on one"
- CUDA graph A recording of a whole series of GPU kernels that the GPU can replay with one launch. It saves the processor the cost of sending each kernel on every step. Taught in "Kernels and FlashAttention"
- Decode The second half of a request, where the model writes the answer one token per pass through its weights. It is slow per token, because every token reads all the weights again. Taught in "Prefill and decode"
- Engine core vLLM's second process. It runs the scheduler and, on one GPU, the worker that runs the model. Taught in "How vLLM is built"
- Feed-forward, or MLP The half of a layer where each token's numbers are widened, bent and narrowed again, by themselves, without looking at other tokens. Taught in "Inside a transformer, in plain words"
- Fine-tuning Training an existing model a little more, so that it does one job its own way. Taught in "Serving many fine-tuned versions on one GPU"
- FlashAttention An attention kernel that works on small tiles kept on the chip, so the big table of scores is never written to the GPU's main memory. It does the same maths with fewer trips to memory. Taught in "Kernels and FlashAttention"
- FP8 A format that stores each number in 8 bits, half the bytes of 16-bit BF16. Taught in "Why writing is limited by memory speed"
- GGUF llama.cpp's model file format, with a header that lists every tensor in the file. Taught in "Why writing is limited by memory speed"
- GiB, MiB and KiB Units of 1,024 bytes (KiB), 1,024 KiB (MiB) and 1,024 MiB (GiB). A GiB is a little more than a GB, about 1.07 GB, and vLLM and CUDA count in GiB. Taught in "Timing one answer from llama.cpp, step by step"
- Goodput The number of requests a second that met their speed targets, such as a first token within 2 seconds. A request that misses its target is wasted work for the customer, so goodput is a truer measure of useful capacity than raw tokens a second. Taught in "The latency numbers that matter"
- GPU A graphics processing unit: a chip with thousands of small cores that do the same kind of step on many numbers at once, with fast memory beside them. Taught in "What a GPU is, and why models run on one"
- GPU memory (VRAM) The memory beside a GPU, where the model's numbers must sit before the GPU can use them. Taught in "What a model is"
- Greedy decoding Always taking the most likely next token. The same start then always gives the same text, and temperature 0 does exactly this. Taught in "How a language model writes, one token at a time"
- Grouped-query attention (GQA) A design where several query heads share one key and one value, which makes the KV cache several times smaller. Taught in "Why newer models need a smaller cache"
- HBM High-bandwidth memory: the faster kind of GPU memory used on data-centre cards such as the H100. Taught in "What a GPU is, and why models run on one"
- Inference Using a finished model's learned numbers to answer a new question. Nothing is learned and nothing in the model changes while it answers. Taught in "What a model is"
- Inter-token latency (ITL) Every single gap between two tokens of an answer, each counted on its own, so one long pause in the middle shows up instead of being averaged away. Taught in "The latency numbers that matter"
- Kernel One small program that runs on the GPU, such as one multiplication of a layer's table, carried out by thousands of threads at once. Writing one token takes hundreds of kernels. Taught in "Kernels and FlashAttention"
- KV cache The keys and values a model keeps for every earlier token in every layer, its notes on the conversation, so they are worked out only once. It grows with every token and decides how many conversations fit on a GPU. Taught in "The KV cache"
- Language model A model whose input is text and whose output is a score for every possible next piece of text. Taught in "What a model is"
- Large language model (LLM) A model trained on a great deal of text to predict the next piece of text. Chat assistants are built on one. Taught in "Introduction: what this course builds"
- Layer One stage of the model's arithmetic, with its own tables of numbers. A layer has an attention half and a feed-forward half, and a model is a stack of identical layers. Taught in "Inside a transformer, in plain words"
- llama.cpp An open-source program that runs language models. It comes with commands to ask a model a question, run a model server and measure speed. Taught in "Before you start"
- Logits The raw score the model gives every token in its vocabulary, each time it is about to write one. A higher score means a better fit. Taught in "How a language model writes, one token at a time"
- LoRA (low-rank adaptation) Fine-tuning that freezes the model's tables and trains two thin tables beside each one. The small adapter file is loaded beside the unchanged base model, so one GPU can serve many fine-tuned versions. Taught in "Serving many fine-tuned versions on one GPU"
- Matrix A table of numbers with rows and columns that turns one list of numbers into another. Multiplying by these tables is the main work inside a model. Taught in "Inside a transformer, in plain words"
- Maximum concurrency vLLM's count, in its start-up log, of how many conversations of the full allowed length would fit in the KV cache at once. It is a worst case, not a limit on users. Taught in "Sizing GPU memory"
- Memory bandwidth (memory speed) How many bytes a second a GPU's memory can deliver to its cores. For writing an answer it matters more than how fast the GPU can calculate. Taught in "Why writing is limited by memory speed"
- Memory-bound Limited by how fast memory can deliver bytes, while the arithmetic waits. Writing an answer is memory-bound. Taught in "Why writing is limited by memory speed"
- Model server A program that loads a model once, keeps it in memory and answers questions sent to it over the network. Taught in "Timing one answer from llama.cpp, step by step"
- Multi-head latent attention (MLA) A design where each layer keeps one short compressed list of numbers per token, from which the keys and values are rebuilt, so the cache is much smaller. Taught in "Why newer models need a smaller cache"
- OpenAI-compatible API A server that accepts the same requests, at the same addresses and in the same format as OpenAI's service, so the OpenAI library can call it unchanged. Taught in "Serving a model with vLLM"
- Page, or block, of the KV cache One fixed piece of the KV cache, 16 tokens by default in vLLM, so the tokens that fit are always a multiple of 16. Taught in "PagedAttention"
- PagedAttention vLLM's way of storing the KV cache in small pages of a fixed number of tokens, taken as they are needed and listed in a block table, so memory is not reserved and wasted. Taught in "PagedAttention"
- PCIe The link between the GPU and the rest of the computer. On the RTX 4090 it moves at most 32 GB a second each way, far slower than the GPU's own memory. Taught in "vLLM memory settings"
- Percentile (p50, p90, p99) The value that a given share of the measurements are at or below. The p50 is the median, and the p99 is the wait that 99 in 100 requests beat, which shows the slow users an average hides. Taught in "The latency numbers that matter"
- Pod RunPod's name for one rented machine. A running pod bills every second, including while you read, so you switch it off when you finish. Taught in "Before you start"
- Preemption Pausing a running request when the KV cache is full, freeing its memory, and working its notes out again later. The paused person waits much longer. Taught in "PagedAttention"
- Prefill The first half of a request, where the model reads all the question's tokens in one pass and writes its notes on them into the KV cache. It is fast per token, and it decides the wait for the first word. Taught in "Prefill and decode"
- Prefix caching Keeping the KV cache pages of earlier requests and reusing them for a new request that starts with the same tokens, so that beginning is not read again. Taught in "Prefix caching"
- Quantization Storing a model's numbers in fewer bits, for example 8 instead of 16. The model takes less memory and writes faster, as long as the quality of its answers survives. Taught in "Quantizing a model's weights"
- Query, key and value The three lists of numbers attention uses for each token. The query is what the token is looking for, the key lets later tokens find it, and the value is what it hands over once found. Taught in "Inside a transformer, in plain words"
- Roofline A chart of a GPU's two limits, a sloping memory roof and a flat arithmetic roof, with each job placed under them by its arithmetic intensity. Taught in "When a GPU waits for memory and when it waits for maths"
- Sampling Choosing the next token by chance, in proportion to its probability, instead of always taking the most likely one. Taught in "How a language model writes, one token at a time"
- Scheduler The part of vLLM's engine core that decides, at every step, which conversations the model works on and how many tokens of each. Taught in "The vLLM scheduler"
- Seed The number a random draw starts from. The same seed gives the same draws, so a sampled answer can be repeated. Taught in "How a language model writes, one token at a time"
- Softmax The step that turns a row of raw scores into probabilities that add up to 100%, keeping their order. Taught in "How a language model writes, one token at a time"
- Speculative decoding Letting a small, fast model guess several tokens ahead while the large model checks them all in one pass. It is the one way to write faster for one person than the memory limit allows. Taught in "Speculative decoding: guessing ahead"
- Static batching Sending a group of requests through together, so the whole group finishes only when its slowest member does. Taught in "The vLLM scheduler"
- Streaming Sending the answer piece by piece as it is written, instead of all at once at the end. It changes how long a person feels they wait. Taught in "Timing one answer from llama.cpp, step by step"
- Streaming multiprocessor (SM) One of the groups of cores a GPU is built from; the cores in a group share a small, fast local memory. Taught in "What a GPU is, and why models run on one"
- System message The instructions sent before the user's message, which the model reads first. Taught in "Prefix caching"
- Temperature The number every score is divided by before softmax. Below 1 makes the choice more careful, above 1 makes it bolder, and 0 means greedy. Taught in "Sampling parameters"
- Tensor A table of numbers with any number of dimensions, such as one layer's table of weights. A model file holds many tensors. Taught in "What a model is"
- Tensor core A unit inside a GPU that multiplies two small tables of numbers and adds the result to a third in one step. Most of a GPU's arithmetic speed comes from them. Taught in "What a GPU is, and why models run on one"
- Tensor parallelism Splitting each of a model's large tables across several GPUs, so every layer is shared out and the GPUs talk to each other at every step. Taught in "Tensor parallelism" (coming)
- TFLOPS Trillions of arithmetic operations on decimal numbers each second, the unit of a GPU's arithmetic speed. Taught in "What a GPU is, and why models run on one"
- Throughput The whole server's output: the answer tokens written each second for everyone together. Taught in "Batching: serving many people at once"
- Time per output token (TPOT) The average gap between the tokens of one answer after the first one. It is the speed a reader feels once the answer has started. Taught in "Prefill and decode"
- Time to first token (TTFT) How long a person waits after sending a question before the first word appears. It is mostly the queue plus the prefill. Taught in "Prefill and decode"
- Token A small piece of text that a model reads and writes: often a whole common word with its space, sometimes part of a word, a digit or a mark. Every price, limit and speed in this course is counted in tokens. Taught in "Tokens: how a model reads and writes text"
- Token table, or embedding table The table with one row of numbers for each token the model knows. Reading a token starts by looking up its row. Taught in "Inside a transformer, in plain words"
- Tokenizer The program, with its fixed list of pieces, that cuts text into tokens and turns each one into its number, and turns numbers back into text. Taught in "Tokens: how a model reads and writes text"
- Tokens a second How speeds are measured in this course, for reading a prompt and for writing an answer. Taught in "Tokens: how a model reads and writes text"
- Top-p and top-k Two ways to narrow the choice before drawing a token. Top-k keeps only the k most likely tokens; top-p keeps the smallest group of most likely tokens whose chances add up to at least p. Taught in "Sampling parameters"
- Training Finding a model's numbers by adjusting them on a huge number of examples. It is done once, before the model is used, and it is very expensive. Taught in "What a model is"
- Transformer The design of almost every language model today: a token table, a stack of identical layers, and an output step that scores every possible next token. Taught in "Inside a transformer, in plain words"
- Utilisation The share of the paid hours in which the GPU is doing useful work. The hours when nobody asks anything still cost money. Taught in "Why inference is where the money goes"
- vLLM An open-source program that serves language models to many people at once on GPUs, and answers in OpenAI's format. Part 3 of this course opens it up. Taught in "Serving a model with vLLM"
- Vocabulary The tokenizer's list of pieces. A token's number is simply its place in this list. Taught in "Tokens: how a model reads and writes text"
- Weights, or parameters The numbers a model learned in training. They sit in the GPU's memory while the model works, and every pass through the model reads them. Taught in "What a model is"
- Working space, or peak activation The memory the model needs for its arithmetic, besides the weights and the KV cache. vLLM measures it with a trial run at start-up and sets it aside before it sizes the cache. Taught in "Sizing GPU memory"