Contents of Hands-On Inference Engineering: Serve LLMs on GPUs with vLLM and SGLang
The whole course in one measured request, what you should already know, and how to rent a GPU without a surprise bill. The book has 75 pages: the opening pages and 8 parts. 15 of them are published so far. The free pages open for everyone; the rest open when you buy the course.
See the early-access price and what the course covers
Your progress0 of 6 checks answered right
It fills only when a prediction matches the measured result. Reading a page does not move it. It is kept in this browser.
Start here
The opening pages
The whole course in one measured request, what you should already know, and how to rent a GPU without a surprise bill.
2 of 3 pages published
- Introduction: what this course buildsOne real request followed from the keyboard to the GPU and back, who the course is for, and how each page's text, code and knowledge check fit together. Free
- Before you startThe Python and terminal basics this course assumes, a five-minute self-check, and renting an RTX 4090 on RunPod, setting it up, asking a model your first question and switching the GPU off. Free
- Renting a GPU without a surprise billOpening a RunPod account with a spending limit, asking AWS for GPU quota on the first day, and the stopped-disk charge that catches most beginners.Coming
Part 1
Inference from zero
Models, tokens, transformers and GPUs in plain words, then your first model on a rented GPU, first with llama.cpp and then with vLLM.
8 pages, all published
- What a model isA model as a function with billions of learned numbers, what training does, and what inference means: using those numbers to answer. Locked
- Tokens: how a model reads and writes textTurning text into tokens with a real tokenizer, counting them, and why every price and speed in this course is measured in tokens. Locked
- How a language model writes, one token at a timeThe loop at the heart of every answer: read everything so far, pick the next token, add it, repeat, and why that loop decides the speed. Locked
- Inside a transformer, in plain wordsWhat happens in one layer, what attention does, and which parts of the model are large, so the later maths has something to stand on. Locked
- What a GPU is, and why models run on oneThousands of small cores, fast memory beside them, and the two numbers on a GPU data sheet that matter most for inference: memory size and memory speed. Locked
- Why inference is where the money goesTraining happens once, answering happens millions of times a day; the cost of one answer worked out from a GPU's hourly price. Locked
- Timing one answer from llama.cpp, step by stepServing Qwen3-8B with llama-server on an RTX 4090 and timing every part of one answer as it streams in, from the first word to the last. Locked
- Serving a model with vLLMInstalling vLLM, starting it with one command, calling it with the OpenAI Python library, and watching it serve 64 people at once on one RTX 4090. Locked
Part 2
The maths of one request
Prefill and decode, memory speed, the KV cache, batching and the latency numbers, each worked out on paper and then measured.
5 of 10 pages published
- Prefill and decodeReading the question all at once is fast; writing the answer one token at a time is slow. Measured on a rented RTX 4090: the same GPU read a prompt about 56 times faster per token than it wrote the answer. Free
- Why writing is limited by memory speedEvery new token reads all the model's weights once; dividing memory speed by model size predicts the writing speed, and the measurement agrees.Coming
- When a GPU waits for memory and when it waits for mathsArithmetic intensity in plain words, the ratio printed on every GPU's data sheet, and why reading a prompt and writing an answer sit on opposite sides of it. Locked
- The KV cacheWhy the model keeps notes on every earlier token, how to work out their size from a model's config file, why long chats slow down, and the cache budget vLLM prints for Qwen3-8B on a 24 GB GPU. Free
- Sizing GPU memoryA worked sizing for a real model on a 24 GB and a 48 GB card, checked against what vLLM reports when it starts.Coming
- Why newer models need a smaller cacheHow sharing keys and values between attention heads cuts the cache, read straight from real model configs. Locked
- Batching: serving many people at onceMeasured on an RTX 4090: total writing speed rises more than eight times with 32 people at once, while each person's own speed falls to about a quarter.Coming
- The latency numbers that matterTTFT, TPOT, inter-token latency and end-to-end time, why an average hides the slow users, and what a p99 is.Coming
- Kernels and FlashAttentionWhat a GPU kernel is, why fusing steps saves time, and what FlashAttention changed, without writing CUDA. Locked
- Long context: 128K tokens and beyondHow the KV cache and the prefill grow with a long prompt, measured at several lengths, and the settings and model features that keep it affordable.Coming
Part 3
vLLM in depth
The engine, PagedAttention, the scheduler, prefix caching, and the important engine arguments, each one changed and measured.
11 pages, coming
- How vLLM is builtFollowing one request through vLLM's processes, so every later setting has a place to live.Coming
- PagedAttentionThe memory problem PagedAttention solves, the block table, and the KV cache size vLLM prints at start-up.Coming
- The vLLM schedulerHow vLLM adds and removes requests between steps, and what happens when the KV cache is full.Coming
- Chunked prefillSplitting a long prompt into pieces so writers keep writing, with the token budget that controls it.Coming
- Prefix cachingShared system prompts and chat history served from cache, measured as a lower time to first token and read from vLLM's hit counters.Coming
- vLLM memory settingsThe settings that decide how much fits on the card, each one changed and its effect measured.Coming
- vLLM serving settingsThe settings that make vLLM safe and useful as an API, and the browser setting that changed in recent versions.Coming
- Sampling parametersHow the next token is chosen, and which choices cost extra time.Coming
- Calling vLLM well from your codeThe client side of latency: why streaming changes what a user feels, and how to send requests in parallel safely.Coming
- Serving many fine-tuned versions on one GPULoading small adapters beside one base model, and what each extra adapter costs.Coming
- Start-up time and cold startsWhy a new vLLM server takes minutes to be ready, what enforce-eager and the compilation settings trade, and how to shorten the wait.Coming
Part 4
Smaller and faster models
Number formats, quantizing weights and the KV cache, checking the quality loss, and letting a small model guess ahead.
7 pages, coming
- Number formats from 32 bits to 4FP32, BF16, FP8, INT8, INT4 and FP4 drawn bit by bit, and why fewer bits means faster writing.Coming
- Quantizing a model's weightsMaking a smaller copy of a real model, serving it with vLLM, and measuring memory, speed and the users that now fit.Coming
- Quantizing the KV cacheStoring the conversation notes in fewer bits, and when it hurts the answers.Coming
- Checking that quality survivedRunning the same test questions on the original and the smaller model, and deciding what loss is acceptable.Coming
- Speculative decoding: guessing aheadWhy checking several guessed tokens at once costs almost the same as writing one, and the acceptance rate that decides the win.Coming
- Speculative decoding in vLLMTurning each method on with the speculative settings, and measuring the speed-up and the acceptance rate.Coming
- When speculative decoding makes things slowerMeasured at low and high load, so you know when to switch it off.Coming
Part 5
SGLang
SGLang's server and RadixAttention, its settings, and a fair head-to-head benchmark against vLLM on the same GPU.
10 pages, coming
- Your first SGLang serverStarting SGLang on the same model and GPU, and a side-by-side table of the settings you already know.Coming
- RadixAttentionHow SGLang finds shared beginnings across many requests, measured on a multi-turn chat and an agent workload.Coming
- SGLang settings that change a numberEach setting changed and measured, with the version-specific names checked against the SGLang docs.Coming
- Structured output at serving timeForcing valid JSON from the engine itself, and what it costs in speed.Coming
- A fair vLLM and SGLang benchmarkThe rules for a comparison you can defend, run for real, with the results written down even where they surprise us.Coming
- Choosing a serving engineWhat each one is good at, what it needs, and a short decision list for a real team.Coming
- TensorRT-LLM and NVIDIA TritonBuilding an engine for a real model, serving it behind Triton, and comparing its speed and effort with vLLM on the same GPU.Coming
- llama.cpp, GGUF and OllamaWhat the GGUF format and its quantization levels are, how llama.cpp and Ollama serve a model without a big GPU, and how fast they really are.Coming
- Serving embedding and reranker modelsWhy these models are served differently from a chat model, batching them for throughput, and measuring cost per million texts.Coming
- Serving vision and multimodal modelsHow an image becomes tokens, why it makes the prefill much heavier, and the settings that limit it, measured with a real vision model.Coming
Part 6
Many GPUs
Tensor, pipeline, data and expert parallelism, the links between GPUs, and what each split costs in speed.
7 pages, coming
- Why split a model across GPUsThe two different reasons, and the four ways to split, drawn before any code.Coming
- Tensor parallelismServing one model on two GPUs, measuring the speed-up, and seeing the communication it adds.Coming
- Pipeline parallelismWhen splitting by layers beats splitting inside layers, especially across machines.Coming
- More copies instead of bigger splitsTwo copies of a model on two GPUs against one model split over both, measured on the same workload.Coming
- Mixture-of-experts models and expert parallelismHow experts are spread across GPUs, and why these models change the memory and speed maths.Coming
- The links between GPUsWhy the same split runs at very different speeds on different machines, with the numbers from NVIDIA's data sheets.Coming
- AMD GPUs and ROCmRunning vLLM on an AMD GPU, what is the same and what is different from NVIDIA, and the measured speed and cost side by side.Coming
Part 7
Serving at scale with Kubernetes and llm-d
Running many model servers as one service: smart routing by cache and load, separate prefill and decode, autoscaling and cache tiers.
8 pages, coming
- vLLM on KubernetesThe plain way first, on a local kind cluster with the inference simulator, then on one GPU node.Coming
- Why ordinary load balancing fails for LLMsRound robin ignores the cache and the queue; the Gateway API InferencePool and an endpoint picker that does not.Coming
- How llm-d is builtThe parts of llm-d version 1, what each one decides, and which guide needs which hardware.Coming
- llm-d on your laptopBuilding the full routing stack locally for free, and watching it pick a server for each request.Coming
- Routing by what each server has cachedPrefix-cache-aware scoring in llm-d, measured as a higher cache hit rate and a lower time to first token.Coming
- Separating prefill and decodeDisaggregation in plain words, the KV cache transfer between servers, and a small real run on two GPUs.Coming
- Autoscaling model serversWhy CPU usage is the wrong signal, scaling on queue length with KEDA and Prometheus, on a small GKE cluster.Coming
- Keeping the KV cache beyond the GPUOffloading and sharing the cache so returning users skip the prefill, and what it costs.Coming
Part 8
Measure, watch, pay and prove it
Honest benchmarks, service levels, cost per million tokens, dashboards, reliability and security, then a real model served at a measured cost.
11 pages, coming
- Benchmarking honestlyMicro benchmarks against workload benchmarks, the settings that fool people, and a report you could show a manager.Coming
- Service levels and load testsSetting a time-to-first-token and time-per-token target, and finding the load where they break.Coming
- Cost per million tokensThe formula worked from our own measured runs, and the comparison with a hosted API's price list.Coming
- Watching a model serverThe metrics vLLM exports by name, the five panels every serving dashboard needs, and alerts that mean something.Coming
- Tracing and profiling a slow requestFollowing one slow request across the gateway and the engine, and seeing where the GPU time went.Coming
- Reliability and security for model servingRate limits, API keys, keeping one customer from slowing another, and replacing a model while it serves.Coming
- Managed endpoints: Bedrock, SageMaker, Vertex AI and NIMWhat each managed option runs for you, what it costs per token or per hour, and what you give up in control.Coming
- Host it yourself or use an APIHosted APIs, managed endpoints and your own GPUs compared on cost, control, privacy and effort.Coming
- Capstone: plan a real serviceChoosing a model, a GPU, an engine and settings on paper, with a predicted speed and cost.Coming
- Capstone: serve it and prove the costRunning the plan for real, comparing the measurements with the prediction, and writing the report.Coming
- Inference engineer interview questionsThe questions interviewers ask about memory, speed, batching, quantization and cost, each answered the way a strong candidate would, using numbers you measured yourself.Coming