Inference Engineering / Part 1 · Inference from zero / Serving a model with vLLM Contents Part 1 · Inference from zero Contents of the book Start here The opening pages Introduction: what this course builds free Before you start free Renting a GPU without a surprise bill coming Part 1 Inference from zero What a model is Tokens: how a model reads and writes text How a language model writes, one token at a time Inside a transformer, in plain words What a GPU is, and why models run on one Why inference is where the money goes Timing one answer from llama.cpp, step by step Serving a model with vLLM Part 2 The maths of one request Prefill and decode free Why writing is limited by memory speed coming When a GPU waits for memory and when it waits for maths The KV cache free Sizing GPU memory coming Why newer models need a smaller cache Batching: serving many people at once coming The latency numbers that matter coming Kernels and FlashAttention Long context: 128K tokens and beyond coming Part 3 vLLM in depth How vLLM is built coming PagedAttention coming The vLLM scheduler coming Chunked prefill coming Prefix caching coming vLLM memory settings coming vLLM serving settings coming Sampling parameters coming Calling vLLM well from your code coming Serving many fine-tuned versions on one GPU coming Start-up time and cold starts coming Part 4 Smaller and faster models Number formats from 32 bits to 4 coming Quantizing a model's weights coming Quantizing the KV cache coming Checking that quality survived coming Speculative decoding: guessing ahead coming Speculative decoding in vLLM coming When speculative decoding makes things slower coming Part 5 SGLang Your first SGLang server coming RadixAttention coming SGLang settings that change a number coming Structured output at serving time coming A fair vLLM and SGLang benchmark coming Choosing a serving engine coming TensorRT-LLM and NVIDIA Triton coming llama.cpp, GGUF and Ollama coming Serving embedding and reranker models coming Serving vision and multimodal models coming Part 6 Many GPUs Why split a model across GPUs coming Tensor parallelism coming Pipeline parallelism coming More copies instead of bigger splits coming Mixture-of-experts models and expert parallelism coming The links between GPUs coming AMD GPUs and ROCm coming Part 7 Serving at scale with Kubernetes and llm-d vLLM on Kubernetes coming Why ordinary load balancing fails for LLMs coming How llm-d is built coming llm-d on your laptop coming Routing by what each server has cached coming Separating prefill and decode coming Autoscaling model servers coming Keeping the KV cache beyond the GPU coming Part 8 Measure, watch, pay and prove it Benchmarking honestly coming Service levels and load tests coming Cost per million tokens coming Watching a model server coming Tracing and profiling a slow request coming Reliability and security for model serving coming Managed endpoints: Bedrock, SageMaker, Vertex AI and NIM coming Host it yourself or use an API coming Capstone: plan a real service coming Capstone: serve it and prove the cost coming Inference engineer interview questions coming Part 1 · Inference from zero
Serving a model with vLLM on a rented GPU: one command, the OpenAI library, and many people at once Installing vLLM, starting it with one command, calling it with the OpenAI Python library, and watching it serve 64 people at once on one RTX 4090.
This page is part of the paid Inference Engineering course The early-access price is ₹4,444 or $149 , one payment. It rises to ₹9,999 or $249 once every part is released, and buying now gives you every part as it is released, at no extra cost, with lifetime access.
Free to read first: Introduction: what this course builds , Before you start , Prefill and decode and The KV cache .
Already bought it? Sign in to read this page.