Contents Part 7 · Serving at scale with Kubernetes and llm-d

Contents of the book

Start hereThe opening pages

Introduction: what this course buildsfree
Before you startfree
Renting a GPU without a surprise billcoming

Part 1Inference from zero

Part 2The maths of one request

Prefill and decodefree
Why writing is limited by memory speedcoming
When a GPU waits for memory and when it waits for maths
The KV cachefree
Sizing GPU memorycoming
Why newer models need a smaller cache
Batching: serving many people at oncecoming
The latency numbers that mattercoming
Kernels and FlashAttention
Long context: 128K tokens and beyondcoming

Part 3vLLM in depth

How vLLM is builtcoming
PagedAttentioncoming
The vLLM schedulercoming
Chunked prefillcoming
Prefix cachingcoming
vLLM memory settingscoming
vLLM serving settingscoming
Sampling parameterscoming
Calling vLLM well from your codecoming
Serving many fine-tuned versions on one GPUcoming
Start-up time and cold startscoming

Part 4Smaller and faster models

Number formats from 32 bits to 4coming
Quantizing a model's weightscoming
Quantizing the KV cachecoming
Checking that quality survivedcoming
Speculative decoding: guessing aheadcoming
Speculative decoding in vLLMcoming
When speculative decoding makes things slowercoming

Part 5SGLang

Your first SGLang servercoming
RadixAttentioncoming
SGLang settings that change a numbercoming
Structured output at serving timecoming
A fair vLLM and SGLang benchmarkcoming
Choosing a serving enginecoming
TensorRT-LLM and NVIDIA Tritoncoming
llama.cpp, GGUF and Ollamacoming
Serving embedding and reranker modelscoming
Serving vision and multimodal modelscoming

Part 6Many GPUs

Why split a model across GPUscoming
Tensor parallelismcoming
Pipeline parallelismcoming
More copies instead of bigger splitscoming
Mixture-of-experts models and expert parallelismcoming
The links between GPUscoming
AMD GPUs and ROCmcoming

Part 7Serving at scale with Kubernetes and llm-d

vLLM on Kubernetescoming
Why ordinary load balancing fails for LLMscoming
How llm-d is builtcoming
llm-d on your laptopcoming
Routing by what each server has cachedcoming
Separating prefill and decodecoming
Autoscaling model serverscoming
Keeping the KV cache beyond the GPUcoming

Part 8Measure, watch, pay and prove it

Benchmarking honestlycoming
Service levels and load testscoming
Cost per million tokenscoming
Watching a model servercoming
Tracing and profiling a slow requestcoming
Reliability and security for model servingcoming
Managed endpoints: Bedrock, SageMaker, Vertex AI and NIMcoming
Host it yourself or use an APIcoming
Capstone: plan a real servicecoming
Capstone: serve it and prove the costcoming
Inference engineer interview questionscoming

Part 7 · Serving at scale with Kubernetes and llm-d

Autoscaling model servers: the right signal, slow start-ups and scaling to zero

Why CPU usage is the wrong signal, scaling on queue length with KEDA and Prometheus, on a small GKE cluster.

Code zip with this page

This page is being written and is not published yet

Pages are published in book order. Everything published later is included when you buy the course.

See the contents of the book or read what the course covers.