Inference Engineering / Part 8 · Measure, watch, pay and prove it / Inference engineer interview questions Contents Part 8 · Measure, watch, pay and prove it Contents of the book Start here The opening pages Introduction: what this course builds free Before you start free Renting a GPU without a surprise bill coming Part 1 Inference from zero What a model is Tokens: how a model reads and writes text How a language model writes, one token at a time Inside a transformer, in plain words What a GPU is, and why models run on one Why inference is where the money goes Timing one answer from llama.cpp, step by step Serving a model with vLLM Part 2 The maths of one request Prefill and decode free Why writing is limited by memory speed coming When a GPU waits for memory and when it waits for maths The KV cache free Sizing GPU memory coming Why newer models need a smaller cache Batching: serving many people at once coming The latency numbers that matter coming Kernels and FlashAttention Long context: 128K tokens and beyond coming Part 3 vLLM in depth How vLLM is built coming PagedAttention coming The vLLM scheduler coming Chunked prefill coming Prefix caching coming vLLM memory settings coming vLLM serving settings coming Sampling parameters coming Calling vLLM well from your code coming Serving many fine-tuned versions on one GPU coming Start-up time and cold starts coming Part 4 Smaller and faster models Number formats from 32 bits to 4 coming Quantizing a model's weights coming Quantizing the KV cache coming Checking that quality survived coming Speculative decoding: guessing ahead coming Speculative decoding in vLLM coming When speculative decoding makes things slower coming Part 5 SGLang Your first SGLang server coming RadixAttention coming SGLang settings that change a number coming Structured output at serving time coming A fair vLLM and SGLang benchmark coming Choosing a serving engine coming TensorRT-LLM and NVIDIA Triton coming llama.cpp, GGUF and Ollama coming Serving embedding and reranker models coming Serving vision and multimodal models coming Part 6 Many GPUs Why split a model across GPUs coming Tensor parallelism coming Pipeline parallelism coming More copies instead of bigger splits coming Mixture-of-experts models and expert parallelism coming The links between GPUs coming AMD GPUs and ROCm coming Part 7 Serving at scale with Kubernetes and llm-d vLLM on Kubernetes coming Why ordinary load balancing fails for LLMs coming How llm-d is built coming llm-d on your laptop coming Routing by what each server has cached coming Separating prefill and decode coming Autoscaling model servers coming Keeping the KV cache beyond the GPU coming Part 8 Measure, watch, pay and prove it Benchmarking honestly coming Service levels and load tests coming Cost per million tokens coming Watching a model server coming Tracing and profiling a slow request coming Reliability and security for model serving coming Managed endpoints: Bedrock, SageMaker, Vertex AI and NIM coming Host it yourself or use an API coming Capstone: plan a real service coming Capstone: serve it and prove the cost coming Inference engineer interview questions coming Part 8 · Measure, watch, pay and prove it
Inference engineer interview questions, answered with this course's own measured numbers The questions interviewers ask about memory, speed, batching, quantization and cost, each answered the way a strong candidate would, using numbers you measured yourself.
This page is being written and is not published yet Pages are published in book order. Everything published later is included when you buy the course.
See the contents of the book or read what the course covers .