Inference Engineering Interview Pack
GPU memory, KV cache, batching, quantization, speculative decoding, parallelism and cost per token.
120 questions, each with why it is asked, a full answer, what the interviewer listens for, follow-ups, common mistakes and a diagram. By Roni Das.
About 20% junior, 35% mid, 30% senior and 15% staff.
Sections, from small ideas to big ones
- GPU memory and the KV cache
- static and continuous batching
- paged attention
- quantization
- speculative decoding
- tensor, pipeline and expert parallelism
- serving stacks
- latency vs throughput and cost per token
Coming soon
5 free questions from this pack
The free sample questions are being added.