Inference Engineering interview question · Speculative decoding · senior level
When does speculative decoding stop helping or make things worse?
Free to read in full. This is 1 of the 120 questions in the Inference Engineering interview pack, with 141 diagrams. Get the pack, ₹999 · $20
Why it is asked
Speculative decoding is often switched on blindly. The interviewer checks that you know it spends idle GPU compute, and that the idle compute disappears at high batch or low acceptance.
The answer
Speculative decoding stops helping when the GPU is no longer idle during a normal decode step: at large batch sizes, with a low acceptance rate (the chance each draft token is accepted), or with a draft that costs too much. Past that point it makes each token slower.
Why it works at all: at small batch a decode step is memory-bound, limited by reading the weights from HBM (the GPU's main memory), so checking 5 tokens costs about the same as producing 1. Speculation spends that spare compute.
Where it breaks:
- Large batch. In a simple roofline model of an 8B target on one H100 (BF16, acceptance 0.8, 4 draft tokens, draft cost 5% of a target step), the speedup is 2.80x from batch 1 to 32, 2.62x at 64, 1.42x at 128 and 0.74x at 256. Each sequence sends 5 tokens to the target, so verification turns compute-bound near batch 59 (295 / 5). Break-even is near batch 187.
- Low acceptance: high-temperature sampling, creative text, or a draft from a different domain. At batch 1, acceptance 0.3 gives only 1.19x; with a heavier draft costing a quarter of a target step per drafted token, it is 0.71x, a slowdown.
- Hidden costs: rejected tokens are wasted compute, and a draft model's weights and KV cache take memory that would otherwise hold a bigger batch.
What to measure: acceptance rate per traffic type, speedup against batch size on your hardware, TPOT (time per output token) p50 and p99, and goodput. Then switch it off, or shorten the draft, above a measured batch threshold.
What the interviewer listens for
- It spends spare compute that exists only while decode is memory-bound
- Model: 2.80x up to batch 32, 1.42x at 128, 0.74x at 256
- Low acceptance or an expensive draft can make it slower even at batch 1
- Draft memory reduces KV room and so maximum batch
- Gate it by measured batch size and acceptance rate
Follow-ups the interviewer may ask
- Why does the curve fall near batch 59 and not 295? Hint: each sequence verifies 5 tokens, so the target processes 5 times the tokens per step.
- Does speculation help time to first token? Hint: no, it only speeds up decode; the draft also has to process the prompt.
- How would you adapt it at runtime? Hint: shorten the draft or turn it off as the running batch grows.
Common mistakes
- Benchmarking at batch 1 and enabling it for a busy multi-tenant endpoint
- Ignoring the draft's own memory and KV cache
- Assuming acceptance measured on one dataset holds for all traffic
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson LLM Inference Optimization: Serving More Tokens Per GPU.
One question is not enough for the interview.
This is 1 of the 120 questions in the Inference Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (141 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)