GenAI and LLM interview question · Decoding · senior level
How does speculative sampling keep exactly the target model's distribution?
Free to read in full. This is 1 of the 150 questions in the GenAI and LLM Engineering interview pack, with 173 diagrams. Get the pack, ₹999 · $20
Why it is asked
Speculative decoding is a standard speed-up; senior candidates must show why it is lossless, not just that it is fast.
The answer
It accepts each draft token with probability min(1, p(x)/q(x)) and, on rejection, resamples from the leftover distribution max(0, p - q), normalized; this makes every output token follow exactly the target model's distribution p.
Setup (Leviathan et al. 2023; Chen et al. 2023): a small draft model q proposes gamma tokens one by one (gamma is the number of draft tokens). The large target model p then scores all of them in one forward pass, which costs about one normal decode step because decode is memory-bound.
Verification, left to right for each draft token x: 1. Accept with probability min(1, p(x)/q(x)). If the target likes the token at least as much as the draft did, it is always kept. 2. On the first rejection, sample a replacement from max(0, p - q), renormalized, and stop this round. 3. If all gamma tokens pass, sample one bonus token from p at the next position.
Why the output equals p. Let residual(x) = max(0, p(x) - q(x)), normalized to sum to 1. Token x can come out two ways: The two add up to p(x). The code checks this exactly and by simulation: acceptance rate 0.70, output equal to p.
- accepted draft: q(x) x min(1, p(x)/q(x)) = min(p(x), q(x));
- rejection, then resample: P(reject) x residual(x) = p(x) - min(p(x), q(x)).
Speed: with alpha, the chance one draft token is accepted, and gamma drafts, expected tokens per target pass is (1 - alpha^(gamma+1)) / (1 - alpha): 2.31 for alpha 0.6 and gamma 4, 3.36 for alpha 0.8. Under greedy decoding the rule reduces to accepting a draft token only if it equals the target's argmax.
What the interviewer listens for
- Accept with min(1, p/q); on reject sample from norm(max(0, p - q))
- Output distribution equals the target's exactly
- Target checks all drafts in one forward pass
- Bonus token when all drafts pass
- Expected tokens per pass (1 - alpha^(gamma+1)) / (1 - alpha)
Follow-ups the interviewer may ask
- Why is the verification pass cheap? Hint: decode is memory-bound; scoring gamma + 1 positions reads the weights once.
- What happens at high temperature? Hint: draft and target disagree more, acceptance falls, speed-up shrinks.
- Why does speculative decoding help less at large batch sizes? Hint: decode becomes compute-bound, so verifying extra draft tokens is no longer free; draft-free variants such as Medusa heads and EAGLE reduce the overhead.
Common mistakes
- Saying speculative decoding approximates the target and loses quality
- Accepting only exact matches when sampling, which changes the distribution
Written by Roni Das, who builds distributed systems and AI systems for a living.
One question is not enough for the interview.
This is 1 of the 150 questions in the GenAI and LLM Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- Why are LLMs bad at counting letters or reversing a word? (junior)
- Write out scaled dot-product attention with the shape of every tensor. (mid)
- What is GRPO, and how do verifiable rewards train reasoning models? (senior)
- Design a system that answers questions from 300-page contracts accurately. (staff)
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)