Speculative Decoding Explained: Faster Answers Without Changing Them
What is speculative decoding?
Speculative decoding lets a cheap guesser propose the next few tokens while the large model checks them all in one pass, the same pass it would spend writing one token. Accepted guesses are free tokens, and the model's own choice replaces the first wrong one, so the answer is unchanged. In our run, Llama 3.1 70B on one H100 rewriting code went from 26.7 ms to 7.3 ms a word, 3.65 times faster, with 70.5% of guesses accepted. It helps few users most and can hurt a busy server.
Save your progress and get the AI Engineering cheat sheet PDF
This guide explains how speculative decoding makes a language model write faster for one person without changing what it writes, how much faster it was in a real run on an H100, and why it can slow a busy server down. It answers 15 questions in plain words.
15 questions · 4 sections · answers from the course
These answers come from our course Hands-On Inference Engineering: Serve LLMs on GPUs with vLLM and SGLang: 43 lessons ready so far, each one run and measured on a rented NVIDIA GPU. The first lessons are free to read, with no card needed.


The short version
Writing an answer waits on memory, so with one user the GPU's arithmetic is mostly idle. Speculative decoding spends that idle arithmetic checking guessed tokens. The better the guesses, the more tokens each pass writes. When the server is busy there is no idle arithmetic left, so guess fewer tokens or none.
If this helped, the full hands-on course goes much deeper.
In the course you rent the GPU, learn every vLLM setting for speculative decoding, and plan a real 70B service where guessing ahead is the only way to keep the speed promise.
1. The idea in plain words
Speculative decoding is a way to make one person's answer arrive faster, using arithmetic the GPU would otherwise leave idle.

What is speculative decoding?
It is a way of writing an answer faster without changing it. Something cheap, called the drafter, guesses the next few tokens (a token is a small piece of text, often part of a word). The large model, called the target, then checks all the guesses in one pass through its weights, the same pass it would have spent writing a single token.
Every guess the target agrees with is a token written without an extra pass. At the first guess it disagrees with, the round stops and the target's own choice takes that place, so even a bad round still writes one token.
Why is there spare room on the GPU for checking guesses?
Because writing an answer is limited by memory speed, not by arithmetic. Every new token needs a pass that reads all of the model's weights from the GPU's memory. With one person, the GPU's arithmetic units spend most of that time waiting for the weights to arrive.
On a rented RTX 4090, Qwen3-8B writing 164 tokens a second for one person meant reading about 767 GB of weights a second, about 76% of the card's rated 1,008 GB a second. Checking five tokens in that pass instead of one adds arithmetic, which is the part that was idle.
What we measured: Qwen3-8B on an RTX 4090: 164.0 tokens a second for one person, about 767 GB a second of weights read, 76% of the rated memory speed.
Does speculative decoding change the model's answer?
No, with the usual checking rule. The target checks every guess, and a guess it would not have chosen is thrown away. In vLLM the default rule, called standard rejection sampling, keeps the answers following the same probabilities as the target alone. At temperature 0, where the model always takes its most likely token, that means the same words.
At normal temperatures two runs can pick different words, with or without speculative decoding, because each choice is a random draw. vLLM also offers rules that relax the check on purpose for more speed; keep the standard one for customer answers.
How is this related to prefill and decode?
Every request has two halves. Prefill reads the whole question in one pass, which is fast per token. Decode writes the answer one token per pass, which is slow per token. On our RTX 4090 a small model read at 35,796 tokens a second and wrote at 642.
Checking guesses looks like a small prefill: several tokens go through the model together and share one read of the weights. Speculative decoding moves part of the slow writing into that faster, shared kind of pass.
2. A measured speed-up
Here is one real run from the course's GPU planning game, where each plan was also checked on a rented GPU with vLLM 0.33.
How much faster does speculative decoding make an answer?
In our run, 3.65 times for one person. Llama 3.1 70B, stored in 8 bits, served one person on one H100 who asked it to rewrite real Python files. Without guessing it wrote a new word every 26.7 ms. With n-gram guessing of 4 tokens it wrote one every 7.3 ms.
That was the only way to keep the ticket's promise of a word within 15 ms on one card. The speed-up depends heavily on the text, so measure it on your own requests.
What we measured: Llama 3.1 70B FP8 on one H100 SXM, vLLM 0.33, one person rewriting code: 26.7 ms a word without guessing, 7.3 ms with n-gram guessing 4 tokens, 3.65 times faster.
What is the acceptance rate?
It is the share of guessed tokens the target agreed with. vLLM counts it. In our run the target accepted 1,892 of 2,684 guesses, 70.5%.
Acceptance decides almost everything. A guesser that is usually right lets one pass of the large model write several tokens. A guesser that is usually wrong costs extra work and writes one token a round, as before.
What we measured: Llama 3.1 70B FP8 on one H100 with n-gram guessing: 1,892 of 2,684 guesses accepted, 70.5%.
How do you estimate the speed-up before you measure it?
With a simple formula. If each guess is right with chance a, and the drafter guesses k tokens a round, the large model writes on average (1 - a^(k+1)) / (1 - a) tokens per pass. With a at 70% and k at 4 that is about 2.77 tokens a pass, so at most about 2.77 times faster when the guesser costs nothing, as n-gram nearly does.
Our measured 3.65 beat that estimate. The formula treats every guess as equally likely to be right. One likely reason the real run did better is that a code rewrite copies long stretches of the original, so right guesses tend to come together. Use the formula as a cautious first estimate, then measure.
How many tokens should the drafter guess?
It depends on how often the target agrees, which depends on the text. More guesses win more when they are right, because one check can then accept many tokens. They waste more work when they are wrong, because every rejected guess was computed for nothing.
Text that repeats long stretches of the question, such as code edits, can afford more guesses than short, varied chat replies. In vLLM the count is num_speculative_tokens; try a few values on your real requests and keep the fastest.
Does speculative decoding need extra GPU memory?
It can. A separate draft model needs its own room, and so do its notes. n-gram needs none, because it only looks back through the text. In our 70B run the memory was tight already: the server needed --max-num-seqs 4 and --max-num-batched-tokens 2048 to start at all, which left 2.94 GiB for the KV cache, the notes kept for every token.
That is acceptable for a single-user plan like this one, and it is a reminder that speculative decoding suits few users with a fast answer, not many users at once.
What we measured: Llama 3.1 70B FP8 on one H100: needed --max-num-seqs 4 --max-num-batched-tokens 2048 to start, KV cache 2.94 GiB.
If this helped, the full hands-on course goes much deeper.
In the course you rent the GPU, learn every vLLM setting for speculative decoding, and plan a real 70B service where guessing ahead is the only way to keep the speed promise.
3. The kinds of guesser
The methods differ in where the guesses come from. vLLM 0.33 accepts eleven method names; these are the ones most teams try.
What is n-gram speculative decoding?
n-gram looks for the last few tokens earlier in the prompt and the answer so far, and proposes whatever followed them there. It needs no extra model and runs on the processor, so it costs almost nothing.
It works best when the answer copies the question, as in code edits, rewrites and answers that quote a document. It was the method in our measured 3.65 times run. In vLLM: --speculative-config '{"method": "ngram", "num_speculative_tokens": 4}'.
What is a draft model?
A separate, much smaller model from the same family writes the guesses, for example Qwen3-0.6B for Qwen3-8B. It must use the same vocabulary as the target, so every guessed token means the same thing to both models.
It guesses well on new text that n-gram cannot copy, but it needs its own room in GPU memory and its own time for every guess.
What are EAGLE-3 and multi-token prediction?
Both use small trained layers instead of a whole second model. An EAGLE-3 head reads the target's own internal numbers from several of its layers and guesses the next tokens; it must be trained for that exact target model.
Multi-token prediction (MTP) uses extra prediction layers that some models are trained with and ship inside their own checkpoint, so there is nothing separate to download. In vLLM 0.33 both are chosen with the method field of --speculative-config.
4. When it helps, and when it hurts
Speculative decoding is not a free switch. It trades spare arithmetic for speed, and a busy server has no spare arithmetic.
Why does speculative decoding help one user the most?
Because with one person the GPU's arithmetic is mostly idle while it waits for the weights, so checking extra guesses is almost free. It is the one way to write faster for a single person than the memory speed allows.
Late at night, for a coding assistant, or for any product that promises a fast answer to a few people at a time, it can be the difference between keeping and breaking that promise, as in our 70B run.
Can speculative decoding make a server slower?
Yes, on a busy server. With many people in the batch, one read of the weights already writes a token for each of them, so the arithmetic is no longer idle. Rejected guesses then take arithmetic away from real tokens for other people, and the total can fall. That is how the method works, not a fault of one engine.
vLLM can guess fewer tokens as the batch grows, with a table of batch sizes in num_speculative_tokens_per_batch_size, or none above a size you choose. The course's hands-on lessons on speculative decoding, which will measure this slowdown, are still being written.
How should I test speculative decoding before turning it on?
Measure at your real traffic level, not only with one user. Compare the gap between words, the wait for the first word and total tokens a second, with and without guessing, on requests like your real ones, and read the acceptance rate from vLLM's metrics.
Keep the standard checking rule so the answers stay the same, and repeat the test at the busiest hour as well as the quietest.
Where to go next
If this helped, the full hands-on course goes much deeper.
In the course you rent the GPU, learn every vLLM setting for speculative decoding, and plan a real 70B service where guessing ahead is the only way to keep the speed promise.
Every number on this page comes from Hands-On Inference Engineering: Serve LLMs on GPUs with vLLM and SGLang, where you rent the GPU yourself, run each command and compare your numbers with ours. Start with the free introduction and the free lessons on prefill and decode and the KV cache. If you are also preparing for interviews, the two big courses together and the Inference Engineering interview pack are below.
These questions are a start. The interview asks more.
Get all 120 questions in the Inference Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (141 diagrams in all). A PDF you keep, junior to staff level.
Pay by UPI, cards or netbanking through Razorpay. Pay by card or PayPal, in US dollars. Download at once from My packs.
