free tool · no sign-up · every formula shown
Speculative Decoding Simulator: a fast guesser and a careful checker
A small model guesses the next few tokens, and the big model checks them all in one step. How much faster that is depends on how often a guess is right and how cheap a guess is. Run rounds and watch the guesses being kept or thrown away.
Model sizes, GPU specifications and prices were read from their official sources on 13 November 2026, and every number says where it came from.
Speculative decoding: a fast guesser and a careful checker
speculative decoding simulatorA small model guesses the next few tokens. The big model checks all of them in one step, keeps the guesses up to the first wrong one, and adds one token of its own. Checking several tokens costs about the same as writing one, because decode waits on memory, not on maths.
Dashed: guesses waiting to be checked. Green: guesses the big model agreed with. Red, crossed out: the first wrong guess, thrown away with the faded guesses after it. When every guess is right there is no red one. The last token, in the accent colour, is the big model's own.
Tokens the big model adds per check, on average
2.77
plain decoding: 1
Tokens per check in your rounds so far
none yet
0 rounds
Speed-up over plain decoding
1.98x
faster
The number of guesses that is fastest here
4
1.98x
Expected tokens per check = (1 - ak+1) / (1 - a) = (1 - 0.705) / 0.30 = 2.773
Speed-up = expected tokens / (1 + k x c) = 2.773 / (1 + 4 x 0.10) = 1.981
The assumptions behind these numbers
- The formula is theorem 3.8 of Leviathan, Kalman and Matias (2023). It assumes every guess is right with the same chance, independently, which real text only roughly does.
- It assumes the big model's check of k + 1 tokens costs the same as one normal step. That holds while decode waits on memory. On a busy server that already waits on maths, the check costs more and speculation can make things slower.
- With greedy decoding the answer is the same text the big model alone would write, apart from tiny floating-point differences. With sampling, the answers follow exactly the big model's own probabilities. Speculation changes the speed, not what the model would say.
Three things to try
Lower the acceptance rate to 30%
Most guesses are thrown away, and guessing many tokens only adds cost. The best number of guesses drops to one or two.
Make guesses free
Set the cost of a guess to 0%, as with n-gram guessing. Now more guesses never hurt; they just stop helping.
Make guesses expensive
At 40% of a big step, speculation can be slower than plain decoding, which the result tile tells you.
Questions people ask
What is speculative decoding?
A small, fast model (or a cheap trick such as copying text that appeared earlier) guesses several next tokens. The big model checks them in one step and keeps the guesses up to the first one it disagrees with, then adds one token of its own. With greedy decoding the answer is the same text the big model would write alone; with sampling it follows exactly the big model's own probabilities.
What is the acceptance rate?
It is how often the big model agrees with a guess. With an acceptance rate a and k guesses, the big model writes on average (1 - a to the power k + 1) / (1 - a) tokens per step, from Leviathan, Kalman and Matias (2023).
When does speculative decoding make things slower?
When guesses are often wrong, when a guess is expensive, or when the server is already busy enough that a step waits on maths rather than memory. Then checking several tokens costs more than one normal step.
The other inference tools: GPU Memory Calculator for LLMs, LLM Latency and Throughput Estimator, LLM Cost per Million Tokens Calculator, KV Cache and PagedAttention Visualizer, Static and Continuous Batching Visualizer. They come from the course Hands-On Inference Engineering, where each sum is worked on paper and then measured on a real GPU. To see a tool used on a real question, try the free exercises with worked answers.