free tool · no sign-up · every formula shown
LLM Latency and Throughput Estimator: prefill, decode and the roofline, in plain words
Reading the prompt is limited by how fast a GPU calculates, and writing the answer by how fast it reads its memory. This works out both from the GPU's datasheet, plays one request through at that speed, and shows what happens to each person's speed and the total as more people share the GPU.
Model sizes, GPU specifications and prices were read from their official sources on 13 November 2026, and every number says where it came from.
How fast will it answer?
prefill and decode estimatorEvery request has two halves. Prefill reads your whole prompt at once and is limited by how fast the GPU calculates. Decode writes the answer one token at a time and is limited by how fast it reads its memory. This estimates both from the GPU's datasheet.
Time to the first token
92 ms
prefill waits on the maths
Time for each next token
25 ms
decode waits on memory
Each person's speed
40 tok/s
30 words a second, roughly
Total for 8 people
320 tok/s
whole answer in 6.5 s
Press play to send one request.
What happens as more people share the GPU
Solid line: each person's speed. Dashed line: the total. While decode waits on memory and the KV cache is still small next to the weights, extra people are nearly free, because one read of the weights serves them all. The shaded part on the right is out of reach: the KV cache for that many people does not fit in memory. The left scale is each person's speed and the right scale the total.
Decode, one step = the larger of (bytes read / memory speed) and (FLOPs / compute speed)
bytes = 17.27 GB (the weights, plus 8 KV caches) / (864 GB/s x 80%) = 25 ms
FLOPs = 8 x 2 x 8.03 billion active parameters, plus attention = 0.133 TFLOP, / (362 TFLOPS x 50%) = 0.7 ms
Prefill = 16.72 TFLOP for 1,024 tokens = 92 ms of maths, against 23 ms to read the weights once
Check the method against a real RTX 4090
On 13 November 2026 we ran Qwen3-8B (a 5.03 GB file, 4-bit) on an NVIDIA GeForce RTX 4090 (24 GB) rented on RunPod with llama.cpp. NVIDIA rates its memory at 1,008 GB/s. The same sum as above, at 80% of that speed:
| Tokens already in the conversation | Estimate | Measured |
|---|---|---|
| 0 | 160.4 tok/s | 164.1 tok/s |
| 2,048 | 151.3 tok/s | 155.1 tok/s |
| 8,192 | 129.3 tok/s | 135.0 tok/s |
| 16,384 | 108.3 tok/s | 114.9 tok/s |
With an empty conversation the GPU read its memory at about 82% of the rated speed, which is where the default of 80% comes from: 5.03 GB x 164 tokens a second is 824 GB/s. As the conversation grows, the estimate and the measurement stay within about 6% of each other, because for an 8B model each new token's time is mostly spent reading the weights and the notes. The estimate is still the best case, never a promise. Batching on the same GPU, with the smaller Qwen3-0.6B at 8 bits, showed the same shape as the chart above: 586 tokens a second for one person, 4,854 in total for 32 people, and 152 each.
The assumptions behind these numbers
- Memory and compute speeds are NVIDIA's datasheet numbers without sparsity, read on 13 November 2026. Real engines reach only part of them, which is what the two sliders are for.
- Every decode step reads all the weights once and every person's KV cache at the middle of their answer.
- Each active parameter costs 2 FLOPs per token, plus attention, which grows with the length of the conversation.
- 4-bit and MXFP4 weights are unpacked and the maths runs in BF16. FP8 uses the GPU's FP8 speed where it has one; the A100 and A10G do not.
- Not counted: time in the queue, the network, the tokenizer, the time GPUs spend talking to each other, and prefills of other people slowing the decodes. Each of these makes the real numbers worse.
Three things to try
Add people until the total stops growing
Move the people at once from 1 to 256. The total climbs while decode waits on memory and flattens when the maths becomes the limit.
Swap an L4 for an RTX 4090
They have the same 24 GB, but the 4090 reads its memory more than three times faster, and the time per token falls by about as much.
Try FP8 on an A100 and on an H100
The H100 has FP8 maths and the A100 does not, so the prompt is read faster only on the H100. Both write faster, because there are fewer bytes to read.
Questions people ask
What are prefill and decode?
Prefill is the model reading your whole prompt at once; it does a lot of maths for each weight it reads, so it is limited by compute. Decode writes the answer one token at a time; each token reads every weight once for very little maths, so it is limited by memory speed.
How do I estimate tokens per second from memory bandwidth?
Divide the memory speed by the bytes read for each token: the model's weights plus the KV cache. A 16 GB model on a GPU that reads 1,000 GB a second can write at most about 60 tokens a second for one person. Real engines reach part of that: with Qwen3-8B on an RTX 4090, llama.cpp reached about 82% of the rated 1,008 GB a second counting the whole 5.03 GB file, or about 76% counting only what each token really reads.
Why does batching help so much?
While decode waits on memory, the GPU reads the weights once per step and can use them for many people at the same time. So the total speed rises almost in step with the number of people, until the maths for all of them takes longer than the read.
What is TTFT and TPOT?
TTFT is the time to the first token: how long a person waits before anything appears. TPOT is the time per output token after that. A chat feels quick with a TTFT under a second and a TPOT well under 100 milliseconds.
How accurate is this estimate?
It is a best case. It leaves out queueing, the network, the time GPUs spend talking to each other and other people's prompts slowing your answer. On a rented RTX 4090 with Qwen3-8B it came within about 6% of the measured speed, both for a short conversation and for one of 16,384 tokens. Measure with a benchmark before you plan a budget.
The other inference tools: GPU Memory Calculator for LLMs, LLM Cost per Million Tokens Calculator, KV Cache and PagedAttention Visualizer, Static and Continuous Batching Visualizer, Speculative Decoding Simulator. They come from the course Hands-On Inference Engineering, where each sum is worked on paper and then measured on a real GPU. To see a tool used on a real question, try the free exercises with worked answers.