free tool · no sign-up · every formula shown
GPU Memory Calculator: will this language model fit on this GPU?
A model on a GPU needs room for its weights, for the KV cache that holds every conversation, and for some working space. Pick a model from its real Hugging Face config, a number format, a conversation length and how many people use it at once, and see which GPUs it fits on.
Model sizes, GPU specifications and prices were read from their official sources on 13 November 2026, and every number says where it came from.
Will this model fit on this GPU?
GPU memory calculatorA model on a GPU needs room for three things: its weights, the KV cache that holds every conversation, and some working space. Change the numbers and watch the card fill up.
Pick a GPU. Each card shows how many of it this setup needs.
Yes. It fits on one NVIDIA L40S, with room for 24 people at 8,192 tokens each.
One NVIDIA L40S
Sold as a 48 GB card. The driver reports 45 GiB, and vLLM uses 41.4 GiB of it.
- weights 15.0 GiB
- KV cache 16.0 GiB
- working space 2.0 GiB
- free 8.4 GiB
The weights
15.0 GiB
8.0 billion numbers
KV cache for each token
128 KiB
131,072 bytes
KV cache for each person
1.0 GiB
8,192 tokens
KV cache for everyone
16.0 GiB
16 people
KV cache per token = 2 (a key and a value) x layers x KV heads x head size x bytes
= 2 x 32 layers x 8 KV heads x 128 head size x 2 bytes = 131,072 bytes per token
Weights = 8,030,261,248 parameters x 2 bytes = 15.0 GiB
Usable on one card = 0.92 x 45 GiB = 41.4 GiB (a 48 GB card is 44.7 GiB on paper; the driver reports 45 GiB)
The assumptions behind these numbers
- Memory on this page is in GiB (1 GiB is 1,073,741,824 bytes), the unit nvidia-smi and vLLM's log use. GPUs are sold in decimal GB, so a 48 GB card holds 48 billion bytes, which is 44.7 GiB. The card names keep the GB they are sold by.
- vLLM uses 92% of a GPU's memory by default (gpu-memory-utilization 0.92 in vLLM 0.33.0). The card sizes are what the driver usually reports, a little under the datasheet number on some cards.
- The working space is a flat guess you can change. vLLM measures the real number when it starts and prints how many tokens of KV cache it could make room for: trust that line over this page.
- More than one GPU means tensor parallelism inside one machine of up to 8 GPUs, then pipeline parallelism across machines. The number of GPUs must divide the model's 32 attention heads. The time GPUs spend talking to each other is not counted here.
- 8-bit and 4-bit formats keep the embedding table and output head in BF16, as AWQ, GPTQ and FP8 checkpoints do. 4-bit files also store a scale for every group of weights, a few percent more than shown.
- Model sizes come from each model's config.json on Hugging Face, and GPU sizes from NVIDIA's datasheets, both read on 13 November 2026.
Checked against vLLM: with 4 GiB set aside for the cache, this formula gives Qwen3-0.6B room for 37,449 tokens. vLLM 0.33 printed 37,440 on a rented RTX 4090 on 13 November 2026, which is that number rounded down to 2,340 whole blocks of 16 tokens.
Three things to try
Make the conversations longer
Move the tokens from 8,192 to 131,072 and watch the KV cache grow until it is bigger than the model itself.
Switch the KV cache to FP8
It halves the cache, so about twice as many people fit on the same card. The course checks what that does to the answers.
Compare gpt-oss with Llama
gpt-oss-20b keeps a short window on half its layers, so its cache grows far more slowly than a dense model's.
Questions people ask
How do I work out the KV cache size of a model?
Multiply 2 (a key and a value) by the number of layers, the number of KV heads, the head size and the bytes per number. That is the cache for one token. Llama 3.1 8B has 32 layers, 8 KV heads and a head size of 128, so in BF16 it needs 2 x 32 x 8 x 128 x 2 = 131,072 bytes, 128 KiB, for every token of every conversation.
Why does vLLM say a different number of tokens fit?
vLLM measures its real working space when it starts, keeps 8% of the GPU free by default, and rounds the cache down to whole blocks. Its start-up log prints the KV cache size in tokens, and that line is the one to trust. On a rented RTX 4090, with 4 GiB set aside for the cache, this formula gave Qwen3-0.6B room for 37,449 tokens and vLLM printed 37,440: the gap is vLLM rounding down to whole blocks of 16 tokens.
Does a 4-bit model also shrink the KV cache?
No. Quantizing the weights shrinks the weights only. The cache stays in BF16 unless you also set the KV cache to FP8, which is a separate setting in vLLM.
What is the working space for?
Activations while a batch runs, CUDA graphs that vLLM records to start steps quickly, and the runtime itself. It depends on the engine and the settings, so this page uses a guess you can change, 2 GiB per GPU to start.
Why must the number of GPUs divide the attention heads?
Tensor parallelism gives each GPU an equal share of the attention heads, so the heads must split evenly. A model with 28 heads can run on 1, 2 or 4 GPUs this way, but not on 8.
The other inference tools: LLM Latency and Throughput Estimator, LLM Cost per Million Tokens Calculator, KV Cache and PagedAttention Visualizer, Static and Continuous Batching Visualizer, Speculative Decoding Simulator. They come from the course Hands-On Inference Engineering, where each sum is worked on paper and then measured on a real GPU. To see a tool used on a real question, try the free exercises with worked answers.