Think of a restaurant kitchen. The big ovens are bolted in and always take the same space, but the counter space left over decides how many orders the cooks can work on at once. On a GPU, the model's weights are the ovens. The KV cache is the counter: every conversation in progress needs a piece of it, and when it is full, new requests have to wait.
For every token in a conversation, every layer of the model stores a key and a value, so it never has to recompute them. That store is the KV cache, and its size decides how many people one GPU can serve at once and how long their conversations can be.
Until version 0.33, vLLM decided the cache size before the server was fully ready. It took a share of the GPU's memory, 92% by default, set by gpu_memory_utilization. From that it subtracted the weights and an estimate of the working memory the server would need later, for example for CUDA graphs, which are recorded GPU work that makes each step faster. Whatever the estimate left over became the cache. If the estimate was too high, memory sat unused. If you raised the share too far, the server could run out of memory.
Version 0.34 turns this around. The source code describes it plainly: vLLM reserves the address range for the cache using the GPU's virtual memory, finishes CUDA graph capture and warmup, and only then backs the cache with real memory, using what is actually free. Because the measurement comes after start-up, the default share is now 1.0, the whole card.
The release notes give one example: MiniMax-M3 on an NVIDIA H200 went from 433,000 tokens of cache to 672,000. That is 55% more room for conversations on the same GPU. There is a cost, which the notes also state: on some models the time between output tokens rose by 6 to 18% on an H200 with 128 requests at once.
You can still turn the new behaviour off. Pass --gpu-memory-utilization 0.9 to set a share yourself, or --no-enable-extensible-kv-cache to go back to estimate-based sizing. vLLM also falls back to the old way by itself on a driver without virtual memory support, and when you fix the cache size with kv_cache_memory_bytes.