What is vLLM? A plain explanation, then a real install on a GPU
What is vLLM?
vLLM is free, open-source software that serves large language models on GPUs. You start it with one command and a model's name, and it answers on the same web addresses as OpenAI's API, so code written for OpenAI works with it once you change the address. It is fast when many people use it at once because of two ideas: PagedAttention, which stores the model's notes on each conversation in small pages so little GPU memory is wasted, and continuous batching, which lets new requests join between steps instead of waiting for a whole batch to finish. On one rented RTX 4090, vLLM 0.33 served 64 people at once at 12,087 tokens a second in total.
The commands and numbers on this page come from the lesson "Serving a model with vLLM" in our Inference Engineering course, run with vLLM 0.33.0 on one NVIDIA RTX 4090 that we rented on RunPod on 13 November 2026.
Try it yourself: 4 lessons of the course are free to read, with no payment.
What vLLM does, in plain words
A language model on its own is a large set of numbers in a few files. To answer questions, something has to load those numbers onto a GPU, take requests from apps over the network, run the model one token at a time, and send the answer back. That something is a serving engine, and vLLM is one of them. It is open source, so you can read every line of it and run it on your own GPUs at no charge.
Its main job is to serve many people from one GPU without wasting memory or time. Writing an answer is slow, one token after another, so a GPU that serves one person at a time spends most of its power waiting. vLLM keeps it busy by working on many conversations in the same step.
Why vLLM is fast: PagedAttention and continuous batching
For every token of every conversation, a model keeps notes so it never has to read the conversation again. These notes are the KV cache, and on a busy GPU they can take more memory than the model itself. PagedAttention stores them in small fixed-size blocks, the way an operating system stores memory in pages, so a conversation takes only the blocks it really needs and little space is left empty.
Continuous batching decides who runs in each step. Instead of waiting for a whole group of requests to finish before starting the next group, vLLM adds new requests and removes finished ones between steps, so the GPU is never kept waiting for the slowest answer in a group.
Both ideas are about memory and waiting, which is why vLLM, out of the box, reserves 92% of the GPU's memory in version 0.33 and gives everything the weights do not need to the cache. On the 24 GB RTX 4090 with the small model Qwen3-0.6B, the weights took 1.12 GiB and the cache got 19.43 GiB, room for 181,936 tokens.
How to install vLLM on a GPU machine
You need a Linux machine with an NVIDIA GPU. We used a rented RTX 4090 and the fast Python installer uv. First make a Python environment of its own, then install the exact version you want:
uv venv --python 3.12 ~/vllm-env
source ~/vllm-env/bin/activate
uv pip install vllm==0.33.0On a fresh machine the install took 1 minute 0.7 seconds and put 199 libraries in place, among them PyTorch and OpenAI's own Python library. Leave room on the disk: the environment alone measured 9.0 GB. Pin the version, because the settings and their defaults change between releases.
Start a model, then call it with the OpenAI library
One command downloads a model from Hugging Face by its name and starts serving it on port 8000:
vllm serve Qwen/Qwen3-0.6BOur first start was ready after 2 minutes 50 seconds. Loading the model itself took 10.1 seconds; most of the rest was vLLM preparing the GPU, which its documentation describes as compiling the model and capturing CUDA graphs. Later starts are quicker. Then any program that uses OpenAI's library can talk to it, with a new address and the model's name:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed-for-a-local-server")
reply = client.chat.completions.create(
model="Qwen/Qwen3-0.6B",
messages=[{"role": "user", "content": "Why might a parcel's tracking page not change for a day?"}],
)In our run, a 35-token question got its first word back after 0.167 seconds, and vLLM sent its own token counts at the end, the way a paid API does for its bill.

Hands-On Inference Engineering, founding price
Learn vLLM setting by setting, on a real GPU
Hands-On Inference Engineering starts from what a model and a GPU are, serves a model with vLLM, then goes inside it: PagedAttention, the scheduler, chunked prefill, prefix caching and the memory and serving settings, each changed and measured on a GPU you rent by the hour.
36 of the 81 planned lessons are out today, about 32 hours, and the rest are being written. Every lesson added later is included.
₹2,222 or $49 once, lifetime access. It rises to ₹9,999 or $249 after the founding period.
Serving 1, 16, 32 and 64 people at once
The real test of a serving engine is many people at the same moment. We sent 1, 16, 32 and then 64 questions at once, each answer 200 tokens long:

One person alone got 359 tokens a second. Sixty-four people got 12,087 tokens a second in total, about 34 times as much, while each of them was only about 1.9 times slower than alone, and all 64 answers were back after 1.06 seconds. That trade, a little speed for each person in exchange for far more in total, is what vLLM is for.
The vLLM settings people change first
vLLM 0.33 has hundreds of settings, but a handful decide most of what you see. Each is added to vllm serve as a flag:
| Setting | What it decides |
|---|---|
--gpu-memory-utilization | The share of the GPU's memory vLLM may use. It is 0.92 by default in vLLM 0.33. |
--max-model-len | The longest prompt plus answer one request may have. A shorter limit leaves room for more people. |
--max-num-seqs | The most requests vLLM works on in one step. |
--max-num-batched-tokens | The most tokens vLLM processes in one step, which limits how much of a long prompt is read at once. |
--enable-prefix-caching | Whether work for a prompt that starts the same way, such as a shared system prompt, is reused. |
--kv-cache-dtype | The number format of the KV cache. A smaller format, such as fp8, fits more tokens. |
--quantization | Loading a model stored in fewer bits, so it takes less memory. |
--tensor-parallel-size | How many GPUs one model is split across. |
--api-key | A key that requests must carry. The environment variable VLLM_API_KEY does the same and keeps the key off the command line. |
--served-model-name | The name your apps use for the model. |
--enforce-eager | Skipping CUDA graphs: a faster start, and slower writing once it runs. |
Our Inference Engineering course explains every one of vLLM 0.33's settings in its reference lessons, with its real default and when an engineer changes it, and measures the important ones on a GPU.
Where vLLM fits next to laptop tools such as Ollama
| vLLM | Ollama | |
|---|---|---|
Built for | Serving many people from a GPU server | Running a model for yourself on your own computer |
Typical machine | A Linux server with an NVIDIA GPU | A laptop or desktop, including a Mac |
How you start it | vllm serve and a Hugging Face model name | ollama run and a model name |
Talks to apps through | OpenAI's API format, on port 8000 | Its own API, with an OpenAI-compatible option |
If you want a model on your laptop, start with our guide to running an LLM locally, which measures Ollama, LM Studio, llama.cpp and vLLM side by side. If you want to serve a team or a product from a GPU you control, read how to self-host an LLM.
What to learn next
vLLM's speed makes sense once you see the two halves of every request. The free lesson Prefill and decode measures them on a rented GPU: the same card read a prompt about 56 times faster per token than it wrote the answer. The free lesson Before you start shows how to rent the GPU, ask a model your first question and switch the GPU off so nothing keeps billing.
Questions people ask
Is vLLM free?
Yes. vLLM is open-source software that you can download and run on your own GPUs at no charge. You pay only for the GPU it runs on, whether you own it or rent it by the hour.
Is vLLM compatible with the OpenAI API?
Yes. vLLM answers on the same addresses as OpenAI's API, such as /v1/chat/completions and /v1/models, in the same format, so the OpenAI Python library works with it once you change the address and the model's name. In our run on an RTX 4090, the first word came back after 0.167 seconds.
How do I install vLLM?
On a Linux machine with an NVIDIA GPU, make a Python environment and install a pinned version, for example uv pip install vllm==0.33.0. On a fresh rented RTX 4090 that took about a minute, and the environment used 9.0 GB of disk. Then start a model with vllm serve and its Hugging Face name.
How long does vLLM take to start?
Our first start of Qwen3-0.6B on an RTX 4090 took 2 minutes 50 seconds: loading the model took 10.1 seconds, and most of the rest was spent preparing the GPU. The flag --enforce-eager skips part of that preparation for a faster start and slower writing.
How many requests can vLLM serve at once?
It depends on the model and on the room in the KV cache. With Qwen3-0.6B on one RTX 4090, 64 requests at once got 12,087 tokens a second in total, about 189 each, and all 64 answers were back after 1.06 seconds.
Hands-On Inference Engineering, founding price
Learn vLLM setting by setting, on a real GPU
Hands-On Inference Engineering starts from what a model and a GPU are, serves a model with vLLM, then goes inside it: PagedAttention, the scheduler, chunked prefill, prefix caching and the memory and serving settings, each changed and measured on a GPU you rent by the hour.
36 of the 81 planned lessons are out today, about 32 hours, and the rest are being written. Every lesson added later is included.
₹2,222 or $49 once, lifetime access. It rises to ₹9,999 or $249 after the founding period.
Written by Roni Das. Published 2026-11-16, checked against the lessons on 2026-11-16.