Self-hosted LLM: how to run an open model on a GPU you control
What is a self-hosted LLM?
A self-hosted LLM is an open language model, such as Qwen or Llama, that runs on GPUs your team controls, either in your own data centre (on premise) or in a cloud account that belongs to you. Your apps send prompts to your own server instead of to an outside AI company, so the prompts and the answers stay with you. You pay for the GPU by the hour, or buy it, instead of paying for each token. Doing it well comes down to three questions: will the model fit in the GPU's memory, how do you serve it to many users at once, and how do you lock the server so only your own apps can use it.
Every number on this page comes from a lesson of our Inference Engineering course, measured on one NVIDIA RTX 4090 that we rented on RunPod on 13 November 2026. You learn on a rented GPU, the same way you would run it on your own servers: the commands do not change, only the machine does.
Try it yourself: 4 lessons of the course are free to read, with no payment.
What self-hosted means, and what private means here
When you call a hosted AI service, your prompt travels to that company's servers, their model writes the answer, and they bill you for every token. When you self-host, you download an open model's files once, from a site such as Hugging Face, and run them with serving software on a GPU you control. Your apps then talk to your own server.
People use three names for this. On-premise means the GPU sits in your own building or data centre. Self-hosted is wider: it also covers a GPU server you rent in a cloud account that belongs to you. Host your own LLM is how most people phrase the question. The software and the commands are the same in all three cases.
Private means that no outside AI company sees your prompts or your answers. It does not mean the server is safe on its own. A model server you start with one command will answer anyone who can reach its port, so locking it is part of the job, and a later section shows how.
When hosting it yourself is worth it, and when an API is cheaper
Hosting a model yourself is usually worth it in three situations:
- Your data must not leave your control, because of a customer contract, a regulator or your own policy.
- You have steady traffic that keeps a GPU busy most of the day, because you pay for the GPU every hour whether it is answering or idle.
- You need a fixed model version that nobody changes under you, or settings a hosted service does not offer.
An API is usually cheaper when your traffic is small or comes in bursts, when you need the very largest models, which need many GPUs, or when nobody on the team can run servers. Work out your own numbers before you decide. The course's full lesson on this choice, "Host it yourself or use an API", is still being written.
Will the model fit? Size the GPU memory first
A GPU's memory has to hold two things: the model's weights, which have a fixed size, and the KV cache, the notes the model keeps on every token of every conversation it is serving. The cache is what decides how many people one GPU can serve at once.
Here is a real example from the free lesson The KV cache. We served Qwen3-8B with vLLM on a 24 GB RTX 4090. Its weights in 16 bits took 15.27 GiB. Each token of each conversation needs 144 KiB of cache, worked out from four numbers in the model's config file. vLLM gave the cache what was left, 4.56 GiB, which is room for 33,216 tokens: about 16 conversations of 2,000 tokens at the same time.
You can do the same sum for your own model and GPU with our free GPU memory calculator. If the weights alone do not fit, you need a bigger GPU, a smaller model, a model stored in fewer bits, or a model split across several GPUs.

Serve it with one command, and call it like OpenAI
This page serves the model with vLLM, a free, open-source serving engine. On a fresh rented GPU, we made a Python environment and installed it with uv:
uv venv --python 3.12 ~/vllm-env
source ~/vllm-env/bin/activate
uv pip install vllm==0.33.0
vllm serve Qwen/Qwen3-0.6BThe install took 1 minute 0.7 seconds on a fresh machine, and the first start was ready after 2 minutes 50 seconds, most of it spent preparing the GPU. The server then answers on port 8000, at the same web addresses as OpenAI's API, so your existing code only needs a new address and a model name:
from openai import OpenAI
client = OpenAI(base_url="http://127.0.0.1:8000/v1", api_key="not-needed-for-a-local-server")
reply = client.chat.completions.create(
model="Qwen/Qwen3-0.6B",
messages=[{"role": "user", "content": "Why might a parcel's tracking page not change for a day?"}],
)Hands-On Inference Engineering, founding price
Learn to run it on your own servers: practise on a rented GPU first
Hands-On Inference Engineering teaches you to size a model for a GPU, serve it with vLLM, lock it with a key, and measure how many people it serves and what it costs. Every lab runs on a GPU you rent by the hour and switch off at the end, with the same commands you would use on your company's own servers.
36 of the 81 planned lessons are out today, about 32 hours, and the rest are being written. Every lesson added later is included.
₹2,222 or $49 once, lifetime access. It rises to ₹9,999 or $249 after the founding period.
Lock it so only your own apps can call it
vLLM can require a key. In the lesson "vLLM serving settings" we made a random key, put it in the environment variable VLLM_API_KEY so it never appears in the list of running programs, and started vLLM again. The model list then answered 401, which means unauthorized, without the key, and 200 with it:

Look at the first two lines: /health and /metrics answered without any key. vLLM 0.33 checks the key only on paths that start with /v1, /v2, /inference and /cohere. vLLM's own security guide says it plainly: "Do not rely exclusively on --api-key for securing access to vLLM." So keep the server on 127.0.0.1 or a private network, and put a reverse proxy such as nginx or Envoy in front of it that exposes only the addresses your apps need. The settings for HTTPS certificates and the rest of the server's security are explained one by one in the course's reference lesson "The API server and its security".
How many people one GPU can serve at once
A serving engine is built to share one GPU between many people. With the small model Qwen3-0.6B on the RTX 4090, one person on their own got 359 tokens a second. When 64 people asked at the same moment, the GPU wrote 12,087 tokens a second in total, about 189 for each person, and all 64 answers were back after 1.06 seconds.

A bigger model is slower and leaves less room for the cache, so measure your own model on your own GPU before you promise a number to anyone. The free lesson Prefill and decode explains why reading a prompt is so much faster than writing an answer, which is where most of this time goes.
What it costs to learn this on a rented GPU
You do not need to buy a GPU to learn this. We rented an RTX 4090 on RunPod's secure cloud at $0.89 an hour, and our whole session for every lesson in the first part of the course, plus setup, took 2 hours 18 minutes and cost $2.01. Every lab ends by switching the GPU off, so nothing keeps running and billing. The free lesson Before you start walks you through renting one, asking a model your first question and switching it off.
Running a model for real users is a different sum: a GPU that serves people around the clock is billed for every hour of the month. Measure your speed first, then divide the GPU's monthly price by the tokens it can really write, and compare that with an API's price list.
A laptop or a server?
If you want a model for yourself on your own computer, tools such as Ollama and LM Studio are the easy way, and our guide to running an LLM locally compares them with measured numbers. A self-hosted LLM for a team or a product is a different job: many people ask at once, so you want a serving engine such as vLLM on a GPU server, with a key, a private network and someone watching it.
What this page does not cover yet
This page covers one model on one GPU. Splitting a large model across several GPUs, shrinking a model with quantization, and running many model servers on Kubernetes are later parts of the course that are still being written. We will add them here when the lessons that measure them are out.
Questions people ask
Is a self-hosted LLM private?
It keeps your prompts and answers away from outside AI companies, because the model runs on a GPU you control. It is not secure on its own: a model server answers anyone who can reach its port. Give it a key, keep it on a private network, and put a reverse proxy in front of it. vLLM's own security guide says not to rely on the key alone.
What GPU do I need to self-host an LLM?
Enough memory for the model's weights plus the KV cache for the people you serve. On a 24 GB RTX 4090, Qwen3-8B in 16 bits took 15.27 GiB for its weights and left room for 33,216 tokens of conversation, about 16 chats of 2,000 tokens. Our free GPU memory calculator does the same sum for other models and cards.
Is self-hosting an LLM cheaper than using an API?
Only when a GPU stays busy. You pay for a GPU every hour, busy or idle, while an API charges per token. Steady, heavy traffic and data that must stay with you favour self-hosting; small or bursty traffic and the very largest models usually favour an API. Measure your own speed before you decide.
What is the difference between a self-hosted LLM and a local LLM?
A local LLM usually means a model on your own laptop for one person, with a tool such as Ollama or LM Studio. A self-hosted LLM usually means a model on a GPU server your team controls, serving many people or a product, with an engine such as vLLM.
What does on-premise LLM mean?
An on-premise LLM runs on GPUs inside your own building or data centre. It is one kind of self-hosting. A GPU you rent in a cloud account you own is self-hosted too, and the serving software and commands are the same in both cases.
Hands-On Inference Engineering, founding price
Learn to run it on your own servers: practise on a rented GPU first
Hands-On Inference Engineering teaches you to size a model for a GPU, serve it with vLLM, lock it with a key, and measure how many people it serves and what it costs. Every lab runs on a GPU you rent by the hour and switch off at the end, with the same commands you would use on your company's own servers.
36 of the 81 planned lessons are out today, about 32 hours, and the rest are being written. Every lesson added later is included.
₹2,222 or $49 once, lifetime access. It rises to ₹9,999 or $249 after the founding period.
Written by Roni Das. Published 2026-11-16, checked against the lessons on 2026-11-16.