Introduction: what this inference engineering course builds, and how to use it

This page is free to read, and you do not need to install or run anything on it. Everything you see in it comes from a real run on an NVIDIA RTX 4090 GPU that we rented by the hour on 13 November 2026, and the next page, "Before you start", shows you how to rent the same GPU, install the same tools and ask the same model your first question yourself.

After this page you will be able to:

• say in plain words what a model is, what a large language model is, and what a token is;
• follow one real request from the moment you press Enter to the moment the last word arrives, and say where the time went;
• explain why answering questions, not training, is where most of the money in AI is spent;
• say what this course builds, part by part, and where each part's hands-on work runs.

Time: about 30 minutes.

The problem: a pause, and then the words arrive one by one

Think of the last time you asked a chat assistant a question. You typed it and pressed Enter. For a moment nothing happened. Then the first word appeared, and the rest followed one after another, a little like someone typing quickly on the other side of the screen.

Behind that small pause is a large, expensive machine. Somewhere a computer with a graphics card, a GPU, read your question and wrote the answer, one small piece at a time. Someone rents that GPU by the hour, and it costs the same whether it answers one person or fifty. So the questions that matter are very practical: the reason for the pause before the first word, the reason the words arrive one by one and not all at once, how many people one GPU can answer at the same time before everyone notices it slowing down, and what one answer really costs.

Answering those questions, and then making the answers faster and cheaper, is the work of an inference engineer. This course teaches that work from the first idea, with real runs on real hardware at every step.

Why the wait for an answer, and its cost, matter

• This is where most of the money goes. A model is trained once, but it answers questions every second of every day after that. Each answer runs on a GPU that someone pays for by the hour. We work this out with real numbers later on this page.
• Users feel every millisecond. People judge a chat assistant by two things: how long the first word takes, and how fast the rest of the words arrive. Both are decided by the engineering in this course, not by the model alone.
• A company that serves its own models needs this skill, and you can prove it with numbers. A company that serves its own models needs someone who can say "this GPU will serve 40 people at once with the first word in under half a second, at this cost per million words", and then show the measurement. By the end of this course you can.

Four words you need first: model, language model, token and the writing loop

Before we follow a real request, we need four words. Each one is simpler than it sounds.

What a model is

Imagine a music mixing desk with a very large number of small dials. Turn the dials one way and the desk makes one sound; turn them another way and it makes a different sound. A model is like that desk, except that the dials are numbers stored in a file, and instead of sound it turns an input into an output.

The numbers start out random. Training is the long process of showing the model a huge number of examples and nudging every dial a tiny amount each time, so that its outputs get closer to the right ones. When training ends, the dials stay where they are. The file of numbers is the model.

Inference is what happens after that: using the finished numbers to answer a new question. Nothing is learned during inference. The model only reads its numbers and does arithmetic with them. That is the part this course is about.

The model we use on the free pages, Qwen3-0.6B, holds about 596 million of these numbers. Its file stores each number in about 8.5 bits and is 610 MiB. (A MiB, a mebibyte, is 1,048,576 bytes, a little more than a million, and it is the unit Linux tools such as du use for file sizes; a Mac's Finder shows the same file as about 639 MB.) That is small for a language model. The models companies serve every day hold billions or hundreds of billions of numbers.

What a large language model is

A large language model, an LLM, is a model that was trained on a very large amount of text to do one simple-sounding job: given some text, say which piece of text is most likely to come next. That is all it does. It answers a question by writing the most likely next piece, adding it to the text, and doing the same again.

What a token is

A model does not read letters or whole words. It reads tokens: small pieces of text, often a whole common word, sometimes part of a word or a single character. Before a question reaches the model, a program called a tokenizer cuts it into tokens and turns each token into a number from the model's list of known pieces.

Here is our model's own tokenizer cutting a plain question into tokens. The number on the left is the token's place in the model's list; the text on the right, in quotes, is the piece it stands for. Notice that most pieces carry the space in front of the word. We ran it in the inference-lab folder on our rented GPU, where "Before you start" puts the model file:

llama-tokenize -m models/Qwen3-0.6B-Q8_0.gguf -p "Why is my parcel late? It was due on Monday." --show-count --log-disable

 10234 -> 'Why'
   374 -> ' is'
   847 -> ' my'
 29309 -> ' parcel'
  3309 -> ' late'
    30 -> '?'
  1084 -> ' It'
   572 -> ' was'
  4152 -> ' due'
   389 -> ' on'
  7014 -> ' Monday'
    13 -> '.'
Total number of tokens: 12

Ten words and two marks became twelve tokens, because every word here is common. Now look at what happens to a parcel code the model has never seen:

llama-tokenize -m models/Qwen3-0.6B-Q8_0.gguf -p "Parcel QX-7781-B for Mrs Ghoshal" --show-count --log-disable

 27743 -> 'Parcel'
  1207 -> ' Q'
    55 -> 'X'
    12 -> '-'
    22 -> '7'
    22 -> '7'
    23 -> '8'
    16 -> '1'
  7671 -> '-B'
   369 -> ' for'
 17618 -> ' Mrs'
 23908 -> ' Gh'
  9267 -> 'osh'
   278 -> 'al'
Total number of tokens: 14

Five words became fourteen tokens. The code QX-7781-B alone took eight: one for each digit of 7781, and four more for its letters and hyphens. Unusual text costs more tokens, and as you will see in a moment, every token costs time and money.

How a model writes: one token at a time

A model does not write a whole answer in one go. It works in a loop. It reads all the text so far, picks the most likely next token, adds that token to the end, and goes round again, until it writes a special token that means "I have finished" or reaches the limit you set. An answer of 100 tokens is 100 turns of this loop. Keep this loop in mind: it is the reason for almost everything in this course.

One real request, followed from your keyboard to the GPU and back

Now we can follow a real request. On a rented GPU, an NVIDIA GeForce RTX 4090 with 24 GB of memory on RunPod, we started llama-server, a small model server from the open-source llama.cpp project, with the same Qwen3-0.6B model. A model server is a program that keeps a model loaded in memory and answers questions sent to it over the network. It speaks the same request format as OpenAI's chat API, which most model servers, including vLLM, also speak.

We started it in the background and asked whether it was ready:

llama-server -m models/Qwen3-0.6B-Q8_0.gguf -c 4096 --port 8080 --reasoning off --alias qwen3-0.6b > server.log 2>&1 &

[1] 1573

--alias qwen3-0.6b only sets the model name the server reports in its replies. The line [1] and a number is the shell telling you the server's job number and process number.

curl -s http://127.0.0.1:8080/health

{"status":"ok"}

The question we sent is in a small file, request.json, which we wrote in the same folder; cat shows what is in it. It holds one message from the user, a limit of 60 tokens for the answer, and one setting, cache_prompt, which we turned off so that the server reads the whole question from scratch. (The page "The KV cache" shows what that cache saves.)

cat request.json

{
  "messages": [
    {"role": "user", "content": "In one sentence, why might a parcel arrive a day late?"}
  ],
  "max_tokens": 60,
  "cache_prompt": false
}

Then we sent it with curl, a command that sends a web request from the terminal, and passed the reply to jq, a small tool that picks parts out of a JSON reply (on Ubuntu you install it with apt-get install jq; it was already on our GPU machine). We asked it for three parts: the answer, the token counts, and the server's own timings.

curl -s http://127.0.0.1:8080/v1/chat/completions -H "Content-Type: application/json" -d @request.json | jq '{answer: .choices[0].message.content, usage, timings}'

{
  "answer": "A parcel might arrive a day late due to unexpected circumstances, such as weather delays or a delivery error.",
  "usage": {
    "completion_tokens": 22,
    "prompt_tokens": 25,
    "total_tokens": 47,
    "prompt_tokens_details": {
      "cached_tokens": 0
    }
  },
  "timings": {
    "cache_n": 0,
    "prompt_n": 25,
    "prompt_ms": 35.192,
    "prompt_per_token_ms": 1.40768,
    "prompt_per_second": 710.3887247101615,
    "predicted_n": 22,
    "predicted_ms": 58.415,
    "predicted_per_token_ms": 2.7816666666666667,
    "predicted_per_second": 359.49670461354106
  }
}

Read the reply slowly, one part at a time.

• The answer is one plain sentence, 22 tokens long (completion_tokens).
• The question became 25 tokens (prompt_tokens). On its own it is 13 tokens: we counted it afterwards with the same llama-tokenize command, which ended Total number of tokens: 13. The server wraps every message in a few extra tokens that mark who is speaking and where the message ends, the model's chat template. You pay for those tokens too.
• Reading the question took 35.2 milliseconds (prompt_ms). The model read all 25 tokens in one pass. This first half of every request is called prefill.
• Writing the answer took 58.4 milliseconds for 22 tokens (predicted_ms), about 2.8 milliseconds for each token (predicted_per_token_ms). This second half, the loop that writes one token per turn, is called decode.

So the request spent about 38% of its time reading and 62% writing, even though the question and the answer were about the same length. Reading all 25 question tokens took less time than writing the 22 answer tokens one by one. That surprises almost everyone the first time. It is the most important pattern in this course, and the free page "Prefill and decode" measures it properly, with longer questions and many runs.

Note: A very short question like this one is the hardest case for a fair comparison, because the GPU has so little to do in prefill that its fixed start-up costs dominate. With a question of a few thousand tokens, prefill on this same GPU reads tokens dozens of times faster than decode writes them. You measure exactly that in "Prefill and decode".

Why inference, not training, is where the money goes

Training a large model is famously expensive, and the bill arrives once. Inference is cheaper per question, but the questions never stop. Every answer occupies a GPU for a fraction of a second, and a popular service answers millions of questions a day, every day, for as long as the model is in use. For a model that many people use, the answering bill keeps growing long after the training bill is paid, and it can end up the larger of the two.

The arithmetic of one answer is simple enough to do on paper:

1. A GPU is rented by the hour. On 13 November 2026, RunPod's price page listed an NVIDIA RTX 4090, the card most of this course uses, at $0.34 an hour on its community cloud, which is machines that other people rent out through RunPod, and $0.89 an hour on its secure cloud, which is machines in data centres RunPod works with. The one we rented for this page cost $0.89 an hour, because the community cloud had none free that morning.
2. In one hour, the GPU writes some number of tokens. That number depends on the model, the GPU, the engine, and, above all, how many people it serves at once.
3. The cost of a million tokens is the hourly price divided by how many millions of tokens it wrote in that hour.

Step 2 is where the engineering lives. Here is a real example of how much it can change, measured on the same rented RTX 4090. In "Prefill and decode" we ask our small model to write for one person at a time, and then for 32 people at once. Here is what the GPU did, and what a million written tokens cost at the $0.89 an hour we paid for it:

• One person at a time, the GPU wrote 586 tokens a second. At that speed, at $0.89 an hour, a million tokens written cost about $0.42.
• 32 people at once, it wrote 4,854 tokens a second in total, about 8.3 times more, although each person now got only about 152 tokens a second. At that speed, a million tokens written would cost about $0.05.

These sums count the answer's tokens only: the time spent writing, and not the time spent reading the questions. The speeds are real measurements on the GPU we rented, and the price is what we paid for it. Nothing about the machine or its price changed between the two lines. The only thing that changed was how the work was shared out.

That is the whole job in one picture. The same hardware at the same price can cost several times more or less per answer, depending on how the work is arranged. Learning to arrange it well, and to prove it with measurements, is what this course is for.

What this course builds

The course is laid out like a book: "Start here", then eight parts. Each part ends with something you have built and measured yourself.

1. Inference from zero. What a model is, tokens, the loop that writes an answer, a transformer in plain words, and what a GPU is. You run a model on a rented GPU, first with llama.cpp and then with vLLM.
2. The maths of one request. Prefill and decode, why writing is limited by memory speed, the KV cache, batching, and the latency numbers every serving team watches. Each number is worked out on paper first and then measured.
3. vLLM in depth. How vLLM is built and every important setting, each one changed on a real GPU and its effect measured.
4. Smaller and faster models. Quantization, which stores the model's numbers in fewer bits, and speculative decoding, where a small fast model guesses ahead and the large model checks.
5. SGLang. A second serving engine, and a fair benchmark of vLLM against SGLang on the same GPU.
6. Many GPUs. Splitting one model across GPUs, and when copies are better than splits.
7. Serving at scale with Kubernetes and llm-d. Kubernetes is the standard system for running programs across many machines, and llm-d is a router built on it. This part routes each request to the server that already has its work cached, first on a free local cluster with simulated GPUs, then on real GPUs.
8. Measure, watch, pay and prove it. Honest benchmarks, service levels, cost per million tokens and dashboards, ending in a capstone where you serve a real model and prove its cost.

Where the hands-on work runs, and what it costs you

Every page says where its hands-on part runs, before you start it.

• On one rented NVIDIA GPU, an RTX 4090, for every page that measures anything: the free pages "Before you start", "Prefill and decode" and "The KV cache", and Parts 1 to 5 and 8. Every number on these pages comes from a GPU like this one.
• On two rented GPUs for Part 6, and a small cloud cluster for one page in Part 7.
• On a free local Kubernetes cluster on your own computer for most of Part 7, where the GPUs are simulated so you can learn the routing for nothing. "Before you start" shows you how to rent the GPU and switch it off, and the page "Renting a GPU without a surprise bill", in Part 1, goes further, with other clouds and ways to keep spending inside the credit you add. Every lab ends with a step that switches the GPU off. At RunPod's community price in November 2026, we estimate all the GPU labs together at about $10 to $30; at the secure price, which we paid for the free pages, it is about two and a half times as much.

How to use each page

Every page follows the same order, because that is the order in which people learn:

1. The problem, as a short story, before any new word.
2. Why it matters, in practical terms.
3. The idea, explained in plain words with a diagram.
4. The hands-on part, where you run every command yourself. Each command is shown with its real output under it, copied from a recording of our own run, and with a screenshot of our terminal. When your numbers differ from ours, and they will, the page tells you which parts should match and which depend on your hardware.
5. A knowledge check, with the answers and the reason each wrong answer is tempting.

Pages with code come with a zip file that holds the code, a START-HERE.md file with every step and its expected output, the page's diagrams, and the whole page as a PDF. Pages are called by their names, like "The KV cache", and every page links to the next.

The versions we teach. vLLM 0.33, SGLang 0.5.23 and llm-d 1.0, checked against their official documentation on 12 November 2026. The free pages use llama.cpp's ready-built copy for NVIDIA GPUs, because its small built-in tools time each half of a request on its own. Each page is checked against the official documentation again before it is published.

Knowledge check

Answer all five questions before you read the answers below them.

1. Which of these happens during inference?

• A. The model's numbers are nudged a little after every question, so it learns from each user.
• B. The model's numbers stay the same, and are used to work out an answer to a new question.
• C. The model downloads new numbers from the internet to answer recent questions.
• D. The model is trained again on the question before it answers.

2. Our tokenizer cut "Why is my parcel late? It was due on Monday." into 12 tokens, and "Parcel QX-7781-B for Mrs Ghoshal" into 14. Why does the shorter note need more tokens?

• A. Codes, digits and rare names are cut into small pieces, because the tokenizer has no single piece for them.
• B. Capital letters always take two tokens.
• C. The tokenizer adds a token for every hyphen and nothing else.
• D. Short texts are padded with extra tokens to a fixed length.

3. In our real request on the RTX 4090, prefill took 35.2 milliseconds for 25 tokens and decode took 58.4 milliseconds for 22 tokens. What is the best explanation?

• A. The question was easy to read, but the answer needed difficult thinking.
• B. Decode always takes about twice as long as prefill, whatever the lengths.
• C. The network was slow while the answer was being sent back.
• D. The model reads the whole question in one pass, but writes the answer one token per turn of its loop.

4. A GPU rents for the same price per hour whether it serves one person or thirty-two. On our rented GPU, serving 32 people at once wrote about 8.3 times more tokens per second in total than serving one. What does that do to the cost of a million tokens?

• A. Nothing, because the hourly price did not change.
• B. It goes up, because more people use the GPU.
• C. It falls to roughly one eighth of the one-person cost, because the same hour produces about 8.3 times as many tokens.
• D. It falls to a thirty-second, because there are 32 people.

5. Why can inference, rather than training, end up as most of the money a popular model costs?

• A. Inference uses older, more expensive GPUs.
• B. Training happens once, but inference runs for every question, every day, for as long as the model is in use.
• C. Training is free on most clouds.
• D. Inference has to train the model again for each user.

Answers

1. B. During inference the model's numbers stay exactly as training left them; the model only does arithmetic with them. A is the tempting one, because chat assistants can seem to learn from you; any remembering you see is done by the app sending your earlier messages again, not by the model changing. C and D describe things that do not happen when a model answers.

2. A. A tokenizer has a fixed list of pieces, made from text it saw often. Common words are one piece each; a code like QX-7781-B and a rare name like Ghoshal are not in the list as whole pieces, so they are cut small, and each digit of 7781 became its own token. B and C are guesses from the look of the text, and our run shows "Parcel" with a capital P as one token. D does not happen: a request is never padded to a fixed length.

3. D. Prefill reads all the question's tokens together in one pass, so its time grows slowly with the question's length. Decode writes one token per turn of the loop, and each turn has to go through the whole model, so 22 turns took longer than one pass over 25 tokens. A is tempting because writing feels harder than reading, but the model does the same kind of arithmetic in both halves. C is wrong: our request never left the GPU machine, because the server and curl ran on the same rented machine, which RunPod calls a pod. B is wrong: the split depends on the lengths, and with a long question and a short answer, prefill can take longer.

4. C. The cost of a million tokens is the hourly price divided by the millions of tokens written in that hour. The price stayed the same and the hour produced about 8.3 times as many tokens, so each million costs roughly one eighth as much. A forgets that the tokens per hour changed. D assumes the total speed grew 32 times, but it grew only about 8.3 times, because each person's own speed fell. That trade-off is the subject of "Batching: serving many people at once".

5. B. Training is one large bill; inference is a smaller bill for every question, paid over and over for as long as people use the model. A, C and D are not true: inference runs on the same kinds of GPU, training is never free, and inference does not retrain anything.

Words from this page

• Model: a very large set of numbers, learned from examples, that turns an input into an output.
• Training: the long process that sets a model's numbers, done once before the model is used.
• Inference: using a finished model's numbers to answer a new question. Nothing is learned.
• Large language model (LLM): a model trained on a great deal of text to predict the next piece of text.
• Token: a small piece of text, often a word or part of a word, that a model reads and writes.
• Tokenizer: the program that cuts text into tokens and turns them into numbers, and back again.
• Chat template: the few extra tokens a server adds around each message to mark who is speaking.
• Model server: a program that keeps a model loaded and answers requests sent to it over the network. llama-server, vLLM and SGLang are model servers.
• Prefill: the first half of a request, where the model reads the whole question in one pass.
• Decode: the second half, where the model writes the answer one token per turn of its loop.

What to remember from this page

• A model is a large file of numbers. Training sets them once; inference uses them for every question.
• Text reaches the model as tokens. Common words are one token; codes and rare names are cut small.
• Every request has two halves: prefill reads the question in one pass, and decode writes the answer one token at a time.
• A GPU is paid for by the hour, so the cost of an answer depends on how many tokens that hour produces. Serving many people at once can change it several times over.

Next: "Before you start" lists what you should know, shows you how to rent the GPU, installs the three tools the free pages use (uv, llama.cpp and the hf command), and runs your first model on it.

Sources

Checked on 13 November 2026.

• Our own run on a RunPod pod with one NVIDIA GeForce RTX 4090 (24 GB), secure cloud, $0.89 an hour, llama.cpp build 12110 for CUDA 12.8, model Qwen/Qwen3-0.6B-GGUF, file Qwen3-0.6B-Q8_0.gguf. The recording is in this page's run folder.
• llama.cpp server documentation, for the /health endpoint, the OpenAI-compatible chat endpoint and the cache_prompt option: https://github.com/ggml-org/llama.cpp/blob/master/tools/server/README.md
• Qwen3-0.6B on Hugging Face (Apache 2.0 licence): https://huggingface.co/Qwen/Qwen3-0.6B-GGUF
• RunPod pricing, RTX 4090 community cloud $0.34 an hour and secure cloud $0.89 an hour: https://www.runpod.io/pricing
• Our batching run on the same GPU, from the page "Prefill and decode".