Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Foundation · LLM and GenAI Ops
A setting that controls how random a language model's word choice is. Low temperature picks the most likely next token almost every time; high temperature spreads the choice wider.
In short
LLM temperature is a sampler setting that sharpens or flattens a language model's odds for the next token before one token is drawn at random. Temperature 0 always takes the top token, so the same prompt gives the same text; higher values spread the choice, and too high breaks the text.
In short
LLM temperature is a sampler setting that sharpens or flattens a language model's odds for the next token before one token is drawn at random. Temperature 0 always takes the top token, so the same prompt gives the same text; higher values spread the choice, and too high breaks the text.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers LLM Temperature across a run of lessons, not one page. These 4 alone are about 222 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
The model gives odds; a sampler picks one token. Measured with qwen2.5:3b on 3 prompts, 40 runs per setting: at temperature 0 every run wrote the same text; at 0.7 almost every run was different. For 'The capital of France is', 40 of 40 runs began with Paris at temperature 0 and only 5 of 40 at 1.5, where the text broke into other scripts and symbols. The same seed gave the same text every time.
A language model does not write sentences. At every step it gives odds for every possible next token, and something else picks one. Measured with qwen2.5:3b on 24 prompts: for 12 facts the top next token had a median probability of 0.70; for 12 open prompts only 0.17. The same five words with and without the hidden chat template gave ' Paris' at 0.62 or 'The' at 0.999.
A language model never sees letters or words. It sees tokens: small pieces of text, each with a number. This lesson shows how real tokenizers cut text, measured with OpenAI's tiktoken. This course's 1.27 million words come to 1.29 tokens per word. Numbers are cut into pieces of up to three digits. One sentence written in eight languages took 11 tokens in English but up to 100 in other languages with GPT-2's tokenizer, and 12 to 19 with GPT-4o's. That difference is paid in money and in context space.
I tested twelve changes to the same 0.5B ticket model, one setting at a time: the seed, the number of training examples, the number of passes, the learning rate, and whether the loss counts the prompt. First I measured the wobble, what the seed alone does: the totals were 93, 92 and 93 whole tickets of 131, but 15 to 21 tickets flipped between right and wrong between any two seeds. Against that, two changes clearly mattered: more examples (26, 58, 78, 93 for 50, 100, 200 and 500 rows) and a learning rate far too small (58). A higher learning rate (96) and masking the prompt (97) beat the one-ticket spread of the totals, but not the flips, and the sign test cannot tell either from luck.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Temperature and Sampling: How One Token Gets Pickedpay once,
yours for life
A language model does not write words by itself. At every step it gives a list of odds, one for every possible next token. A token is a small piece of text, such as a word or part of a word. The odds are numbers from 0 to 1, and together they add up to 1.
Something must look at that list and choose one token. That part is called the sampler. It is ordinary code, not part of the model. After it picks a token, the model runs again on the text with that token added.
Temperature is one of the sampler's settings. It is a dial that changes the odds before the pick. A low temperature makes the likely tokens even more likely. A high temperature makes the odds more even, so rare tokens get picked more often.
A good picture is a jar of marbles, mostly red, with a few blue, green and yellow ones. Draw without looking and you usually get red. Temperature changes the mix before you draw. Turn it down and the jar becomes almost all red. Turn it up and the colours even out.
The simplest rule is to always take the top token. This is called greedy choosing. It is fast, and it gives the same answer every time. For a fact, a number or a classification, that is what you want.
Greedy choosing also has a cost. The model can only ever write one text for a given prompt. Ask for five product names and you get the same name five times. Every story starts the same way.
In the lesson's lab, a small model (qwen2.5:3b, run locally in Ollama) continued "The capital of France is". At temperature 0 it wrote "Paris. The capital of Germany is Berlin" in all 40 runs. That is not wrong, but nobody asked about Germany. The model simply followed its likeliest path every time.
So samplers usually draw at random, but following the odds. A token with odds of 0.62 gets picked about 62 times in every 100 draws. Most draws still pick a likely token. Now and then a less likely one wins, and the text branches.
The rule is short. Each token's odds can be written as a log probability, which is the natural logarithm of the probability. The sampler divides each log probability by the temperature, turns the results back into ordinary numbers, and rescales them so they add up to 1 again.
There is a simpler way to see it. Dividing by T is the same as raising every probability to the power 1 ÷ T. At temperature 0.5 every probability is squared. At temperature 2 every probability is replaced by its square root. Squaring pushes big numbers far ahead of small ones. Square roots pull them closer together.
The lesson works this through by hand. After "The capital of France is", the token "Paris" had odds of 0.6219 and a blank "__" had 0.0434. So Paris was about 14 times as likely. At temperature 0.5 it becomes about 205 times as likely. At temperature 2 it is only about 3.8 times as likely.
At exactly 1 nothing changes: you draw from the model's own odds. At 0 you cannot divide by zero, so samplers treat 0 as a special case and just take the top token. That is why temperature 0 and greedy choosing mean the same thing.
Arithmetic on ten tokens is only half the story. The lesson also ran the real model. It used three prompts: a fact, an open prompt ("My favourite food is") and a story. Each setting ran 40 times with seeds 1 to 40, writing 12 tokens each time. A seed is a number that fixes the random draws, so a run can be repeated.
The fact prompt shows the cost most clearly. Runs that began with Paris fell from 40 of 40 at temperature 0 and 0.3, to 31 at 0.7, 21 at 1.0 and only 5 at 1.5.
Why is 1.5 so much worse than the ten-token arithmetic suggests? The real model's list has 151,936 slots. A high temperature lifts every rare token a little. Each one is still unlikely, but there are so many that together they win many draws. This group of rare tokens is called the long tail. One strange token then makes the next step strange too, so the errors grow.
Top-k and top-p exist to stop that drift. Both are filters. A filter removes some tokens before the draw, so they cannot be picked at all.
Top-k keeps only the k most likely tokens, where k is a whole number such as 5 or 40. With top_k 5 after "The capital of France is", the five kept tokens add up to 0.7347. After rescaling, Paris gets about 0.85 of the draw. The other 151,931 tokens can no longer win.
Top-p keeps the most likely tokens in order until their odds add up to at least p, a number between 0 and 1. With top_p 0.5, Paris alone holds 0.6219, which is more than half. So Paris is the only token kept, and it is drawn every time. In the lab, all 40 runs of the fact prompt began with "Paris.".
The key difference: top-k always keeps the same number of tokens, whether the model is sure or not. Top-p adapts. It keeps one token when the model is sure and many when it is not. That is why top_p 0.5 cut the fact prompt to 26 different texts but left all 40 story texts different.
Note that top_k 1 keeps only the top token, so it behaves exactly like temperature 0. The lab got 1 text out of 40 for every prompt.
The sampler runs in four steps. The model gives odds. Temperature reshapes them. Top-k and top-p cut off the unlikely end. Then one token is drawn, and the seed decides which random numbers are used. Real tools run temperature and the filters in different orders, so check your tool's docs if the order matters to you.
Computers do not use truly random numbers here. They use a formula that makes a long sequence of numbers that look random, and the seed is where that sequence starts. In the lab, the story prompt at temperature 1.0 with seed 7 gave exactly the same text twice. That makes a sampled result something you can test and debug. The lesson checked this on one computer and one Ollama version only.
One trap catches people who debug. The odds Ollama reports with logprobs come from the model, before temperature and the filters. They do not change when you move the dial. To see what temperature does, read the text the model writes.
Start with one question: is there one right answer?
Temperature 0 makes a model consistent, not correct. It wrote the same unwanted sentence about Berlin all 40 times. It does not stop a hallucination; it only repeats the same one.
The full lab, a playground where you turn the dial on real odds yourself, and the quiz are in the Temperature and Sampling lesson. If the idea of a model giving odds is new, start one step earlier with what a model outputs.
Ollama
Takes temperature, top_k, top_p and seed in a request's options. Its documented defaults are 0.8, 40 and 0.9, which the lesson's lab overrode so its runs could be repeated.
Chat playgrounds and LLM APIs
Most expose a temperature slider or a temperature field. The lesson's advice applies to all of them: set it on purpose, with a seed when the tool supports one.
Python
The lesson's playground applies temperature to ten real odds with math.exp and draws 1,000 tokens with random.choices, so you can watch Paris fall from about three in four draws to about half.
It means there is no random draw. The sampler always takes the single most likely token, which is called greedy choosing. In the lesson's lab, all 40 runs at temperature 0 wrote the same text, even with 40 different seeds.
When there is one right answer: a fact, a number, a classification or JSON output. Then variety only adds mistakes. Remember that 0 makes the model consistent, not correct: it repeats the same answer, right or wrong.
Use 0 or close to it when there is one right answer. When you want variety, start around 0.7 and read ten samples before deciding. In the lesson, 1.5 broke the text in many runs, so very high values are rarely useful.
Top-k keeps a fixed number of the most likely tokens, such as 40, at every step. Top-p keeps the most likely tokens until their odds add up to p, so it keeps few when the model is sure and many when it is not. Both remove rare tokens before the draw.
Yes, in effect. top_k 1 keeps only the single most likely token, so there is nothing left to choose between. The lesson's lab got exactly one text out of 40 runs with top_k 1, as it did at temperature 0.
Token
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
Hallucination
When a language model produces fluent, confident text that is not supported by any source and is often simply false. Retrieval, grounding, and evaluation are the main defenses.
Structured Output
Constraining a model to return valid JSON matching a schema, so downstream code can rely on it. The foundation of reliable tool calling.
Prompt Engineering
Writing and structuring the instructions given to a language model to get reliable output. In 2026 the craft has moved from clever wording to schemas, context layout, and evaluation.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Temperature and Sampling: How One Token Gets Picked" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Temperature and Sampling: How One Token Gets Picked lessoncourse 2
Take models from notebook to production, with labs on real models.