Draw One Without Looking
Picture a jar of marbles, mostly red, with a few blue, green and yellow. Reach in without looking and you will usually draw a red one. Now picture a dial that changes the mix before you draw: turn it one way and the jar becomes almost all red; turn it the other way and the colours even out.
Lesson 1 showed that a language model's output, at every step, is a list of odds for every possible next token. It never writes a word on its own. Something has to look at that list and choose one token, and then the model runs again on the text with that token added. The part that chooses is called the sampler, and it has settings. You have almost certainly met them already: the "temperature" slider in a chat playground, or a temperature field in an API request.
Those settings decide whether the same question gets the same answer every time or a different one, whether a reply stays sensible or turns into nonsense, and whether you can repeat a result later to check it. Many people copy a setting from an example, or leave it at a default they have never looked at. In this lesson I measure what each setting really does to the text a small model writes, and I work through the arithmetic by hand so that you can predict the effect before you touch the dial.
The Words You Need First
Sampling. Picking the next token at random, but following the odds, so likely tokens are picked more often. If a token has odds of 0.62, it is picked about 62 times in every 100 draws.
Temperature. A dial that sharpens or flattens the odds before the pick.
top_k. Keep only the k most likely tokens and drop the rest before the pick. k is a whole number, such as 5 or 40.
top_p. Keep the most likely tokens, in order, until their odds add up to p, and drop the rest. p is a number between 0 and 1.
Seed. A number that fixes the random draws, so the same run can be repeated exactly.
Greedy. Always take the single most likely token. Temperature 0 does this.
Log probability and odds mean the same as in lesson 1: odds are numbers from 0 to 1 that add up to 1 at each step, and a log probability is the same value written as its natural logarithm. Python's math.exp turns a log probability back into odds.
Filter. A step that removes some tokens before the draw, so they cannot be picked. top_k and top_p are the two filters in this lesson.
Rescale. Divide every number by their total, so that together they add up to 1 again.
␣ marks a space at the start of a token, as in lesson 1. "␣Paris" is the token " Paris".