Who Decides When a Reply Is Finished?
Lessons 1 and 2 showed how a model picks each next token. Lesson 3 showed that it writes one token at a time. But none of that says when it should stop. A model has no sense that it has "said enough". Left alone, it would keep writing tokens forever.
So something outside the model has to end a reply. There are only three things that can do it, and each leaves a different kind of reply behind. Knowing which one ended a reply tells you whether you received a finished answer or a piece of one.
Think of a person who has been asked to keep marking sheets until told to stop. Nobody comes to tell them. They carry on long after the useful work is done, repeating the same motions. A model that is not given a clear way to stop behaves much the same, and this lesson measures how often.
It also measures the common fixes: a repeat penalty, a stop sequence and a length limit. Each helps with one problem and can cause another.
Words You Need First
Loop. A reply that writes the same run of words again and again. The lab counts a reply as looping if some run of 8 words appears in it at least 3 times.
Repeat penalty. A setting, repeat_penalty in Ollama, that makes tokens used recently a little less likely. 1.0 means no penalty.
Stop sequence. A piece of text, such as a line break, that ends the reply as soon as the model writes it. In Ollama it is the stop option.
num_predict. The most tokens a reply may have. When it is reached, the reply is cut.
done_reason. A field in every Ollama reply: "stop" if the reply ended by an end marker or a stop sequence, "length" if it was cut by num_predict.
Greedy. Always taking the single most likely next token, from lesson 2. It is what temperature 0 does.