The Same Job, Said Seven Ways
Imagine a front desk in a busy office. Every morning the manager tells the new helper what to do with the notes that customers hand in. On Monday she says: "Put each note into one of these trays: billing, delivery, returns, account." On Tuesday she says: "You sort the notes. The trays are billing, delivery, returns and account." On Wednesday: "Is this note about billing, delivery, returns or account?" The meaning is the same every day. A good human helper would sort the notes the same way on all three days, because a person listens for the meaning, not the exact words.
A language model does not listen for meaning in the same way. It reads the exact words you give it, one token at a time, and the next token it writes depends on all of them. So a question worth measuring is this: if I say the same thing in different words, do I get the same answers?
This lesson answers it with a measurement. I took the sorting instruction from lesson 1, wrote it seven ways that all mean the same thing, and sent each version with the same 40 customer messages to two small models. I also kept the original words and only changed the order in which the four categories are listed. Then I counted, message by message, which answers stayed the same and which moved.
Five Words for This Lesson
Wording. The exact words of a prompt. Two wordings can mean the same thing to a person and still be different text to a model. In this lesson every wording keeps the same two jobs from lesson 1: an instruction that names the four categories, and a format rule that asks for one word.
List order. The order in which the instruction names the categories. Lesson 1 lists them as billing, delivery, returns, account. The order carries no meaning here, so in theory it should not matter.
Spread. The lowest score and the highest score across several wordings of one prompt. If seven wordings score between 32 and 39, the spread is 32 to 39.
Steady. A message that got a usable answer under every one of the seven wordings. A message that is usable under some wordings and not under others is changing.
Fixed, broken. When you move from one wording to another, a message is fixed if it was not usable before and usable after, and broken if it was usable before and not usable after. Lesson 7 used the same two counts to compare direct answers with step-by-step ones.
As in lesson 1, an answer is usable when the whole reply, in lower case and without spaces at either end, is exactly the correct category word. It is right when its first word, after an optional "Category:", is the correct category, even if more text follows.