Four New Staff, One Briefing
Imagine a shop that takes on four new people for its customer service desk on the same Monday. On the first morning, the manager gives all four the same briefing. She reads out the house rules for writing up a customer's problem: which team it goes to, what the customer is asking for, the order number copied exactly, the product named in one plain word, and whether it is truly urgent. Everyone hears every rule. Everyone gets a printed copy to keep on the desk.
By Friday, the manager looks through the week's write-ups. They are all neat and in the right format, because the form on the screen only accepts the right format. But the content is another matter. One person writes "the blue ceramic mug" where the rule says "mug". Another marks every angry customer as urgent. The most experienced of the four gets almost everything right, and still, when a customer writes in only to say thank you, writes it up as a question.
The useful question for the manager is not "who is best?" It is "which rules are being dropped, and by whom?" A rule that the newest person drops but the experienced person keeps will fix itself with time. A rule that all four drop in the same way is a different problem: the briefing is not enough for it, and something else has to change.
This lesson does exactly that with four language models instead of four people. Each one gets the full house rules from lesson 2 as its instructions, and I check what it writes, field by field, on the 140 test messages. I wanted to know where a prompt alone stops being enough, because that is where training a model starts to be worth its cost.