Reading the Rule Book, or Knowing the Job
Think of two people on a customer service desk. The first one started today. She has a thick printed rule book, and before she writes up each customer's problem she opens it and reads the whole thing again: which team gets which problem, how to write the product name, what counts as urgent. She is careful, and she follows most of it. But reading the full book for every single customer is slow, and some rules still slip past her, because a rule you have only read is not the same as a habit.
The second person has been on the desk for months. She has written up hundreds of these problems, and her manager corrected the ones she got wrong. Her rule book is on the shelf behind her. She does not need it, because the rules are now in how she works. She will still make mistakes. They will just be different mistakes from the new starter's.
The last lesson was the first person. Four models were handed the full house rules with every message, and I checked which rules they kept and which they dropped. This lesson is the second person. I took three small models, showed each of them the 500 training examples from lesson 2 three times, and then gave them only one short line of instructions. The rules were never shown to them in words. They had to learn the rules from the examples.
Then I asked the same question as the manager: is the experienced one better? The answer turned out to be more interesting than yes or no. The smallest trained model scored exactly the same as the biggest prompted one, and they did not get the same tickets right.