The New Starter Who Learned the Shop by Heart
Imagine a new person starts at the help desk of a home shop. On the first day, the manager gives them a printed sheet of shop facts: how long returns take, what delivery costs, where the showroom is. There are two ways the new starter can use that sheet. They can keep it on the desk and read from it when a customer asks. Or they can learn it by heart over the weekend and put the sheet away.
Both can work. The one who learned it by heart answers quickly, and may even answer better when a customer asks in a strange way, because the facts are now part of how they think. But learning by heart has two risks. Under pressure, they may mix two facts up, and say the price-match window when the customer asked about returns. And when a customer asks something that was never on the sheet, a person who has only ever practised giving answers may simply give one, instead of saying "let me check".
This lesson runs that same experiment on two small language models. I wrote 30 facts about a shop that does not exist. I gave them to the models in the two ways: pasted into the instructions the model reads, or trained into the model. Then I asked questions in different words from the training questions, and questions the facts do not answer.
The chapter plan had a title ready for this lesson: " does not add facts reliably". That is not what the data showed, so I changed it. Training did teach the facts, and in one case scored higher than the pasted sheet, not beyond luck. And asked something no fact answers, no set-up, trained or not, made these small models reliably say "I don't know".