Months on One Kind of Card
Think of a man who used to keep the accounts for a small shop. He added up bills in his head, worked out change, and sorted the post into piles for the right people. Then the shop gave him a new job. For months he did nothing but fill in the same kind of card, again and again, in exactly the house style, until he could do it half asleep.
One day someone hands him a bill and asks him to add it up. Can he still do it? Maybe he is exactly as good as before. Maybe he is a little rusty. Or maybe he answers in the card's style out of habit, short and clipped, even when the question wants something else. You would not know until you asked him, and you would not really know from one question either. You would have to ask him the same questions you used to ask him, and compare.
That is this lesson. In lesson 4 I trained two small models to do one narrow job: turn a customer message into a support ticket. The training worked for that job. Here I ask what it did to everything else. I took questions the ticket training never showed, questions the models answered before, and asked them again with the ticket training switched on.
The short answer is that the smaller model got fewer right and the larger one got the same number right. The longer answer is more interesting, because the counts are small, and when I read the replies, the drop with the most lost questions, 9 on sorting, turned out to be something other than forgetting.