The Card Index Before the Training Course
Imagine you start a new job answering letters from customers. Each letter has to go into one of four trays. On your first morning, your manager offers you two ways to learn the job. The first is a training course: two weeks of study, after which you should be able to sort any letter on your own. The second is much simpler. Next to your desk there is a drawer full of cards. Each card holds an old letter and the tray it went into. When a new letter arrives, you find the card with the most similar old letter and put the new letter in the same tray.
The training course sounds more serious. It costs time and money, and at the end you have learned something. The card drawer sounds almost like cheating. But before paying for the course, a sensible manager would ask one question: how well does the drawer already work? If the drawer gets nearly every letter right, the course has very little left to improve. If the drawer fails often, the course has a real job to do.
This chapter is about the training course. In machine learning it is called , and it is often the first thing people reach for when a model does not do what they want. This first lesson does something less exciting on purpose. I trained a small model on labelled examples, and then I tried the card drawer too, on exactly the same test. The drawer did as well as the course. The rest of this lesson shows how I measured that, what it means, and why it changes which task the rest of the chapter uses.