The Number on the Leaderboard Is About Someone Else's Documents
You are about to pick an model. You open a leaderboard. One model scores 64.2. Another scores 62.8. The first one costs money. The second one runs on your laptop for free. The gap is small, but it looks real. So you pick the paid one, because a bigger number feels like a safer choice.
Here is the problem. That 64.2 was measured on a pile of documents that is not yours. It was measured with questions your users did not ask. The number is real and the benchmark is honest. It still may tell you nothing about what will happen when you point that model at your own support tickets.
This lesson does not argue that from theory. It runs it. Every number below was measured on one laptop and one AWS account, with two real models. The scripts are in the lab folder, so you can run them and get your own numbers.
Three findings come out of it:
- On an easy set of documents, a free local model and a paid cloud model scored exactly the same. The test could not tell them apart.
- Growing the pile from 15 documents to 2,015 cost some accuracy, but not much.
- Adding just 30 documents of one particular kind cost more than all 2,000 of them.
Those 30 documents are the lesson. By the end you will know what kind they were, and how to write your own set in an afternoon. That set will tell you more about which model to buy than any leaderboard can.
First, two words you will need.
A corpus is just the set of documents you search. Ours starts at 15.
Recall@1 answers one question: how often did the correct document come back first? If you ask 10 questions and the right document is first for 7 of them, recall@1 is 0.70. Recall@3 asks whether the right document was anywhere in the top three.
Look at the two kinds of bar. On the left, the pile grows by 2,000 documents and recall@1 falls by 20 points. On the right, the pile grows by only 30 documents and recall@1 falls by 30 points. Per document added, those 30 did about a hundred times more damage.
The model is the same in every bar. The questions are the same in every bar. Only the other documents changed.
That is what people mean when they say benchmarks do not transfer. A retrieval score is not a property of a model on its own. It comes from three things together: the model, your documents, and your questions. A leaderboard only controls the first one.