The Question Is Not How Many. It Is Which.
Your system produced a hundred thousand traces last month. A trace is one saved request: the question, the documents fetched, the answer, and whether it was right. You are going to read about thirty of them.
The drawers are the hundred thousand traces. The folder is the thirty you will read. Thirty out of a hundred thousand is three hundredths of one percent, so every conclusion you draw from that folder is a claim about the wall behind it.
Everyone accepts that part. Thirty read carefully beats a thousand skimmed, and nobody has time for more. The part nobody looks at is which thirty.
Both columns describe the same afternoon. The left one is the part people plan; the right one is the part that decides what the afternoon can find. "How many" is a budget question and it has an easy answer. "Which" is a method question and it has no default, so something answers it for you.
So this is a sampling decision, like any study. Decide in advance. Write it down. Report it with your findings.
Before any of the numbers below, here is what a trace in this chapter's lab actually is, because a sampling rule can only select on fields you recorded.
One request in, one answer out, and the call carries the documents that were put in front of the model. That is why a trace can record gold_in_context at all. It is also why a rule like "only two-document questions" is something you apply before reading, not a judgement you make afterwards.
Here is what this chapter's lab contains, so the numbers below have something concrete behind them. It is not a sample: it is a fixed eval set of 40 questions, run in full. That distinction matters, and this lesson is about it.
Ten wrong answers out of forty. Because the whole set was run, 25 percent is the true rate here, not an estimate of one. That is what makes this lab useful. You can watch what sampling rules do to a number you already know.
Most teams never make the decision at all. They open the trace store and read what is in front of them, and what is in front of them is almost never a random sample of anything.