Thirty Is a Property of the File
Every eval conversation starts the same way. "We run thirty cases."
That number then goes into every interval, every power calculation and every argument about whether a change is real.
Thirty is the length of a list. Whether it is also the amount of evidence depends on what you are measuring. This lesson is one file where the two answers are very far apart.
Why does this matter so much? Because the size of a sample decides how sure you are allowed to be. With a small sample, a pass rate of 70% could easily come out much lower or much higher on a fresh set of cases. With a large one, it is pinned down tightly. You will see exactly how wide the range is for this file, worked out by hand, later in the lesson. Every tool this chapter has used to express that, the confidence interval, the power calculation, the argument about whether a two-point change is real, takes the sample size as an input. Give it a number that is too big, and every one of those tools gives you an answer that is too confident, while looking perfectly correct.
The previous lesson found that this file's rows come in groups of three, and that splitting them as if they were independent hides half the spread. This lesson asks the next question: if the rows are not independent, how many independent observations does the file really hold? The answer I measured is that it depends on the metric. For a retrieval metric, 10. For a generation metric, up to 22. Same thirty rows.