Every eval conversation starts the same way. "We run thirty cases."

That number then goes into every interval, every power calculation and every argument about whether a change is real.

Thirty is the length of a list. Whether it is also the amount of evidence depends on what you are measuring. This lesson is one file where the two answers are very far apart.

The trace file this chapter uses was built by running ten questions three times each. Look at what came back.

Not similar. Identical, to three decimal places, on all ten questions.

That should not be surprising once it is said out loud. Retrieval is a function of the query and the index. Neither changed between the runs, so neither did the result. Runs two and three cost money and told you nothing new about whether retrieval works.

Ten squares of evidence, twenty squares of duplication.

Nothing about how those traces were collected was a mistake. Running three times was the right call, for a reason the next slide gets to. The mistake is only in counting them three times afterwards.

Now compute a different metric on exactly the same thirty rows.

Same question. Same retrieved context, because retrieval was deterministic. Different text.

Five questions wrote something new every time. Three repeated themselves word for word.

Note the "up to". Two answers that differ by a word are two strings, and whether they are two observations depends on your grader. An exact-match check sees two. A judge reading for meaning may well see one. Twenty-two is the ceiling, not the answer.

The whole measurement is counting distinct things. No model is called.

Read the last block. The same measured 70%, with the interval computed at the row count and at the honest count, and a computed verdict rather than one written in advance.

The code below does the same counting on rows you can paste in.

Take the measured retrieval pass rate from earlier in this chapter, 70%, and compute its interval twice.

Plus or minus 16.4 points, or plus or minus 28.4. Nothing about the measurement changed. Only the count of how many things were measured.

The honest interval runs from about 42% to about 98%. That is an uncomfortable range, and it is the correct one for a retrieval metric computed from ten distinct results.

Three numbers earlier in this chapter used thirty. All three were computed correctly. All three were computed against the wrong number.

None of this is new, and it is not specific to language models.

Survey statisticians have divided by a factor for this since long before anyone was writing evals. Two people in the same household are not two independent opinions. Three runs of one query are not three independent retrievals.

The formula is a ratio of variances and the output is called the effective sample size. You do not need the formula to use the idea. Counting distinct inputs, as the lab does, gets you the same place for the deterministic case.

The honest position is not that this file's n is ten. It is that the file does not have an n at all until you say what you are measuring.

Read the bottom two carefully. Run-to-run variation is a question about the repeats, so every row is evidence. Cost per query is measured per call, so every call counts.

Those two metrics are not being cheated by the repeats. They are the reason the repeats exist.

Two questions, and the second one is worth writing into your eval code rather than remembering. Retrieval is deterministic today. It stops being so the first time somebody adds a tie-break or turns on a reranker with sampling.

There is a wrong conclusion available here and it is worth blocking.

Keep the repeats. They are cheap, and a failure that shows up one time in five will never appear in a single-run eval.

This lesson is about arithmetic, not about running fewer things. The fix costs one column on a results table and no compute at all.

sort_keys=True is not decoration. Two dictionaries with the same contents in a different order are the same retrieval result, and without it you would count them as two.

Print both numbers. A reader who disagrees with your count will now say so, which is the outcome you want.

The power table three lessons ago is still correct. The question is which row you read off it, and a set of ten reads a very different row from a set of thirty.

Step one is the only one that needs thought. The other two are a line each.

If you remember one picture from this lesson, make it that one. Ten filled squares and twenty outlines, and the file calls all thirty of them cases.

One file, two metrics, two sample sizes. The file will not tell you which one you are using, so the row has to.