82% Against 79%: What a Hundred-Case Eval Can Actually Detect

You change the prompt. You run the eval. The old version scored 79%, the new one scores 82%.

Three points better. You ship it.

That is the most common decision in applied AI work, and on a hundred-example set it is close to a coin toss. The eval is not badly built. The model is not being difficult. A hundred examples simply cannot resolve three points.

By the end you will know what a set of your size can see. You will know which changes are worth measuring at all, and what to write on the results table instead of a bare percentage.

One thing to keep in mind from here on. Every detection rate below assumes each version was scored on its own separate sample. That is the least favourable case, and most teams do something better without knowing it. We come back to that near the end.

Why does this matter so much? Because the decision to ship is where evals earn their cost. If the eval cannot tell a better version from a worse one, every decision made on it is partly luck. Worse, it feels like evidence, so nobody goes looking for a better way. This lesson gives you the arithmetic to know, before you run anything, which differences your set can see and which it cannot.

This lesson borrows a few words from statistics. Each one has a plain meaning, and here they are before any numbers.

Eval set. The fixed list of test cases you run your system on. Each case is scored pass or fail. The set size, written n, is how many cases it holds.

Pass rate, or score. The share of cases that passed. 80 passes out of 100 is an 80 percent score.

Sample. Your eval set is a small selection from all the questions your users could ask. A different selection of the same size would give a slightly different score. That difference is not a bug. It is what "sample" means.

Point estimate. The single number you measured, such as 80 percent. It is your best guess, not the truth.

Standard error. How much the score would typically move if you drew a fresh sample of the same size. It gets smaller as the set gets bigger.

95 percent interval. The point estimate plus and minus about two standard errors (1.96 of them, to be exact). It is the range of true values that are reasonably consistent with what you measured.

Significant. A difference is called significant when it is large compared with its standard error, large enough that luck alone would rarely produce it. The usual cut-off, and the one used here, is the 5 percent level.

p-value. How often luck alone would produce a difference at least this large, if the two versions were really equal. A small p-value, below 0.05 here, is what "significant" means.

Real gain, or effect size. How much better the new version truly is, in points. In the lab this is set on purpose, so that we know the right answer and can check whether the eval finds it.

Power, or detection rate. If there really is a gain of a given size, the share of eval runs that would find it significant, in the right direction. Power is a property of your set size and the gain, not of one run.

Paired and unpaired. Unpaired means each version was scored on its own separate sample. Paired means both versions were scored on the same cases, so you can compare them case by case.

Discordance. In a paired comparison, the share of cases where the two versions disagree: one passes and the other fails.

Those twelve words are enough for the whole lesson.