You have an eval file. You want a half to build against and a half to check yourself with. So you write the line everyone writes.

random.shuffle(rows)
dev, test = rows[:15], rows[15:]

Rows are what the file holds. They are rarely what is independent, and this lesson is about what happens in the gap between those two facts.

The trace file every lab in this chapter reads holds thirty rows. Count what is actually in it.

Ten questions, each run three times. Thirty rows in the file, ten pieces of evidence about the system.

Three runs of one question are not three facts. If the retriever finds the right document for Q1, it will probably find it on all three runs. The cells in a column agree with each other far more than two different questions ever would.

That is the whole mechanism. Everything below follows from it.

Two ways to cut the same thirty rows.

On the left, run one of Q1 shapes your thinking and run two grades it. On the right, Q1 is wholly on one side of the line.

Nearly eight of the ten questions, on an average shuffle. In one shuffle out of ten, all of them.

Every question in the lower box has a sibling in the upper one. Whatever you learned from Q1 while building, you are about to check against Q1.

That is not a held-out set. It is the same exam with the questions shuffled.

Here is where this lesson stops agreeing with what you have probably read.

The story everyone tells about leakage is that it inflates your score: the model saw the answers, so the number comes out too high. That is true when a model can memorise. Check whether it is true here.

Seventy point zero against seventy point three. Four tenths of a point, across four thousand splits of each kind. The leakage did not inflate anything.

This is a retrieval eval, not a fine-tune. Nothing in the pipeline memorises Q1's answer between the dev half and the held-out half, so there is no mechanism for the score to be flattered. The obvious story does not apply, and the lab was written to report that either way.

The damage is somewhere else, and it is worse than a flattered number.

Four thousand splits of each kind, on the real file. The verdict at the bottom is computed, not decided in advance.

The code below is the same measurement on a smaller sample, so it runs in the page. Change TRACES to your own rows and group_of to whatever your grouping column is.

Same file, same system, same definition of a pass. The only thing that changes is where the line goes.

The leaky split moves 8.5% from one shuffle to the next. The clean one moves 15.1%.

Think about why. When the same questions sit on both sides of the line, the two halves cannot disagree with each other very much. They are looking at the same ten things. That forced agreement shows up as a number that barely moves when you re-split, and a number that barely moves is exactly what people read as trustworthy.

Both bars have the same centre. One of them is honest about what a single split of this file can tell you.

Two lessons ago you worked out that a hundred-example set cannot resolve three points. This is the same problem arriving from the other direction: the set is smaller than it looks, and the split hides how much smaller.

Understating the spread is enough to cause real damage on its own. The score never has to be wrong.

The third one is the most expensive. Think of a threshold, a reranker cutoff, a chunk size. Anything tuned on the dev half and then confirmed on a held-out half that holds the same questions will confirm itself.

This is the trap. A team who has heard that leakage inflates scores goes looking for an inflated score, does not find one, and concludes the split is fine.

The symptom to check is the spread.

Step two is the bug, and there is nothing to see. Both halves are the right size. Both contain a mix of questions. Nothing in the file records that three rows belong together unless you go and look at the column that says so.

Step four is where it gets believed, and it gets believed for a reason that sounds like good practice: the number is stable when you re-run it.

There is one question to ask before writing a split, and it is not about code.

If learning something from this row tells you about other rows, those rows are one unit. Split the units.

The last box is the one that catches careful teams. If you generated two hundred eval cases from twenty prompt templates, you have twenty units. The wording varies, the underlying thing does not, and a row-level split puts the same template on both sides.

A week after people learn this rule they start applying it to everything, so it is worth marking the edge.

Repeating is not leaking. Three runs of Q1, all of them in the held-out half, are perfectly fine. They are just worth one question rather than three, which matters for your sample size and not for your split.

One extra argument if you use scikit-learn, three lines if you would rather not. The code is not the hard part.

The library's own words are worth reading. It splits "according to a third-party provided group". That group "can be used to encode arbitrary domain-specific groupings of the samples".

Third-party means you. The library cannot know that three of your rows are one question, because nothing in the data says so. That is why the safe thing is not the default, and why this bug survives code review.

Two questions, both answerable from the file without re-running anything.

There is a third check that costs nothing and needs no knowledge of your data at all.

Re-split ten times and look at where the answers land. A held-out number that barely moves across ten different splits of a small set is telling you something, and it is not that the measurement is good.

Run the third one first. It costs one loop, and it works even when you do not yet know what your groups are. That is exactly the situation you are in when the bug is present.

Step four is the one that pays forward. Once your eval reports "30 rows, 10 units", every interval, every power calculation and every argument about set size in this chapter gets asked about the right number.

A held-out half that shares its questions with the half you tuned on will look calm and tell you very little. Calm is the symptom.