The Line Everybody Writes Without Thinking
You have an eval file. You want a half to build against and a half to check yourself with. So you write the line everyone writes.
random.shuffle(rows)
dev, test = rows[:15], rows[15:]
Rows are what the file holds. They are rarely what is independent, and this lesson is about what happens in the gap between those two facts.
Why split an eval set at all? Because you are going to change things while you look at it. You try a new chunk size, you look at the failures, you adjust a prompt, you look again. After a few rounds your choices are fitted to the cases you have been looking at. A system can look good on those cases simply because you tuned it on them. So you keep some cases back, and you check against them only at the end. If the system still does well on cases you never tuned against, you have real evidence that it works on new questions, and not only on the ones you studied.
That only works if the kept-back cases are really new. This lesson measures what happens on this chapter's own trace file when they are not. The result surprised me. The usual warning about leakage is that it makes your score too high. On this file it did not. It did something else, which I think is worse because nobody checks for it.