You have thirty real cases and you want three hundred. The obvious move is to describe the job to a model and ask for more. Here are the kinds of question our users ask. Write two hundred like these.

What comes back is usually good. The questions are fluent, varied and on topic. The set gets ten times bigger and the pass rate barely moves.

That is worth being suspicious about. A set ten times larger reporting the same number means one of two things. Either the system is genuinely fine, or the set grew in a direction where nothing was ever going to go wrong.

There is a way to tell which, and it takes one measurement on the cases you already have.

Take the same thirty traces from the previous lesson. For each one, look at what the correct document had to beat.

The corpus behind those traces contains near-duplicates on purpose: D01n2 is a near-duplicate of D01, D02n0 and D02n1 of D02. Real corpora are full of these. The same policy in three regions, an old and a new version of one page, a summary and the document it summarises.

In 24 of the 30 traces, the document sitting directly behind the gold one, the runner-up, was a near-duplicate of the gold itself. Not an unrelated document that happened to score well. A copy of the right answer, competing with the right answer.

Remove the near-duplicates from the candidate set and look at what the correct document was actually competing against.

In 18 of the 30 traces there is nothing left at all. The top-k contained the gold document and copies of it, and no third opinion. The competition was entirely the corpus arguing with itself.

For the 12 traces that do still have a rival, the gap widens:

Now the point.

A synthetic case generator writes questions. It can paraphrase, lengthen, add typos, change the register, invent new intents. Every one of those changes the left-hand side of the problem.

It does not write your index. It cannot add a near-duplicate document. It cannot make two of your pages say the same thing in different words. It cannot create the gap between an old policy and the version that replaced it.

On this set that means a question generator has no access to the thing that made 24 of 30 cases hard. You could generate two thousand questions. The documents they are answered from would be the same documents, in the same chunks, with the same copies sitting behind them.

This is not an argument against synthetic cases. It is an argument about which axis they grow.

The code below reads the same thirty traces and finds the near-duplicates by their ids. It counts how many traces have no rival left once they are removed, and recomputes the gap for the ones that do.

Read section two before section three. Section three is a margin over the twelve traces that still have a runner-up, and it is the smaller half of the finding. Section two is the larger half: on the other eighteen there was nothing to have a margin against.

Generate against the corpus, not against the spec. The weak prompt is "write questions like these". The useful one is "here are two documents from our index that are nearly the same. Write a question only one of them answers". That produces a case whose difficulty comes from the place difficulty actually lives.

Go and find the duplicates first. You do not need a model for this. Cluster your index by embedding, or hash shingles of each document, and look at the pairs that come back closest. Every tight pair is a hard case waiting to be written.