Two Hundred More Cases, None of Them Hard
You have thirty real cases and you want three hundred. The obvious move is to describe the job to a model and ask for more. Here are the kinds of question our users ask. Write two hundred like these.
What comes back is usually good. The questions are fluent, varied and on topic. The set gets ten times bigger and the pass rate barely moves.
That is worth being suspicious about. A set ten times larger reporting the same number means one of two things. Either the system is genuinely fine, or the set grew in a direction where nothing was ever going to go wrong.
There is a way to tell which, and it takes one measurement on the cases you already have.
Why do people generate cases at all? Because real cases are slow to collect. Each one needs a real question, a real answer and a person who checks what the right answer was. Thirty real cases can take days. A model can write two hundred questions in minutes, and they read well. So generation is a sensible idea, and this lesson is not against it. The question is narrower: when you generate cases, which part of the problem gets bigger, and is that the part where your system actually fails?
To answer it I use the same thirty real traces as the previous lesson. I do not generate anything. Instead I look at where the difficulty in the real cases came from. If it came from the questions, a question generator can make more of it. If it came from somewhere else, no number of generated questions will reach it.