Who Wrote Your Test Questions?

Let me start with a picture from school. Say a teacher writes an exam by copying sentences straight from the textbook. A student who only matches words can find every answer, without understanding anything. The exam looks easy. Real life is not like that exam.

Now here is one result from a small experiment I ran. I will call the experiment "the lab". After this slide, I will explain every word.

I took 60 documents. For each document, a language model wrote two questions that the document answers. (A language model is the kind of AI that writes text, like the one behind ChatGPT.)

The first question uses the document's own words. It sounds like a test question an engineer writes after reading the page.

The second question asks about the same problem in everyday words. It sounds like a real person who does not know the technical words yet.

Then I asked three search systems to find the right document for each question. A search system is a program that takes a question and returns the documents most likely to answer it.

With questions in the document's own words, all three systems found the right document every single time. 100 percent.

With questions in everyday words, the same systems found it between 35 and 63 percent of the time.

The documents were the same. The search systems were the same. Only the questions were different.

What does that mean for your own tests? If the questions in your test were written by someone who read the documents first, your score is measuring the first kind of question. Your real users ask the second kind.