The Effect That Shrank When I Looked Again

The previous lesson padded answers with filler and the filler lost. While I was there I ran a second treatment: the same answer with the same words, split one sentence per line. That one won.

Two numbers, ten items, neither of them significant on its own. But the gap between them looked like something: length did nothing and structure did everything. That is a clean, quotable sentence and I wanted it to be true, so I wrote twenty more items and ran it again.

The effect was still there. It was less than half the size, and the test on the new items came back undecided.

This lesson is about what to do with that, because what most people do with it is the thing that makes a p value stop meaning anything.

Here are the two numbers that started it. Both come from the ten tie items the previous lesson used. A tie item is one where the two answers make the same claim, so nothing about quality can decide the verdict.

Neither number on its own says anything. Twenty verdicts landing at 30% or at 65% is well inside what a fair coin does.

The interesting quantity is the difference between them, because the two treatments differ in a way that maps onto a real question.

Here is where the two numbers come from. Ten items, each judged twice (once with my answer in slot A and once in slot B), gives 20 verdicts per treatment. The padded answer won 6 of its 20, which is 6 ÷ 20 = 30%. The bulleted answer won 13 of its 20, which is 13 ÷ 20 = 65%. A gap of 65 − 30 = 35 points is large. But 20 verdicts is a very small sample, and a small sample can produce a large gap by luck alone. That is the question the rest of this lesson has to answer.