Take ten retrieved documents. One of them holds the answer, and it is sitting sixth in the list. Ask the model which document answers the question, and it gets it right four times out of fourteen.

Now move the right document from the middle of the list to the end. Same ten documents, same question, same model at temperature zero. It gets it right fourteen times out of fourteen.

Nothing about the retrieval changed. The same retriever returned the same ten documents for the same question. Nothing about the model changed. The only difference is which seat the right answer sat in.

That is the lesson. The rest is how it was measured, and why the highest number in the run is the one you should trust least.

An arrangement is one ordering of the ten documents. The retrieval runs once per question, and the same ten documents are reused in all three arrangements. The gold document is pulled out and re-inserted at slot 1, at slot 6, or at slot 10. The other nine keep their order.

So a difference between the three rows cannot be a difference in the question, because it is the same question. It cannot be a difference in retrieval, because the retrieval ran once. It cannot be a difference in which documents were available, because they are the same documents.

The list of things held still is longer than the list of things that moved, and that is deliberate. A lab where two things change at once cannot tell you which one did the work.

Three lessons in this chapter have shown how far a judge moves when you nudge it. Reword the rubric and the headline shifts. Re-run the same eval unchanged and it shifts again. Tell the judge which answer is yours and a quarter of the verdicts follow.

A judge here would be a second thing that responds to position, and the two could not be separated.

So there is no judge here.

Each question is a known-item question: it was written against one specific document, and that document's id was recorded before anything ran. The model is asked which of the ten documents answers the question. Its answer is a slot. The slot's id is compared to the recorded id, as a string.

That comparison is arithmetic. It cannot be swayed by length, by tone, by which answer is listed first, or by any of the other things this chapter has spent ten lessons measuring.

Ten documents, one of them right, and the end-to-end answer rate moved by ten questions out of fourteen on nothing but the ordering.

The middle being worst is what the published work predicts. The two ends not being equal is not, and fourteen questions is not enough to claim that asymmetry is real. The next figure is the test that says so.

Every question produces three scores, not one. The three columns are the same fourteen questions three times over, which means a question that is simply hard is hard in all three columns and cancels out.

That is what makes this a paired comparison, and it is why the test below counts only the questions that changed.

A question that was right in both arrangements, or wrong in both, carries no information about which arrangement is better. Only the ones that flipped do, and an exact McNemar test counts exactly those.

Both comparisons against the end slot run entirely one way. Seven questions changed between first and last, and all seven went right only with the gold at the end. Ten changed between middle and last, and all ten did the same. Not one went the other way.

First against middle is four and one, p = 0.3750. That is not a finding.

It would be easy to write "we reproduced the U-shaped curve" and move on. The numbers even look like it. But the test on this sample cannot separate the first slot from the middle. Claiming a U here means importing somebody else's larger study and presenting it as this run's result.

What this run earned is narrower and still useful: the end slot beat both of the others, decisively, on the same questions.

One hundred and twenty-six calls: fourteen questions, three arrangements, three seeds each.

The --report flag replays the stored verdicts rather than calling the model again. So the capture above, the figures in this lesson and the playground further down all read the same run. That matters more than it sounds. Generation is not stable across runs even at temperature zero, so a number recomputed by calling the model again is a different experiment wearing the same name.

The first control is the context window, and it is the one that decides whether any of this is about position at all.

A prompt that gets silently trimmed produces a table with exactly this shape, for a completely different reason. So the window is set explicitly and the token count is recorded on every call. The longest prompt in the run was 1189 tokens against a window of 8192.

The second control is the decoder, and it did not fully hold.

Every arrangement was run three times at temperature zero with three different seeds. Greedy decoding should ignore the seed entirely, so any disagreement means the harness is looser than it claims. One arrangement out of forty-two disagreed: question q15 with the gold document in the middle, where the three seeds chose slot 6, slot 10 and slot 10.

That is one call in a hundred and twenty-six. It is small, and it is the honest size of the noise floor under every number here. It moved the middle row by one question, and the middle row is the row this lesson leans on least.

Everything so far is an accuracy table, and an accuracy table is the same three numbers whichever documents the wrong answers named. The run recorded which slot the model actually chose, and that is where the useful part is.

Forty-two selections. Three slots used. Slot 6 appears only because the gold document sat there in one of the three arrangements, and slot 10 carries twenty-nine of the forty-two.

Seven of the ten documents were never named once. Not named less often. Never.

Seventeen of the forty-two selections were wrong. All seventeen named slot 1 or slot 10.

This is a different claim from "the middle is read badly". Reading the middle badly would look like wrong answers scattered across slots 2 through 9. Not one wrong answer landed there.

The three columns with any height are the three slots the gold document was ever placed in. Everything between them is flat. Those are documents the model was shown, could read, and never once named, in any arrangement, for any question.

The denominator matters and it is easy to get wrong. A wrong answer cannot name the gold's own slot, so the space is nine slots, not ten. How many of those nine are ends depends on where the gold sat: one when the gold is at an end, two when it is in the middle.

Multiply the per-row probabilities and the chance that all seventeen land on an end is about six in a hundred trillion.

That number says the pattern is not chance. It does not say how large the effect is in your system, and seventeen observations is not a law.

Now go back to the tallest bar.

The model's habit, when it has not found the answer, is to name slot 10. It did that in fifteen of the twenty-eight arrangements where the gold document was somewhere else.

The "last" arrangement puts the gold document in slot 10.

Skill and habit produce the same score sheet here. A model that reads carefully and finds the document at slot 10 scores fourteen out of fourteen. A model that shrugs and says "slot 10" also scores fourteen out of fourteen.

Every arrangement from the run is below: the question, where the right document was put, and which slot the model named. Nothing in the data is a summary. The summary is computed from it so you can check every number in this lesson.

Change a row and the assertion at the bottom is what tells you the lesson's central claim no longer holds. That is the point of putting it there rather than in a sentence.

Reordering is what the reranking step is for. Hold the set fixed and reorder it, which is the case drawn above. Recall@k cannot move: the right document was in the set before and it is in the set after, which is all recall@k asks.

Most shipped rerankers do a little more than that. They score a larger pool and keep the best k, so they can change the set as well as the order, and then recall@k does move. That makes the problem harder to see, not easier. The retrieval metric now moves for one reason while the answer rate moves for two, and the ordering half is still unrecorded. Context precision does move, and it moves in the direction everybody calls an improvement, because the best match is now at the top.

On the model measured here, the top is the weaker end.

So a change that improves two retrieval metrics can lower the answer rate, and no retrieval metric is capable of showing you that. The only number that would have caught it is the end-to-end one, and by then the change has already been attributed to something else.

None of the four changes at the top of that list is exotic. They are the ordinary week-to-week work of a retrieval pipeline, and every one of them moves the position of the right document without touching its presence.

The last row is why this is an evals lesson rather than a prompting one. The eval recorded the question, the answer and the score. It did not record the order. So when the score moves, nothing in the log can rule the order in or out. The investigation goes somewhere else: the embeddings, the chunker, the model version, the prompt.

The effect is not new and this lesson does not claim it is. Liu et al. measured it in 2023 across several models and reported the U-shaped curve that the agents chapter already teaches.

What a curve of accuracy against position cannot show is where the wrong answers went, because a curve only plots how often the model was right. Running it yourself on a model you control gives you the other half of the run, and the other half is where the fallback is.

This is one model, one corpus, one value of k, one task and fourteen questions. A larger model may not behave this way. Longer documents almost certainly change the numbers. The distance between "the middle" and "the end" is itself a function of k, and this run holds k fixed at ten.

It also says nothing about the mechanism. Attention over long inputs, recency in the training data, and the order the documents were serialised into the prompt all predict the same table. This design cannot choose between them. That is a real limitation, not a modest one.

What the run does establish, on this system, is that the order is a variable with a large effect and no column in the eval.

The third item is the one to do first, because it is the only one that produces a measurement rather than a capability. Run your existing eval a second time with the retrieved documents in reverse order. If the score does not move, you have ruled out a whole class of explanation for every future regression. If it does move, part of the number you report to your team belongs to your ordering step rather than to your system.

Either answer is worth twenty minutes.

Ten of the fourteen questions changed answer on the seating alone. Write the order into the eval row. The next time a score moves you can rule that out with one query, instead of spending a week on the embeddings.