Instructions Around a Long Document: Where to Put the Rules

Imagine a colleague walks over with a very long printout. Somewhere on it is a short note: "Which department should this customer letter go to?" If the note is clipped to the very top, you read it first and then skim the rest. If it is buried halfway down, you might read two thousand words before you find out what you are meant to do. If it is at the very bottom, you read everything, and the question is the last thing on your mind when you answer.

A person copes with all three, because a person can go back and look again. A language model reads the whole printout once, in order, and then writes its answer straight after the last word it read. So it is fair to ask whether the place you put your instructions, relative to a long document, changes the answer.

This lesson measures it. I took the sorting prompt from lesson 1, pasted a long piece of text into it in four different places, and sent it with the same 40 customer messages to two small models. Then, because the answer turned out to be "not much, in this test", I show the case where placement matters a great deal: a document too long for the window.

Document. A long text pasted into a prompt. In real apps it is often a help article, a contract or a web page that the model should use. In this lab it is on purpose a text that has nothing to do with the task, so that any change in the answers comes from its length and position, not from what it says.

Placement. Where the document sits in the prompt. There are three parts that stay the same (the rules, the message, and the word "Category:" that asks for the answer) and the document moves around them.

Control. The same prompt with no document. Every placement is compared with it, message by message.

Window. The most tokens a model reads in one request, prompt and reply together. Ollama calls it num_ctx. The previous chapter's lesson 7 measured it.

Silent cut. When a prompt is longer than the window, Ollama shortens it and still returns a normal reply, with no error. The same lesson 7 showed which part it keeps.

As in lesson 1, a reply is usable when the whole reply, in lower case and without spaces at either end, is exactly the correct category word.