What If the Prompt Is Too Big?
Lesson 5 showed that a model makes room for a fixed number of tokens, its context window, and that a bigger window reserves more memory. This lesson asks the question that matters most in practice: what happens when you send more than fits?
There are only a few things a server could do. It could refuse the request with an error. It could make the window bigger. Or it could cut the prompt down and carry on. Each choice has a very different effect on your app, and you cannot tell which one happened from the reply's text alone.
Think of the person in the picture. They can hold a few sheets, not the whole wall of boxes. If you hand them more than they can hold, some pages fall to the floor. The important questions are which pages fall, and whether anyone tells you.
This lesson measures both with Ollama on my laptop. It finds that Ollama cuts the prompt without an error, that it keeps the end and drops the start, and that it keeps much less than the window could hold. Then it shows how to check for this in your own code.
Words You Need First
Context window. The most tokens a model handles in one request, the prompt and the reply together. Lesson 5 measured what it costs in memory.
num_ctx. Ollama's name for the window size. You can send it with each request, in the options of the call.
Truncation. Cutting a prompt down so that it fits. The word comes from the Latin for "cut off".
Default window. The window Ollama uses when a request does not say. It is not the largest window the model supports.
The code. To test what the model can still see, the lab hides a made-up six-letter word, such as ZAKIMO, in a sentence like "The code for box 7 is ZAKIMO." and asks for it at the end. The model cannot know the word from its training, so it can only answer correctly if that sentence is inside what it was given.