Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Foundation · LLM and GenAI Ops
The maximum number of tokens a language model can take in at once, counting the prompt, any documents and its own reply.
In short
A context window is the most tokens a language model handles in one request, counting your prompt and its reply together. Anything past that limit is not read. On one common local server, a prompt that does not fit is cut silently, with no error in the reply.
In short
A context window is the most tokens a language model handles in one request, counting your prompt and its reply together. Anything past that limit is not read. On one common local server, a prompt that does not fit is cut silently, with no error in the reply.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Context Window across a run of lessons, not one page. These 4 alone are about 198 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Measured with Ollama and qwen2.5:3b: a prompt of about 5,900 tokens sent with a 4,096-token window was cut to 2,050 tokens, the first 4 and the last 2,046, with no error in the reply. A code placed at the start was found 0 times in 5; at the end, 5 in 5. At three window sizes the kept part was exactly half the window plus 2. Without num_ctx, this laptop's window was 4,096.
Measured on a laptop with qwen2.5:3b over 30 shuffled calls: the first word came back after a median 1.0 s for a 256-token prompt and 36.6 s for about 7,900 tokens. With the laptop's own slowing taken out, writing got about 1.1 tokens a second slower for every 1,000 prompt tokens. A bigger context window made Ollama report more memory: 2.05 GB at 2,048 tokens, 2.79 GB at 32,768.
Lesson 1's sorting rules with 2,000 words of unrelated text pasted in four places: nowhere, before the rules, between the rules and the message, or after the message. On qwen2.5:3b the scores were 40, 40, 40 and 36 of 40; on llama3.2:3b 36, 38, 35 and 34. No placement differed from the no-document control by more than luck. Every prompt fitted the 8,192-token window. A document that does not fit is the real danger: in the demo, a cut prompt with the document last lost both the rules and the question.
A language model never sees letters or words. It sees tokens: small pieces of text, each with a number. This lesson shows how real tokenizers cut text, measured with OpenAI's tiktoken. This course's 1.27 million words come to 1.29 tokens per word. Numbers are cut into pieces of up to three digits. One sentence written in eight languages took 11 tokens in English but up to 100 in other languages with GPT-2's tokenizer, and 12 to 19 with GPT-4o's. That difference is paid in money and in context space.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with The Context Window: What Happens When a Prompt Does Not Fitpay once,
yours for life
A language model reads text as tokens. A token is a small piece of text, often a word or part of a word. The context window is the most tokens the model makes room for in one request. It covers everything at once: the instructions, any documents you paste in, the earlier turns of a chat, and the reply the model writes.
Think of a person holding a few sheets of paper in front of a wall of archive boxes. They can hold a few sheets, not the whole wall. Hand them more than they can hold and some pages fall to the floor. For a model, the important questions are which pages fall, and whether anyone tells you.
The window is not free. A bigger window reserves more memory when the model loads, whether you use it or not. The lesson's lab measured that on a laptop: each extra 8,192 tokens of window reserved about 0.177 GB. That cost is why servers do not simply set every window to the largest size a model supports.
The K means thousand, and the unit is tokens, not words or characters. A 32K window holds about 32,000 tokens. A 200K window holds about 200,000 tokens. That limit covers the prompt and the reply together.
How many words is that? It depends on the text and on the model's tokenizer. On this course's own English text, the lesson measured 4,500 words as about 5,894 tokens, which is roughly 1.3 tokens a word. At that rate a 32K window holds about 25,000 words, and a 200K window about 150,000 words. Code and unusual words take more tokens each, so they fit less.
So do not guess from word counts. Count tokens with the model's own tokenizer before you send. That is the only number the window cares about.
A server has only a few choices. It can refuse the request with an error. It can make the window bigger. Or it can cut the prompt down and carry on. Cutting a prompt to fit is called truncation. You cannot tell which one happened from the reply's text alone.
The lesson measured what one popular local server, Ollama, does. It built a prompt of about 5,894 tokens and sent it to a small model, qwen2.5:3b, with a window of 4,096 tokens. Ollama's setting for the window size is called num_ctx.
The reply came back normally. There was no error and no warning in it. But the count of prompt tokens the model read, a field called prompt_eval_count, was 2,050. About 3,844 tokens, roughly 65% of the prompt, were never read.
Two things are surprising here. First, the cut was silent. Second, the prompt was cut to far less than the window. The window had room for 4,096 tokens, and only about half of it was used.
To find out which part was dropped, the lab hid a made-up word, such as ZAKIMO. It sat in a sentence like "The code for box 7 is ZAKIMO." Then it asked for the code at the end. The model cannot know a made-up word, so it can only answer if that sentence was inside what it read.
With the code at the start of the long prompt, the model found it 0 times in 5. With the code at the end, 5 times in 5. Both prompts kept 2,050 tokens. With a window of 8,192, where the same prompts fit, the model got all 10 right. So it could find the code whenever it could see it.
Ollama's own server log explains the shape. It keeps the first 4 tokens, then fills the rest of the room with the end of the prompt. So the question at the end survives, and the model always has something to answer. That is exactly why the cut is easy to miss.
Most prompts put the instructions first. A support bot sends its rules, then the customer's message. A summary job sends "Summarise this", then the document. When a long prompt is cut from the start, the instructions are the first thing to go.
The reply still looks fine. The support bot still answers the customer; it just no longer follows its rules. The summary still reads like a summary; it just covers only the end of the document. Nothing in the reply tells you.
The lesson also measured how much was kept at three window sizes. A 2,048 window kept 1,026 tokens, a 4,096 window kept 2,050, and an 8,192 window kept 4,098. Each is exactly half the window plus 2. A prompt that fits is not cut at all. The lesson did not find this rule written in Ollama's documentation, so check it on your own version.
Many programs never set the window size. Then they get the server's default, which is not the largest window the model supports. Ollama's help text says the default is 4k, 32k or 256k tokens, based on the memory of the graphics processor. On the lesson's laptop it was 4,096.
So a program that sends a 6,000-token prompt without setting num_ctx got it cut to 2,050 tokens there. In that test the code sat in the middle and was cut away, and the model answered with a made-up "B7". A program that works on a big machine, where the default may be 32k, can quietly lose most of its prompt on a smaller one.
Ollama does notice the cut. It writes a "truncating input prompt" warning to its own server log. But your app does not read that log. The reply has 11 fields, and none of them says the prompt was cut. The only sign is the token count.
Other servers behave differently. Some return an error when a prompt is too long, and hosted APIs publish their own limits. The lesson tested one model on one version of Ollama, 0.32.14.
The rule is simple: never let the server decide what to drop. Decide it yourself.
A prompt that fits is still not a promise that every part is used equally well. In the lesson, one made-up word was found at every position in a fitting prompt, 40 of 40. But finding one odd word is an easy task, and harder tasks can behave differently. That gradual loss of quality in long prompts is called context rot.
The full lab, a script you can run to check your own server, and a worked example of trimming a chat are in The Context Window lesson. For the memory and speed cost of long prompts, see what a long prompt really costs.
Ollama
Sets the window per request with num_ctx. In the lesson's lab, a prompt that did not fit was cut to half the window plus 2 tokens, keeping the first 4 tokens and the end, with no error in the reply.
Ollama /api/chat
Behaves differently from one long prompt: with six long turns that did not fit, it dropped whole old turns and kept the system message, and the model answered 3 of 3 correctly.
Python
The lesson's 29-line window_check.py sends one prompt at two window sizes and flags a cut when the kept count equals num_ctx divided by 2, plus 2.
It means the model can handle about 200,000 tokens in one request, prompt and reply together. On the lesson's English course text, about 1.3 tokens made a word, so that is roughly 150,000 words of similar text. Code and rare words use more tokens.
About 32,000 tokens for the prompt and the reply together. At the lesson's measured rate of roughly 1.3 tokens a word on English prose, that is about 25,000 words. Always count tokens with the model's own tokenizer rather than guessing from words.
Yes. The window is shared by the instructions, any documents, the chat history, the new message and the reply. Leave room for the longest reply you allow when you size the window.
It depends on the server. Some return an error. In the lesson's lab, Ollama cut the prompt silently: 5,894 tokens sent to a 4,096 window became 2,050 tokens, the first 4 and the last 2,046. The reply looked normal.
Checked against these on .
What a Long Prompt Really Costs: Wait, Writing Speed and Memory
Measured on a laptop with qwen2.5:3b over 30 shuffled calls: the first word came back after a median 1.0 s for a 256-token prompt and 36.6 s for about 7,900 tokens. With the laptop's own slowing taken out, writing got about 1.1 tokens a second slower for every 1,000 prompt tokens. A bigger context window made Ollama report more memory: 2.05 GB at 2,048 tokens, 2.79 GB at 32,768.
Instructions Around a Long Document: Where to Put the Rules
Lesson 1's sorting rules with 2,000 words of unrelated text pasted in four places: nowhere, before the rules, between the rules and the message, or after the message. On qwen2.5:3b the scores were 40, 40, 40 and 36 of 40; on llama3.2:3b 36, 38, 35 and 34. No placement differed from the no-document control by more than luck. Every prompt fitted the 8,192-token window. A document that does not fit is the real danger: in the demo, a cut prompt with the document last lost both the rules and the question.
What a Token Is: How a Language Model Reads Text
A language model never sees letters or words. It sees tokens: small pieces of text, each with a number. This lesson shows how real tokenizers cut text, measured with OpenAI's tiktoken. This course's 1.27 million words come to 1.29 tokens per word. Numbers are cut into pieces of up to three digits. One sentence written in eight languages took 11 tokens in English but up to 100 in other languages with GPT-2's tokenizer, and 12 to 19 with GPT-4o's. That difference is paid in money and in context space.
The Silent Cut: How Much of Your Text an Embedding Model Really Reads
Send an embedding model a long text and it may read only the start, with no error. Measured with Ollama on this laptop: nomic-embed-text and bge-m3 both stopped at 2,048 tokens by default, although bge-m3's own page says 8,192. A sentence hidden past that point changed the score by exactly 0.000. Raising two settings let bge-m3 read 8,192 tokens, and then the hidden sentence was read but barely moved the score: 0.332 against 0.328.
MCP in Production: What the Protocol Buys and Costs
MCP turns N times M tool integrations into N plus M, but every attached server rents context-window tokens on every request. When it pays, when it does not.
Token
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
Context Rot
The drop in a language model's accuracy as its input gets longer, even when the answer is in the text and well inside the context window.
Context Engineering
Deciding what goes into a model's limited context window and in what order. Production data shows quality degrades well before the advertised token limit, so what you leave out matters as much as what you put in.
KV Cache
The stored keys and values of every token processed so far, kept so attention does not recompute the whole history each step. It grows with every token and is what limits how many conversations a GPU can serve.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "The Context Window: What Happens When a Prompt Does Not Fit" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the The Context Window: What Happens When a Prompt Does Not Fit lessoncourse 2
Take models from notebook to production, with labs on real models.