Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
Deciding what goes into a model's limited context window and in what order. Production data shows quality degrades well before the advertised token limit, so what you leave out matters as much as what you put in.
In short
Context engineering is deciding what text goes into a model's context window on each call, in what order, and what is left out. The model remembers nothing between calls, and it reads a long, noisy window worse than a short, clean one, so these choices set both cost and accuracy.
In short
Context engineering is deciding what text goes into a model's context window on each call, in what order, and what is left out. The model remembers nothing between calls, and it reads a long, noisy window worse than a short, clean one, so these choices set both cost and accuracy.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Context Engineering across a run of lessons, not one page. These 4 alone are about 252 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Agents have no memory between calls. What the model sees is the transcript you resend every step, and what it forgets is whatever you cut to fit. How to manage both.
A model's recall falls as its window fills, and worst from the middle. Measure the fill level where your agent starts to rot, and re-run it per model version.
Most of an LLM bill is tokens you resend, not tokens users type. Decompose the spend, then cut it with caching, slimming, output caps, and batching.
Measured on a laptop with qwen2.5:3b over 30 shuffled calls: the first word came back after a median 1.0 s for a 256-token prompt and 36.6 s for about 7,900 tokens. With the laptop's own slowing taken out, writing got about 1.1 tokens a second slower for every 1,000 prompt tokens. A bigger context window made Ollama report more memory: 2.05 GB at 2,048 tokens, 2.79 GB at 32,768.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
A language model is stateless: it carries nothing from one call to the next. The context window is the text it reads on a single call, and it has a hard limit measured in tokens (small pieces of text, roughly three quarters of a word). In a chat or an agent, your code resends everything the model needs on every call: the instructions, the tool descriptions, the history, and any documents it should use. The model does not remember the task. It rereads a transcript you hand it.
That makes two decisions yours: what goes into the window, and what you leave out. Context engineering is the name for making those decisions on purpose instead of sending everything. Anthropic describes it as designing and curating what goes into the window rather than filling it.
It differs from prompt engineering in scope. Prompt engineering is about how the instructions are written. Context engineering covers every token on the call, and in a long agent run the instructions are a small slice next to tool results and history that grow on every step.
On one call, several things share the same token budget: a system message with rules, the schema of every tool on offer, facts retrieved from a long-term store, a summary of older turns, and the recent turns kept word for word. The fixed parts at the top are often small. The recent turns, full of tool output, are usually the largest part, and they are where most cutting decisions happen.
Tool descriptions are not free either. In one measured lab on llama3.2:3b, each tool added about 57 tokens to the prompt: 5 tools made a 428-token prompt and 100 tools made 5,862. The median reply time grew from 2.7 seconds to 35.5 seconds on the same laptop, to pick the same tool. Every tool you offer is read on every call, whether it is used or not.
Long pasted documents can dominate in the same way. In a lab that pasted 2,000 unrelated words into a short sorting prompt, the document was 2,569 of 2,707 tokens, about 95 percent of the prompt. The rules and the message, the parts that decide the answer, were the other 5 percent.
A 40-turn agent loop was simulated with real tool outputs at realistic sizes, counting every character sent. The tools produced 83,783 characters in total. Keeping everything in the transcript sent 1,769,810 characters over the run, 21 times what the tools produced, and the last turn cost 16.5 times the first. Nothing ran twice. The history was simply resent on every turn.
Keeping only the last few results sent 560,638 characters (6.7 times). Summarising older turns every few steps sent 491,738 (5.9 times), and this is the optimistic case, because a real summary costs a model call of its own. Storing each full result outside the window and keeping only a short head plus an id, so the agent can fetch the rest when needed, sent 292,800 (3.5 times). These are characters, not tokens, but no fixed conversion changes the ranking.
Each option loses something. Cutting the oldest turns is cheap but silently deletes whatever fact lived there, such as an account id given once on turn two. A summary keeps the gist and can drop the detail that mattered. Keeping only relevant turns needs a relevance score that can miss a turn that mattered. Store-and-refer throws nothing away, which is why it is often the best default for large tool results.
Models read long input worse than short input, well before the advertised limit. Chroma's Context Rot study tested 18 current models on simple tasks and found every one degraded as the input grew, and degraded further when the extra text looked similar to the answer. Liu and others showed in 2023, in Lost in the Middle, that models use facts near the start and end of a long input better than facts in the middle. A fact can be in the window, paid for, and still be effectively unread.
This failure is silent. The call still succeeds, and the only symptom is a higher share of wrong answers. Some servers add a second silent failure: Ollama, when a prompt is longer than its window, keeps the first 4 tokens and the end of the prompt, drops the rest, and returns a normal reply with no error. Its default window on the tested setup was 4,096 tokens.
Order decides what survives such a cut. In one demo, a long document placed after the question was cut so that only document text remained, and qwen2.5:3b replied with a comment about the document, never having seen the rules or the question. With the document placed first and the rules and question last, both answers stayed right.
Prune tool output right after the tool returns. Keep the fields the agent will use plus an id to fetch the rest. Collapse finished sub-tasks into a short result summary, but only at a stable boundary, not in the middle of a step where the exact error text may be needed. Show the model only the tools the current step needs.
Protect what must never be lost. A rule such as "never delete without confirming" stated once in the conversation can be dropped by a summariser that sees it as chatter. Keep hard constraints in the system message or a protected block that compaction never touches. Put long documents first and the live instruction near the end, where the model reads best and where a cut is least likely to reach.
Watch the cost of editing the front of the prompt. Providers cache an unchanged prefix so it is cheap to resend. Rewriting old turns changes that prefix and loses the cache. Keep the stable parts early and unchanged, and do your trimming near the end. Finally, measure: on your own traces, check how accuracy changes as the window fills, and re-run that check when the model version changes.
Anthropic and OpenAI prompt caching
Both providers can cache an unchanged prompt prefix so that resending it costs less, which is why keeping the start of the context stable saves money.
Claude Code /compact
A command in Claude Code that replaces the conversation history with a summary to free up room in the context window.
Ollama num_ctx
The option that sets the context window size in Ollama; a prompt longer than it is cut without an error.
Chroma Context Rot report
A public study by Chroma that measured model performance falling as input length grew, on deliberately simple tasks.
Prompt engineering is how you word the instructions. Context engineering is choosing every piece of text the model sees on a call, including history, tool results and documents, and what to leave out. For a single short call the two overlap. For an agent that runs many steps, context engineering is most of the work.
You can, but accuracy usually drops well before the limit, and nothing warns you. Studies such as Chroma's Context Rot found every tested model degraded as input grew. Measure accuracy against how full the window is on your own data, and stay below the point where it starts to fall.
Context rot is the drop in a model's accuracy as its input gets longer, even when everything fits. It gets worse when the extra text looks similar to the answer, such as repeated or near-duplicate tool results. The fix is to send less and cleaner text, not a bigger window.
Cutting the oldest messages is cheapest but silently loses early facts. Summarising keeps the gist but can drop details and costs a model call. For large tool results, storing them outside the window with an id the agent can fetch often wins. Whatever you choose, keep hard rules in a protected block that is never summarised.
For long prompts, put bulk material such as documents and tool results first and the live instruction near the end. Models tend to read the end of a long input better than the middle, and if the prompt is ever cut, the end is the part that survives on servers like Ollama.
Prompt Engineering
Writing and structuring the instructions given to a language model to get reliable output. In 2026 the craft has moved from clever wording to schemas, context layout, and evaluation.
AI Agent
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
Token
The unit a language model reads and writes, roughly a few characters. Cost and context limits are counted in tokens, not words, so token accounting is how you budget an LLM system.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Agent Memory and Context: What the Model Sees, and What It Forgets" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Agent Memory and Context: What the Model Sees, and What It Forgets lessoncourse 2
Take models from notebook to production, with labs on real models.