Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Intermediate · LLM and GenAI Ops
How an AI agent keeps what it needs across steps and across conversations. The model itself remembers nothing, so the app decides what to resend each step and what to save in a store for later.
In short
AI agent memory is how an agent keeps what it needs across the steps of one task and across separate conversations. The language model itself remembers nothing between calls, so the app does the remembering. Short-term memory is the recent conversation the app resends on every step. Long-term memory is facts the app saves in a store, such as a database, and searches again in a later conversation.
In short
AI agent memory is how an agent keeps what it needs across the steps of one task and across separate conversations. The language model itself remembers nothing between calls, so the app does the remembering. Short-term memory is the recent conversation the app resends on every step. Long-term memory is facts the app saves in a store, such as a database, and searches again in a later conversation.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Agent Memory across a run of lessons, not one page. These 4 alone are about 190 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Agents have no memory between calls. What the model sees is the transcript you resend every step, and what it forgets is whatever you cut to fit. How to manage both.
An AI agent is a language model in a loop, and its reliability compounds: 95 percent per step over 20 steps is only 36 percent. Why agents fail on long tasks, and the levers that fight it.
A language model does not write sentences. At every step it gives odds for every possible next token, and something else picks one. Measured with qwen2.5:3b on 24 prompts: for 12 facts the top next token had a median probability of 0.70; for 12 open prompts only 0.17. The same five words with and without the hidden chat template gave ' Paris' at 0.62 or 'The' at 0.999.
Keyword search matches words; meaning search matches embeddings. Combining the two rankings (rank fusion) was measured twice. On 470 quiz questions that reuse the lessons' own words, keyword search alone found the right lesson in the top 5 for 453, meaning search with bge-m3 for 439, and both together for 455. On lesson 4's 60 questions written in other words and six languages, keyword search found 5, meaning 27, and both 22. Where keyword search matched no word at all, combining changed nothing; where it matched words in the wrong lessons, it pulled good answers down.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
An AI agent is a program that lets a large language model (LLM), the kind of model behind ChatGPT, choose its next step in a loop: think, use a tool (call a program, such as a search or a calculator), look at the result, and repeat. See AI agent.
Here is the surprising part. The model remembers nothing between calls. Each call is like meeting a person with no memory: you must tell them everything again. The agent memory lesson puts it plainly: what the model sees is a transcript that the app resends from the top, every time.
That transcript has to fit inside the context window: the most text, counted in tokens, that the model can read in one call. A token is a small piece of a word. Long conversations, tool results and documents fill the window quickly. So the app must decide what to keep, what to shorten and what to save somewhere else. Those decisions are what people mean by agent memory.
Most agent frameworks split memory in two, and the split is about scope: how long the memory lives and who can see it.
The two solve different problems. Short-term memory is a budget problem: the conversation grows until it no longer fits. Long-term memory is a search problem: out of everything saved, which few facts belong in this step? That second problem is the same as retrieval in RAG: finding the right saved text and adding it to what the model reads.
Many articles list three types of long-term memory. They come from how psychologists describe human memory, and a 2023 research paper called CoALA (Cognitive Architectures for Language Agents) mapped them to agents. The LangGraph docs use the same three.
There is also a choice about when to write a memory. Writing in the hot path, the work done while the user waits, means the agent saves a fact while it answers. The fact is ready at once, but that reply takes longer. Writing in the background means a separate job reads finished conversations later and pulls out facts. Replies stay fast, but the memory is not there straight away.
The simplest way to keep a conversation inside the window is to trim it: send only the last few messages. It is cheap and easy, and it forgets without any warning. This short example shows it. A user mentions an allergy in the first message, then chats about a trip. The window keeps the last 8 messages.
trim_forgets.py
chat = [("user", "Hi, I am allergic to peanuts.")]
for i in range(1, 11):
chat.append(("user", f"Question {i} about my trip"))
chat.append(("assistant", f"Answer {i}"))
chat.append(("user", "Suggest a snack for the flight."))
KEEP = 8 # the window only fits the last 8 messages
seen = chat[-KEEP:] # what the model is sent on this step
print("messages so far:", len(chat), "| sent to the model:", len(seen))
print("model can see the allergy:", any("peanuts" in text for _, text in seen))Output, captured 2026-10-21
messages so far: 22 | sent to the model: 8
model can see the allergy: FalseThe fact that mattered most was in message 1 of 22, so it was the first thing cut. Summarising old turns helps, but a summary can drop exactly the detail you needed, or a safety rule. Research on long inputs adds a second problem: models use information at the start and end of the context best and miss more of what sits in the middle (Liu and others, "Lost in the Middle", 2023). Even when a fact fits, it is not always used. The context rot lesson measures this on a real agent.
The fix for facts that must last is to save them outside the window and fetch them when needed. In LangGraph you compile the graph (turn the list of steps into a program that can run) with a store. Any step can then call store.put() to save a fact and store.search() to read facts back, in any thread. This example has no LLM, so it runs the same way every time. It ran on langgraph 1.2.5.
store_remembers.py
from typing import TypedDict
from langgraph.checkpoint.memory import InMemorySaver
from langgraph.graph import END, START, StateGraph
from langgraph.store.base import BaseStore
from langgraph.store.memory import InMemoryStore
class State(TypedDict, total=False):
user_id: str
message: str
reply: str
def agent(state: State, *, store: BaseStore) -> dict:
ns = (state["user_id"], "facts") # one folder of facts per user
if "allergic" in state["message"]:
store.put(ns, "allergy", {"text": state["message"]}) # write long-term memory
return {"reply": "Noted."}
facts = [item.value["text"] for item in store.search(ns)] # read it back, any thread
return {"reply": f"I will remember: {facts}"}
g = StateGraph(State)
g.add_node("agent", agent)
g.add_edge(START, "agent")
g.add_edge("agent", END)
app = g.compile(checkpointer=InMemorySaver(), store=InMemoryStore())
app.invoke({"user_id": "u-42", "message": "I am allergic to peanuts."},
{"configurable": {"thread_id": "monday-chat"}})
out = app.invoke({"user_id": "u-42", "message": "Suggest a snack."},
{"configurable": {"thread_id": "friday-chat"}}) # a NEW conversation
print(out["reply"])
other = app.invoke({"user_id": "u-7", "message": "Suggest a snack."},
{"configurable": {"thread_id": "friday-chat-2"}})
print("another user sees:", other["reply"])Output, captured 2026-10-21
I will remember: ['I am allergic to peanuts.']
another user sees: I will remember: []
The fact was saved in the Monday thread and found in a brand new Friday thread. That is what long-term means. A different user found nothing, because the namespace includes the user ID. Getting that namespace right is a security matter, not a detail: an agent that searches the wrong folder tells one customer another customer's facts.
InMemoryStore and InMemorySaver, used here, lose everything when the program stops. In production you use a database-backed checkpointer and store, such as PostgreSQL, and usually add embeddings (number lists that capture meaning) so the store can find facts by meaning, not only by an exact name.
In a system design interview, or in a real build, these are the decisions to say out loud.
For a full design that uses all of this, read the AI agent interview guide and the customer support agent interview guide.
LangGraph
Short-term memory through checkpointers and thread IDs; long-term memory through a store with namespaces, put() and search().
MemGPT (now Letta)
A 2023 research system that manages an LLM's limited context like an operating system manages memory, moving information in and out of the window.
OpenAI
ChatGPT's memory feature saves facts about a user across chats, which the user can view and delete.
PostgreSQL
A common backing database for both the checkpointer and the long-term store in production agents.
It is how an agent keeps information across the steps of a task and across separate conversations. The model itself remembers nothing between calls, so the app resends recent messages every step (short-term memory) and saves lasting facts in a store that it searches later (long-term memory).
Short-term memory is the current conversation, saved per thread and resent to the model each step; it ends when the conversation ends. Long-term memory is facts saved outside the conversation, in a store, so a new conversation can find them again. Short-term memory is limited by the context window; long-term memory is limited by how well you can search it.
The common split is semantic memory (facts, such as a user's preferences), episodic memory (past events, such as how an earlier task went) and procedural memory (rules and instructions for doing the task). These come from human memory research and were mapped to agents in the CoALA paper; the LangGraph docs use the same three.
Because the model only sees what the app sends on each call, and that must fit the context window. When the conversation grows, older messages are trimmed or summarised, and facts in them disappear. Save facts that must last in a long-term store and fetch them when needed.
Compile the graph with a store, for example InMemoryStore while testing or a PostgreSQL-backed store in production. In a node, call store.put(namespace, key, value) to save a fact and store.search(namespace) to read facts back. Put the user ID in the namespace so one user never sees another's memories.
Checked against these on .
AI Agent
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
Context Window
The maximum number of tokens a language model can take in at once, counting the prompt, any documents and its own reply.
Context Engineering
Deciding what goes into a model's limited context window and in what order. Production data shows quality degrades well before the advertised token limit, so what you leave out matters as much as what you put in.
LangGraph
An open-source framework for building AI agents as a graph of steps that share one state. Nodes do the work, edges pick the next step, and a checkpointer saves the state so a run can pause, resume and remember.
Context Rot
The drop in a language model's accuracy as its input gets longer, even when the answer is in the text and well inside the context window.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
This page explains the idea. The "Agent Memory and Context: What the Model Sees, and What It Forgets" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Agent Memory and Context: What the Model Sees, and What It Forgets lessoncourse 2
Take models from notebook to production, with labs on real models.
Asked about Agent Memory in an interview?
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (173 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: card or PayPal. Download at once from My packs.
