Loading...
System Design and AI Engineering, by Roni Das. 976 interactive lessons.
Start LearningAI Engineering · Intermediate · LLM and GenAI Ops
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
In short
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question. It handles harder questions, but every extra step is another model call, so it costs more and takes longer.
In short
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question. It handles harder questions, but every extra step is another model call, so it costs more and takes longer.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers Agentic RAG across a run of lessons, not one page. These 4 alone are about 265 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Most RAG misses are query failures: the question is worded unlike the chunk. Rewrite, expand, HyDE, decompose, or route the query before it hits the index.
Some questions ask for all of something: which lessons mention Google? Google is in 47 of 818 lessons, and a search that returns 10 can never find more than 10. A lab on 48 real names compares top-10 search, a name index built by a small model (part of the first step of GraphRAG), and a plain text filter. For common names, search found 22 to 34 percent. The model-built index found 93 percent. The plain text filter, with no model at all, found every one.
An AI agent is a language model in a loop, and its reliability compounds: 95 percent per step over 20 steps is only 36 percent. Why agents fail on long tasks, and the levers that fight it.
The wrap of the ML and AI lifecycle chapter. I walk the loop once, in the order a team meets it, from the baseline to the answers that arrive late, and for each step give what it is for, the failure its lesson measured on the electricity data, the check that saw it or the fact that nothing did, and what the lesson could not show. Persistence 0.8484 beat every model; 20 seeds gave 0.7179 to 0.7906; a day-old cached input cost 7.04 points and looked normal; a missing seed passed the data check and failed the answers check 19 times in 19; and retraining every week clearly helped only when labels came back within a day (+10.26 and +8.96 points over the same run's first model kept). A one-page checklist, and the limits of the chapter.
all part of
AI Engineering
206 lessons · about 213 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
pay once,
yours for life
Start with plain RAG, short for retrieval-augmented generation. A RAG system searches your documents for the parts that fit a question. It puts those parts into the prompt (the text sent to the model). Then the model answers from them. The search step is called retrieval. In plain RAG it runs once, the same way, for every question: retrieve, then generate.
An AI agent is a language model in a loop. It decides an action and your code runs it. The result comes back, and the model decides again until the task is done. Agentic RAG puts retrieval inside that loop. The model decides whether to search at all. It rewrites the question into a better search and picks which source to search. It reads what came back, and searches again if the text does not answer the question yet.
So the difference is who holds the plan. In plain RAG your code fixes the steps in advance. In agentic RAG the model chooses the next retrieval step based on what it has seen so far. Anthropic's guide to building agents describes the same building block. It is a model with retrieval and tools that can write its own search queries and pick the tools it needs.
Plain RAG works well when a question is short, stands on its own, and has its answer written in one place. Our query transformation and routing lesson lists the questions where one fixed search falls short. Each one is a decision an agentic system can make:
A plain RAG pipeline can do some of this with fixed rules. It becomes agentic when a model makes these choices step by step, and can go back for more text when the first search was not enough.
The loop has a few moves. The first four are taught in the query transformation lesson; the budget that stops the loop comes from the agents lessons. Each move changes what is searched, where, or how often.
Frameworks package several of these moves. The lesson names three. LangChain's MultiQueryRetriever does rephrasings. LlamaIndex offers query decomposition in one step or many, and HyDE, which searches with a made-up answer instead of the question. Azure AI Search has query rewriting built in.
Every move in the loop is a model call on the path to the answer. That adds latency, the time the user waits. The query transformation lesson says: retrieval cannot start until the rewrite finishes. A multi-hop question adds its delay once per hop. So the lesson keeps the loop for the questions that need it, not for every question.
The cost of each step also grows. Our LLM agents in production lesson explains why. The model does not remember earlier calls, so on every step the whole transcript so far is sent again. A run that takes ten steps rereads a growing history ten times, and the later reads cost the most.
Reliability compounds too. The agent loop lesson shows the arithmetic. Say every step succeeds 95 percent of the time, and all of them must succeed. Then five steps succeed about 77 percent of the time, and twenty steps about 36 percent. The 95 percent is a teaching example, not a measured rate for any RAG system. But the pattern is real: every decision the model makes is one more chance to go wrong. Anthropic's guide says agentic systems often trade latency and cost for better results.
Use plain RAG first. If most questions stand alone and the answer sits in one place, one search and one answer is cheaper and faster, and its failures are easier to find.
Traces are logs of each step a request took. Reach for agentic steps when your traces show a specific failure: follow-ups that miss, compound questions, multi-hop questions, or questions sent to the wrong source. The query transformation lesson gives three rules that keep the cost down.
Keep the control flow in code where you can. Our deterministic scaffolding lesson argues for letting code own the steps. Call the model only for the parts that need language, such as sorting a question into a type or writing the answer. Set a step budget, a spend budget and a time budget on every run. Log the raw question next to each rewritten query. Then you can tell a bad rewrite from a missing document.
In an agentic RAG system the retrieval sources are often tools the model can call: a search over documents, a SQL query, a search over another index. MCP, the Model Context Protocol, is one standard way to offer such tools to an AI app. So a search tool on an MCP server can be one of the sources the loop picks from.
MCP does not decide what is relevant and does not build the prompt; it only connects the app to the tool. The full comparison is on RAG vs MCP, and the protocol itself is in MCP explained.
LangChain MultiQueryRetriever
Generates several rephrasings of a question with a model, retrieves for each, and returns the merged, de-duplicated results. Named in our query transformation lesson.
LlamaIndex query transformations
Documents HyDE and single-step and multi-step query decomposition. In the multi-step form, follow-up questions are asked against the index based on earlier answers.
Azure AI Search query rewriting
Microsoft's docs: the search service sends the query to a generative model that writes alternative queries, then uses the original and the rewritten queries to retrieve results. The feature is marked preview.
It is RAG where a model decides how to search, step by step. Plain RAG runs one search and then answers. Agentic RAG lets the model rewrite the question and choose where to search. It can search more than once. It checks whether the text it found answers the question before it writes the answer.
In RAG, your code runs a fixed retrieve-then-generate pass for every question. In agentic RAG, a model runs a loop and chooses each retrieval step from what it has seen so far. Agentic RAG copes better with follow-ups, compound and multi-hop questions, but it makes more model calls, so it is slower and costs more.
No. For questions that stand alone and have their answer in one place, plain RAG is cheaper and faster. Each agentic step is another model call, the delay stacks per step, and every model decision is one more chance to go wrong. Use agentic steps only where your traces show plain retrieval failing.
Agentic RAG is an AI agent whose main tools are searches. MCP, the Model Context Protocol, is one standard way to offer those search tools to an AI app. So an MCP server can be one of the sources an agentic RAG loop searches. MCP connects the tools; it does not decide what to retrieve.
Give every run hard budgets: a maximum number of steps, a maximum spend and a maximum time, checked before each model call. When one runs out, the run stops and says what it could not find. Our LLM agents in production lesson calls the part that enforces this the governor.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
AI Agent
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
MCP (Model Context Protocol)
An open protocol that lets AI apps such as Claude, ChatGPT, Cursor and VS Code use outside tools and data. An MCP server offers the tools once, and any app with an MCP client can find and call them.
Tool Calling
When a model emits a structured request to run a function, and an orchestrator executes it and feeds the result back. The model never runs anything itself, so the orchestrator validates every call.
Reranking
A second retrieval stage where a cross-encoder scores each candidate chunk against the query directly. Slower than the first-stage retriever but far more precise, so it runs only over the top few dozen results.
ReAct Agent
An AI agent that alternates reasoning and acting: it writes a thought, calls a tool, reads the result, and repeats until it can answer. ReAct is short for reason plus act.
This page explains the idea. The "Query Transformation and Routing: Fixing Retrieval Before It Runs" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 206 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Query Transformation and Routing: Fixing Retrieval Before It Runs lessoncourse 2
Take models from notebook to production, with labs on real models.