Loading...
System Design and AI Engineering, by Roni Das. 980 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
A language model placed in a loop: it decides an action, calls a tool, reads the result, and repeats until done. The hard part is that per-step errors compound over many steps.
In short
An AI agent is a language model placed in a loop: it picks one action, your code runs it as a tool, the result is fed back, and it picks again until the goal is met or a budget runs out. Errors multiply across steps, so a 95 percent step gives only about 36 percent success over 20 steps.
In short
An AI agent is a language model placed in a loop: it picks one action, your code runs it as a tool, the result is fed back, and it picks again until the goal is met or a budget runs out. Errors multiply across steps, so a 95 percent step gives only about 36 percent success over 20 steps.
Learn it in the course
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers AI Agent across a run of lessons, not one page. These 4 alone are about 267 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
An AI agent is a language model in a loop, and its reliability compounds: 95 percent per step over 20 steps is only 36 percent. Why agents fail on long tasks, and the levers that fight it.
One end-to-end score tells you something broke, not what. Split RAG and agent quality into retrieval, generation, tool, and trajectory evals that localize failures and score the path, not just the answer.
The operator's view of running LLM agents: deployment topology, budgets and governors, tracing, canary rollout, human escalation, and the golden signals you page on.
When an agent acts, whose authority does it use? Give agents scoped, short-lived, delegated identity with OAuth token exchange, audience-bound tokens, and workload identity instead of one broad ambient key.
all part of
AI Engineering
210 lessons · about 219 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with The Agent Loop: Why AI Agents Fail as Tasks Get Longerpay once,
yours for life
An AI agent is a language model put in a loop. The model looks at a goal and the history so far, and it proposes one action. An orchestrator, which is ordinary code you write, runs that action, takes the result, and hands it back. The model looks again and proposes the next action. This repeats until the model says the goal is met, or until a limit stops it.
A few words carry the rest of the idea. A tool is a function the agent may ask for, such as a database query, a web search or sending an email. A step is one full turn: one action proposed, run, and its result returned. A side effect is a change in the world that stays after the call, like a charge or a sent email. A budget is a hard limit on steps, tokens or seconds.
The key point is who does what. The model never runs anything. It only writes text that proposes an action. The orchestrator executes, and it owns the loop, the budget and the stop condition. That gives you seams you can control: how many steps are allowed, which tools exist, what happens on an error, and whether a proposed action may run at all.
For a task to succeed, every step has to succeed. If each step works 95 percent of the time and steps fail independently, the whole task succeeds 0.95 multiplied by itself once per step. For 5 steps that is about 77 percent. For 10 steps about 60 percent. For 20 steps about 36 percent. The same agent that looked great on a 3-step demo, about 86 percent, fails most 20-step tasks.
Raising each step helps a lot. At 99 percent per step, 20 steps succeed about 82 percent of the time. At 99.9 percent, about 98 percent. A quick check for small failure rates: the task fails about n times f of the time, where n is the number of steps and f the failure rate per step, as long as n times f stays well under 1.
These numbers assume each step fails independently and at the same rate. Real agents break that assumption, usually in the bad direction: the history grows every step, so later steps have more to track and fail more often. Treat the formula as a way to reason about length, and measure your own agent on tasks as long as your users bring.
Sort failures by cause before fixing them. A transient tool error, such as a rate limit or a timeout, is not the model's fault, and a retry fixes it. The model's own failures are different: invalid JSON, the wrong tool, an invented order number, or a rule from fifteen steps ago that it forgot. Those need validation, clearer tool schemas, checks, or a shorter task. Retrying a permanent error only burns budget.
Retries have a limit. In a worked example where a step works 90 percent of the time and 70 percent of its failures are transient, one retry lifts the step to 96.3 percent and a 20-step task from about 12 percent to about 47 percent. Two or three retries barely move it further, because the permanent failures set a ceiling near 96.8 percent. Those two input numbers are a teaching example, not a measurement.
A retry also replays side effects. If the first attempt charged a card and only the reply was lost, a naive retry charges it again. The fix is an idempotency key: one stable id per logical action, sent on every attempt, so the service runs the action at most once. Stripe accepts such a key on its API requests for this reason.
The model is stateless, so the orchestrator resends the whole history every step. In a simulated 40-turn run with real tool outputs, keeping every result in the history sent 21 times as much text as the tools produced, and turn 40 cost 16.5 times turn 1. Cost grows faster than the task.
A long, noisy history also makes each step less reliable. Models read long input worse, and facts in the middle of a long input worse still. That feeds back into the compounding math. A per-step rate that falls from 0.97 to 0.93 because the history is cluttered takes a 20-step task from about 54 percent to about 23 percent. You did not add steps. You fed each step worse input.
So managing what the agent sees is a reliability lever, not only a cost lever. Prune large tool results, keep only the tools the current step needs, and keep hard rules in a protected place that summarising never removes.
An agent fits when the next step truly depends on what earlier steps found, for example a support case where the second lookup depends on the first, or research where you cannot list the searches ahead of time. A fixed program cannot know that path, and a model that picks the next step is useful.
An agent is a poor fit when you already know the steps. If every run is fetch the order, check the refund rule, write the reply, write those steps as normal code and call the model only for the part that needs language. That fixed pipeline, often called a workflow, has fewer places to go wrong and is easier to test.
The cheapest reliability win is usually fewer steps, not a smarter model. Split long goals into short stages with a verified checkpoint between them, validate every tool call before it runs, set step, token and time budgets, and put irreversible actions such as moving money or deleting data behind a check or a human approval.
LangGraph
An open-source library from the LangChain team for building agents as graphs of steps with saved state between them.
OpenAI Agents SDK
OpenAI's open-source library for building agents that call tools in a loop and hand work between agents.
Claude Agent SDK
Anthropic's library for building agents on the same loop and tool system that Claude Code uses.
Stripe idempotency keys
Stripe's API accepts an idempotency key header so a retried request does not create a second charge.
A chatbot answers once and a person reads the answer. An agent runs a loop: it proposes actions, your code runs them as tools, and the results feed the next decision. Because the output of one step becomes the input of the next, one wrong action can send the whole run off course.
Because step success rates multiply. At 95 percent per step, a 20-step task succeeds only about 36 percent of the time, assuming independent failures. Real agents often do worse, because the growing history makes later steps harder.
In a workflow, your code fixes the order of steps and the model fills in only the parts that need language. In an agent, the model chooses the next step. Use a workflow when you know the steps, and an agent only where the path truly cannot be known in advance.
It helps a little with the model's own errors, but it is the most expensive lever and does nothing for transient tool failures. Shorter tasks, validated tool calls, classified retries with idempotency keys, and checkpoints between stages usually move reliability more for less money.
No. The model only writes text proposing a tool call. Your orchestrator reads it, checks it, runs the real function, and feeds the result back. That is why every call can be validated and stopped before it touches anything real.
Telegraf
InfluxData's plugin-driven metrics agent, collect metrics from 300+ sources with a single binary
Design an AI Agent Platform
Design a production AI agent orchestration platform - MCP, A2A protocol, tool calling, memory systems, multi-agent workflows, cost management, and safety guardrails
Evaluating RAG and Agents: One Score Detects, Components Localize
One end-to-end score tells you something broke, not what. Split RAG and agent quality into retrieval, generation, tool, and trajectory evals that localize failures and score the path, not just the answer.
LLM Agents in Production: The Model in a Loop
The operator's view of running LLM agents: deployment topology, budgets and governors, tracing, canary rollout, human escalation, and the golden signals you page on
Agent Identity and Authorization: Whose Authority Is It?
When an agent acts, whose authority does it use? Give agents scoped, short-lived, delegated identity with OAuth token exchange, audience-bound tokens, and workload identity instead of one broad ambient key.
Tool Calling
When a model emits a structured request to run a function, and an orchestrator executes it and feeds the result back. The model never runs anything itself, so the orchestrator validates every call.
MCP (Model Context Protocol)
An open protocol that lets AI apps such as Claude, ChatGPT, Cursor and VS Code use outside tools and data. An MCP server offers the tools once, and any app with an MCP client can find and call them.
Context Engineering
Deciding what goes into a model's limited context window and in what order. Production data shows quality degrades well before the advertised token limit, so what you leave out matters as much as what you put in.
Prompt Injection
An attack where untrusted text, often returned by a tool, carries an instruction the model obeys, because the model cannot reliably tell instructions from data. The top security risk for agents, and unsolved by prompting alone.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Agentic RAG
Agentic RAG is RAG where a language model runs the search in a loop. It decides whether to search, what to search for and where. Then it checks if the text it found answers the question.
This page explains the idea. The "The Agent Loop: Why AI Agents Fail as Tasks Get Longer" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 210 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the The Agent Loop: Why AI Agents Fail as Tasks Get Longer lessoncourse 2
Take models from notebook to production, with labs on real models.