systemdesign.academyAI EngineeringFree lessonsSign inGet the course
  1. Home
  2. AI Engineering: Data, RAG and Agents

Where this shows up in interviews

This idea carries a full system design question on its own. Each walks through the full answer.

  • →Design a Multi-Agent System
Previous lessonTool Calling Is a Contract, and the Model Will Break ItNext lessonAgent Memory and Context: What the Model Sees, and What It Forgets

System Design

  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Pricing
  • Privacy
  • Terms
  • Refunds
  • Delivery
  • Contact
  • Agents in Production
  • Durable Execution: An Agent Run Is a Workflow, Not a Request

    • AI Engineering · Agents In Production

    Durable Execution: An Agent Run Is a Workflow, Not a Request

    Free preview: the first part of this 15-part lesson.

    Already bought the course? Sign in to read the whole lesson.

    this lesson is part of

    AI Engineering

    Unlock this lesson and the 199 other paid lessons

    • 210 lessons in 16 chapters
    • 118 lessons built on a lab that really ran
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the AI Engineering course

    The rest of this lesson

    1. The Words You Need First
    2. Why a Request and a Run Need Different Recovery
    3. What Durable Execution Actually Is
    4. The Mistake: You Cannot Re-Run a Model Call to Recover
    5. Record on the Way Down, Replay on the Way Back
    6. Run the Record-and-Replay Loop Yourself
    7. The Replay Code, Traced by Hand
    8. Suspend, Retry, Resume: The States a Run Can Sit In
    9. Retries Are Dangerous Without Idempotency Keys
    10. The Small Schema That Holds It All Up
    11. What Durable Execution Costs You
    12. Common Mistakes
    13. How This Looks in Real Systems
    14. Knowledge Check

    read the rest of this lesson in

    AI Engineering

    Unlock this lesson and the 199 other paid lessons

    • 210 lessons in 16 chapters
    • 118 lessons built on a lab that really ran
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the AI Engineering course

    Continue this lesson

    in Agents In Production

    This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

    One-time payment. Lifetime access. No subscription.
    Global
    $49/lifetime

    One-time payment via PayPal

    Get Lifetime Access
    Foundation lessons free, try before you buy
    India Special
    India
    ₹999/lifetime

    One-time payment: UPI, card or netbanking

    Get Lifetime Access
    Foundation lessons free, try before you buy
    Every ML Platform lesson, Foundation to Advanced
    Real architecture diagrams with real tool logos
    MLOps, LLMOps, and data engineering in depth
    Quizzes with instant feedback
    Built and reviewed by senior engineers
    Lifetime access, all future content included

    Foundation lessons of this course are free, no payment needed. Try them now

    What Happens When Your Agent Crashes at Step 40?

    Picture a run that is 40 steps deep. It has read three files, called an internal API twice, drafted an email and sent it, and kicked off a refund. On step 41 the process dies. A deploy rolls the pod, or the node is reclaimed, or the code hits an unhandled exception. The run is gone from memory.

    Now what. If you restart the run from the top, you re-read the files (harmless), re-call the API (maybe harmless), send the email again (not harmless), and issue the refund a second time (expensive, and now a support ticket). If you drop the run entirely, the user's task is half done with no record of where it stopped. Neither option is acceptable, and the reason is that an agent run is not a request. It is a long-lived process that changes the outside world as it goes.

    The thing to notice before we go further is how ordinary the causes of that crash are. None of them require a bug in your agent. A run that lasts minutes or hours will sit through a routine deploy, a spot-instance reclaim, and a transient tool blip as a matter of course, and any one of them ends the process. Sort those causes and you can see what a recovery mechanism actually has to survive.

    Failure-mode taxonomy for a durable agent run titled what actually kills a run at step 41, a five-row table listing a deploy rolling the pod as a planned infra event, a node reclaimed as an unplanned infra event, an unhandled exception as a code crash, a transient tool error such as a 429 or 503 as an environment blip, and waiting for a human approval as a by-design pause, each paired with how it shows up and what durable execution does about it, from a new worker resuming from the last recorded step to the run suspending and resuming when the approval lands

    The first four rows are why crashes have to be survivable. The fifth is why a long task has to be affordable to leave running. This is the problem durable execution solves. It is the fourth lever from the first lesson in this chapter, recoverable steps, made concrete. The other three shorten the chain, raise per-step reliability, and add verification so failures are rarer. This one makes the failures that still happen survivable.