systemdesign.academyAI EngineeringFree lessonsSign inGet the course
  1. Home
  2. AI Engineering: Data, RAG and Agents
  3. Agents in Production
  4. Deterministic Scaffolding: The LLM Explains, It Does Not Arbitrate

AI Engineering · Agents In Production

Deterministic Scaffolding: The LLM Explains, It Does Not Arbitrate

Free preview: the first part of this 16-part lesson.

Already bought the course? Sign in to read the whole lesson.

this lesson is part of

AI Engineering

Unlock this lesson and the 199 other paid lessons

  • 210 lessons in 16 chapters
  • 118 lessons built on a lab that really ran
  • An AI tutor beside every lesson that answers and draws
  • Lifetime access, every new lesson included

$49 once, by PayPal

See the AI Engineering course

The rest of this lesson

  1. Two Ways to Wire the Same Agent
  2. Every Decision Splits Into Two Questions
  3. The State Machine Owns the Workflow. The Model Fills Slots.
  4. What Belongs to the Model, What Belongs to Code
  5. Why Shrink the Model's Surface at All
  6. The Reliability Math of Deciding Less
  7. Identity and Amounts Come From State, Not From the Model
  8. A Support Workflow as a State Machine
  9. The Exchange, Step by Step
  10. Run the Harness: Classify, Validate, Contain

read the rest of this lesson in

AI Engineering

Unlock this lesson and the 199 other paid lessons

  • 210 lessons in 16 chapters
  • 118 lessons built on a lab that really ran
  • An AI tutor beside every lesson that answers and draws
  • Lifetime access, every new lesson included

$49 once, by PayPal

See the AI Engineering course

Continue this lesson

in Agents In Production

This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

One-time payment. Lifetime access. No subscription.
Global
$49/lifetime

One-time payment by card or PayPal

Get Lifetime Access

Where this shows up in interviews

This idea carries a full system design question on its own. Each walks through the full answer.

  • →Design an AI Agent for Customer Support
Previous lessonReflection and Decomposition: Making Agents Check Their Own WorkNext lessonStructured Output Costs Right Answers: One JSON Box, Measured

System Design

  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Pricing
  • Privacy
  • Terms
  • Refunds
  • Delivery
  • Contact
Why the Hostile Case Was Contained
  • Where the Model Still Earns Its Place
  • What This Costs
  • How Teams Actually Draw This Line
  • Knowledge Check
  • Foundation lessons free, try before you buy
    India Special
    India
    ₹999/lifetime

    One-time payment: UPI, card or netbanking

    Get Lifetime Access
    Foundation lessons free, try before you buy
    Every ML Platform lesson, Foundation to Advanced
    Real architecture diagrams with real tool logos
    MLOps, LLMOps, and data engineering in depth
    Quizzes with instant feedback
    Built and reviewed by senior engineers
    Lifetime access, all future content included

    Foundation lessons of this course are free, no payment needed. Try them now

    The LLM Explains. It Does Not Arbitrate.

    An agent that can issue a refund is one confident misclassification away from issuing one it should not. The model reads a message that says "this is outrageous, I want my money back for order 1001," decides on its own that this is a refund, and calls the refund tool. Most of the time it is right. The time it is wrong, because the order was already refunded, or the message was quoting someone else, or the customer is not who they claim to be, the money is gone and you find out from finance.

    The reflex is to make the model better at that decision: a sharper prompt, a bigger model, an example of when not to refund. The reflex is wrong, and the earlier lessons in this chapter say why. The lesson showed that a good model still emits bad calls on ordinary inputs. The first lesson showed that per-step reliability caps out below 1.0, so a decision the model gets wrong some fraction of the time will be wrong some fraction of the time no matter how you prompt it.

    Senior agent systems do something else. They take the decision away from the model. The model still reads the message and still says what it thinks it is: a refund request. But it does not decide whether the refund happens. Code decides that, by looking up the order, checking it is refundable, and confirming the customer owns it. The model explains. Code arbitrates.

    Responsibility architecture map for deterministic scaffolding: a deterministic left zone where code owns the control flow with a state machine and router, a precondition gate, and an executor plus audit log, an arbitration boundary down the centre, and a stochastic right zone where the model fills labeled slots by classifying into an intent, extracting fields, and drafting a reply, with the slot value crossing the boundary as data and the binding decision staying entirely in code

    That word, arbitrate, is the whole lesson. To arbitrate is to make the binding decision that changes the world: money moves, a row deletes, an email sends, the workflow takes one branch instead of another. Everything here is a rule for keeping that decision in code and leaving the model with the part only language can do.