systemdesign.academyAI EngineeringFree lessonsSign inGet the course
  1. Home
  2. AI Engineering: Evaluation, LLM Ops and Security

Where this shows up in interviews

2 system design questions lean on this idea. Each walks through the full answer.

  • →Design a Multi-Agent System
  • →Design an AI Agent for Customer Support
Previous lessonValidating Your Judge: TPR, TNR, and the Human CeilingNext lessonOffline vs Online: Evals Verify, Experiments Validate

System Design

  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Pricing
  • Privacy
  • Terms
  • Refunds
  • Delivery
  • Contact
  • LLM Evaluation and Error Analysis
  • Evaluating RAG and Agents: One Score Detects, Components Localize

    • AI Engineering · Evals

    Evaluating RAG and Agents: One Score Detects, Components Localize

    Free preview: the first part of this 18-part lesson.

    Already bought the course? Sign in to read the whole lesson.

    this lesson is part of

    AI Engineering

    Unlock this lesson and the 199 other paid lessons

    • 210 lessons in 16 chapters
    • 118 lessons built on a lab that really ran
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the AI Engineering course

    The rest of this lesson

    1. Break 'Is This Good' Into Layers That Fail Separately
    2. End-to-End Detects. Components Localize.
    3. The RAG Triad, and Where It Sits Inside an Agent
    4. Retrieval: Recall@k and Context Precision
    5. Generation: Faithfulness, Checked One Claim at a Time
    6. A Tool Call Is Four Checks, Not One
    7. A Right Answer Reached the Wrong Way Is Still a Bug
    8. Outcome-Level Scoring vs Step-Level Scoring
    9. A Trajectory Rubric You Can Score Without a Golden Path
    10. Attribute the Failure to the Step That Caused It
    11. Multi-Turn: Score Every Turn and the Whole Session
    12. Final-State Judging vs Process Evaluation
    13. Roll Facet Pass Rates Up Into a Task Score, and a To-Do List
    14. Quality Is Not the Only Axis: Cost and Latency Are Evals Too
    15. What Component Evals Cost, and When to Skip Them
    16. How Teams Actually Do This at Scale
    17. Knowledge Check

    read the rest of this lesson in

    AI Engineering

    Unlock this lesson and the 199 other paid lessons

    • 210 lessons in 16 chapters
    • 118 lessons built on a lab that really ran
    • An AI tutor beside every lesson that answers and draws
    • Lifetime access, every new lesson included

    $49 once, by PayPal

    See the AI Engineering course

    Continue this lesson

    in Evals

    This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

    One-time payment. Lifetime access. No subscription.
    Global
    $49/lifetime

    One-time payment by card or PayPal

    Get Lifetime Access
    Foundation lessons free, try before you buy
    India Special
    India
    ₹999/lifetime

    One-time payment: UPI, card or netbanking

    Get Lifetime Access
    Foundation lessons free, try before you buy
    Every ML Platform lesson, Foundation to Advanced
    Real architecture diagrams with real tool logos
    MLOps, LLMOps, and data engineering in depth
    Quizzes with instant feedback
    Built and reviewed by senior engineers
    Lifetime access, all future content included

    Foundation lessons of this course are free, no payment needed. Try them now

    The Score Dropped Seven Points and Nobody Knew Why

    Your support assistant scored 71 percent end-to-end task success this week. Last week it was 78. That single number is real, and it is nearly useless. It cannot tell you which layer moved. Did retrieval start missing the right documents? Did the model begin ignoring the context it was given? Did a tool change its API? Did someone edit the prompt on Tuesday? Seven points down, and the number points at nothing.

    So the week goes the way these weeks go. Someone pulls fifty failing traces and reads them by hand. Someone else has a theory about the model. A third person is sure it is the new prompt. Three days later the actual cause turns out to be a dependency that quietly renamed a tool argument, and every tool call after the upgrade was malformed. The end-to-end score saw the damage. It just could not say the word "tool."

    This is the tax you pay for measuring a RAG or agent system with one number. The number is a smoke alarm. It tells you there is a fire and gives you the whole building to search. DoorDash felt this hard enough to do something about it. Their Dasher support assistant cut by about 90 percent. The way they got there was not one cleverer judge. It was several narrow ones, each checking a different property, so a failure came with a location. They report this in their engineering writeup, Path to high-quality LLM-based Dasher support automation.

    This lesson is the evals-chapter treatment of both systems that fail this way. A RAG pipeline has two internal stages, retrieval and generation, and the earlier lesson on RAG evaluation drills into the retrieval-side metrics. An agent adds a third layer, tool calls. It also adds a fourth concern no RAG metric touches at all: the trajectory, meaning the sequence of steps the agent took to reach its answer.

    The move is to stop asking "is this good". Ask smaller questions that each have a checkable answer. And for agents, grade the path and not only the destination. Do that and a seven-point drop arrives with an address.