systemdesign.academyAI EngineeringFree lessonsSign inGet the course
  1. Home
  2. AI Engineering: Evaluation, LLM Ops and Security
  3. LLM and GenAI Ops
  4. Reliability for Stochastic Systems: Error Budgets for Probabilistic Output

AI Engineering · Llm Genai Ops

Reliability for Stochastic Systems: Error Budgets for Probabilistic Output

Free preview: the first part of this 13-part lesson.

Already bought the course? Sign in to read the whole lesson.

this lesson is part of

AI Engineering

Unlock this lesson and the 199 other paid lessons

  • 210 lessons in 16 chapters
  • 118 lessons built on a lab that really ran
  • An AI tutor beside every lesson that answers and draws
  • Lifetime access, every new lesson included

$49 once, by PayPal

See the AI Engineering course

The rest of this lesson

  1. First Job: Make 'The Answer Was Good' a Countable Event
  2. Quality Is the Fourth Layer of the SLO Stack, and the Silent One
  3. Where the SLI Number Physically Comes From
  4. The Error Budget: 8 Percent, Spent Deliberately
  5. Alert on Burn Rate, Act on Exhaustion
  6. Design the Failure: The Graceful Degradation Ladder
  7. What the Ladder Buys, Computed
  8. Guardrails Are the Control Plane the Budget Reads Through
  9. Set the SLO by the Cost of a Wrong Answer, Not by Pride
  10. Multi-Step Flows: The SLO Lives at the User, the Budget Lives Per Step

read the rest of this lesson in

AI Engineering

Unlock this lesson and the 199 other paid lessons

  • 210 lessons in 16 chapters
  • 118 lessons built on a lab that really ran
  • An AI tutor beside every lesson that answers and draws
  • Lifetime access, every new lesson included

$49 once, by PayPal

See the AI Engineering course

Continue this lesson

in Llm Genai Ops

This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

One-time payment. Lifetime access. No subscription.
Global
$49/lifetime

One-time payment by card or PayPal

Get Lifetime Access
Previous lessonLLM Observability and Tracing: Seeing Inside a Nondeterministic SystemNext lessonWas the Cheap Model Sure? Choosing Which Questions a Cascade Passes On

System Design

  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Pricing
  • Privacy
  • Terms
  • Refunds
  • Delivery
  • Contact
  • What Ships Today, and What Nobody Has Solved
  • Knowledge Check
    • Foundation lessons free, try before you buy
    India Special
    India
    ₹999/lifetime

    One-time payment: UPI, card or netbanking

    Get Lifetime Access
    Foundation lessons free, try before you buy
    Every ML Platform lesson, Foundation to Advanced
    Real architecture diagrams with real tool logos
    MLOps, LLMOps, and data engineering in depth
    Quizzes with instant feedback
    Built and reviewed by senior engineers
    Lifetime access, all future content included

    Foundation lessons of this course are free, no payment needed. Try them now

    An LLM Product Can Be Down in a Way No Dashboard Shows

    Availability 99.97 percent. p95 1.4 seconds. Error rate 0.02 percent. Every panel green, and the support queue is filling with users who were told, politely and in perfect JSON, that a refund policy exists which does not.

    That gap is not a monitoring bug. It is a definitional mismatch. Classic reliability engineering was built for systems where a request is either served correctly or fails in a way the infrastructure can see: a 500, a timeout, a refused connection. Google's SRE book defines the whole discipline on that assumption, availability as good requests over total requests, counted from signals the serving stack emits (Embracing Risk). An LLM violates the assumption at its core. Its characteristic failure is a request that succeeds by every infrastructure measure and is wrong in its content. The 200 arrives on time, parses cleanly, reads confidently, and misinforms.

    Reliability dashboard mock beside the answer a user received: a dark production panel shows availability 99.97 percent, p95 latency 1.4 seconds, 5xx error rate 0.02 percent, and throughput 2,000 a day, every tile green and within SLO, while a red-bordered panel on the right shows a fluent reply citing a 60-day refund policy that does not exist, flagged as a semantic failure invisible to every tile, and a coverage strip below marks availability, latency, and validity as covered but quality as carried by nothing

    So uptime is the wrong metric, but the instinct to throw out SRE with it is also wrong. Error budgets, burn-rate alerts, and pre-agreed consequences are exactly the machinery a probabilistic component needs, arguably more than a deterministic one, because a component that is wrong 8 percent of the time by design needs a formal answer to the question "how wrong, for how long, before we act." What has to change is the event being counted. This lesson rebuilds the SRE stack on a new unit of failure: the answer that fails a semantic check.

    The rate limits lesson handled the layer below this one, whether the API answers at all. This lesson assumes that layer holds and asks the question that remains when it does: the system is up, fast, and well-formed. Is it right often enough, and what happens when it is not?