systemdesign.academyAI EngineeringFree lessonsSign inGet the course
  1. Home
  2. AI Engineering: Evaluation, LLM Ops and Security
  3. LLM and GenAI Ops
  4. Model Routing and Cascades: Running Three Models Without Chaos

AI Engineering · Llm Genai Ops

Model Routing and Cascades: Running Three Models Without Chaos

Free preview: the first part of this 14-part lesson.

Already bought the course? Sign in to read the whole lesson.

this lesson is part of

AI Engineering

Unlock this lesson and the 199 other paid lessons

  • 210 lessons in 16 chapters
  • 118 lessons built on a lab that really ran
  • An AI tutor beside every lesson that answers and draws
  • Lifetime access, every new lesson included

$49 once, by PayPal

See the AI Engineering course

The rest of this lesson

  1. A Stack Is a Tier of Models, Not One Model
  2. Two Ways to Use More Than One Model: Route or Cascade
  3. Static Rules: Predictable, and Blunt
  4. Learned Routers: Predict Difficulty, Pay in Labels and Drift
  5. Cascades: Try Cheap, Escalate on Evidence
  6. The Escalation Signal Is Where Cascades Live or Die
  7. What a Cascade Costs Per Thousand Requests, and Where It Stops Paying
  8. The Gateway: One Door for Every Model Call
  9. The One Thing the Gateway Must Not Do
  10. Every Model Swap Is a Deploy, and Route Changes Are Too

read the rest of this lesson in

AI Engineering

Unlock this lesson and the 199 other paid lessons

  • 210 lessons in 16 chapters
  • 118 lessons built on a lab that really ran
  • An AI tutor beside every lesson that answers and draws
  • Lifetime access, every new lesson included

$49 once, by PayPal

See the AI Engineering course

Continue this lesson

in Llm Genai Ops

This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.

One-time payment. Lifetime access. No subscription.
Global
$49/lifetime

One-time payment by card or PayPal

Get Lifetime Access
Previous lessonToken Cost Engineering: Where the LLM Bill Actually GoesNext lessonRate Limits and Provider Failure: The Number One Production Error

System Design

  • Foundation
  • Intermediate
  • Advanced
  • Capstone

AI Engineering

  • Foundation
  • Data, RAG and Agents
  • Evaluation, LLM Ops and Security

systemdesign.academy

  • Home
  • Glossary
  • Interview prep
  • Reviews
  • About
  • Pricing
  • Privacy
  • Terms
  • Refunds
  • Delivery
  • Contact
When to Route, When to Cascade, When to Just Pick One
  • How Teams Actually Run This
  • Knowledge Check
    • Foundation lessons free, try before you buy
    India Special
    India
    ₹999/lifetime

    One-time payment: UPI, card or netbanking

    Get Lifetime Access
    Foundation lessons free, try before you buy
    Every ML Platform lesson, Foundation to Advanced
    Real architecture diagrams with real tool logos
    MLOps, LLMOps, and data engineering in depth
    Quizzes with instant feedback
    Built and reviewed by senior engineers
    Lifetime access, all future content included

    Foundation lessons of this course are free, no payment needed. Try them now

    One Prompt, Three Models, and Nobody Owns the Switch

    The same request can cost you sixteen times more depending on which model answers it. At list prices in 2025, OpenAI's gpt-4o-mini bills about 0.15 dollars per million input tokens while gpt-4o bills about 2.50, and the output side keeps the same ratio, roughly 0.60 against 10.00 dollars, so the strong model is around sixteen times the price of the cheap one per token. Check the current numbers on the pricing page, because they move, but the shape is stable, and Anthropic's lineup has the same shape, with Haiku a fraction of Sonnet (pricing). That gap is the entire reason model routing exists. Send every request to the strong model and most of your bill is easy questions answered by an expensive model that a cheap one would have gotten right.

    The trouble shows up on the way to using that gap. Summarization calls the cheap model from one service. The code assistant calls the strong model from another. Someone wired in a specialist for extraction last quarter, straight into a third code path. No single place decides which model handles a request, no single place records what it cost, and no single place lets you change a model. When the strong model shipped a new version, three teams found out separately, and the extraction eval that was pinned to the old version quietly started grading a different model. The cost of that sprawl is not only money. It is that you cannot answer the two questions a finance review will ask, which model served this traffic and what did it cost, because the answer lives in three log formats across three services.

    Before and after governance of model calls: on the left, a summarizer, a code assistant, and an extraction job each call a different model from their own code path with no shared cost ledger and no shared switch; on the right, all three route through one LLM gateway that centralizes auth and keys, retries and fallback, the cost ledger, budgets and quotas, and model selection, which then fans out to the cheap, specialist, and strong tiers

    Two ideas clean this up. Model selection, which decides per request which model runs, and a gateway, which is the one layer every model call passes through. This is a different decision from the one in the lesson on fine-tuning versus RAG versus prompting. That lesson picks how you adapt a model before you ship. This one picks which model answers each live request, and then makes sure the picking happens in a place you can see and gate.