Loading...
System Design and AI Engineering, by Roni Das. 972 interactive lessons.
Start LearningAI Engineering · Advanced · LLM and GenAI Ops
A component that decides which language model should answer each request, so easy requests go to a cheap model and hard ones to a strong model.
In short
An LLM router is a component that decides, for each request, which language model should answer it, so easy requests go to a cheap, fast model and hard ones go to a strong, expensive model. It exists because the price gap between models is large, and most traffic does not need the strongest one.
In short
An LLM router is a component that decides, for each request, which language model should answer it, so easy requests go to a cheap, fast model and hard ones go to a strong, expensive model. It exists because the price gap between models is large, and most traffic does not need the strongest one.
This page is the free part.
₹999 in India$49 everywhere elseonce, for the whole course
The AI Engineering course covers LLM Router across a run of lessons, not one page. These 4 alone are about 285 minutes of step-by-step reading, every one with code you run in the browser, all with a quiz.
Cheap models for easy traffic, strong models for hard cases. How to route, when a cascade stops paying, and the one thing your gateway must never do.
A cascade lets a cheap model answer first and passes some questions to a strong model. Everything depends on the clue that picks which ones. On 175 maths problems with known answers, the cheap model said it was 100 out of 100 sure on almost every answer, so asking how sure did exactly as well as random. Asking it three times beat random clearly, but mostly by spotting hard questions, a free rule on the question did about as well, and in tokens it cost as much as the strong model alone.
Does batching help a small model on a CPU, and what does it cost in latency? On a quiet 1-vCPU EC2 box, predict_proba on 1,024 rows took 5.24 times as long as on one row, so one row in a big batch was 195 times cheaper. Then I put a micro-batcher into the chapter's FastAPI service and sent it random (Poisson) arrivals from half to 2.5 times the most it answered with no batching (S, 1,098.4 a second): 17 settings, 7 rates, 5 rounds, every setting compared with no batching on the same arrivals. With no batching the service fell behind at S and gave about 1,045 answers a second at most. Batches of up to 32 with no timed wait kept up at every rate, gave 2.62 times as many answers at 2.5 x S, and even at half of S cut the p99 from 5.26 ms to 2.50 ms. Every timed wait of 1 ms or more raised the median at that low rate. All 7,179,151 scores were equal to the bit to single-row scores. Every AWS step is recorded so you can build the box yourself.
A new model goes live in one of two ways: a canary, where a share of real requests goes to it, or shadow, where it answers every request silently next to the old one. I measured both on electricity prices, with the right answer arriving half an hour later. On a known 4.09-point regression, shadow raised the alarm in all 20 starts (median 241.5 half-hours) and a 20% canary in 12. Checking after every half-hour turned a 1% false-alarm rate into 6 to 10% for a canary. What a rollout is, why comparing the same rows needs fewer of them, what shadow costs, and two faults in my own first design, kept on the record.
all part of
AI Engineering
203 lessons · about 209 hours · 10 free to read
₹999in India, by UPI
$49everywhere else, by PayPal
You own this course
Continue with Model Routing and Cascades: Running Three Models Without Chaospay once,
yours for life
An LLM, a large language model, is the kind of model behind a chatbot: text goes in, text comes out. Most companies do not use just one. They use a small set of models with different prices and different quality.
An LLM router sits in front of those models. A request comes in. The router looks at it and picks one model to answer. The rest of your code does not need to know which model was chosen.
Why bother? Because the price difference is big. The lesson uses OpenAI's 2025 list prices. gpt-4o-mini cost about 0.15 dollars per million input tokens. gpt-4o cost about 2.50 dollars. A token is a small piece of text, roughly part of a word. On output the ratio was the same, about 0.60 against 10.00 dollars. So the strong model was around sixteen times the price per token. Prices move, so check the provider's page, but the shape stays the same.
If every request goes to the strong model, most of your bill pays for easy questions that a cheap model would answer just as well. A router is how you stop paying that.
The lesson describes three tiers of model. Each tier earns its place on a different slice of traffic.
The saving comes from how traffic is split between them. At a sixteen to one price ratio, moving sixty percent of traffic from the strong tier to the cheap tier cuts the blended bill by more than half. The router's whole job is to make that split well, without making answers worse.
There are two ways to use more than one model. They differ on one thing: when the decision happens.
A router decides up front. It reads the request before any model runs and picks one model. That costs one model call per request, with no waste. But the router is guessing how hard the request is before it has seen any answer.
A cascade decides after it sees evidence. It sends the request to the cheap model first and checks the answer. If the answer passes the check, it returns it. If not, it escalates, meaning it sends the request on to the strong model. The decision rests on a real answer, not a guess. The cost is a second model call every time it escalates.
So routing is cheaper per request but has to guess. Cascading guesses less but pays twice when it escalates. The lesson notes that most mature systems use both. A coarse router in front handles obvious traffic. A cascade inside a tier catches the cases the router could not separate.
Based on a figure in the lesson Model Routing and Cascades: Running Three Models Without Chaos.
Once you route up front, you still have to choose how the router makes its choice. The lesson compares three kinds.
RouteLLM, from LMSYS and Berkeley, is a public example of a learned router. It trains a router that chooses between a weak and a strong model. Its paper reports large cost cuts at matched quality on its benchmarks. The lesson's warning applies: those are benchmark results, not your traffic.
A cascade is only as good as the check that decides accept or escalate. The lesson is direct about which check to trust.
Do not escalate on the model's own confidence. If you ask the cheap model how sure it is, the number often does not match whether the answer is right. Models tend to be overconfident.
Escalate on a validator. A validator is a check in code on the cheap answer. Does the JSON parse? Does the code compile and pass its tests? Does the field match the schema? Is the claim supported by the source document? In the lesson's illustrative example of 1,000 requests, both signals escalate the same 220. The validator catches 180 of the 200 real failures. Self-reported confidence catches 60. The cost is the same, and the validator catches three times as many.
Cost is easy to write down. Count one cheap call as 1 and one strong call as R, the price ratio. Every request pays 1. A fraction p escalates and also pays R. So a cascade costs 1 + p × R per request, against R for sending everything to the strong model.
In the lesson's illustrative model, with R at 16 and 90 percent easy traffic, the cascade cost about 1.35 dollars per thousand requests against 7.20 for all-strong, with accuracy of 0.931 against 0.963. As traffic gets harder, p rises. Near 48 percent easy traffic, the cascade escalates about 44 percent of requests and costs more than half the all-strong bill. At that point the cheap first pass is mostly wasted work. A cascade is a bet that most requests are easy.
These two terms are often mixed up. A router is the decision: which model answers this request. A gateway is the place where that decision runs.
An LLM gateway, also called an AI gateway, is one API in front of every model provider. Every model call in the company goes through it. The lesson lists what it centralises: API keys and authentication, retries and fallbacks when a provider fails, logging and a cost ledger for every call, budgets and quotas, and model selection.
So the router usually lives inside the gateway. That is not an accident. The router can only govern the traffic it sees, and the gateway is the one component that sees every request. Without a gateway, each service picks its own model in its own code, and nobody can answer which model served this traffic or what it cost.
Open examples named in the lesson are LiteLLM, which puts one OpenAI-style API in front of many providers, Cloudflare AI Gateway and Kong AI Gateway.
A gateway that can switch models can also break your measurements. Say a caller asks for gpt-4o, and your quality tests were measured on gpt-4o. If the gateway quietly serves a cheaper model instead, every test result now describes a model that is not running.
The lesson's rule: the model that served a request is part of the contract. A fallback to another model is fine when a provider hard-fails, but it must be recorded and shown to the caller. When a request or an evaluation pins a model, honor the pin or fail loudly.
For the same reason, every route change is a deploy. Swapping a model version, raising an escalation threshold or moving twenty percent of traffic between tiers changes what users get, even with no code change. Keep the route config in version control and run your evaluation set in CI on every change, as described in the prompt CI lesson.
Use one model when volume is low, or when a wrong cheap answer is not acceptable. Build a router or cascade only after you have measured that a cheap model really handles a meaningful share of your traffic at acceptable quality. If it does not, the routing machinery is overhead on top of a strong-model bill.
Routing pairs well with other cost tools, such as LLM caching, which avoids the model call entirely for repeated requests. For more on escalation checks, see the cascade escalation signals lesson, and for what the gateway does when a provider fails, see rate limits and provider failure.
To build the whole system, with the runnable cost model, the signal comparison and the CI gate for route changes, the next step is the Model Routing and Cascades lesson.
RouteLLM (LMSYS and Berkeley)
An open framework that trains a router to choose between a weak and a strong model per request, using human preference data and public benchmarks.
FrugalGPT (Stanford)
Formalised the LLM cascade: try a cheap model, score the answer, and move to a stronger model only when the score is not good enough.
LiteLLM
An open-source gateway that puts one OpenAI-style API in front of many providers, with retries, fallbacks and cost tracking.
Cloudflare AI Gateway
A hosted gateway that sits in front of model providers to log, cache and rate-limit model calls from one place.
It is the component that picks which model answers each request. It reads the request, for example its task type or length, and sends easy requests to a cheap model and hard ones to a strong model. The goal is a bill close to the cheap price with quality close to the strong model.
A router is a decision: which model should answer this request. A gateway is one API that every model call passes through. It handles keys, retries, fallbacks, logging, cost tracking and budgets. Model selection is one of its jobs, so the router usually runs inside the gateway, where it sees every request.
It gives every service one door to all model providers. In one place it handles API keys, retries and fallback when a provider fails, a log and cost record for every call, budgets and quotas, and model selection. It must never silently swap the model a caller pinned.
A router picks one model before any answer exists, so it costs one call but has to guess difficulty. A cascade runs the cheap model first, checks the answer, and escalates to the strong model only if the check fails. It decides on evidence but pays for a second call on every escalation.
Checked against these on .
Inference Optimization
The engineering that serves more tokens per GPU-second: KV caching, continuous batching, quantization, prefix caching, and speculative decoding. Serving cost is dominated by how well these are done.
LLM Evaluation
Measuring whether a model or pipeline actually works, using error analysis, labeled datasets, and validated judges rather than a glance. The single most in-demand and least-taught AI engineering skill.
LLM Caching
Reusing earlier work to avoid paying for the same model call twice: exact caches for identical requests, semantic caches for similar ones, and prompt caching on the provider's side.
Model Serving
Exposing a trained model behind an API that answers predictions within a latency budget, with batching, autoscaling, and versioning. Where most of the production cost and the request-path complexity lives.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
This page explains the idea. The "Model Routing and Cascades: Running Three Models Without Chaos" lesson builds it up step by step, with diagrams. It is part of the AI Engineering course, which has 203 lessons. 10 of them are free to read. You pay once and keep it for life, with no subscription.
Open the Model Routing and Cascades: Running Three Models Without Chaos lessoncourse 2
Take models from notebook to production, with labs on real models.