The Assistant Who Was Always Sure
Imagine a busy help desk. A young assistant sits at the front and answers people quickly. Behind a glass wall sits an expert. The expert is slower and costs more, but is almost always right.
The plan is simple. The assistant answers every question. When the assistant thinks an answer might be wrong, they pass that question back to the expert. That way, easy questions cost little, and hard ones still get the expert.
This plan only works if the assistant knows when to pass a question back. If the assistant is sure about everything, including the wrong answers, the expert never sees the questions that needed them.
AI apps use exactly this plan to save money. A small, cheap model answers first. A big, expensive model gets the questions the small one might get wrong. In this lesson I measure the most important part of that plan: how do you know which answers to pass on?
The Words You Need First
Model. Here, a language model: an AI that reads text and writes text, like the one behind ChatGPT.
Cheap model and strong model. A small model is fast and costs little to run, but it makes more mistakes. A bigger model is slower and costs more, but it makes fewer mistakes.
Cascade. A plan where the cheap model answers every question first, and some questions are then passed on to the strong model. It is called a cascade because questions flow down from one model to the next, like water down steps.
Escalate. To pass a question on to the strong model. Every escalated question costs more, because you pay for both models.
Signal, or clue. What you use to decide which questions to escalate.
Call. One request to a model: one question in, one reply out.
Token. A small piece of text, roughly three quarters of a word. Models read and write in tokens, and model companies charge by the token, so a long reply costs more than a short one.