Imagine a busy help desk. A young assistant sits at the front and answers people quickly. Behind a glass wall sits an expert. The expert is slower and costs more, but is almost always right.

The plan is simple. The assistant answers every question. When the assistant thinks an answer might be wrong, they pass that question back to the expert. That way, easy questions cost little, and hard ones still get the expert.

This plan only works if the assistant knows when to pass a question back. If the assistant is sure about everything, including the wrong answers, the expert never sees the questions that needed them.

AI apps use exactly this plan to save money. A small, cheap model answers first. A big, expensive model gets the questions the small one might get wrong. In this lesson I measure the most important part of that plan: how do you know which answers to pass on?

Model. Here, a language model: an AI that reads text and writes text, like the one behind ChatGPT.

Cheap model and strong model. A small model is fast and costs little to run, but it makes more mistakes. A bigger model is slower and costs more, but it makes fewer mistakes.

Cascade. A plan where the cheap model answers every question first, and some questions are then passed on to the strong model. It is called a cascade because questions flow down from one model to the next, like water down steps.

Escalate. To pass a question on to the strong model. Every escalated question costs more, because you pay for both models.

Signal, or clue. What you use to decide which questions to escalate.

Call. One request to a model: one question in, one reply out.

Token. A small piece of text, roughly three quarters of a word. Models read and write in tokens, and model companies charge by the token, so a long reply costs more than a short one.

How could you tell that the cheap model's answer might be wrong? This lab starts with three clues.

1. Ask it how sure it is. After answering, the cheap model gives a number from 0 (a guess) to 100 (certain). You escalate the answers with the lowest numbers. Engineers call this stated confidence.

2. Ask it again, and see if it agrees with itself. A model has a setting called temperature, which controls how much randomness goes into its answer. At temperature 0 there is none: it always gives its first choice. The cheap model answers each question once at temperature 0, then twice more at temperature 0.8, which allows a fair amount of randomness. If all three answers match, the model is probably steady on this question. If they do not, you escalate it. Engineers call this self-consistency.

3. Pick at random. You escalate questions with no clue at all. A clue that does no better than random is not telling you anything.

To be fair, every clue gets the same budget: the same number of questions it is allowed to escalate. Later, a fourth clue turns up that I had not planned.

Before the main test, I ran two smaller versions on easy maths problems, each with a right answer known exactly. They taught me something I did not expect.

In the first version, I made the cheap model reply with just two numbers and nothing else: its answer, and how sure it was. It had no room to write out its working, meaning the steps of the solution, like a student's rough work. It got only 2 of 200 problems right. On 194 of its 198 wrong answers it said it was 100 out of 100 sure, and on the other 4 it said 0.

In the second version, I let the cheap model write out its working first, then give the two numbers. It got 197 of 200 right. And it said 100 out of 100 sure on every single answer: all 197 right ones, and all 3 wrong ones.

There is a second lesson here. In the first version almost every answer was wrong, and in the second almost every answer was right. Neither leaves enough of both kinds to test a clue. So the main test mixes easy problems and hard ones.

1. The problems. Short maths word problems made by a small program, so the right answer to each one is known. 100 are easy, like "Asha has 40 books and buys 12 packs of 2. How many now?". 75 are hard, like multiplying two 3-digit numbers, or finding what is left over after sharing 4,438 marbles among 37 children.

2. The cheap model. A small model called llama3.2:3b. ("3b" means about 3 billion internal numbers, small enough to run on a laptop.) For each problem it works the answer out in words, then writes its answer and how sure it is.

3. Two more tries. The cheap model answers each problem twice more, at temperature 0.8, so we can see if its three answers agree.

4. The strong model. A slightly bigger model called qwen3:4b, which writes much longer working. It answers every problem once.

5. The mark. An answer is right if its number matches the known answer. No AI grades anything.

Then, for each clue, the lab acts out a cascade: it escalates some problems, takes the strong model's answer for those, keeps the cheap model's answer for the rest, and counts how many the whole cascade got right.

One kind of problem was set aside. I first built 200 problems. For 25 of them, all about a shop selling tickets, the question could be read two ways, and my answer key only accepted one. The strong model noticed: in its working it argued about which reading I meant, and mostly picked the other one. A reviewer then found that every problem both models got "wrong" was one of these. The mistake was mine, not the models'. Those 25 are left out of every number from here on, which leaves 175.

First, how good is each model without any cascade?

The cheap model got 110 of 175 right: 100 of the 100 easy problems and 10 of the 75 hard ones. The strong model got every one of the 175 right.

That makes the question very clean. Every problem a cascade passes on gets fixed, so the only thing that matters is which problems it picks.

This is a real recording of the lab's report, printed on the laptop where the lab ran.

In this recording, "confidence" means asking the model how sure it is, "agreement" means asking it again, "random" means picking at random, and "biggest" is a fourth clue, explained on the slide "Was It Spotting Wrong Answers, or Hard Questions?". "Fixed" counts wrong answers the strong model then got right, and "broke" counts right answers it made wrong. Each "p =" number is the "by luck" chance explained on the slide "Same Budget, Three Clues", written as a decimal: 3e-06 means about 3 in a million.

Here is what the cheap model said when I asked how sure it was.

It said 100 on 174 of its 175 answers. On the remaining 1, its working ran past the 1,500-token limit the lab allows per reply, and it never gave a number. On right answers it said 100. On wrong answers it also said 100. So choosing "the least sure" answers is just choosing at random.

Why does this happen? One likely reason: a language model writes the words that usually come next, and after a confident piece of working, "Confidence: 100" is what usually comes next. So the number may describe how confident the text sounds, not whether the maths is right. This lab did not test that reason directly. Researchers have measured the same problem: models often state a high confidence that does not match how often they are right, and checking whether a model agrees with itself usually works better than asking it (Xiong and others, 2024).

Now the second clue. The cheap model answered each problem three times. How often did the three answers disagree?

The three answers disagreed on 76 problems.

• Of the 65 wrong answers, 60 came with disagreement.
• Of the 110 right answers, only 16 did.

So disagreement looks like a useful warning. If one of the three replies could not be read at all, the lab counts that as disagreement too.

It is not perfect. 5 wrong answers came out the same all three times. For example, on one problem the right answer was 26,208, and all three tries said 26,168.

Now the fair test. Each clue may escalate exactly 65 problems, the number the cheap model got wrong. A perfect clue would escalate exactly the wrong ones.

Asking the model how sure it was tied with random: 26 each. That is what a number that is always 100 gives you.

Asking it again caught 50. Why not all 60 of the wrong answers that disagreed? Because 76 problems disagreed and the budget was 65, so some were left out.

Compared problem by problem with random, asking again caught 26 wrong answers that random missed, and random caught 2 that it missed. Treat each of those problems as a coin flip. If the two clues were equally good, each problem would go either way half the time, and a split as uneven as 26 to 2 would happen by luck about once in 329,773 tries.

So asking again beats random, clearly. The next slide asks what it was actually spotting.

Here is something I had missed until a reviewer pointed it out, and it changes how to read the last slide.

Every one of the cheap model's wrong answers was on a hard problem. It got all 100 easy ones right. And disagreement mostly happened on hard problems: 9 times on the easy ones, against 60 plus 7 on the hard ones.

Now look inside the hard problems only. Disagreement flagged 60 of the 65 wrong answers, 92 percent. But it also flagged 7 of the 10 right ones, 70 percent. That gap is far smaller than the gap between easy and hard problems, and with only 10 right answers to judge by, it is too small to trust. Most of what it caught, it caught by noticing that a question was hard.

That suggested a fourth clue, one that needs no model at all: escalate the problems with the biggest numbers in the question first. Hard problems here have bigger numbers. I chose this rule after seeing the results, so treat it as an example of a free check, not a discovery. At the same budget it caught 56 wrong answers, and the cascade was 94.9% right, against 91.4% for asking again. Against asking again the split was 12 to 6, which could happen by luck about 24 times in 100, so I would not call it better. But it needs no extra calls.

In a real app, hard questions do not always show it so plainly, and asking again may earn more of its keep. The rule to take away: test a clue against random, and also against a free rule that only reads the question.

You do not have to escalate exactly as many problems as are wrong. You can escalate more or fewer. More escalation means more right answers, and more cost.

At the left edge, nothing is escalated, so the cascade is just the cheap model: 110 of 175, or 62.9% right. At the right edge, everything is escalated, so it is just the strong model: 100% right.

A good clue lifts the line quickly while only a few problems are escalated. With just 10 percent of problems escalated, 18 problems, the same number the playground below asks you to try, asking again gave a cascade that was 70.3 percent right, the biggest-numbers rule 72 percent, and random 66.3 percent.

A clue is not free, and here the price decides everything. The lab counted every token each model wrote.

The cheap model wrote about 140 tokens per reply. The strong model wrote about 916, because it writes out very long working. Asking again means three cheap replies on every problem, before any escalation. Here is what each plan wrote in total:

If a strong token costs the same as a cheap one, asking again costs 101% of simply sending everything to the strong model, and gets fewer right. In a real app you could not set the budget to the number of wrong answers, because you would not know it, so you would escalate every disagreement: 76 problems here, 97.1% right, 176,580 tokens. That is more than the strong model alone.

This box holds the real results for all 175 problems. For each one: was the cheap model right, did its three answers agree, was the strong model right, and the biggest number in its question. Press Run to see how many wrong answers each clue catches, and how often the whole cascade is right.

Then change BUDGET, the number of problems each clue may escalate. Try 18, then 120.

Maths problems only. Every answer here is one number, so it is either right or wrong. Many real questions, like "summarise this email", have no single right answer, and asking again works differently there, because two good summaries can use different words.

Difficulty was easy to see. Easy and hard problems looked different from the question alone, which is why a free rule did so well. Real traffic is messier.

One temperature. The two extra answers used temperature 0.8, and each was right less often than the first answer. A different setting would flag a different set of problems. I did not test other settings.

One cheap model and one strong model. Other models may state confidence differently. The strong model also ran with a 3,000-token limit per reply, and with its thinking mode, a setting that lets it reason at greater length before answering, switched off.

Tokens are not prices. The lab counts tokens written. Real costs depend on each model's price per token, and on the tokens each prompt reads too.

One run. Another run, with other problems or other random tries, would give somewhat different numbers.

1. Label a sample. Take a few hundred real questions from your app, and write down the right answer for each. Read the questions too: this lab lost 25 of its problems to a question that could be read two ways.

2. Run both models on all of them. Count where each one is right. Check that your sample has plenty of both right and wrong cheap answers, or you cannot test a clue.

3. Test every clue against random, and against a free rule. Random tells you whether the clue knows anything. A free rule that only reads the question, like its length or the size of its numbers, tells you whether the clue knows more than the question already shows.

4. Count the cost in tokens. Include the tokens the clue itself needs, and remember that a strong model may write far more per reply. Then compare against simply using the strong model for everything.

5. Prefer a check you can run in code. If the answer can be checked by a program, use that first. For example: does the reply follow the fixed data format your app expects, or does the code the model wrote pass its tests?

Those three numbers are the lesson. The model's own confidence caught no more than random. Asking again caught more, and a rule that never called a model caught more still.