We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
AI Engineering · Evals
Free preview: the first part of this 15-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 203 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 203 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI coursein Evals
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
You have started reading failures. You are writing short labels for what went wrong. At some point somebody asks the obvious question: how many do you need to read?

The answers people give are all numbers. Fifty. A hundred. One percent of traffic. Whatever fits in an afternoon. Each of those is a guess that sounds like a method.

The right place to stop does not depend on how many traces you read. It depends on whether you are still learning anything.
There is a name for it. It comes from research that works with words instead of numbers, and it is called theoretical saturation. Stop when reading more traces stops teaching you new kinds of failure.
Read that again, because one word is doing all the work. Failures never stop appearing. You will find a failure in trace five hundred. What stops appearing is a failure of a kind you have not seen before.
That is not a feeling. It is a curve you can plot, and it either flattens or it does not.

That is the real run from this chapter's lab. Forty traces, thirteen distinct failure codes, and the verdict comes down to a single trace of margin.

Look at the shape, because it has three parts and each one teaches something.
It sits flat for six traces. Nothing at all. Those answers were correct, and a correct answer gets no code. A saturation curve only moves when something fails, so a flat start means the system was working, not that you were reading badly.
Then it climbs in steps all the way to trace 26. Two codes, then two more, then one, then one, then two, then two, then two more at trace 26. This is what an unfinished curve looks like: the gaps between steps are short, and every few traces teach you a new word.
Then one last code at trace 31, and nothing for the final nine traces.

Nine quiet traces out of forty is a fifth of the reading, which is just enough to call this saturated. Only just. If the last code had arrived at trace 35 instead, the honest answer would have been "keep reading".