We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
AI Engineering · Evals
Free preview: the first part of this 16-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 203 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 203 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI coursein Evals
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
The is finished. Forty traces were read, ten of them failed, and the thirteen codes those failures carried were merged into four categories. Each category is named after a repair somebody could actually make.

The obvious next step is to put all four on a dashboard and watch them weekly. That is how you end up with a dashboard nobody looks at.
Not because the categories are wrong. They are real, they are well named, and the work that produced them was worth doing. They are just not big enough to carry a weekly number. And there is a one-line piece of arithmetic that says so, before you spend a sprint building the charts.

That closing line is the whole lesson. Being a real category and being a weekly number are two separate qualifications. The first one is a judgement about the failures. The second one is mostly arithmetic, and you can run it in about a second.
Here it is, run on the labels from grader A over the forty traces.

Read the last column. To watch "answer is incomplete" weekly you would need to read 67 traces every week. For the smallest category you would need 400. All four numbers are bigger than the forty traces the categories were found in.
That is a genuinely useful finding, and it is worth saying plainly instead of dressing it up. Keep the categories. Keep reading traces. Build the metrics later, when the counts can carry one.