We use analytics cookies.We use cookies to analyze site traffic and improve your experience.
Loading...
AI Engineering · Evals
Free preview: the first part of this 15-part lesson.
Already bought the course? Sign in to read the whole lesson.
this lesson is part of
Generative AI and Agentic AI Engineering
Unlock this lesson and the 203 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI courseread the rest of this lesson in
Generative AI and Agentic AI Engineering
Unlock this lesson and the 203 other paid lessons
$35 once, by PayPal
See the Generative AI and Agentic AI coursein Evals
This lesson is part of the AI Engineering course. Unlock the full course, a separate one-time payment, for lifetime access.
One-time payment by card or PayPal
Get Lifetime AccessOne-time payment: UPI, card or netbanking
Get Lifetime AccessFoundation lessons of this course are free, no payment needed. Try them now
Your system is producing wrong answers. Somebody schedules a meeting to agree what the failure categories are, so everyone labels consistently.
That meeting is where the finding gets decided, before anybody has read a failure.

Seven items on the board and nobody has opened a trace. Whatever list comes out of that hour is a guess about your system, written by people reasoning from memory. Every trace that does not fit lands in "other". And "other" is where new failure kinds are never looked at again, because nobody re-reads a bucket whose name means "did not match".

Five against thirteen is the whole argument in two numbers. The five are a guess made in advance and the thirteen are what one grader actually needed to describe ten failures. Four of the thirteen would have had nowhere to go on the left-hand side.
Open coding is the other order. You read failures and write a short description of each one in whatever words come to mind. No list. No dropdown. No consistency requirement.
Then, and only then, you build the categories from what you wrote. That is the next lesson. This one is about doing the messy part well, because the messy part is where the information is.