LLM Evaluation
Measuring whether a model or pipeline actually works, using error analysis, labeled datasets, and validated judges rather than a glance. The single most in-demand and least-taught AI engineering skill.
What is LLM Evaluation?
Measuring whether a model or pipeline actually works, using error analysis, labeled datasets, and validated judges rather than a glance. The single most in-demand and least-taught AI engineering skill.
LLM Evaluation is a advanced concept that sits in the LLM and GenAI Ops area of system design. Engineers reach for it whenever they need to reason about real-world trade-offs in that space, not just for textbook correctness, but because real production systems at companies like Netflix, Amazon, and Google make these decisions every day.
If you want to go deeper than this definition, with diagrams, code, and a quiz to lock it in, work through the "Evaluating LLMs in Production" lesson linked below. It walks through the why, the mechanism, the trade-offs, and how the giants actually use it in production.
Learn LLM Evaluation in depth
Full interactive lesson with diagrams, code examples, real-world references, and a quiz.
Open the Evaluating LLMs in Production lessonRelated lessons
Lessons that touch on LLM Evaluation as part of a larger topic.
RAG Evaluation: Localizing Failure Between Retrieval and Generation
Recall@k, context precision, and faithfulness split every wrong RAG answer into a retrieval bug or a generation bug, each with a different fix.
ml-intermediate · retrieval rag
Why Evals Are the Job: Moving From Vibes to Measurement
Why evals, not observability, decide if your LLM feature is good, and why measuring quality is the core skill of AI engineering in 2026.
ml-advanced · evals
Error Analysis: Reading Failures Before Building Metrics
Read real failures before you build a single metric. Open coding, axial coding, and saturation turn a vague feeling into a ranked failure taxonomy.
ml-advanced · evals
Building a Golden Dataset: The Set That Catches Regressions
A golden dataset is a curated set of inputs and known-good answers you run before and after every change, so a prompt tweak that quietly breaks things cannot ship.
ml-advanced · evals
LLM-as-Judge and Its Failure Modes
The full catalogue of ways an LLM judge fails, the mechanism behind each one, the fix that removes it, and how to validate a judge against human labels before you trust its scores.
ml-advanced · evals
See also
Related glossary terms you might want to look up next.
LLM-as-Judge
Using a language model to score another model's output. Fast and scalable, but it has real failure modes such as self-preference and position bias, so the judge itself must be validated against human labels.
Error Analysis
Reading real failure traces and coding them into categories to find where a system actually breaks, before building any metric or dashboard. Most of eval effort should go here, not into tooling.
Golden Dataset
A curated set of inputs with known-correct expected outputs, used to measure quality and catch regressions when you change a prompt, a model, or the retrieval stack.
RAG (Retrieval-Augmented Generation)
A pattern that retrieves relevant text from an external index and puts it in the model's prompt so the answer is grounded in current, verifiable facts rather than the model's memory.
Embedding
A list of numbers that represents the meaning of a piece of text, so that similar meanings sit close together in vector space. The same model must embed both the corpus and the query.
ANN (Approximate Nearest Neighbor)
Search that finds vectors close to a query without scanning every item, trading a small chance of missing a true neighbor for a large gain in speed. HNSW and IVF are the common indexes.