AI Engineering interview question · LLM-as-judge and online evals · mid level
How do you check that an LLM judge agrees with human reviewers?
Free to read in full. This is 1 of the 150 questions in the AI Engineering (LLM applications) interview pack, with 180 diagrams. Get the pack, ₹999 · $20
Why it is asked
The interviewer wants to know if you treat the judge as a model that must itself be tested, and whether you know why raw agreement is a misleading number.
The answer
You test the judge like any other classifier: humans label a set of real outputs, the judge labels the same set, and you compare the two case by case before you trust a single judge score.
How to do it:
- Take 100 to 200 real traces, with plenty of failures in them. A domain expert labels each one pass or fail, using the same written rubric the judge will get.
- Split the labeled set into a development part, where you tune the judge prompt, and a test part that you open only at the end. Otherwise you tune the prompt to the test cases and the result looks better than it is.
- Run the judge on the test part and count four cells: both pass, both fail, judge pass but human fail, judge fail but human pass.
Do not report raw agreement alone. If 90% of cases pass, a judge that always says "pass" agrees 90% of the time and catches nothing. Report two rates instead: the true positive rate (of the cases humans passed, the share the judge passed) and the true negative rate (of the cases humans failed, the share the judge failed). Cohen's kappa, a score that removes the agreement you would get by pure chance, is a useful single number.
Example with 200 cases: both pass 120, both fail 50, judge pass but human fail 20, judge fail but human pass 10. Raw agreement is 85%, but the judge catches only 50 of 70 real failures (71%), and kappa is 0.66.
Repeat the check whenever the judge model or prompt changes, and on a fresh sample every few weeks.
What the interviewer listens for
- Human labels on real traces, same rubric as the judge
- Separate dev and test splits so the judge prompt is not overfit
- True positive and true negative rates, not raw agreement
- Cohen's kappa corrects for chance agreement
- Re-validate after any judge model or prompt change
Follow-ups the interviewer may ask
- Can you correct the judge's pass rate using these numbers? Hint: true pass rate = (observed pass rate + TNR - 1) / (TPR + TNR - 1).
- What if two humans disagree with each other? Hint: measure human-human agreement first; the judge cannot be expected to beat it.
- How many labeled cases do you need? Hint: enough failures to estimate TNR, so aim for at least 30 to 50 failing cases.
Common mistakes
- Reporting 90% agreement on a dataset where 90% of cases pass
- Tuning the judge prompt on the same cases used to report its accuracy
- Letting the judge define the rubric instead of a domain expert
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Validating Your Judge: TPR, TNR, and the Human Ceiling.
One question is not enough for the interview.
This is 1 of the 150 questions in the AI Engineering (LLM applications) pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (180 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (junior)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (mid)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (senior)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (staff)
Free questions from the other packs
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)