Agentic AI interview question · Agent evals · mid level
What are pass@k and pass^k, and why do they matter for agents?
Free to read in full. This is 1 of the 120 questions in the Agentic AI Engineering interview pack, with 153 diagrams. Get the pack, ₹999 · $20
Why it is asked
These two metrics measure opposite things and are often confused. The interviewer checks the definitions, the math, and which one matches the product.
The answer
pass@k is the probability that at least one of k attempts succeeds; pass^k is the probability that all k attempts succeed. The first measures whether the agent can do a task, the second whether it does it every time.
For one task with success rate p and independent attempts:
- pass@k = 1 - (1 - p)^k
- pass^k = p^k
With p = 0.75:
- k = 1: both are 75%.
- k = 2: pass@2 = 93.75%, pass^2 = 56.25%.
- k = 4: pass@4 = 99.6%, pass^4 = 31.6%.
- k = 8: pass@8 is almost 100%, pass^8 = 10.0%.
Why it matters: pass@k came from code generation, where a person or a test suite can pick the one working sample. A customer support agent does not get to pick. Each customer gets one run, and a business with thousands of customers sees every failure. pass^k was introduced with tau-bench by Sierra in 2024 to measure exactly this consistency.
Two practical notes:
- On a benchmark, compute the metric per task and then average across tasks. Averaging p first and then raising it to k gives the wrong answer, because tasks have different success rates.
- Estimate from n runs per task where n is at least k. With c successes out of n, the standard unbiased estimate (right on average) of pass^k is C(c, k) / C(n, k), where C(a, b) is the number of ways to choose b items from a.
Quote pass@k for "can it solve this with retries and a checker"; quote pass^k for "will users get a consistent result".
What the interviewer listens for
- pass@k: at least one of k succeeds, 1 - (1 - p)^k
- pass^k: all k succeed, p^k
- At p = 0.75, k = 8: pass^k is only 10%
- Compute per task, then average
- pass^k matches user-facing reliability
Follow-ups the interviewer may ask
- What does a big gap between pass@1 and pass^8 tell you? Hint: the agent can do the task but is inconsistent; look for flaky steps.
- Is pass@k useful at all for agents? Hint: yes, when a verifier, such as unit tests, can pick the good attempt.
Common mistakes
- Quoting a pass@k number as if it were the user's success rate
- Computing p^k from the average success rate across tasks
Written by Roni Das, who builds distributed systems and AI systems for a living.
Learn it properly
This is taught step by step in the course lesson Agent Reliability Math: pass@k, pass^k, and Why 95 Percent Fails.
One question is not enough for the interview.
This is 1 of the 120 questions in the Agentic AI Engineering pack
Each question has why it is asked, a full answer, what the interviewer listens for, the follow-ups, the common mistakes and a diagram (153 diagrams in all). A PDF you keep, junior to staff level.
India: UPI, cards or netbanking through Razorpay. Elsewhere: PayPal. Download at once from My packs.
More free questions from this pack
Free questions from the other packs
- What do temperature and top-p control, and what values would you use for data extraction versus brainstorming? (AI Engineering)
- What is hybrid search, and how do you merge keyword and vector results with Reciprocal Rank Fusion? (AI Engineering)
- Users ask "list all X" questions and RAG returns only five items. Why, and what do you do? (AI Engineering)
- How do you check that an LLM judge agrees with human reviewers? (AI Engineering)
- Cut the cost of an LLM feature serving 10 million requests a day by 60% without hurting quality. Walk through your plan. (AI Engineering)
- Why are LLMs bad at counting letters or reversing a word? (GenAI and LLM)
- Write out scaled dot-product attention with the shape of every tensor. (GenAI and LLM)
- How does speculative sampling keep exactly the target model's distribution? (GenAI and LLM)
- What is GRPO, and how do verifiable rewards train reasoning models? (GenAI and LLM)
- Design a system that answers questions from 300-page contracts accurately. (GenAI and LLM)
- What happens, step by step, when an LLM server handles one request? (Inference Engineering)
- Derive the KV cache size formula and work it out for Llama 3 70B. (Inference Engineering)
- What is PagedAttention? (Inference Engineering)
- When does speculative decoding stop helping or make things worse? (Inference Engineering)
- Design a service to serve Llama 3 70B at 10 QPS with latency targets. (Inference Engineering)
- A model scores 95% accuracy offline but users say it is useless. What do you check first? (ML Engineering)
- How do you train on a dataset that does not fit in memory? (ML Engineering)
- A long distributed training job dies at hour 20. How do you make training fault tolerant? (ML Engineering)
- How does request batching on a GPU trade latency for throughput? (ML Engineering)
- An online model metric dropped 8% overnight. How do you debug it? (ML Engineering)
- You need to show delivery times for 20 cities. Would you use histograms, box plots, violin plots or ECDFs, and why? (Data Science)
- A test is 99% accurate and you test positive. What is the chance you have the disease? (Data Science)
- What share of first-time buyers make a second purchase within 30 days? Write the SQL. (Data Science)
- How would you forecast daily orders for the next four weeks, and how do you know the forecast is any good? (Data Science)
- A coupon A/B test shows a small average lift. How do you find which users the coupon actually works for, and who should get it next time? (Data Science)
- Compute the cosine similarity of a = (3, 4) and b = (4, 3) by hand. What does the result mean? (Maths for ML)
- A batch of 32 MNIST images goes into a dense layer with 128 units. What are the shapes, and what is the rule for when two matrices can be multiplied? (Maths for ML)
- A disease affects 1% of people. A test catches 90% of sick people (sensitivity) and wrongly flags 9% of healthy people (false positive rate). You test positive. What is the chance you are sick? (Maths for ML)
- Why do we not use mean squared error for classification with a sigmoid output? Compare the gradients numerically. (Maths for ML)
- Compute scaled dot-product attention by hand for two tokens with d_k = 2. (Maths for ML)
- A customer says: "We want AI for our support team." What do you do in the first meeting? (Forward Deployed Engineer)
- The customer's admin offers you a global admin account "to save time". What do you ask for instead, and why? (Forward Deployed Engineer)
- Your connector writes tickets into the customer's system. A request timed out, the connector retried, and now there are duplicate tickets. How do you make the writes safe to retry? (Forward Deployed Engineer)
- The customer wants RAG over 10 million documents, all behind their firewall. Size it for me on the whiteboard. (Forward Deployed Engineer)
- One deployment of your AI platform must serve 50 business units of the same enterprise. How do you design multi-tenancy? (Forward Deployed Engineer)