Two Numbers, One Character Apart, Pointing Opposite Ways
A vendor benchmark reads 90 percent. A support automation that runs the same workflow every day fails one user in three. Both numbers can be true about the same agent, because they are measuring different things, and the difference is one character.
pass@k is the probability that at least one of k attempts succeeds. pass^k is the probability that all k attempts succeed. The first rises as you add attempts, because you only need one winner. The second falls as you add attempts, because one loss is enough to spoil the set. A benchmark leaderboard usually quotes something close to the first. A user who depends on the agent lives inside the second. When the two get confused, a team ships against the wrong number and finds out in production.
This lesson is the arithmetic the rest of the chapter rests on. The agent loop lesson showed that reliability compounds and named the levers; here we derive the formula both directions, turn it into a per-step engineering budget you can test against, and separate the interventions that move it from the ones that only feel like they do. No new mechanics beyond that. The whole subject is one equation and its consequences.
Read the two definitions side by side and the shape of every later decision is already visible. pass@k measures whether the agent can do the task with enough tries: a capability question. pass^k measures whether it does the task every time: a consistency question. They bracket the single-attempt rate from opposite sides, and the right one to quote depends entirely on which question your product is actually asking. The divergence above is the whole lesson in one picture. A vendor sitting on the green curve and a user living on the red one can both describe the same agent honestly and disagree by fifty points, because they are reading the same p through two different exponents pointed in opposite directions.