How many test cases an LLM eval needs to support a decision

Why 80% against 83% on 100 cases is not evidence

Suppose a new prompt lifts your eval from 80 to 83 passing cases out of 100. It looks like progress, and someone will want to ship it. Before that conversation, put an interval around each number.

We computed the 95% interval for 80 of 100 in two ways. The normal approximation gives a standard error of 4 points and an interval from 72.2% to 87.8%. The Wilson score interval, which behaves better with small samples and rates near 0% or 100%, gives 71.1% to 86.7%. For 83 of 100, Wilson gives 74.5% to 89.1%. The two ranges overlap heavily. This eval cannot tell the prompts apart, and fresh cases could easily reverse the order.

A 100-case suite is still worth having. Husain and Shankar describe CI test sets as small and purpose-built, in many cases 100 examples or more, and a set that size can catch a change that clearly breaks something. It cannot rank two versions a few points apart, which is usually what a prompt or model switch asks of it.

Standard errors, and why grouped questions widen them

Evan Miller's paper on error bars for evals, written at Anthropic, treats an eval as an experiment and starts with the basics: compute the standard error of the mean using the central limit theorem. For pass or fail scores, that is the square root of p × (1 - p) / n, and the 95% interval is the mean plus or minus 1.96 standard errors. This is the normal calculation above, and it assumes every case is an independent draw.

Real test sets often break that assumption. If ten questions ask about the same contract, or forty cases come from one support topic, the model tends to get them right or wrong together. The set then holds fewer independent pieces of evidence than its row count suggests. Miller recommends clustered standard errors whenever questions come in related groups, and reports that they can be more than three times larger than the naive ones. On the DROP benchmark, his clustered standard error was 1.34 against a naive 0.44.

In practice, give every case a group key when you build the set, such as the source document, customer, topic or conversation, and cluster the intervals on it. If the interval widens a lot, the set needs more groups, not more questions per group.

Compare both versions on the same cases

When two prompts or models run on the same cases, comparing their two averages throws information away. Miller recommends inference on the question-level paired differences instead: for each case, did version B do better, worse or the same as version A? Only the cases that flipped carry information about the difference.

A hypothetical support assistant shows how this works. On 100 test cases, prompt A passes 80 and prompt B passes 83. Cross-tabulated, the results look like this.

B newly passes 8 cases and newly fails 5, a net gain of 3. We computed the paired 95% interval for the difference as -4.1 to +10.1 points, against -7.8 to +13.8 if the two runs are treated as unrelated samples. A paired bootstrap that resamples cases gives almost the same range. An exact McNemar test on the 13 flipped cases gives p = 0.58.

At this size, even the cleanest result would not settle it. If B fixed 3 cases and broke none, the exact McNemar p would be 0.25, the chance of three coin flips all landing the same way. Now take a hypothetical 400-case set where A passes 81% and B 84%, with 16 cases newly passing and 4 newly failing. The same 3-point gain then has a paired interval of 0.8 to 5.2 points and p of about 0.01. What decides the question is how many cases flipped and how one-sided the flips were.

How many cases a given difference needs

Turn the question around before you build the set: what is the smallest difference you would act on? Miller gives a power calculation for this. In his worked example, detecting a 3-point difference at 5% significance with 80% power takes about 969 questions, and he suggests that evals should have roughly 1000 questions or more. Halving the difference roughly quadruples the number of cases.

When you cannot collect more cases, run each one several times. Miller shows that with 198 questions, sampling each answer 10 times instead of once lowered the minimum detectable effect from 13.2% to 7.5%. Repeats help less than new cases, because they reduce the noise in the model's answers but not the luck of which questions ended up in the set.

Miller also advises against lowering the temperature just to reduce variance. Our reading: if production runs at a higher temperature, an eval at temperature 0 measures a configuration your users never see. Resample instead.

Repeated runs: pass@k and pass^k

Running each case several times also shows something an average hides: whether the system is consistent. Two metrics summarise repeated trials in opposite ways. pass@k counts a case as solved if at least one of k attempts succeeds. pass^k, introduced with the τ-bench benchmark, counts it only if all k attempts succeed. Anthropic's guide to agent evals uses the same pair.

They answer different questions. pass@k fits work where someone or something picks the good attempt, such as generated code that must pass tests before anyone uses it. pass^k fits a customer-facing agent, where each conversation is another trial and every failure reaches a customer.

τ-bench shows how far apart these can be. In its retail domain, gpt-4o succeeded on about 61% of tasks in a single trial, while pass^8, success on all eight trials, fell below 25%. Part of that spread comes from the simulated customer. τ-bench uses a language model to play the user and samples its replies at temperature 1.0, while the agent runs at temperature 0, so each trial is a differently worded conversation. pass^k there measures how the agent copes with different phrasings, not only its own randomness.

When you run repeats, keep the case as the unit. Five runs of 100 cases are not 500 independent cases. Following Miller's logic, cluster the standard errors on the case.

How we report eval numbers

These are the rules we follow when an eval result has to support a decision.

• Every pass rate comes with its 95% interval and the number of cases behind it, for example 80% (71% to 87%, n = 100)
• Versions are compared on the same cases, with the flips shown: newly passing, newly failing, unchanged. Someone reads every newly failing case
• Results are broken down by slice, such as language, product or case type. One case in a 20-case slice is worth 5 points, so slices get intervals too
• Each case carries a group key, and intervals are clustered on it
• The smallest difference worth acting on is agreed before the run, and the set is sized for it
• The small CI suite catches breakage, and a larger paired run decides the switch

The same logic applies to sampled production traffic. Husain and Shankar recommend tracking confidence intervals for production metrics and investigating when the lower bound crosses your threshold.

LLM evaluation for AI productsTesting a fine-tuned model against the model it would replaceWhat an AI pilot must prove before you fund the next stage