Acceptance criteria for AI systems: when is an LLM application good enough?

Criteria as failure types

A classic acceptance criterion says what the right output is. For an AI system it works better to say what must not go wrong, because there are many right answers and a countable number of ways to fail. Typical criteria for a customer service assistant: it states a wrong price, amount or date; it contradicts the policy in the source document; it answers a question outside its scope instead of declining; it misses a request to speak to a person; it reveals another customer's data.

Find these failure types by reading real or pilot outputs with a domain expert first. Shreya Shankar and colleagues found that people refine their criteria while they grade outputs, which they call criteria drift. So read a sample, agree the criteria from what you found, and freeze them before the acceptance test.

Error analysis for LLM apps

Severity and thresholds, set before testing

Give each failure type a severity and a required pass rate. IMDA's starter kit for testing LLM applications says each organisation must determine its own thresholds and should set them before testing, with a threshold of zero for egregious harms. Setting them first keeps the numbers from being argued over after the results are in.

These figures illustrate the format; they are not a recommendation. The right requirement depends on what a failure costs you, whether a person would catch it, and how the process performed before the AI system.

A criterion that allows no failure cannot be proven by testing alone. A clean run only bounds the failure rate: with 300 cases and none failing, the 95% interval for the failure rate still reaches up to 1.3%. Critical criteria therefore also need controls outside the model, such as access rules that make the failure impossible.

Measure each rate with an interval

A pass rate from a test set is an estimate. If 90 of 100 cases pass, the true rate for this kind of question could be noticeably lower or higher. The Wilson score interval, which behaves well with small samples and with rates near 100%, puts the 95% interval for 90 of 100 at 82.6% to 94.5%.

That gives three outcomes for each criterion. The criterion is met if the lower end of the interval reaches the requirement. It is not met if the upper end is below it. If the requirement lies inside the interval, the test has not decided yet, and more cases will narrow it. With a requirement of 85%, 90 of 100 is undecided; at exactly the same pass rate, 196 cases would settle it, and in whole cases 180 of 200 does.

The check treats cases as independent. If several cases come from the same document or conversation, the real interval is wider, and our guide to eval sample sizes shows how to account for that.

Eval sample sizeAI system testing

Repeated runs and consistency

Run each case several times at production settings. A case that passes on one run and fails on the next is not a flaky test to delete; it tells you how often a user would see the failure. Count the case as the unit: five runs of 100 cases are not 500 independent cases.

For a system that acts, such as an agent that issues refunds, consider requiring that every run of a case passes. In the original 2024 τ-bench paper, gpt-4o solved about 61% of retail tasks in a single trial, but fewer than 25% of tasks in all eight trials. A requirement on the single-run rate would have hidden that.

Putting the criteria into a contract

If an agency or vendor builds the system, agree the failure types, the requirements and the way they are measured before the build starts, and keep the test set on your side. The vendor can get a sample to develop against, while the cases used for acceptance stay unseen, so the system is not tuned to the exact test. At acceptance, run the set through the interface your users will have, and record for each failure the input, the output and the criterion it broke.

Agree up front what happens after an undecided result: usually a larger test run, or a fixed period of supervised use with a sample checked by hand. How formal acceptance works legally depends on your type of contract, so check that with your legal adviser; this is not legal advice.