Regression testing for LLM applications: what to run when a prompt or model changes

What counts as a change

In ordinary software, a regression run follows a code change. In an LLM application, behaviour also changes when someone edits a prompt in a configuration file, re-indexes documents or adjusts a setting, and when a provider updates the model behind an alias. Put all of these under version control, or at least in a change log, and trigger the suite on each of them.

• Prompt and system prompt edits, including those made outside the code repository
• A new model or model version, and alias updates by the provider
• New or changed documents in the retrieval index, and changes to chunking or embeddings
• New tools, and changed tool descriptions or permissions
• Settings such as temperature, maximum output length or the number of retrieved chunks

What goes in the regression suite

Anthropic's engineering team distinguishes capability evals, which measure progress on tasks the system cannot yet do well, from regression evals, which should have a nearly 100% pass rate. For a regression suite that means cases the current version handles, covering every failure type you have seen and fixed, plus the critical criteria from acceptance. When a bug is fixed, its case joins the suite, so the bug cannot quietly return.

Order the checks by cost. Deterministic checks run first and fail fast: schemas, required fields, tool arguments, citations that point to existing documents, refusals for out-of-scope requests. Then come the LLM judges, each validated against a domain expert's labels, for the criteria that need judgement. Keep the suite small enough to run on every change; Hamel Husain and Shreya Shankar describe CI sets as small and purpose-built, in many cases 100 examples or more.

Validating an LLM judgeAI evals engineering

Compare versions case by case

Run the old and the new version on the same cases, which is back-to-back testing in the terms of the ISTQB CT-AI syllabus, and look at the cases that flipped. A new prompt can raise the average and still break five cases that used to work, and those five may be the ones a customer notices. Report three numbers: cases newly failing, cases newly passing and the net change.

Whether a net change is real needs a paired test, not a comparison of two averages. Our guide to eval sample sizes shows the method with a worked example: on 100 cases, 8 fixed and 5 broken is still within noise.

Eval sample size

When to stop a release

• Any new failure in a critical criterion stops the release, whatever the average does.
• A deterministic check that newly fails stops the release, because it is a bug.
• For statistical criteria, stop when the pass rate falls clearly below the acceptance requirement, with the upper end of its interval below it, and review by hand when it is close or when cases flipped in both directions.
• A judge that changed, with a new model or a new prompt, is validated again against the expert's labels before its numbers count.

Model changes deserve a second step. Before you switch production to a new model, run it in shadow mode on a sample of live traffic, with its answers logged but not shown, and compare them with the current version. Shadow runs catch the questions your test set does not contain yet.

Keeping the suite fast and affordable

A suite of 300 cases with three runs each and two judges per run means 900 answers and 1800 judge calls per change. Run the full suite on model and prompt changes and a smaller smoke set on minor edits, reuse stored answers from the unchanged version as the baseline, and move a judge to a cheaper model only after you have validated that model against the expert's labels.

AI system testing