How to test AI systems: a test strategy for LLM applications, RAG and agents

Testing AI systems is not the same as testing with AI

Search for AI testing and most results are about using AI to generate or maintain tests for ordinary software. This guide is about the other direction: testing software whose behaviour comes from a language model. ISTQB separated the two in 2026. Version 2.0 of its Certified Tester AI Testing syllabus concentrates entirely on testing AI-based systems and now covers generative AI and large language models; content about using AI for testing was removed.

The system under test is the application, not the model. A support chatbot is a model plus a system prompt, retrieval over your documents, tools, filters and the code around them, and a failure can start in any of those parts. Singapore's IMDA therefore separates output testing, which treats the app as a black box, from component testing inside the pipeline, in its starter kit for testing LLM applications.

AI system testing

Why one expected result stops working

A classic test case compares the output with an expected result. A question like "Can I return a sale item after 30 days?" has many correct answers in different words, and a few wrong ones that look similar. Exact matching fails good answers and lets through bad ones that happen to use the same words. This is the test oracle problem: deciding whether an output is correct needs a method of its own.

There are three usable oracles. Code, for anything with a fixed answer: an amount, a date, a schema, a tool call, a cited document that exists. A person with domain knowledge, who labels a sample by hand. And an LLM judge, a second model that answers one narrow pass or fail question and is only as good as its agreement with that person. Use them in that order, and validate every judge before you trust its numbers.

Validating an LLM judge

Test levels: deterministic and statistical

Much of an AI system can still be tested the old way. Split the suite by whether a check has one right answer.

Keep the deterministic part fast and strict: a failure there is a bug. Treat the statistical part as measurement: report a rate, compare it with the threshold and read the failing cases.

Techniques that carry over from the CT-AI syllabus

The CT-AI syllabus lists adversarial, metamorphic, drift, A/B and back-to-back testing among its model testing techniques, and exploratory testing and red teaming for generative AI. For an LLM application they look like this.

• Metamorphic tests: ask the same question in different words, languages or orders and expect the same facts. You need no expected answer, only consistency.
• Back-to-back tests: run the current and the candidate version on the same cases and look at the cases that changed. This is the core of regression testing for prompt and model changes.
• Adversarial tests and red teaming: prompt injection hidden in documents or emails the system reads, requests outside its scope, attempts to get at other users' data.
• Drift tests: rerun a fixed set on a schedule, because documents change and a model alias can start pointing to a new version.
• Exploratory testing: a tester with domain knowledge talks to the system for an hour and writes down every odd answer. It finds failure types nobody thought to put in a test set.

If your organisation follows ISO/IEC TS 42119-2, published in November 2025, the same techniques fit its application of the ISO/IEC/IEEE 29119 test process to AI systems.

Regression testing for LLM applications

Where the test cases come from

The best cases come from real use: conversations, tickets or tasks the system has already handled, read by someone who knows the domain. Hamel Husain and Shreya Shankar recommend starting with this kind of error analysis rather than writing evaluators up front, because the failure types only become clear once you read outputs. Shankar and colleagues call the effect criteria drift: grading outputs is how people work out their criteria.

That does not conflict with fixed acceptance criteria. Read a pilot sample first, agree the criteria and thresholds from what you found, then freeze them for the acceptance test. Before launch, when there is no traffic yet, start with cases your experts write and synthetic cases along explicit dimensions such as topic, customer type and difficulty. If cases contain personal data, pseudonymise them before they go into a repository.

IMDA suggests a minimum of 30 to 50 prompts for a narrow topic and around 100 to 200 for a wider one. Whether that is enough depends on how close the measured pass rate is to your threshold, which the acceptance check in our guide to acceptance criteria calculates.

Acceptance criteria for AI systemsError analysis for LLM apps

Who owns the quality of an AI system

Three roles meet here. The domain expert decides what a correct answer is and labels cases. Engineering owns the deterministic tests, the judges and the gate in CI. QA owns the test strategy, the acceptance criteria and the release report, and keeps asking what is not covered yet. In a small team one person may hold two of these roles. What matters is that the domain expert is not left out, because nobody else can say whether an answer is right.

What the AI Act asks of testing

For high-risk AI systems, Article 9 of the AI Act requires testing against metrics and probabilistic thresholds defined in advance, as part of risk management, and Article 15 requires appropriate levels of accuracy, robustness and cybersecurity. After the Digital Omnibus, published on 24 July 2026, these requirements apply from 2 December 2027 for high-risk systems under Annex III and from 2 August 2028 for those under Annex I. Most chatbots and internal assistants are not high-risk, but the same evidence (criteria, cases, results and decisions) is what customers and auditors tend to ask for. This is not legal advice.