Which past cases belong in an agent test set, and which stay out of tuning

Which past cases qualify

A past case is useful only if you know the right result and can check the agent against it. Use cases that meet all of these conditions:

• The case is closed and its final outcome is recorded, such as an invoice approved, held or rejected with a reason
• Two people who know the process would reach the same verdict from the same documents, the test Anthropic uses for an unambiguous eval task
• The decision can be made from information that existed at intake. If a clerk resolved the case after a phone call that left no trace, the agent cannot be expected to match it
• You can reconstruct the inputs: the documents, the relevant records as they were at the time and the policy that applied

Recorded outcomes are not automatically correct. Before a case enters the set, a domain expert confirms or corrects the result and notes the reasoning. If capable agent versions fail a case across many runs, check the case and its grader before blaming the agent. Anthropic's guide treats a near-zero pass rate as a common sign of a broken task, but the case may also be genuinely hard.

Draw a mix that matches the work

Sample most of the set in proportion to real volume by case type. Then add cases on purpose, because the expensive failures are rare and random sampling misses them.

• Routine cases by type, in roughly the proportions of the last few months
• Known exceptions such as partial deliveries, currency mismatches, duplicate submissions and documents in another language
• Cases where the correct action is to stop and pass the case to a person
• Cases where a tool or data source returns nothing, an error or stale data
• Later, every confirmed production failure, reduced to the minimum needed to reproduce it

Anthropic recommends balanced sets that test both when a behaviour should happen and when it should not, so a set that only rewards completion does not push the team towards an agent that never escalates.

A hypothetical example: for an invoice-matching agent, a first set of 80 cases might hold 44 standard matches, 12 deviations within tolerance, 8 deviations outside tolerance that must be held, 8 invoices without an order that must be escalated, 4 duplicates and 4 cases where the ERP lookup fails. Your proportions come from your own case history.

Because the deliberate additions are oversampled, the overall pass rate of this set is not the rate you should expect in production. Report each category separately, or weight the categories by their real frequency before combining them.

The part nobody tunes against

Split the cases before the first prompt is written. The development set is for reading transcripts, adjusting prompts, choosing examples and fixing tools. The held-out set is only scored and summarised by category; nobody edits the agent while looking at its cases.

Machine learning teams apply the same discipline to test data. Google's Machine Learning Crash Course notes that test sets wear out with repeated use and that duplicates between training and test data inflate results.

How to split depends on what the held-out set should measure. If the agent must handle suppliers or document templates it has never seen, keep all cases of a supplier or template on one side, so the score is not helped by familiarity. If production will mostly bring known suppliers, cases from the same supplier on both sides are a fair test, but remove exact or near-exact copies of the same document. These actions count as tuning against the held-out set:

• Copying a held-out case, or a close paraphrase of it, into the prompt as an example
• Adding a rule to the prompt after reading why a specific held-out case failed
• Iterating on two agent versions by checking again and again which one passes more held-out cases

Once a held-out case has influenced the agent, move it into the development set and replace it with a fresh case from recent work.

What each case record contains

Store each case as a record the evaluation harness can run without asking anyone. This sketch for the hypothetical invoice agent is not a vendor schema.

case_id: inv-0147
split: held_out             # development | held_out
category: price_deviation_outside_tolerance
group_key: supplier-2291    # used when the split is by supplier
source_ref: ERP document 5100231 (pseudonymised copy)
inputs:
  documents: [invoice.pdf, purchase_order.json, goods_receipt.json]
  system_snapshot: snapshots/inv-0147/erp-2026-06-03.json
  policy_version: ap-tolerance-policy v4
expected_outcome:
  action: hold
  reason_code: PRICE_ABOVE_TOLERANCE
must_not:
  - post_invoice
  - change_vendor_master_data
graders:
  - deterministic: action and reason_code match
  - rubric: hold note names the line and the price difference
reviewed_by: AP team lead, 2026-09-18

The fields that matter most are the expected outcome as checkable state, the forbidden actions and the system snapshot from intake. Without the snapshot, a case can start failing because the order was closed in the meantime.

Personal data in past cases

Past cases often contain names, bank details or health information. Keep only what reproduces the decision, pseudonymise identifiers that do not affect the outcome and record each case's origin so it can be removed on request. Whether real cases may be used for testing at all is for your data protection officer to decide.

When the test set misleads you

• It was built from the cases that were easy to label, so the score describes the easy half of the work
• The process or policy changed and old expected outcomes are now wrong, so cases need a fresh review whenever the policy version changes

Give the set an owner and version it with the agent. A practical first step is to pull the last three months of closed cases for one process, ask its owner to mark the cases they would trust as a reference, and see how many remain.

AI agents and process automation