Reviewing agent exceptions without a queue nobody can finish

What counts as an exception

Routine escalations are part of normal operation and are handled in the operational queue. The review is for events that suggest the agent, its tools or its rules are wrong:

• Automatic outcome checks that failed
• Cases a person corrected after the agent completed them
• Downstream reversals, such as a cancelled posting or a refund
• Complaints that mention a case the agent handled
• Escalations with an unclassified reason, and clusters of tool failures

A fixed budget, not a full read

Split exceptions by severity. Anything where money moved wrongly, data left the organisation or a mandatory rule was broken is reviewed every time. Everything else is grouped by a signature, such as case type plus failing tool plus error type, and a few cases from each group are sampled within a fixed weekly budget. Arize's guide to agent evaluation suggests a similar division: collect clearly defined failures automatically and send ambiguous or high-risk ones to a person first.

A hypothetical budget: two hours a week from one process expert and one engineer together, enough for roughly fifteen to twenty grouped reviews if each takes six to eight minutes. Because groups are sampled evenly rather than by volume, do not read the share of errors in the sample as the production error rate. Weight each group by its real volume, or report the groups separately.

A triage record that leads somewhere

Keep the record short enough to fill in during the review. Five fields are usually enough:

• What happened compared with what should have happened, stated as record values
• Where it went wrong: input data, a tool, the model's reasoning, a gap in the rules or a gap in the test set
• Severity, on the same scale used for sampling
• One owner by name
• One action with a checkable end state, such as a regression case that passes

This borrows from Google's SRE guidance on postmortems, which asks for a single owner, a priority and a tracking item for each action, end states that can be verified, and a blameless tone. It also quotes a leader's remark that a postmortem without subsequent action is indistinguishable from no postmortem.

From exception to regression case

1. 1

   Reduce

   Keep the minimum data that reproduces the failure and remove personal data that does not affect it. Arize gives the same advice: confirm and deduplicate failures rather than copying every one.

2. 2

   Confirm the expected outcome

   The process owner states what the correct result is. If they cannot, the action is to clarify the rule, not to write a test.

3. 3

   Measure before and after the fix

   A deterministic check, such as a tool or validator test, should fail before the fix and pass after it. An end-to-end agent case may fail only in some runs, so run it several times on the current release, record the failure rate, and compare the rate for the outcome that matters after the fix rather than expecting one guaranteed pass.

4. 4

   Keep it, then retire it

   The case joins the regression suite, which Anthropic's guide to agent evals says should pass at close to 100 percent. Retire it when the process it describes changes.

Because the fix was written while looking at this case, it belongs in the development and regression sets, not in a held-out set. To check that the fix generalises, add a few fresh cases of the same kind to the held-out set.

Keeping the queue finite

Low-severity items that are not sampled within a set number of weeks are closed as unreviewed and counted, never silently dropped. Trends replace individual reading for high-volume groups: if a signature keeps growing, it gets a review slot. The SRE postmortem guidance recommends storing records in a searchable repository with metadata and treating recurring incidents as a sign of slow follow-through. The same holds here: a signature that keeps returning means its action item is not working.

Other warning signs are action items open for weeks, a regression suite that only grows because no case is ever retired, and reviews run by engineers without anyone from the process. A good first step is to pull last month's corrected and reversed cases, group them by signature and see how many distinct groups there really are.