Testing an agent on current work before it can change anything

What shadow mode means here

The idea comes from release engineering. Martin Fowler describes dark launching as calling new back-end behaviour in production without users seeing the result, and parallel running as executing old and new implementations side by side and comparing their outputs. Google's SRE workbook describes traffic teeing, where a copy of production traffic goes to a new version and its responses are discarded, and notes that this is harder to set up for systems with state.

A process agent is exactly such a system. Its value lies in the changes it makes, so shadow mode has to separate reading, which it does for real, from writing, which it only proposes.

Making sure it cannot change anything

• Give the agent credentials that are read-only in the target systems themselves. An instruction in the prompt is not a control
• Replace every write tool with a recorder that stores the full proposed call, its arguments and the case ID
• Block outbound email and messaging at the network or gateway level
• Check reads for side effects: marking a message as read, locking a record, consuming an API quota, writing to an audit log
• Prove the setup with a test: the agent attempts a write in the shadow environment and it must fail

Recording writes has a side effect of its own. Later steps that read back what the agent wrote, such as a new posting number or an updated status, will find nothing. Either give the shadow run an isolated copy of the state it changes, such as a sandbox tenant or a simulated record store that answers later reads consistently, or stop each shadow run at its first write and compare only up to that point. Otherwise later steps fail or diverge for reasons that say nothing about the agent.

Fowler's original purpose for dark launching was to measure extra load before release. The same applies here: an agent reading every case adds traffic to production systems, so agree limits with their owners.

Comparing proposals with decisions

Match each recorded proposal to what a person eventually did with the same case. Because both run on the same cases at the same time, you avoid a weakness the SRE workbook points out in before-and-after comparisons, where a change is confused with differences between weeks. Sort every case into one of these groups:

• Agreement: same action and same key values
• Agent escalated where the person completed the case
• Disagreement, to be judged by a second expert who does not know which side proposed what

Do not assume the person was right. Human decisions are a reference, not ground truth, and some disagreements will turn out to be human errors. Agreement is not proof either, because agent and person can be wrong in the same way, so include a random sample of agreements in the blind review as well.

A worked readout

Hypothetical figures for illustration: over three weeks, an invoice agent shadows 400 cases. Its proposal matches the person's decision in 318, it escalates 52 and it disagrees on 30. A blind expert review of the 30 finds the agent wrong in 19, the person wrong in 8 and 3 cases ambiguous. The same review of a random 40 of the 318 agreements finds 2 where both were wrong. Most of the 19 errors involve one supplier group with credit notes.

In this example, the decision follows from the categories, not from the headline agreement of roughly 80 percent. Credit notes stay with people and get added to the test set. Standard matches, which make up most of the agreement, move to the next stage, where the agent acts with sign-off. With numbers this small, each category needs its own count before anyone draws a conclusion.

Criteria for leaving shadow mode

Write them down before the first shadow case runs, and have the process owner agree to them, in the same way the SRE workbook asks stakeholders to agree on an objective before it is used for decisions. A typical list covers a minimum number of cases per category, an acceptable error rate on adjudicated cases drawn from both disagreements and a sample of agreements, no attempted calls outside the agent's allowed tools, cost per case within the agreed budget and an explanation for each escalation reason. Meeting them earns supervised action, not autonomy.

What shadow mode cannot show

It cannot test the write path: retries, duplicate protection and failures that only appear when a call really changes something. It cannot show how customers or downstream processes react, and it doubles the work while it runs. Anthropic's guide to agent evals notes that A/B tests measure real outcomes but need enough traffic and say little about why. Shadow mode sits between offline evals and that stage. A good first step is to list the write tools your agent would use and check whether each can be swapped for a recorder without changing the agent's code.