Signals that show a live agent is getting costlier or worse

The case is the unit, not the call

A single case can involve several runs, dozens of model calls, a pause for approval and a few minutes of a clerk's time. Token cost per call says little on its own. Cost per handled case, including review time and rework, is the figure the process owner can compare with today's manual cost.

That requires a case identifier on every span and log line. The OpenTelemetry semantic conventions for agent spans define attributes such as gen_ai.conversation.id, gen_ai.agent.version and the token counts gen_ai.usage.input_tokens and gen_ai.usage.output_tokens. The conventions are still in Development status, so pin the version you implement and add your own case ID attribute if a conversation does not map one to one to a case.

A signal sheet for one process agent

This is a starting list for a hypothetical invoice agent. Every item is broken down by case type and agent version.

Outcome signals

• Share of cases completed without a person
• Share escalated, by escalation reason
• Error rate in the sampled human review of completed cases
• Downstream corrections, such as postings later reversed

Cost signals

• Cost per handled case, with model, platform and review time shown separately
• Model calls, tool calls and tokens per case, as percentiles

Health and drift signals

• Tool failures by tool and error type, and the number of cases with an unknown write outcome
• Time from intake to completion, and the age of the oldest case waiting for a person
• The mix of incoming case types compared with the mix the agent was tested on

Agree the thresholds before you need them

Google's SRE workbook recommends expressing service level indicators as the ratio of good events to all events, choosing a small number of them, measuring as close to the user as practical and writing down in advance what happens when the error budget runs out. For a process agent, the user is the process, so measure at the system of record rather than inside the agent.

A hypothetical objective: at least 97 percent of sampled auto-completed invoices are correct over a rolling four weeks. The number is an assumption for illustration; the process owner sets it from the error rate of the current manual process. The policy attached to it might say that if the objective is missed, the affected case type returns to approval until the cause is fixed.

Catching cost creep

Cost per case rarely jumps. It drifts as documents get longer, as the agent loops through more tool calls, as retries increase or as more cases reach a person. Watch percentiles rather than averages; the SRE workbook's monitoring chapter points out that means hide slow requests, and the same is true of expensive cases. The same chapter recommends recording intended changes, such as versions and configuration, so that shifts can be matched to rollouts. For agents, that means the model, prompt and tool versions on every case.

Quality needs a sample, not just a dashboard

Run-health telemetry shows that a run ended, not that the invoice was posted to the right account. Automatic outcome checks against the system of record close part of that gap, and their results can be emitted as metrics with their own alerts. For case types they do not cover, keep a small, regular human review of completed cases. Anthropic's guide to agent evals describes production monitoring as useful but reactive, and recommends combining it with automated evals and manual transcript review so each layer catches what the others miss.

What should alert someone

Alert a person only on conditions they can act on: a tool failing for a whole category of cases, any case with an unknown write outcome, an approval queue older than its target, or an objective at risk of being missed. Slow cost drift belongs in a weekly review, not a page. The SRE workbook also recommends testing alert rules with synthetic data, since a rule that never fires may simply be broken. A practical first step is to take the signal sheet above, mark which numbers you can already produce per case today, and start with those.