AI agent evals that check outcomes, tool calls and repeated runs

A run that ends with the right record in your system can still have taken a risky path, and a run that looks tidy can end with the wrong result. We check the outcome first, in code wherever possible: the state of the ticket, the amount refunded, the email that was or was not sent. Then we read the trace to find where a failing run went wrong.

Tool-call checks compare what the agent did with a reference: which tools it called, with which arguments and in how many steps. An agent that reaches the right result in 14 calls when 3 would do has a cost and latency problem, even if every answer is correct.

Agents are not deterministic, so one passing run proves little. We run each case several times at the settings you use in production and report how often every run succeeds, not only whether one did. In the τ-bench benchmark, gpt-4o solved about 61% of retail tasks in a single run, while fewer than 25% succeeded in all eight runs.

For agents that talk to people over several turns, a second model can play the customer with a written goal and constraints. That finds problems with state and memory that single questions miss.

We also test the actions that should not happen: acting without an approval it needs, carrying on when it should hand the case to a person, or repeating a write when a tool times out. These are the cases a demo rarely covers.

The suite runs in CI whenever prompts, models, tools or the orchestration change, with a threshold for each failure type. In production, judges run on a sample of real runs, and the report keeps quality, cost per successful task and latency in one table.

One agent or workflow, in up to two languages, with a domain expert from your side for a few hours a week and access to past runs or cases. Teams that already run agent evals can start with a two-week audit of what they have.

• Failure types with counts, from real runs
• 50 to 150 cases with expected outcomes, versioned in your repository
• Outcome checks, tool-call checks and judges with a report on how well each agrees with your expert
• A release gate in CI and sampling in production
• A baseline report with success across repeated runs, cost per successful task and latency