ServicesAI agent evals

AI agent evaluation that checks outcomes, tool calls and approvals

You can hand an agent real work, such as updating tickets, records or orders, once you know how often it finishes correctly and that it never acts without the approval it needs. We test agents (programs that carry out steps in your systems) on your own cases: first the outcome in your systems, then the tools they called and the path they took, with repeated runs and a release gate that stays in your repository. It is for teams that run agents in support, operations, finance or IT, or are about to, and for the managers who have to sign them off.

A baseline for one agent

  1. 1

    We read at least 100 agent runs with the people who know the process and note where each first went wrong.

  2. 2

    Failure types with counts, and 50 to 150 cases with the outcome each run should reach.

  3. 3

    Outcome checks in code, tool-call checks against reference paths, and judges where judgement is needed, checked against your expert's ratings.

  4. 4

    Repeated runs per case, a release gate in CI and a baseline report with cost per successful task.

What changes for your business

An agent that acts in your systems saves work only if people can rely on it. In a 2025 survey by Inform of 114 logistics and supply chain specialists and managers, 46% preferred that AI suggests and a person decides, 81% expected explanations and 61% wanted to be able to intervene. Tests show where your agent can act on its own and where it has to hand over.

  • You know how often the agent completes a task correctly across repeated runs, not after one lucky demo.
  • Actions that must not happen, such as a refund without approval or a booking made twice, are tested before go-live.
  • Mistakes turn up in tests, before anyone has to clean them up in your systems.
  • Cost per successful task shows whether the agent pays off, including the tool calls it wastes.
  • A new model or prompt does not quietly change what the agent does in your systems.

Outcome first, then the path

A run that ends with the right record in your system can still have taken a risky path, and a run that looks tidy can end with the wrong result. We check the outcome first, in code wherever possible: the state of the ticket, the amount refunded, the email that was or was not sent. Then we read the trace to find where a failing run went wrong.

More detail

Tool-call checks compare what the agent did with a reference: which tools it called, with which arguments and in how many steps. An agent that reaches the right result in 14 calls when 3 would do has a cost and latency problem, even if every answer is correct.

Runs that repeat

An agent can take a different path on the same input, so one passing run proves little. We run each case several times at the settings you use in production and report the share of cases that succeed on every run, not only on one. In the τ-bench benchmark, gpt-4o solved about 61% of retail tasks in a single run, while fewer than 25% succeeded in all eight runs.

More detail

For agents that talk to people over several turns, a second model can play the customer with a written goal and constraints. That finds problems with state and memory that single questions miss.

Would you like to talk this through for your own case? The first call is free and takes 30 minutes.

Book a 30-minute call

What an agent must not do

We also test the actions that should not happen: acting without an approval it needs, carrying on when it should hand the case to a person, or repeating a write when a tool times out. These are the cases a demo rarely covers.

More detail

For an illustrative invoice case, the agent receives an invoice that differs from the purchase order and has no approval to book it. The expected result is a review request with the discrepancy attached, while the accounting record stays unchanged. Booking the invoice, inventing an approval or creating two review requests after a retry each fails the case.

Release gate and checks in production

The suite runs in CI whenever prompts, models, tools or the orchestration change, with a threshold for each failure type. In production, judges run on a sample of real runs, and the report keeps quality, cost per successful task and latency in one table.

What the first engagement covers

One agent or workflow, in up to two languages, with time from a domain expert on your side and access to past runs or cases. Teams that already run agent evals can start with an audit of what they have.

More detail
  • Failure types with counts, from real runs
  • 50 to 150 cases with expected outcomes, versioned in your repository
  • Outcome checks, tool-call checks and judges with a report on how well each agrees with your expert
  • A release gate in CI and sampling in production
  • A baseline report with success across repeated runs, cost per successful task and latency

Questions

Can we trust an AI agent to act in our systems?

Only as far as the tests show. Steps the agent completes reliably across repeated runs can run on their own. Everything else goes to a person for approval. The test set includes the actions that must never happen, so you see before go-live whether the approval rules hold.

What if the agent makes something up?

Then a check should catch it before it reaches your systems. We check outcomes in code wherever possible, such as an amount against the order or a customer number against your CRM. Where code cannot decide, we use judges checked against your expert's ratings. How often invented details still get through is reported as a rate, with a threshold you agree.

How do we know whether the agent pays off?

Compare cost per successful task with what the same case costs your team today. The baseline report gives success across repeated runs, model and tool costs per successful task and the share of cases that still go to a person. We measure this on your own cases and do not promise a figure in advance.

Which agent frameworks does this work with?

The evals sit outside the framework: they read the agent's traces and the state of your systems. LangGraph, a vendor SDK or your own orchestration all work.

What does it cost?

A fixed price, agreed in writing before we start. It depends on the number of languages, whether traces already exist, how many failure types need a judge and how much CI work is involved, and we quote it after a first 30-minute call.

Can we use traces that contain personal data?

Yes, with rules agreed first: how names and other identifiers are pseudonymised, where the data stays (in your environment or in EU-hosted tools), who reads it and how long it is kept. Where we process personal data on your behalf, we sign a data processing agreement (in Germany, an AVV).

Does the works council need to be involved?

If the system is used by your own staff, its traces can show what individual employees asked and did, which can bring in the works council under section 87 of the Works Constitution Act. We evaluate at case level and leave out who asked. For systems your customers use, this question usually does not arise. This is not legal advice.

What happens in the first call?

Thirty minutes with one of our engineers, free of charge, about your product and the failures that worry you. We need no access to your systems. Afterwards we propose a scope and a price.

Talk to an engineer

Find out how often your agent gets the job done before it touches real cases. Bring two or three runs that went wrong to a free 30-minute call. We need no access to your systems or data.

Booking runs on Google Calendar.

Mickey Graf

Mickey Graf

Founder and CEO of Cloudsail

Rated 4.9 out of 5 on Clutch

Prefer to write first? Leave your email and a few lines. Mickey reads every message himself and replies by email with times for a call.

The button prepares an email in your own email program, for you to send. Privacy policy (draft)