ServicesAI evals engineering

AI evals and LLM evaluation: know whether each change made your AI better or worse

Your team can change prompts, models and tools without quality quietly dropping, and move to a cheaper model only when the numbers show it is good enough. We build evaluations (repeatable tests of an AI system on fixed cases) from your own traces and cases, together with the people who know what a good answer looks like, and leave them in your repository as a release gate that runs on every change. We work with product and engineering teams that ship AI products, agents or RAG systems. Our engineers sit in Germany and Poland, and we write tests in English, German and Polish.

A baseline for one product

  1. 1

    We read at least 100 of your traces with a domain expert and note what went wrong.

  2. 2

    The notes become failure types with counts, and a versioned test set of 50 to 150 cases.

  3. 3

    Code checks first, then judges for what code cannot check, each tested against human labels.

  4. 4

    A release gate in CI, sampled checks in production and a baseline report you keep.

What changes for your business

Every change to your AI product comes with a number: better, worse or within the noise. That matters more as AI moves into what companies sell. In Bitkom's 2026 survey, 20% of companies that use AI put it into their own products and services, up from 12% the year before.

  • Releases go out with evidence: a prompt or model change that makes one kind of failure worse is stopped before customers see it.
  • A model update at your provider does not quietly change answers, because the same suite runs again.
  • Model choices rest on cost per successful task, so a cheaper model is used wherever it passes your tests.
  • Product owners and management get one table with quality, cost and latency instead of a feeling about the last release.
  • The suite belongs to you and keeps working when you change tools or vendors.

Tracing is not the same as evaluating

In LangChain's State of Agent Engineering report, published in 2026 and based on responses from over 1300 professionals working with agents, 89% had some form of observability in place, yet 29.5% ran no evaluations at all, and quality was the barrier to production they named most often. A trace tells you what happened in one conversation. An evaluation tells you how often a particular failure happens across all of them, and whether a change moved that number.

More detail

Generic scores such as helpfulness or an overall rating out of ten rarely help. They move up and down without telling anyone what to fix. We start from the failures your system actually produces.

Reading traces before writing tests

First we sample at least 100 traces or past cases and read them with someone who knows the work, such as a support lead or a claims handler. For each one we write a short note on the first thing that went wrong. We keep reading until new traces stop showing new kinds of failure.

More detail

The notes are then grouped into a short list of failure types and counted. The counts decide the work. Some failures are fixed directly in a prompt or a retrieval setting, some get a check in code, and only those that need judgement get an LLM judge.

Would you like to talk this through for your own case? The first call is free and takes 30 minutes.

Book a 30-minute call

A test set that belongs to you

The test set comes from your traces and past cases, tagged by intent, difficulty and language, and versioned next to your prompts so every score can be reproduced. Where a product has no users yet, we generate cases along explicit dimensions instead of simply asking a model for test questions, and your experts add the cases they already know are hard.

More detail

For RAG systems we label which passage should answer each question, so retrieval and the answer are scored separately. For agents we check the outcome first and use the trace to find where a run went wrong.

An illustrative acceptance case starts with a customer requesting a refund for an order that cannot be found. The expected reply asks for the missing order reference and makes no refund promise. We check those conditions separately. An invented order number fails even if the reply reads well. Your domain expert labels acceptable answers before candidate models are compared.

  • Cases from real traffic, tagged and versioned in your repository
  • Rare but costly cases kept in the set even when they are a small share of traffic
  • Questions in each language your users write in
  • Cases for every failure type found in the error analysis

RAG evaluation

When an assistant answers from your documents, a wrong answer has two possible causes: the right passage was never retrieved, or the model misused a passage it had. We label which passages should answer each question and measure retrieval on its own, with recall at k and the rank of the first relevant passage, before we score whether the answer is faithful to what was retrieved.

More detail

The set also covers questions your documents cannot answer, questions whose answer changed in a newer document and questions from people without access to the source, because a demo rarely tests any of them.

Checks tested against your expert

We write checks in code first: valid JSON, required fields, a SQL query that runs, a refund amount within policy. They cost almost nothing and give the same answer every time. Where a failure needs judgement, such as an answer that misstates a contract clause, we build a judge that returns pass or fail for that one failure.

More detail

Each judge is tested against labels from your domain expert on cases it has not seen before. We report how many real failures it catches and how many good answers it wrongly fails, so you know how far to trust its numbers. Judges are tested again whenever the judge model, its prompt or the rubric changes.

Release gates and checks in production

A small suite, mostly checks in code, runs in CI on every change to prompts, models, tools or retrieval settings. Each failure type has its own threshold, so a gain in one area cannot hide a regression in another. In production, judges run on a sample of live traffic, with alerts per failure type.

More detail

The baseline report puts quality, cost per successful task and latency in one table, with confidence intervals. On 100 cases, 80% and 83% are usually within the noise of each other, and we say so instead of declaring a win.

What the first engagement covers

The baseline covers one AI product or workflow, such as a support assistant, an internal RAG tool or a process agent, in up to two languages. It needs time from a domain expert on your side and access to traces or past cases. Before we start, we agree how that data is handled, including pseudonymisation of personal data.

More detail

Teams that already run evals can start with the audit described below.

  • Failure types with counts
  • A versioned test set in your repository
  • Checks in code, judge prompts and a report on how well each judge agrees with your expert
  • Configuration for the CI gate and for sampling in production
  • A baseline report with quality, cost and latency, and a ranked list of fixes

Ongoing maintenance after the initial build

An eval suite decays when the product changes and the tests do not. Ongoing maintenance is a separate agreement with its own scope. A named Cloudsail engineer owns the work at a cadence agreed with you, for example per release or on a fixed schedule, and reports what changed in the suite.

More detail

Offline evals and production monitoring answer different questions, so we report them separately. A failure seen in production or in support tickets becomes a candidate eval case. We work with the minimum data needed, agree access rules with your security and privacy owners, and anonymize or synthesize examples where raw user data should not be stored.

  • Adding cases from production failures, support tickets and product changes.
  • Rechecking judge calibration when models, prompts or rubrics change.
  • Retiring stale cases and keeping held-out sets separate.
  • Comparing candidate models, prompts and tools before you switch.

An eval audit if you already run evals

We check your existing evals against your traces and against the decisions you make with them, and leave a written list of what to keep, fix or drop.

More detail
  • How the failure types were found, and whether anyone read real traces
  • Whether each judge was tested against human labels, with its true positive and true negative rates
  • Whether the test set reflects your traffic, your languages and the rare cases that cost the most
  • Whether changes in the scores are larger than the uncertainty of the test set
  • Whether CI thresholds and production alerts match the failures that matter to you

Tools, and the EU AI Act

We work with the platform you already use, such as Langfuse, LangSmith or Braintrust, or with none. Test sets and judges stay in plain formats in your repository. Evaluation tools have changed owners often over the past two years, and OpenAI made its hosted Evals platform read-only on 31 October 2026 and will shut it down on 30 November 2026, so your evals should not depend on any one of them.

More detail

Under the EU AI Act, the duties to test against defined metrics, declare accuracy and monitor after launch fall on providers of high-risk systems. Under the timetable adopted in 2026, they apply from 2 December 2027 for the high-risk uses listed in Annex III and from 2 August 2028 for AI in products covered by Annex I. For most chatbots, internal assistants and RAG tools, the Act sets no such duty. Evals are good engineering either way, and they can supply much of the test evidence a provider will need if a use case becomes high-risk. We do not assess conformity or give legal advice.

Questions

What does management get from evals?

A baseline report that shows how often the product fails, by kind of failure, next to cost per successful task and latency, with the uncertainty stated. After that, every release comes with the same numbers, so decisions about launches, model switches and spend rest on measurements.

Aren't the model provider's benchmarks enough?

Public benchmarks measure general tasks, not your documents, your customers' questions or your rules. A model that scores higher there can still do worse on your cases. Your own evals answer the question that matters for your product, and they keep answering it when the provider changes the model.

Will evals slow down our releases?

The gate in CI is mostly checks in code, which cost almost nothing and give the same result every time. Judges run on a sample. A release is stopped only by a measured regression on a failure type you care about, and that is what the gate is for. Releases that pass go out as before, with numbers behind them.

Is this LLM evaluation, LLM evals or AI evals?

The same work under different names. LLM evaluation and LLM evals usually mean testing a model's answers; AI evals is the wider term that also covers agents and retrieval. We call the whole job AI evals engineering, and describe AI agent evals and RAG evaluation on their own pages.

Is this the same as AI testing or QA for AI?

If you mean testing an AI system rather than using AI to test software, mostly yes: the methods are the same. Our AI system testing page describes them as a fixed acceptance test with criteria, thresholds and a report, for a QA lead or for accepting a system a vendor built. This page is about building and running the suite over time.

How is this different from your coding agent evaluations?

Coding agent evaluations compare tools, models and settings on changes your engineers have already merged. This offer evaluates the AI product itself: the assistant, agent or RAG system your customers or staff use.

Do we need a particular eval platform?

No. We use the one you have, or plain files and your CI. Everything we build stays in your repository.

What if our product has no users yet?

We start from cases your experts write down and from synthetic cases generated along explicit dimensions, and read the first real traces as soon as the system is in use.

What does it cost?

A fixed price, agreed in writing before we start. It depends on the number of languages, whether traces already exist, how many failure types need a judge and how much CI work is involved, and we quote it after a first 30-minute call.

Can we use traces that contain personal data?

Yes, with rules agreed first: how names and other identifiers are pseudonymised, where the data stays (in your environment or in EU-hosted tools), who reads it and how long it is kept. Where we process personal data on your behalf, we sign a data processing agreement (in Germany, an AVV).

Does the works council need to be involved?

If the system is used by your own staff, its traces can show what individual employees asked and did, which can bring in the works council under section 87 of the Works Constitution Act. We evaluate at case level and leave out who asked. For systems your customers use, this question usually does not arise. This is not legal advice.

What happens in the first call?

Thirty minutes with one of our engineers, free of charge, about your product and the failures that worry you. We need no access to your systems. Afterwards we propose a scope and a price.

How are the initial build and maintenance scoped and priced?

Separately. The initial build is a scoped project with agreed deliverables: datasets, checks, judges, CI integration and a first release report. Maintenance is a separate agreement with its own scope and cadence. Cost depends on how many features and failure types are covered, how much domain review is needed, data access and privacy constraints, and how often your product changes. We propose scope in writing after the engineering call.

Talk to an engineer

Not sure whether your last change made your AI product better or worse? Bring two or three conversations where it got things wrong. We need 30 minutes and no access to your systems or data.

Booking runs on Google Calendar.

Mickey Graf

Mickey Graf

Founder and CEO of Cloudsail

Rated 4.9 out of 5 on Clutch

Prefer to write first? Leave your email and a few lines. Mickey reads every message himself and replies by email with times for a call.

The button prepares an email in your own email program, for you to send. Privacy policy (draft)