LLM evaluation built from your own traces

In LangChain's State of Agent Engineering report, published in 2026 and based on responses from over 1300 professionals working with agents, 89% had some form of observability in place, yet 29.5% ran no evaluations at all, and quality was the barrier to production they named most often. A trace tells you what happened in one conversation. An evaluation tells you how often a particular failure happens across all of them, and whether a change moved that number.

Generic scores such as helpfulness or an overall rating out of ten rarely help. They move up and down without telling anyone what to fix. We start from the failures your system actually produces.

In the first week we sample at least 100 traces or past cases and read them with someone who knows the work, such as a support lead or a claims handler. For each one we write a short note on the first thing that went wrong. We keep reading until new traces stop showing new kinds of failure.

The notes are then grouped into a short list of failure types and counted. The counts decide the work. Some failures are fixed directly in a prompt or a retrieval setting, some get a check in code, and only those that need judgement get an LLM judge.

The test set comes from your traces and past cases, tagged by intent, difficulty and language, and versioned next to your prompts so every score can be reproduced. Where a product has no users yet, we generate cases along explicit dimensions instead of simply asking a model for test questions, and your experts add the cases they already know are hard.

For RAG systems we label which passage should answer each question, so retrieval and the answer are scored separately. For agents we check the outcome first and use the trace to find where a run went wrong.

• Cases from real traffic, tagged and versioned in your repository
• Rare but costly cases kept in the set even when they are a small share of traffic
• Questions in each language your users write in
• Cases for every failure type found in the error analysis

We write checks in code first: valid JSON, required fields, a SQL query that runs, a refund amount within policy. They cost almost nothing and give the same answer every time. Where a failure needs judgement, such as an answer that misstates a contract clause, we build a judge that returns pass or fail for that one failure.

Each judge is tested against labels from your domain expert on cases it has not seen before. We report how many real failures it catches and how many good answers it wrongly fails, so you know how far to trust its numbers. Judges are tested again whenever the judge model, its prompt or the rubric changes.

A small suite, mostly checks in code, runs in CI on every change to prompts, models, tools or retrieval settings. Each failure type has its own threshold, so a gain in one area cannot hide a regression in another. In production, judges run on a sample of live traffic, with alerts per failure type.

The baseline report puts quality, cost per successful task and latency in one table, with confidence intervals. On 100 cases, 80% and 83% are usually within the noise of each other, and we say so instead of declaring a win.

The baseline covers one AI product or workflow, such as a support assistant, an internal RAG tool or a process agent, in up to two languages. It needs a domain expert from your side for a few hours a week and access to traces or past cases. Before we start, we agree how that data is handled, including pseudonymisation of personal data.

If you already run evals, a two-week audit checks whether you can trust them: how the failure types were found, whether the judges were tested against human labels and whether the metrics match the decisions you make with them.

• Failure types with counts
• A versioned test set in your repository
• Checks in code, judge prompts and a report on how well each judge agrees with your expert
• Configuration for the CI gate and for sampling in production
• A baseline report with quality, cost and latency, and a ranked list of fixes

We work with the platform you already use, such as Langfuse, LangSmith or Braintrust, or with none. Test sets and judges stay in plain formats in your repository. Evaluation tools have changed owners often over the past two years, and OpenAI made its hosted Evals platform read-only on 31 October 2026 and will shut it down on 30 November 2026, so your evals should not depend on any one of them.

Under the EU AI Act, the duties to test against defined metrics, declare accuracy and monitor after launch fall on providers of high-risk systems. Under the timetable adopted in 2026, they apply from 2 December 2027 for the high-risk uses listed in Annex III and from 2 August 2028 for AI in products covered by Annex I. For most chatbots, internal assistants and RAG tools, the Act sets no such duty. Evals are good engineering either way, and they can supply much of the test evidence a provider will need if a use case becomes high-risk. We do not assess conformity or give legal advice.