Error analysis for LLM applications: read your traces before writing evals

Why generic scores mislead

Many teams with an AI product in pilot already log every call but have no definition of failure for their own product. Off-the-shelf scores for helpfulness or similarity look like a shortcut, but nobody has checked them against real failures. Husain and Shankar warn that generic evaluations waste time and create false confidence when used as quality measures, and OpenAI's evaluation guide lists overly generic metrics such as BLEU among its anti-patterns.

A support assistant can score well on relevance while quoting a returns period that changed last month. Error analysis replaces the guess with observation: you read real traces, write down in plain words what went wrong, group the notes into failure types and count them. Husain and Shankar describe the method as adapted from qualitative research, where the two stages are called open coding and axial coding. Anthropic's engineers add that you cannot tell whether graders work without reading transcripts and grades from many runs.

How many traces, and who reads them

Husain and Shankar recommend reviewing at least 100 traces and carrying on while you are still learning. The stopping rule matters more than the number: stop when new traces no longer reveal new failure types or change existing ones, which qualitative researchers call theoretical saturation.

Their page on error analysis adds a second number. When an AI agent suggests labels, annotate at least 30 traces yourself before looking at its suggestions, within a working pool of roughly 100 diverse traces. So the review is about 100 traces read to saturation, and 30 is the minimum labelled by hand before an agent helps.

Sample recent traces across every channel, language and main intent. If you add traces users flagged, report them separately so they do not inflate failure rates.

Husain and Shankar are explicit that a domain expert should read: the support lead or claims handler whose work the system imitates. An engineer alone will spot malformed JSON and miss a wrong refund rule. Production traces often contain personal data, so agree pseudonymisation before the session.

One label and one note per trace

For each trace the expert records pass or fail and, for a failure, a sentence or two in their own words. Husain and Shankar observe that annotators on rating scales drift to middle values to avoid hard decisions, and suggest binary labels to learn what bad looks like. OpenAI also recommends pass or fail when a model grades.

In long agent or pipeline traces, note only the first upstream failure. Upstream errors cause downstream ones, Husain and Shankar point out, so fixing an early wrong tool call often removes the confused answer at the end as well. A useful note lets someone else find the same problem again:

• Took order 4812 from the email subject instead of 4821, so the delivery date belongs to someone else's order
• Says the returns period is 14 days; the policy in the knowledge base says 30
• Customer asked twice for a person; the assistant kept offering FAQ links

When several people label

With several labellers, have all of them label the same 20 or 30 traces first. Husain and Shankar recommend measuring agreement with a chance-corrected statistic such as Cohen's kappa. Discussing the disagreements matters more than the number: most come down to an unwritten rule, such as whether a correct answer that addresses a German customer with the informal du is a failure. Write it down. Shankar and colleagues found that criteria shift during grading, which they call criteria drift, so revisit early labels later.

Group the notes and count

Then group the notes into failure types. Husain and Shankar call this the most important step and end it by counting the failures in each category. Good categories are narrow enough to suggest a fix and broad enough to hold several notes. An LLM may draft the grouping; the expert decides.

A hypothetical example

Take a hypothetical German-language support assistant for an online appliance shop. The support lead read 100 traces from four weeks of chat. The last 20 still produced a new failure type, so she read 20 more; they added none, and she stopped at 120. 41 traces failed, each counted under its first failure:

Wrong order lookups and outdated rules make up 20 of the 41 failures, about half, and a generic helpfulness score would have missed both because the answers read well. At this sample size, neither 11 against 9 nor 9 against 7 is a meaningful ranking.

Fix it, check it in code, or build a judge

Not every failure type needs an evaluator. For each row, decide what stops it and how you would notice its return.

1. 1

   Fix it when the cause is obvious

   Remove the outdated document from the sources; repair the context that drops earlier messages. Keep a few original traces as regression cases, and build no judge for a problem that is gone.

2. 2

   Write a code check when the failure shows in the data

   Whether the order tool received the customer's number, whether the reply avoids the informal du, whether an email contains Markdown: a few lines of code test each. Husain and Shankar advise favouring deterministic checks over LLM judges in CI, and Anthropic's documentation calls code-based grading the fastest and most reliable option, though it lacks nuance.

3. 3

   Build an LLM judge only where judgement is needed

   Recognising that a customer wants a person without saying so takes judgement. Build one judge for that failure type that returns pass or fail, as Husain and Shankar recommend, and compare its verdicts with the expert's labels before trusting it. Start with no more than two or three judges.

In the example, that makes three code checks, one judge and two fixes, each traced to failures the expert saw.

What the taxonomy turns into

The labelled traces are your first test set: failed traces with category and note are cases with known failures, and passing traces catch fixes that break ordinary conversations. Anthropic considers 20 to 50 tasks from real failures a good start, so 41 failures are enough to begin.

The categories become your metrics: the rate of each failure type, measured by its check or validated judge, in place of one quality score. Repeat the review after a change of model, channel or product line. Once the product produces exceptions daily, a running review queue keeps the taxonomy current; we describe one for process agents separately.

Where error analysis goes wrong

• Reading only complaints, which hides quiet failures such as a wrong delivery date the customer believed
• Engineers labelling alone, or an LLM labelling before a person has labelled at least 30 traces
• Categories so broad that they suggest no fix, such as hallucination
• Stopping at a fixed number while new types still appear, or reading long after the list stopped changing

A practical first step: export 100 recent traces, sit with the person who knows the work, write one sentence per failure, and compare the list with the metrics you had planned.

LLM evaluation for AI productsTurning AI agent production failures into tests