Can you trust an LLM judge? Test it against human labels

Give each judge one failure to look for

Husain and Shankar recommend using an LLM judge as a classifier that returns Pass or Fail for one specific error, not as a general quality score. In practice that means one judge per failure type found in error analysis. For a hypothetical support assistant: one judge for refunds promised outside policy, another for answers that miss the customer's question.

Binary verdicts also make the human labels more reliable. Their FAQ notes that people labelling on a scale drift to middle values to avoid hard decisions, and OpenAI's evaluation guidance also recommends pass/fail or pairwise comparison.

Ask the judge for a short written critique before the verdict. Anthropic's documentation for Claude advises letting a grader reason before it scores. When judge and expert disagree, the critique helps you see whether the rubric is unclear or the judge misread the conversation. A judge prompt for one failure needs four things:

• The failure in the expert's words, including what does not count
• A few labelled passes and failures from the train split
• Only the parts of the conversation needed to decide
• Critique first, then Pass or Fail

Label by hand, then split three ways

The reference is a set of examples a domain expert has labelled Pass or Fail for that failure, each with a one-sentence reason. Anthropic's agent evals guide likewise wants model graders closely calibrated against human experts. Husain and Shankar suggest splitting the labels three ways.

1. 1

   Train: 10 to 20%

   Examples that may appear in the judge prompt. Pick clear cases on both sides.

2. 2

   Dev: 40 to 45%

   For refining the prompt: run the judge, read where it disagrees with the expert, adjust, repeat.

3. 3

   Test: 40 to 45%

   Scored once, at the end, for the numbers you report; nobody reads it while tuning.

Their FAQ warns that repeated tuning on dev overfits: if the test result is clearly worse than the last dev result, the prompt has probably learned the dev examples rather than the failure.

At a 5% failure rate, a random 80-example test split holds about four failures, and a rate measured on four cases means almost nothing. Add failures found in error analysis until the test split has about 30 of each class. TPR and TNR are measured within one class, so this enrichment does not distort them as long as the added failures resemble the ones production produces, but the share of failures in the labelled set then says nothing about production.

Report TPR and TNR, not accuracy

Husain and Shankar define the true positive rate as the share of actual failures the judge catches and the true negative rate as the share of good outputs it correctly passes. The positive class here is the failure. Accuracy, the share of verdicts matching the expert, mixes the two and is dominated by the larger class.

This example is our own reasoning, with invented numbers. Take 1000 conversations from the support assistant, 50 of them with the failure. Judge A passes everything. Judge B catches 40 of the 50 failures and wrongly flags 95 of the 950 good conversations.

Judge A wins on accuracy and never finds a failure. Judge B is usable, but its 90% TNR has a price: 95 of its 135 flags are false alarms. Husain and Shankar warn that with rare failures even a small false-alarm rate creates many unnecessary reviews. They give no universal targets and suggest setting acceptable levels by the consequences for your application. Missing a refund promise that breaks policy costs more than rereading a polite answer.

Correct the pass rate before you report it

A judge with known error rates reports a biased pass rate. Shankar's judgy library corrects for that bias. With failure as the positive class, as above, the estimate is: (observed pass rate + TPR - 1) / (TPR + TNR - 1).

judgy puts TNR in the numerator because its examples code Pass as 1, making a pass the positive class. Same arithmetic, swapped names. Check your code's convention first: confusing the two rates gives a wrong answer whenever they differ.

Continuing the hypothetical example: the expert labels 200 conversations, enriched with failures, split 40, 80 and 80. Of the 80 test examples, 30 contain the failure. The final judge catches 27 of the 30 (TPR 90%) and passes 46 of the 50 good ones (TNR 92%). In production it passes 1700 of 2000 conversations, an observed pass rate of 85%.

The corrected estimate is (0.85 + 0.90 - 1) / (0.90 + 0.92 - 1) = 0.75 / 0.82, about 91.5%. This judge is stricter than the expert, so the raw score understates the assistant by over six points. Report 85% now, switch to a more lenient judge later, and the improvement you see comes from the judge.

The correction assumes TPR + TNR is greater than 1, meaning the judge beats chance; judgy states this. Also, TPR and TNR are themselves estimates from 30 and 50 examples: had the true TPR been 80%, the same observed rate would correct to about 90.3%. judgy puts a bootstrap interval around the estimate, and that interval belongs in the report.

Test for the biases the research found

Zheng and colleagues studied LLM judges on MT-Bench and Chatbot Arena in 2023. Strong judges agreed with human preferences over 80% of the time, about as often as humans agree with each other. The paper also documents systematic biases. The models tested (GPT-4, GPT-3.5 and Claude-v1) are old, so treat the figures as reasons to test your own judge, not as estimates of it.

• Position bias. Most judges favoured the first answer in a pairwise comparison. With the answers swapped, GPT-4 stayed consistent 65.0% of the time, GPT-3.5 46.2% and Claude-v1 23.8%. The authors' remedy is to call the judge twice with the order swapped and count a win only when the same answer wins both times; anything else is a tie.
• Verbosity bias. In a repetitive list test, which lengthens answers without adding information, Claude-v1 and GPT-3.5 were fooled 91.3% of the time and GPT-4 8.7%. OpenAI's evaluation guidance also advises controlling for response length.
• Self-preference. GPT-4 favoured its own answers with a 10% higher win rate, Claude-v1 with 25%. The authors caution that their data was too limited to tell whether this is a real self-enhancement bias. We still avoid letting a model judge itself.

The first two are cheap to test on your dev split: swap the order, or pad a few good answers with repetition, and see whether the verdict moves.

Expect the criteria to drift, and re-validate

Shankar and colleagues describe criteria drift: people need criteria to grade outputs, but grading outputs is how they work out the criteria. Expect your expert's definition of the failure to change in the first labelling rounds. When it does, rewrite the rubric and relabel earlier examples against it. Otherwise the test split mixes two definitions and the TPR you report belongs to neither.

Then pin the judge. Record the full, dated model identifier, the prompt version, the rubric version and the TPR and TNR measured with that combination. Re-run the validation on the test split whenever any of them changes. A test split reused again and again slowly turns into a dev split, so top it up with newly labelled examples from recent production.

The validation report is short: the failure checked, the model identifier, the counts of failures and good outputs in the test split, TPR, TNR, the corrected pass rate with its interval, and the date. Attach it wherever the judge's numbers appear.

LLM evaluation for AI productsError analysis for LLM applicationsHow many test cases an LLM eval needs