Testing whether an assistant's citations support what it says

What a valid citation proves

Citation features have become more reliable at the mechanical level. Anthropic's Citations API, for example, returns the exact cited text with character, page or block locations, and the documentation states that citations are guaranteed to contain valid pointers to the provided documents. That removes one failure: a quote that does not exist.

It does not tell you whether the quoted sentence supports the claim next to it. A real passage can be attached to a sentence that overstates it, drops a condition or comes from somewhere else entirely. Research on generated citations measures this gap. In the ALCE benchmark from 2023, the authors reported that even the best models of the time lacked complete citation support about half of the time on one long-form dataset. Models have changed since, but the measurement problem has not.

Checking claims one at a time

The method borrows from how groundedness metrics work. Ragas, for example, scores faithfulness by extracting the claims in a response, checking whether each can be inferred from the retrieved context and dividing supported claims by all claims. For citations you narrow the evidence: each claim is judged only against the passages it cites, not against everything that was retrieved.

1. 1

   Split the answer

   Break it into claims that can each be true or false. Keep a number together with the conditions that qualify it, such as customer type, dates or exclusions, and split only assertions that are genuinely independent.

2. 2

   Pair each claim with its citations

   Use the exact cited text, not the whole document. When a claim cites several passages, judge it against them together, because a legitimate claim can combine two sources. A claim with no citation is recorded as uncited.

3. 3

   Give a verdict

   Supported, partially supported, not supported, contradicted, or supported by a different retrieved passage than the one cited.

4. 4

   Keep the verdicts

   Store them per question so that a later change can be compared claim by claim.

A hypothetical answer, checked

The example below is invented. The assistant answers a returns question with three sentences and two citations; a third relevant passage was retrieved but not cited.

Question: Can business customers return opened hardware?   (hypothetical)

Answer sentence                                      Cited   Verdict
1 Customers can return hardware within 30 days.      [1]     partial: [1] gives 30 days for
                                                             consumers, 14 days for business
2 Refunds go to the original payment method.         [2]     supported
3 Opened software licences cannot be returned.       [1]     wrong citation: stated in [3],
                                                             which the answer did not cite

Claims fully supported by their own citations: 1 of 3
Release decision: fail (claim 1 drops the condition that answers the question)

Every link in this answer opens a real document, and a reader skimming it would see nothing wrong. The first claim is the serious failure: the cited policy gives a different period for business customers, which is exactly what the user asked about.

Who does the judging

Start with people. Have two domain experts label the claims for a calibration set of a few dozen answers, and resolve their disagreements; the disagreements usually show where your own rules are vague. Then a language model can judge at scale with the same rubric, and you measure how often it agrees with the human labels before trusting its scores.

Keep grounding and correctness apart. Microsoft's RAG evaluation guide gives the example of a response that is grounded in the context and still factually wrong, because the model combined two separate facts into a false one. A citation check catches unsupported claims. It does not catch a source document that is itself out of date.

Acceptance criteria and their limits

Write the criteria before you see the scores, and let them depend on risk. For a hypothetical HR policy assistant, a team might agree on the following:

• No contradicted claims in answers about pay, leave or termination
• Every number and date traceable to a cited passage
• An agreed minimum share of fully supported claims, set from the calibration set
• Uncited claims allowed only for general wording, never for rules or figures

The approach has costs. Judging claims with a model adds inference cost and latency, however the calls are organised, and the judge makes mistakes of its own. Product constraints also matter: Anthropic documents that its citations feature cannot be combined with structured outputs in the same request. When the criteria fail, look at retrieval first; a claim cannot be supported by a passage that never reached the model.