Finding out whether retrieval or generation caused a wrong answer

Two questions behind every wrong answer

A wrong answer looks the same to the user whatever caused it. For the team, there are two separate questions. Did the evidence needed for the answer reach the model? And given that evidence, did the model use it correctly? Teams that only score final answers end up tuning prompts for failures that started in the index.

Evaluation tools reflect the split. LlamaIndex, for example, separates retrieval evaluation, scored with ranking metrics such as hit rate and mean reciprocal rank, from response evaluation, which judges faithfulness, relevance and correctness of the generated answer.

Label the expected evidence

For each test question, record the identifiers of the passages that answer it, and the expected answer. Note whether those passages are alternatives, where any one answers the question, or are all required together. With IDs in place, retrieval scoring becomes cheap and deterministic. Ragas, for instance, offers an ID-based context recall that compares retrieved IDs with reference IDs without calling a language model. Because it divides by all reference IDs, use it for evidence that is jointly required, and count a hit when any one of several alternatives is retrieved.

Then log a trace for every test run. The fields below are the minimum we find useful; names are illustrative and not tied to any framework.

# One trace record per test question (field names are illustrative)
question_id:        q-017
question:           "Which firmware versions need a manual reset?"
expected_evidence:  all_of [release-notes-4.2#known-issues, runbook-reset#scope]
                    # use any_of [...] when either passage alone answers
in_index:           [true, true]        # did ingestion produce these chunks?
candidate_rank:     [3, 41]             # rank in first-stage search results
in_prompt:          [true, false]       # survived reranking and the top-k cut?
answer_verdict:     incomplete          # judged against the expected answer
first_failure:      retrieval (rank 41 > prompt cut at 10)

Find the first failure

1. 1

   Is the passage in the index?

   If not, the cause is ingestion: a parser that skipped a scanned page, a connector that missed a folder, or a document that was never in scope. Fix the pipeline or the source.

2. 2

   Is it among the first-stage candidates?

   If it ranks far down or not at all, look at the query, including any query rewriting step, and the search method. Exact identifiers often need lexical search next to search by meaning, and chunks without context may not match the question.

3. 3

   Did it reach the prompt?

   A passage can be retrieved and still be dropped by reranking or by the cut to the top few chunks. Compare the candidate rank with the cut-off.

4. 4

   Did the model use it correctly?

   Only now is the problem generation. Check whether the answer is faithful to the passages, complete and correct, and whether conflicting passages confused it.

A useful shortcut for the last step is an oracle test. Give the model the labelled passages directly, bypassing search. If it now answers correctly, the defect is upstream. If it still fails, work on the prompt, the model or the source text itself.

A hypothetical diagnosis

Assume a test set of 60 questions with labelled passages, of which 18 get wrong or incomplete answers. The numbers are invented to show how a breakdown guides work. Four expected passages are missing from the index because a batch of scanned PDFs had no text layer. Seven are found but ranked between 20 and 50, while only the top 10 reach the prompt. Two drop out at reranking. Five reach the model and the answers are still wrong; in three of them, two versions of a policy were retrieved together, which is as much a source problem as a model one.

Thirteen of the 18 failures happen before the answer is generated, so changes to the answer prompt could address at most five. Query rewriting and reranker changes count as retrieval work here. The order of work follows: OCR for the scanned batch, then search changes for the seven and the two, then removing or marking the superseded policy, then the remaining generation errors.

Reading retrieval metrics without fooling yourself

Recall at a cut-off is the metric that maps most directly onto this diagnosis, because it asks whether the expected passage was among the chunks the model could see. Anthropic used one minus recall at 20 as the failure rate in its 2024 contextual retrieval experiments. Choose the cut-off your system actually uses; recall at 50 looks good and means little if only 10 chunks reach the prompt.

Several cases make the diagnosis harder. When a question has alternative valid passages, finding any one should count as success, and ordinary recall over all of them will understate it. Questions that need many documents at once, such as counts across a year of reports, are poorly served by top-k retrieval and may need a different design. Labels also age as documents change, and model-based judges vary between runs, so Microsoft suggests target ranges rather than single target values.