Which documents should your RAG assistant use?

Collect real questions first

The tempting first step is to point an indexer at a SharePoint site or a wiki space and see what comes back. The result is an index of everything, drafts and superseded procedures included, and when an answer is poor nobody can say whether the right document was missing, outdated or buried.

Microsoft's guidance for RAG projects gathers content and test questions in parallel, because the questions must be answerable from the content and the content must answer the relevant questions. We ask the team that will use the assistant for 30 to 50 questions they handled recently, taken from tickets, chat threads or email. Real questions expose the vocabulary, the edge cases and the documents that matter.

Add a few questions the documents should not answer, so the tests can tell a confident wrong answer from a correct refusal.

A source inventory with one row per source

For each candidate source, record a few facts. The rest of the project depends on them, and a spreadsheet is enough.

• Owner: the person or team who can say whether the content is current and who answers when it is wrong
• System and location: where the documents live, such as a wiki space, a document library or a ticketing system
• Authority: whether the source is the approved version, a working draft or informal notes
• Version signal: how you tell the current document from an older one, such as a status field, a date or a naming rule
• Access rules: who may read it in the original system, and whether a connector can read those rules
• Format and condition: text, scanned PDF, spreadsheet or slides, and whether text can actually be extracted
• Change rate: how often documents are added, edited or withdrawn

A source without an owner should not go into the first index. When it produces a wrong answer, nobody corrects the document, and the assistant repeats the error until someone notices.

Trace each question to the passage that answers it

For every question, find the passage that answers it and note the document, section and version. Microsoft's guide describes the same record for a RAG test set: the query, the supporting text with its location, and a valid answer. Starting with a small sample by hand is a direct way to learn what your documents can support.

Each question ends in one of four states.

1. 1

   Answered in one approved source

   Ready for testing. Record the passage as its expected evidence.

2. 2

   Answered, but in conflicting sources

   Two documents disagree, often an old and a new version. Someone has to decide which one is authoritative before indexing, or retrieval decides for you.

3. 3

   Answered only in an informal place

   The answer lives in a chat thread or a personal note. Either an owner writes it into an approved document, or the question stays out of scope for now.

4. 4

   Not answered anywhere

   This is a gap in the documentation, and no retrieval setting will fix it. The assistant should decline, and the gap goes to the owner of that subject.

A hypothetical audit for an internal IT helpdesk

Suppose an IT team wants an assistant for employee questions and collects 40 recent tickets. The figures are invented for illustration.

Twenty-two questions trace to a current page in the IT wiki. Seven trace both to the wiki and to an older PDF handbook that still sits on a shared drive and contradicts the wiki on VPN setup. Five are answered only in the team's chat channel. Six are not answered anywhere, mostly about a laptop model introduced last quarter.

The decision follows from these rows. The wiki goes in. The PDF handbook stays out until its owner withdraws it or corrects the VPN chapter. The five chat answers become two new wiki pages, written by the person who gave them. The six laptop questions become test cases in which the assistant must say it has no documented answer.

Checks on the files themselves

Once the in-scope sources are known, open the files. Some problems break retrieval long before chunk size or model choice matter.

• Scanned PDFs without a text layer need OCR or an image-aware extraction path before a text index can use them, or they stay out
• Headers, footers, tables of contents and disclaimers repeat on every page and crowd out real content
• Long tables lose their header row when they are split, so the values below it lose their meaning
• Identifiers such as part numbers or error codes need exact matching, which search by meaning alone handles poorly

None of these has to be solved during the audit. Write them down, because each one changes how ingestion is built and what it costs.

When the audit says you need less than RAG

Sometimes the useful material is small: a handful of approved policies that rarely change. A September 2024 Anthropic article suggested that a knowledge base under roughly 200,000 tokens, about 500 pages, could go straight into the prompt, with prompt caching to cut cost and latency. Test whether that holds for the context window and pricing of your chosen model. Either way, the audit's questions and expected passages become the test set.

The opposite result also occurs. If most questions end as gaps or conflicts, the first project is documentation work, because an assistant built on those sources would mostly decline or contradict itself.

A practical next step: collect 30 real questions this week and trace ten of them by hand.

RAG development