ServicesRAG evaluation

RAG evaluation: how often your document assistant is right, and why it is wrong when it is

Staff and customers can rely on an assistant that answers from your documents once you know how often it is right, whether it cites the right source and whether it keeps to people's access rights. We test RAG systems (answers drawn from your own documents, with the source shown) on your own questions and documents, scoring the search for the right passage and the answer separately, so when a number drops you know what to fix. The test suite stays in your repository. Typically we test knowledge assistants over SharePoint, file shares, manuals or contracts, run in-house or bought in, in English, German and Polish.

A baseline for one assistant

  1. 1

    We collect real questions and read the answers with someone who knows the documents.

  2. 2

    For each question, the passages that should answer it, plus questions the documents cannot answer.

  3. 3

    Retrieval scored on its own, then answers checked for faithfulness to the source, with judges calibrated on your expert's ratings.

  4. 4

    A release gate for changes to chunking, embeddings, prompts or models, and a baseline report.

What changes for your business

In Bitkom's 2026 survey, 22% of companies that use AI use it for internal knowledge management, up from 11% the year before. A knowledge assistant only saves search time if people trust its answers enough to stop checking every one of them. Testing on your own questions gives them, and management, a reason to.

  • You know how often the assistant answers correctly and cites the right document, measured on your own questions.
  • Nobody is shown content they have no right to see: questions from people without access are part of the test.
  • Outdated answers are caught when a newer document has changed the rule.
  • Questions your documents cannot answer get a clear "not in the documents" instead of an invented answer.
  • Changes to search settings or the model are checked before your staff see them.

Retrieval and answers, scored apart

We label which passages should answer each question and measure retrieval with recall at k and the rank of the first relevant passage. Then we check whether each answer stays faithful to what was retrieved and cites it correctly. When a number drops, you know which half to fix.

More detail

An illustrative test asks which expenses the current travel policy allows. The expected result retrieves the current policy the user may access, answers from its relevant passage and cites that version. An outdated policy is a retrieval failure. A correct passage followed by an unsupported allowance is an answer failure. We record the two outcomes separately.

The questions a demo skips

Most of the failures worth finding sit in cases a demo never tests.

More detail
  • Questions your documents cannot answer, where the assistant should say so
  • Answers that changed in a newer document
  • Questions from people without access to the source document
  • Questions in German or Polish about documents written in English, and the other way round
  • Tables, spreadsheets and exact identifiers such as part numbers

Would you like to talk this through for your own case? The first call is free and takes 30 minutes.

Book a 30-minute call

Changes that need a test run

Chunk size, embedding model, reranker, prompt and the model itself each change retrieval or answers in ways that are hard to predict. The suite runs in CI whenever one of them changes, with separate thresholds for retrieval and for answers, so a better answer score cannot hide worse retrieval.

What the first engagement covers

One assistant over one document collection, in up to two languages, with time from someone on your side who knows the documents. Teams that already score their RAG system can start with an audit.

Questions

How do we know whether an answer is correct?

For each test question, someone who knows the documents marks the passage that should answer it. We then check whether the assistant found that passage, whether its answer says what the passage says and whether the citation points to it. The result is a rate for each, measured on your questions, not a general impression.

Can the assistant show people documents they are not allowed to see?

It can if permissions are not carried through from SharePoint or the file share into the search. That is why the test set includes questions asked by people without access to the source, and any answer that reveals the content counts as a failure with a threshold of zero. Your data protection officer sees which cases were tested and with which results.

Is it worth testing an assistant that already works well in the demo?

A demo uses questions the documents answer well. The failures that cost trust sit elsewhere: questions with no answer in the documents, rules changed in a newer version, tables and part numbers, and people without access. If staff see a few wrong answers, they start checking every answer again and the time saved is gone, so it is cheaper to find those cases first.

Can you evaluate an assistant another vendor built?

Yes, as long as we can send it questions and read its answers and, ideally, the passages it retrieved. Without the retrieved passages we can score answers but not retrieval.

Do you need access to our documents?

We work inside your environment or on a copy you choose, with the same access rules as the original system. How personal data in questions or documents is handled is agreed before we start.

What does it cost?

A fixed price, agreed in writing before we start. It depends on the number of languages, whether traces already exist, how many failure types need a judge and how much CI work is involved, and we quote it after a first 30-minute call.

What happens in the first call?

Thirty minutes with one of our engineers, free of charge, about your product and the failures that worry you. We need no access to your systems. Afterwards we propose a scope and a price.

Talk to an engineer

Find out how often your document assistant is right before your staff stop trusting it. Bring two or three answers it got wrong. The call takes 30 minutes and needs no access to your systems or data.

Booking runs on Google Calendar.

Mickey Graf

Mickey Graf

Founder and CEO of Cloudsail

Rated 4.9 out of 5 on Clutch

Prefer to write first? Leave your email and a few lines. Mickey reads every message himself and replies by email with times for a call.

The button prepares an email in your own email program, for you to send. Privacy policy (draft)