ServicesRAG evaluation
RAG evaluation: how often your document assistant is right, and why it is wrong when it is
Staff and customers can rely on an assistant that answers from your documents once you know how often it is right, whether it cites the right source and whether it keeps to people's access rights. We test RAG systems (answers drawn from your own documents, with the source shown) on your own questions and documents, scoring the search for the right passage and the answer separately, so when a number drops you know what to fix. The test suite stays in your repository. Typically we test knowledge assistants over SharePoint, file shares, manuals or contracts, run in-house or bought in, in English, German and Polish.
A baseline for one assistant
- 1
We collect real questions and read the answers with someone who knows the documents.
- 2
For each question, the passages that should answer it, plus questions the documents cannot answer.
- 3
Retrieval scored on its own, then answers checked for faithfulness to the source, with judges calibrated on your expert's ratings.
- 4
A release gate for changes to chunking, embeddings, prompts or models, and a baseline report.
What changes for your business
In Bitkom's 2026 survey, 22% of companies that use AI use it for internal knowledge management, up from 11% the year before. A knowledge assistant only saves search time if people trust its answers enough to stop checking every one of them. Testing on your own questions gives them, and management, a reason to.
- You know how often the assistant answers correctly and cites the right document, measured on your own questions.
- Nobody is shown content they have no right to see: questions from people without access are part of the test.
- Outdated answers are caught when a newer document has changed the rule.
- Questions your documents cannot answer get a clear "not in the documents" instead of an invented answer.
- Changes to search settings or the model are checked before your staff see them.
Retrieval and answers, scored apart
We label which passages should answer each question and measure retrieval with recall at k and the rank of the first relevant passage. Then we check whether each answer stays faithful to what was retrieved and cites it correctly. When a number drops, you know which half to fix.
The questions a demo skips
Most of the failures worth finding sit in cases a demo never tests.
- Questions your documents cannot answer, where the assistant should say so
- Answers that changed in a newer document
- Questions from people without access to the source document
- Questions in German or Polish about documents written in English, and the other way round
- Tables, spreadsheets and exact identifiers such as part numbers
Changes that need a test run
Chunk size, embedding model, reranker, prompt and the model itself each change retrieval or answers in ways that are hard to predict. The suite runs in CI whenever one of them changes, with separate thresholds for retrieval and for answers, so a better answer score cannot hide worse retrieval.
What the first engagement covers
One assistant over one document collection, in up to two languages, with time from someone on your side who knows the documents. Teams that already score their RAG system can start with an audit.
Questions
How do we know whether an answer is correct?
For each test question, someone who knows the documents marks the passage that should answer it. We then check whether the assistant found that passage, whether its answer says what the passage says and whether the citation points to it. The result is a rate for each, measured on your questions, not a general impression.
Can the assistant show people documents they are not allowed to see?
It can if permissions are not carried through from SharePoint or the file share into the search. That is why the test set includes questions asked by people without access to the source, and any answer that reveals the content counts as a failure with a threshold of zero. Your data protection officer sees which cases were tested and with which results.
Is it worth testing an assistant that already works well in the demo?
A demo uses questions the documents answer well. The failures that cost trust sit elsewhere: questions with no answer in the documents, rules changed in a newer version, tables and part numbers, and people without access. If staff see a few wrong answers, they start checking every answer again and the time saved is gone, so it is cheaper to find those cases first.
Can you evaluate an assistant another vendor built?
Yes, as long as we can send it questions and read its answers and, ideally, the passages it retrieved. Without the retrieved passages we can score answers but not retrieval.
Do you need access to our documents?
We work inside your environment or on a copy you choose, with the same access rules as the original system. How personal data in questions or documents is handled is agreed before we start.
What does it cost?
A fixed price, agreed in writing before we start. It depends on the number of languages, whether traces already exist, how many failure types need a judge and how much CI work is involved, and we quote it after a first 30-minute call.
What happens in the first call?
Thirty minutes with one of our engineers, free of charge, about your product and the failures that worry you. We need no access to your systems. Afterwards we propose a scope and a price.
Related services
AI system testing
Before go-live, acceptance tests on your own cases tell you how often your chatbot, assistant or agent is right, in terms your QA team already uses.
AI evals engineering
Evals built from your own traces show whether each change made your AI product better or worse, with a release gate that stays in your repository.
AI agent evals
Find out how often an agent gets the job done before it acts in your systems, by checking outcomes, tool calls and how reliably runs repeat.
Free AI usage review
One page for management on where your AI money goes, whether it pays off and what to change, from exports you choose to share. Free, no system access.
On-premise AI
AI on offers, contracts and customer files, with the data staying in your company: own servers or own cloud account, tested before go-live.
Data integration for AI
Answers from file shares, SharePoint, ERP and tickets in seconds, for assistants and agents, with permissions carried through and tested.
AI agents and process automation
Agents that take on the routine steps of complex cases and pass decisions to people, with a cost per handled case.
From the blog
Retrieval or generation?
A diagnostic procedure for finding out whether a wrong RAG answer started in ingestion, search, reranking or the model, using labelled evidence and per-question traces.
RAG citation quality
How to test whether the passages an assistant cites actually support each claim it makes, with a claim-level check, a worked example and acceptance criteria.
Multilingual RAG testing
How to test whether an assistant finds the same evidence when people ask in English, German or Polish, with a parallel test set, a language matrix and terminology fixes.
Validating an LLM judge
How to test an LLM judge against hand-labelled examples, report its true positive and true negative rates and correct the pass rate it reports.
Talk to an engineer
Find out how often your document assistant is right before your staff stop trusting it. Bring two or three answers it got wrong. The call takes 30 minutes and needs no access to your systems or data.
Booking runs on Google Calendar.
Prefer to write first? Leave your email and a few lines, and we reply with times.
Look for the draft in your email program
Send it from there, and an engineer replies in person with times for a 30-minute call. If no draft has opened, write to us at miki@cloudsail.com.