ServicesRAG evaluation
RAG evaluation that scores retrieval and answers separately
When an assistant that answers from your documents gets something wrong, either retrieval never found the right passage or the model misused one it had. Scoring only the final answer hides which. We build RAG evaluations from your own questions and documents, score both halves separately and leave the suite in your repository, in English, German and Polish where your users need it.
Four weeks on one assistant
- 1
Week 1: we collect real questions and read the answers with someone who knows the documents.
- 2
Week 2: for each question, the passages that should answer it, plus questions the documents cannot answer.
- 3
Week 3: retrieval scored on its own, then answers checked for faithfulness, with judges tested against your expert.
- 4
Week 4: a release gate for changes to chunking, embeddings, prompts or models, and a baseline report.
Retrieval and answers, scored apart
We label which passages should answer each question and measure retrieval with recall at k and the rank of the first relevant passage. Then we check whether each answer stays faithful to what was retrieved and cites it correctly. When a number drops, you know which half to fix.
The questions a demo skips
Most of the failures worth finding sit in cases a demo never tests.
- Questions your documents cannot answer, where the assistant should say so
- Answers that changed in a newer document
- Questions from people without access to the source document
- Questions in German or Polish about documents written in English, and the other way round
- Tables, spreadsheets and exact identifiers such as part numbers
Changes that need a test run
Chunk size, embedding model, reranker, prompt and the model itself each change retrieval or answers in ways that are hard to predict. The suite runs in CI whenever one of them changes, with separate thresholds for retrieval and for answers, so a better answer score cannot hide worse retrieval.
What the first engagement covers
One assistant over one document collection, in up to two languages, with someone from your side who knows the documents for a few hours a week. Teams that already score their RAG system can start with a two-week audit.
Questions
Can you evaluate an assistant another vendor built?
Yes, as long as we can send it questions and read its answers and, ideally, the passages it retrieved. Without the retrieved passages we can score answers but not retrieval.
Do you need access to our documents?
We work inside your environment or on a copy you choose, with the same access rules as the original system. How personal data in questions or documents is handled is agreed before we start.
What does it cost?
A fixed price, agreed in writing before we start. It depends on the number of languages, whether traces already exist, how many failure types need a judge and how much CI work is involved, and we quote it after a first 30-minute call.
What happens in the first call?
Thirty minutes with one of our engineers, free of charge, about your product and the failures that worry you. We need no access to your systems. Afterwards we propose a scope and a price.
Related services
AI agent evals
Evals for agents that act in your systems: outcomes, tool calls and how reliably a run repeats.
AI automation
Business processes automated step by step with rules, models and agents, tested on past cases and reported as cost per case.
AI chatbots
Chatbots and assistants for customer service and internal support that answer from your data, act through approved tools and hand over to people.
AI consulting
AI strategy, use case assessment and readiness checks from engineers who also build and measure the first system.
AI engineers for your team
Experienced AI engineers who join your team part-time or for longer and build AI features, agents and platforms with you.
AI evals engineering
Evals for AI products, agents and RAG, built from your own traces, with a test set and release gate that stay in your repository.
Enterprise AI
Assistants, agents and private AI built into your systems, with your permissions, audit logs and a platform team that can run them.
MCP server development
MCP servers that connect approved internal APIs to AI assistants, with narrow permissions, approval for changes and tests.
RAG development
Assistants that answer from your internal documents, with permissions, citations and tests on your own questions.
AI engineering and architecture
Assistants, voice interfaces and agents in the products you sell, with the serving and retrieval underneath.
Custom model training
A small open-weight model trained for one task and compared with your current model on cost and accuracy.
From the blog
Retrieval or generation?
A diagnostic procedure for finding out whether a wrong RAG answer started in ingestion, search, reranking or the model, using labelled evidence and per-question traces.
RAG citation quality
How to test whether the passages an assistant cites actually support each claim it makes, with a claim-level check, a worked example and acceptance criteria.
Multilingual RAG testing
How to test whether an assistant finds the same evidence when people ask in English, German or Polish, with a parallel test set, a language matrix and terminology fixes.
Validating an LLM judge
How to test an LLM judge against hand-labelled examples, report its true positive and true negative rates and correct the pass rate it reports.
Talk to an engineer
Bring two or three conversations where your AI system got it wrong. The call takes 30 minutes and needs no access to your systems or your data.
Booking runs on Google Calendar.
Prefer to write first? Leave your email and a few lines, and we reply with times.
Look for the draft in your email program
Send it from there, and an engineer replies in person with times for a 30-minute call. If no draft has opened, write to us at miki@cloudsail.com.