What a RAG supplier must hand over so your team can maintain the system

Why a RAG handover involves more than code

A RAG system behaves according to its sources, its configuration and the models it calls, as much as its code. Sources change, permissions shift, and model versions are retired by their providers. The 2015 paper on hidden technical debt in machine learning systems named data dependencies, configuration and changes in the external world among the reasons such systems are expensive to maintain. A handover that transfers a repository and a diagram leaves those risks with whoever happens to be on call.

The handover inventory

Sources and sync

• Connector code and schedules for every source, running in your accounts
• How each source handles edits, deletions, moves and permission changes, with the measured delay
• The owner of each source and the credentials model, with secrets in your vault rather than in the repository

Retrieval configuration

• Chunking rules, embedding model and version, analyzers, number of retrieved chunks, reranker and prompts, all under version control
• A script that rebuilds the index from an empty state
• Platform settings whose timing matters, such as a deletion policy: some platforms let you add one later, but records deleted before it existed may stay in the index until you reconcile or rebuild it

Evaluation assets

• Test questions with expected passages and answers, including questions to decline and permission cases
• The harness that scores retrieval and answers separately, and the baseline results with the exact configuration that produced them

Operations

• Dashboards for latency, cost per request, sync status and failed items, owned by your team
• Alert rules and where alerts are routed
• A record of who may change prompts, retrieval settings and source scope, and how such changes are reviewed

Situations the runbook must cover

1. 1

   A user reports a wrong answer

   Find the trace, check whether the evidence was indexed, retrieved and passed to the model, and add the case to the test set.

2. 2

   A document must disappear today

   Remove it at the source, force the sync or delete it from the index directly, and confirm with a query that it no longer answers.

3. 3

   A new source is added

   Inventory entry, owner, permissions mapping, connector, test questions for the new content, evaluation run.

4. 4

   A model is retired or replaced

   A new embedding model means rebuilding the index; a new answer model means a full evaluation run against the baseline before switching.

5. 5

   A sync job fails

   Alerts, how to rerun safely, and how to tell which content is now stale.

Rehearse before you sign

A handover is complete when your team has done the critical tasks once without the supplier doing them. Run a rehearsal in staging in which your engineers act and the supplier only answers questions. The checklist below is an example to adapt; the agreed score range should come from the baseline, because model-based scores vary between runs.

Handover rehearsal (run by the client team, supplier observes only)
[ ] Rebuild the index from scratch in staging from the repository and runbook
[ ] Run the evaluation set; scores fall within the agreed range of the baseline
[ ] Add one new test source end to end, including owner and permissions
[ ] Remove one document and show with a canary that it no longer answers
[ ] Diagnose a planted wrong answer from its trace and name the failing stage
[ ] Rotate one connector credential without a code change
[ ] Run a recovery drill: simulate a failed sync or a bad configuration
    change in staging, detect it from alerts and restore service
[ ] Find the cost per request and latency for yesterday on the dashboard
Each item: who ran it, date, result, link to evidence

Items that fail show where knowledge still sits with the supplier. That is more useful than any number of handover meetings, and it gives both sides a clear end point that does not depend on the calendar.

Owners after go-live

Name a person for each of these before launch: answer quality and the test set, each source's content, the sync and index platform, the periodic review of permissions, and the on-call response when the assistant misbehaves. Microsoft's RAG guidance makes the point that evaluation is not a one-time activity, because documents and user questions change, and that keeping the record of past evaluations matters for future work. Someone has to own that record.

If you are about to start a RAG project with an external team, put the handover inventory and the rehearsal into the scope at the beginning, and have repositories, cloud accounts and dashboards live in your organisation from the first day. Contract wording is for your procurement and legal advisers; the engineering deliverables above give them something concrete to write down.