How to update and remove documents without leaving stale answers

Why deleted documents keep answering

Detecting a change is easy, because the changed file is still there to look at. Detecting a deletion requires the sync process to notice that something is missing. Microsoft states this plainly for Azure AI Search: indexers for Azure Storage detect new and changed blobs from their timestamps, but they do not track deletions. Without a soft-delete policy, search documents for removed files stay in the index as orphans.

The timing matters too. The same documentation says the deletion policy has to be in place from the first indexer run; documents deleted before that remain even if you add the policy later and reset the indexer, and the suggested remedy is a new index. For Azure SQL sources, the high-water-mark policy does not detect deleted rows at all, and rows cleared with TRUNCATE are not removed from the index by a reset and rerun. Other platforms have different rules, which is the point: read how yours handles deletion before the first load.

Every edit contains a small deletion

Suppose a policy shrinks from twelve chunks to ten after an edit. If ingestion upserts chunks by position, chunks eleven and twelve from the old version stay in the index and can still be retrieved. The safer pattern replaces every chunk carrying the document ID as a unit. Where the store supports it, write the new version under a version tag and switch to it in one step. Otherwise delete and insert, treat the document as unavailable while that runs, and test what happens when ingestion fails halfway.

Frameworks support parts of this. In LlamaIndex, an ingestion pipeline with a document store keeps a hash per document ID; with a vector store attached, a changed hash causes the document to be reprocessed and upserted, and an unchanged one is skipped. That handles edits efficiently. It does not, by itself, tell you about documents that disappeared from the source, so deletion still needs its own path.

Identifiers must also survive moves. Azure notes that changing document keys or paths breaks the tracking indexers rely on, and that renaming a directory in ADLS Gen2 does not update blob timestamps, so those blobs are not reindexed. Use an ID from the source system where one exists, not a file path.

Five events each source must handle

For every source in scope, write down how each of these events reaches the index and how long it takes:

• Created: a new document appears
• Edited: content changes and the chunk count may change
• Deleted or withdrawn: the document must stop answering
• Moved or renamed: the identity must stay the same
• Permissions changed: access lists in the index must follow

Derived data needs the same treatment. Contextual summaries attached to chunks, answer caches and evaluation sets built from old text all keep stale content alive after the source has changed.

Testing freshness with canary documents

A canary is a harmless test document with a unique fact that no real document contains. Put one in each source and walk it through the events above after every scheduled sync. Check which chunks were retrieved, not only the answer text, because a model may phrase a correct answer while an old chunk still sits in the context.

# Freshness canary run (hypothetical; run after each scheduled sync)
1. create  canary-7f3a.md  "The canary parking code is BLUE-41."
   ask     "What is the canary parking code?"   expect BLUE-41, cites canary-7f3a
2. update  canary-7f3a.md  "The canary parking code is GREEN-12."
   ask     same question                        expect GREEN-12, no chunk says BLUE-41
3. delete  canary-7f3a.md
   ask     same question                        expect decline, zero chunks from canary-7f3a
4. restore canary-7f3a.md from recycle bin / soft delete
   ask     same question                        expect GREEN-12 again
record: time from each source change to the first correct answer

The restore step is there for a reason. Azure documents that an undeleted blob is not always reindexed, because restoring it does not update its LastModified timestamp; resaving its metadata forces a refresh.

When an old version should stay findable

Not every superseded document should disappear. A contract template from last year may still govern existing customers. In that case keep both versions, store effective dates and status as metadata, and filter on them, so that the default question returns the current version and a question about an older agreement can still reach the old one.

What to watch in production

• Documents per source in the index compared with the source system
• Index entries whose IDs no longer exist in the source
• Time of the last successful sync and failed items per source
• Canary results and the measured delay from change to correct answer

If the measured delay is longer than the business accepts, the schedule is not the only lever. Azure, for example, limits SQL indexers to running at most every five minutes and recommends pushing changes directly when updates must appear sooner.