Originally published at Trida AI.

You delete the file. The assistant keeps quoting it.

That is not a model failure, and it is not a retrieval failure either. Retrieval did what it was asked to do: it searched an index. The index still held the document, because nobody told the index the document was gone. A retrieval-augmented generation (RAG) system has at least two copies of every document: the source, where people edit and delete, and the index, where the chunks and embeddings actually get read at query time. Keeping those two copies in agreement is a separate piece of engineering, and in a lot of first builds it is the piece nobody wrote.

The launch version usually handles one event well: adding documents. A folder gets loaded, split, embedded and stored. Edits arrive later and are harder. Deletions arrive last of all, if they arrive at all, because a deletion is an absence. There is nothing left to scan, nothing with a newer timestamp to notice. A sync process that works by looking at what is there will walk straight past what is not.

Change is visible. Deletion is not.

Platform documentation is unusually blunt about this asymmetry. Microsoft's guidance for Azure AI Search indexers says change detection for Azure Storage happens automatically, from built-in timestamps, and then, in the same breath, that deletion detection isn't a given: an indexer doesn't track object deletion in its data sources. The remedy is a soft-delete strategy, and it has to be in place from the very first indexer run. Documents deleted before the policy existed stay in the index even if you add the policy later and reset.

Read that timing rule slowly, because it generalises beyond one platform. Deletion handling is not a setting you retrofit after the first embarrassing answer. The mechanism that notices a removal has to exist before the removal happens, or the index has no record that anything is missing. Creating a new index, with the policy in place from the start, can clean it up. But a new index is a recovery operation, not a sync strategy, and teams that rely on it tend to discover how stale their index was only after a user does.

Edits cause a quieter version of the same problem. Suppose a policy document shrinks. The paragraph that set a warranty period at twenty-four months is cut, and the document now runs to ten chunks instead of twelve. If ingestion adds the new chunks by position and never retires the old ones, chunks eleven and twelve stay searchable. Nothing in them looks wrong on its own. They are fluent, on-topic, and superseded, which makes them worse than an obvious error, because a reviewer scanning the answer sees plausible text and moves on.

Permission changes belong in this list too, even though they rarely get discussed with deletions. A document that moves from a public folder to a restricted one has, from the index's point of view, been edited. If the access label attached to its chunks does not move with it, the document has effectively been deleted for some readers and not for the system that serves them. That is no longer a quality problem.

Five document events and what each one needs from the index before it can be trusted.

Five document events and what each one needs from the index before it can be trusted.

Treat a document as a unit, not as loose chunks

The fix starts with identity. Every chunk needs to know which document it came from, which version of that document, and what happened to that document since. With that in place, an edit stops being "add some new chunks" and becomes "replace this document": the new version's chunks go in, the previous version's chunks come out, and the swap is the smallest unit the sync is allowed to complete. If the swap fails halfway, the document should read as unavailable or as its old self — never as a blend of both, which is the one state where an answer can combine a new rule with a limit the new rule removed.

Frameworks give you parts of this. The LlamaIndex ingestion pipeline keeps a document store alongside the vector store and offers a strategy, UPSERTS_AND_DELETE, that, alongside upserts, deletes stored documents whose IDs are missing from the current run. That last clause carries the catch: it can only notice a missing document if the run is given the complete current collection. Feed it a partial batch and the same logic would treat documents that are still valid as deleted. Deletion by absence is powerful and dangerous in the same breath, so it needs a complete inventory to be safe.

Managed services make a different trade. Amazon Bedrock sync is incremental: it processes only documents added, modified or deleted since the last sync, re-ingesting a changed document and removing a deleted one from the vector store. You give up some control over how the replacement happens, and in return the lifecycle is somebody's explicit job rather than an assumption inside your ingestion script. Either way, the question to ask of any stack is the same, and it is worth asking before the first load, not after: for each source, how does a deletion reach the index, and how long does it take?

Then test it the unfriendly way. Delete a real document from a staging corpus, ask the question it used to answer, and watch what comes back. Shorten a document and check that its tail is gone. Move a document behind a stricter permission and query as a reader who should no longer see it. These are cheap tests. They are also the tests that almost never appear in a launch checklist, because the demo corpus does not change while anyone is watching it.

One warranty policy, before and after an edit

Here is the shape of the failure in a single document, labelled as an example so nobody mistakes it for a customer story. A warranty policy says twenty-four months in version three. Version four cuts that to twelve and deletes the paragraph about buying an extended warranty entirely. The sync upserts the new chunks and leaves two old ones behind. A week later a customer asks about the extended warranty, and the assistant answers from a paragraph that no longer exists anywhere in the source.

An example warranty policy after an edit: the new chunks are in, two old chunks remain, and a question about an extended warranty is answered from a deleted paragraph.

An example warranty policy after an edit: the new chunks are in, two old chunks remain, and a question about an extended warranty is answered from a deleted paragraph.

Notice what would not have caught this. A better embedding model would not. A reranker would not — the stale chunk is genuinely relevant to the question; that is the problem. Even a citation check passes, after a fashion: the answer can point to a chunk that really is in the index. What fails is a question nobody asked the chunk: are you still true? Version and source checks answer that question. Similarity cannot.

There are limits to how far this goes. Some corpora are archives, where nothing should ever leave and a deletion really is an error. Some sources cannot emit deletion events at all, and the honest option there is a scheduled full reconciliation — comparing the index's document list against the source's — with the lag stated plainly instead of hidden. And a hard delete is not always the right response to a withdrawal: regulated work may need the index copy removed while a separate, access-controlled record keeps the fact that the document existed. Those are design decisions. The failure is not choosing one of them; it is never noticing there was a choice.

So before a RAG system goes live, write down five events for every source it reads: created, edited, deleted, moved, and permission-changed. Next to each, write how the index learns about it and how long that takes. If any row says "manual" or is blank, you have found the document that will still be answering in six months. It will not announce itself. It will just be quoted, correctly, from a file that no longer exists.

Work with us

If your assistant is answering from documents your team has already revised or withdrawn, the problem is not the model — it is the lifecycle between your source and your index, and that lifecycle can be mapped event by event before it embarrasses you in front of a customer. Send the corpus through Trida AI's intake and we will work through the five events for each source with you: what reaches the index, what never does, and how you would know.