Short answer: treat a deletion as a versioned event that must reach the source registry, chunk store, vector index, and cache before a document is considered gone. A periodic hard delete alone is too slow for internal engineering documentation, where a removed runbook can remain in an answer after its owner has retired it.
The concrete workload here is a B2B SaaS catalog that aggregates listings from several sources, including an internal engineering documentation feed. Each listing has a stable doc_id, a source revision, and an updated_at timestamp. The retrieval system chunks text for semantic search, then joins the winning chunks back to current metadata. Chunking and freshness are coupled: a fresh parent with stale children is still stale.
What should a retrieval architecture do when engineering documentation is deleted?
Start with an explicit state machine. active means a source revision is eligible for retrieval; deleted means it is excluded immediately and queued for physical cleanup. Keep the tombstone long enough to beat delayed deliveries from connectors. Otherwise, an old “upsert” can resurrect a page that was intentionally removed.
I think the safest event carries the identity and ordering data, not the full text:
from dataclasses import dataclass from datetime import datetime @dataclass(frozen=True) class DocumentEvent: doc_id: str revision: int kind: str # "upsert" or "delete" occurred_at: datetime def apply_event(event: DocumentEvent, registry: dict[str, DocumentEvent]) -> bool: previous = registry.get(event.doc_id) if previous and event.revision <= previous.revision: return False registry[event.doc_id] = event return True The registry is the authority for eligibility. A retriever checks it after vector search, so a chunk that survives briefly in an index cannot become an answer. The filter is cheap, deterministic, and testable without a model.
It failed.
How do chunking and freshness interact across source listings?
Chunk boundaries are an operational decision, not a cosmetic preprocessing step. If a connector replaces one paragraph, recomputing every chunk wastes tokens and makes freshness harder to reason about. Store a content hash for each chunk and derive a stable chunk key from doc_id, revision, and ordinal. On an upsert, unchanged hashes can retain their embeddings; changed hashes get new records. On a delete, every key for that document revision becomes ineligible at once.
For an aggregate listing feed, source clocks will disagree. Use the source revision when it exists, and use a monotonic ingestion sequence as a tie-breaker. Do not compare wall-clock strings from different systems and call the newest one authoritative. Clock skew is normal; resurrection is not.
The loop below keeps retrieval behind a small generic interface. It demonstrates the ordering rule without assuming a particular database or embedding service.
from collections.abc import Iterable from hashlib import sha256 def chunk_key(doc_id: str, revision: int, ordinal: int) -> str: return f"{doc_id}:{revision}:{ordinal}" def refresh(events: Iterable[DocumentEvent], registry, chunks, vector_index, quarantine): for event in events: if not apply_event(event, registry): continue if event.kind == "delete": chunks.mark_ineligible(event.doc_id, event.revision) vector_index.hide_document(event.doc_id) quarantine.record(event) continue # Fetch and chunk the source revision only after it wins ordering. text = chunks.fetch_source(event.doc_id, event.revision) for ordinal, piece in enumerate(chunks.split(text)): digest = sha256(piece.encode("utf-8")).hexdigest() key = chunk_key(event.doc_id, event.revision, ordinal) chunks.put(key, digest, piece, eligible=True) vector_index.upsert(key, piece) The quarantine record gives operators a replayable trail when a connector delivers events out of order. It is not a second source of truth. A failed embedding call should leave the new chunk ineligible, while the old revision remains eligible until the replacement is complete; that policy avoids a freshness gap at the cost of briefly serving older text.
That last choice deserves a concrete timeline. Imagine revision 41 is active at 09:00. At 09:01 the source emits a delete, and at 09:02 the connector retries an upsert for revision 40. The registry records the delete first, so the retry is ignored; the vector index may still contain bytes for revision 41, but the eligibility check blocks them. At 09:03 a sweep can remove those bytes physically. If the sweep runs late, search remains correct, although storage is temporarily larger. This separation between correctness and cleanup is the reason to model deletion as state instead of a one-off maintenance command.
Which deletion semantics should the answer layer enforce?
There are three useful checks, in descending order of protection. First, filter by the registry state before ranking. Second, re-check the winning doc_id and revision immediately before prompt assembly. Third, include the revision in citation metadata and reject a citation whose revision is no longer active. The last check matters for cached answers: a cache key based only on the question can outlive the document it quoted.
A soft delete is appropriate when audit and restore matter. A hard delete is appropriate for retention requirements, but it should follow the eligibility change, not replace it. For sensitive material, redact the text from logs and traces as well; deleting the vector does not delete a copied prompt payload.
The catch is latency. A synchronous fan-out gives the strongest guarantee but couples user-facing writes to every index. An asynchronous fan-out scales better and tolerates connector outages, yet needs a freshness budget and a visible “last indexed revision” metric. Stick with synchronous invalidation when a stale runbook can cause a production change; choose asynchronous processing when eventual consistency is acceptable and replay is reliable.
Evaluation and operations that catch resurrection
An eval harness should include delete-specific cases, not just relevance. Index a page, retrieve it, delete it, then assert that retrieval returns no eligible chunk and that a cached answer is invalidated. Deliver an old upsert after the delete and assert that the tombstone wins. Repeat with two revisions whose timestamps are reversed to expose clock assumptions.
Track counts for active chunks, ineligible chunks, tombstones, replay attempts, and the age of the oldest pending delete. Sample citations in production and verify their revision against the registry. I’m not sure one freshness number fits every team; your mileage may vary, so make the budget a product decision and test it in the same CI job as ranking quality.
The operational checklist is short prose: persist the event before fan-out, make every consumer idempotent, expose revision lag, retain tombstones for the connector’s maximum retry window, and run a scheduled sweep that compares registry membership with index membership. Alert on resurrection, not only on queue depth.
