Deletion is the hard one

The document was deleted. Somebody confirmed it — the file is gone from the shared drive, the page has been removed from the wiki, the record was expunged from the source system on request.

The retrieval system still answers from it. Not from a cache: from records that were derived at ingest time and have no relationship to whether their source still exists.

This is the least forgiving operation in the pipeline, because it is the only one where the failure mode is not merely a worse answer. It is answering from something that was supposed to be gone.

Why deletion propagates badly

Ingestion is a derivation, and derived data does not know about its source.

Text was extracted, cleaned, and written somewhere. That written record is a new object. Nothing about it observes the file it came from, and no mechanism will notice the file’s absence unless something is explicitly looking.

Compounding it: most change-detection mechanisms are structurally incapable of seeing a deletion. A cursor on modification time queries for things that changed — a deleted document does not appear in that result, and its absence is indistinguishable from it being unchanged. The most common freshness mechanism in use cannot report the most consequential event.

So deletion needs a mechanism of its own. It never falls out of the update path for free.

Three mechanisms that actually find deletions

A change feed that emits delete events. The only approach that is both cheap and prompt. If the source offers it, this is the answer, and it is worth substantial integration effort to get.

Enumeration and diff. List every document currently in the source, compare against every document the index holds, and treat anything present in the index and absent from the source as deleted. Cost scales with document count rather than document size, so it is usually affordable even for corpora that cannot be rebuilt. This is the reconciliation pass, and for most systems it is the real safety net.

Soft-delete in the source. Where the source marks records deleted rather than removing them, deletion arrives as an ordinary update and the whole problem collapses into change detection. Worth checking for before building anything, because a surprising number of systems work this way.

What does not work is inferring deletion from failure to fetch. A document that 404s may be deleted, or it may be a permissions change, an outage, a moved file, or a rename — and treating a transient fetch failure as a deletion means an outage silently empties the index. Fetch failures should mark a record as unverified and escalate; only enumeration or an explicit event should conclude deletion.

Removal has to be complete

One source document usually produces more than one downstream record. Deleting one of them is worse than deleting none, because it leaves the system in a state nobody can reason about.

Which means deletion needs the same thing every other operation needed: a stable document identity that every derived record carries. If each record knows which document it came from, deletion is a query. If it does not — if records were written with no durable back-reference — then deletion requires reconstructing the relationship, and the honest options are a full rebuild or living with orphans.

This is the single strongest argument for deciding document identity early rather than letting the file path become it by default. Every derived record should carry the document’s identity from the beginning, because the operation that needs it is the one that cannot be improvised.

The other half of completeness is remembering that a document collapsed from several copies has several locations. Deleting one copy on disk does not mean the content is gone — the other four paths still hold it. Deletion is correctly a decision about the document, evaluated against all its known locations, not about a path.

The pipeline

THE PIPELINE — deletion

  · Source emits a delete event
                    → propagate immediately. Cheapest and
                      promptest path.

  · Enumeration and diff against the index
                    → finds deletions a cursor cannot.
                      The real safety net.

  · Modification-time cursor only
                    → FAILS SILENTLY. A deleted document
                      never appears in the changed set, and
                      its records answer questions forever.

  · Fetch returns not-found
                    → NOT a deletion. May be permissions, an
                      outage, a move or a rename. Mark
                      unverified; do not conclude.

  · One copy of a multi-copy document removed
                    → the document still exists. Deletion is
                      about the document, not the path.

  · Derived records with no back-reference to
    their source
                    → FAILS SILENTLY. Nothing can find them
                      to remove. Orphans until a full
                      rebuild.

  · Whether deletion must be hard or may be a
    status change
                    → REQUIREMENT-DEPENDENT. An expungement
                      obligation and a tidiness preference
                      are not the same operation.

Tombstones, and why they are usually right

The instinct is to delete the record. Often the better move is to keep a tombstone — a record that the document existed, its identity, where it was, and that it is gone — while removing the content.

Three things this buys. Re-ingestion is idempotent: if the file reappears in a listing tomorrow, the system knows it was deleted rather than treating it as new. Enumeration diffs stay cheap, because absence from the index and known-deleted are distinguishable. And somebody asking “why can’t I find the thing I know we had” gets an answer instead of silence, which is the difference between a system with a records story and one without.

The exception matters and it is not a detail: where deletion is required to be genuine — a legal expungement, a rights request, content that must not persist in any form — a tombstone must contain no content, and possibly no metadata that identifies the subject. That is a requirement question rather than a design preference, and it needs answering by whoever owns the obligation, not decided at the pipeline layer.

Access changes are deletions in disguise

The case that surprises teams late: a document nobody deleted, whose permissions changed.

If access is captured at ingest — and it has to be, because the query side cannot reconstruct it — then a permission change is a content-neutral update that alters who may retrieve something. Handled as an update, it works. Ignored, the system keeps serving a document to people who lost access to it, and that failure looks exactly like the deletion failure from the outside.

Two properties are worth stating plainly. Permission changes usually do not touch modification timestamps, so a cursor will not see them either. And they are frequently made at the container level — a folder, a group, a site — so one change can alter the effective access of thousands of documents that were themselves untouched.

Capturing access at ingest is this stage’s job. Enforcing it at query time is not, and the two must not be confused: an ingestion pipeline that records permissions has done its part, and a query path that ignores them has undone it.

What this stage hands on

Records that exist only for documents that still exist and are still permitted, with tombstones for those that do not, and a reconciliation pass that periodically proves it.

That word — proves — is the whole point of treating deletion as a first-class operation. Every other ingestion failure degrades answers, and degraded answers get noticed and fixed. This one produces confident, well-formatted, fully-cited answers from documents that were deliberately removed, and nothing in the system will ever raise a hand about it.