Replacing what an edit invalidated
A paragraph was removed from a policy last month. The document was re-ingested that night, successfully, and the new text is in the index. The removed paragraph is also in the index, and it is the one being quoted back.
Nothing failed. The pipeline wrote records derived from the new version of the document and never had any instruction to do something about the records derived from the old one.
An edit is not an update
Detecting that a document changed is a solved problem with known mechanisms, covered in keeping an index fresh. What happens next is a separate decision and it is the one that gets skipped, because the word “update” makes it sound as though there is a row to modify.
There is not. One document produces a set of downstream records, and an edit produces a different set. The operation is a replacement of one set by another, and the two sets do not have to be the same size, in the same order, or about the same passages.
The default behaviour of a write path is to add. Writing the new set without removing the old one leaves both, and both are retrievable. The document’s current text is present, its previous text is present, and nothing distinguishes them — they carry the same document identity, the same title, the same source path.
Why the old records cannot be matched to the new ones
The natural repair looks like a diff: work out which records changed, update those, leave the rest. It does not survive contact with real edits, for a structural reason worth understanding before building anything on it.
The text handed on from ingestion is cut into pieces by a downstream stage, and the pieces are a function of the whole text. Insert a sentence near the top and every piece after it is drawn from slightly different material. Nothing has changed in the sense of an editable field; the whole partition is different.
So a record’s position in the sequence is not an identity. Piece four of the new version is not a modified piece four of the old one — it is a piece four that happens to be fourth. Keying records positionally and overwriting by position produces a set that is partly new and partly stale in a pattern nobody can predict, which is worse than either extreme because it looks like a successful update.
The alternative that does work is to treat the derived set as disposable in its entirety and to hold identity only at the document level. A record’s key is the document plus something stable about the record itself, and the whole set for a document is addressable and removable as a group. Whether the pieces are recomputed identically or differently stops mattering, because none of them are being preserved.
Replace the whole set, and mind the order
Two ways to sequence a replacement, and they fail differently.
Delete first, then write. Between the two operations the document is absent from the index. Queries in that window get no results from it — a recall gap, bounded in time, and obvious in its shape: a document everyone knows about is briefly unfindable.
Write first, then delete. Between the two operations both versions are present. Queries in that window can get either, including contradictory answers from the same document.
Neither is universally right, and the choice depends on which failure the system can tolerate. What is universally right is that the window must be short and the second step must be guaranteed. A crash between the two steps is the origin of most permanently duplicated documents: the write succeeded, the delete never ran, and the next incremental pass sees a document whose source is unchanged and does nothing about it.
Guaranteeing the second step means recording the intent before starting — the document is mid-replacement, these are the records being superseded — so that a restart can finish the job rather than discovering nothing to do. It is the same reasoning that makes a reconciliation pass necessary rather than optional: any two-step write will sometimes stop after one step, and something has to notice.
The pipeline
THE PIPELINE — replacement on edit
· Edited document, whole derived set
removed and rewritten
→ index holds exactly the current text.
· New records written, old ones left
in place
→ FAILS SILENTLY. Removed passages stay
retrievable, carry the document's
identity, and get cited as current.
· Records overwritten by position in the
sequence
→ FAILS SILENTLY. The new set is a
different partition of different text,
so positions do not correspond.
· Crash between the write and the delete
→ FAILS SILENTLY. Both versions persist
and the source looks unchanged
forever after.
· Text re-derived and found identical
→ no replacement needed. The cheapest
outcome, and worth checking for first.
· Whether to write before deleting or after
→ REQUIREMENT-DEPENDENT. A brief recall
gap and a brief contradiction are
not the same cost.Check whether anything actually changed
Most detected changes do not need a replacement at all, and the check is cheap enough to always run.
A source can report a document as modified because its permissions changed, because a bulk operation touched every timestamp, because it was copied, or because someone opened it and saved it without editing. In all of those the extracted text is byte-identical to what is already stored, and comparing a hash of the new text against the stored one ends the work immediately.
The saving is not just the write. A replacement is the one ingestion operation with a visible window of inconsistency, so not performing an unnecessary replacement is a correctness improvement, not only an efficiency one. A pipeline that replaces the derived set every time a timestamp moves spends a meaningful fraction of its life with documents in a half-written state.
Where the text has changed but only trivially, the honest answer is that ingestion cannot tell trivial from substantive and should not try. A whitespace-only difference has usually been eliminated by normalisation before the hash is taken, which is the right place for that judgement — it is a question about representation, not about meaning.
What the replacement has to carry forward
An edit changes text and it must not change identity or lose facts that were established once.
The document identity stays the same. This is the whole reason it exists. An edited document is the same document, so the identity survives and every new record carries it. Content hashes change on an edit by design, which is exactly why they cannot be the identity.
Version continuity. If the source treats the edit as a new version rather than a mutation, the previous version may be a document in its own right that should be superseded rather than removed. Replacement and superseding are different operations and the source system, not the pipeline, is the authority on which one happened.
Everything captured at ingest that the new pass cannot rediscover. The original ingestion date, the locations the document has been found at, any human decision recorded against it. A replacement that rebuilds the record from scratch quietly resets those, and the loss is invisible because the document looks fine.
What this stage hands on
For an edited document: text and metadata for the current version, with every record derived from any previous version gone, and the identity unchanged so that the next edit has something to target.
The limit of it is worth stating plainly. This makes the index consistent with the source at the moment the replacement completed, which is not the same as consistent now. Between edits the index is correct; during one it is briefly not; and if a replacement ever stops halfway, it is wrong indefinitely with no error anywhere. Which is why the mechanism that matters most here is not the replacement itself but the pass that comes back later and checks that the number of records a document has is the number it is supposed to have.