A document ID that survives a move

Somebody reorganised the shared drive. Nothing was written, nothing was deleted, and the next morning the corpus was four hundred documents larger. The same policy now answers twice, once with the current text and once with a version from eighteen months ago.

The pipeline did exactly what it was told. It identifies a document by where the document lives, a folder rename changed where several hundred of them live, and every one of them arrived looking like something it had never seen before.

Identity is a decision, and it gets made by default

Every ingestion pipeline needs an answer to “have I seen this document before”. The answer is a key, and if nobody chooses one, the key becomes the path — because the path is what the connector handed over.

That default is invisible while the corpus is static. A folder of files nobody reorganises will never expose it. It surfaces on the day someone tidies up, migrates a share, or renames a project, which is usually long after the pipeline was declared finished.

A path is a location. Locations change without content changing, and content changes without locations changing. Neither direction of that statement is true of an identity, which is why the path makes a poor one.

What a move does to an index

A single rename produces two defects at once, and they are easy to diagnose separately and hard to connect.

The document at the new path has no matching record, so it is ingested as new. That is the duplicate: two sets of records, both retrievable, differing by however much the document changed between the two ingestion runs. Retrieval has no way to prefer one.

The record at the old path still exists, and its source does not. That is the orphan. It will not be updated, because nothing at that path will ever change again. It will not be removed, because a modification-time cursor cannot see an absence — the same blind spot that makes deletion its own operation.

The corpus is now permanently wrong in a way that grows with every reorganisation, and the only signal is that answers sometimes cite text that is no longer in the file.

What to use instead

In rough order of preference, and the first one that is available is usually the answer.

A source-native identifier. Most real document systems already have one: a document ID, an object key, a record primary key, a message ID. It is stable across renames and moves by design, because the source system had the same problem and solved it. Pulling it through the connector and storing it is the single highest-value decision in the whole pipeline, and it costs nothing at the time.

A composite of a stable container and a stable name. Where the source has no document ID but does have a stable identifier for a folder, mailbox or site, a key built from that plus the filename survives the common case — the container being renamed — while still breaking if the file itself is renamed. Partial, but a large improvement on the full path.

A path, explicitly chosen, with rename detection beside it. Sometimes there is nothing better; a plain filesystem share offers no durable identifier at all. Then the path is the key by necessity rather than by accident, and it needs the heuristic below to stay honest.

Content hashing is version identity, not document identity

The tempting move, when no stable ID exists, is to key on a hash of the content. It is stable across moves, which is exactly the property that was missing.

It is also wrong, and wrong in a way that takes a while to notice. A document that has been edited is still the same document. A content hash says it is a different one. Key on the hash and every edit becomes a new document plus an orphan — the same defect as the rename, arriving through the other door, and now triggered by ordinary editing rather than by occasional reorganisation.

Content hashes are genuinely useful, and their job is elsewhere: deciding whether a document actually changed, and detecting that two documents are copies of each other. Both of those are questions about versions and copies. Neither is the question “which document is this”.

The pipeline

THE PIPELINE — document identity

  · Source exposes a stable document ID
                    → use it. Survives moves, renames and
                      re-parenting for free.

  · Full path used as the key
                    → FAILS SILENTLY. A folder rename
                      re-ingests every file under it as new
                      and orphans every old record. Nothing
                      errors, and the corpus answers twice.

  · Content hash used as the key
                    → FAILS SILENTLY. Every edit becomes a
                      new document plus an orphan. Looks
                      correct until anyone edits anything.

  · Same content appears at a new path and
    vanishes from the old one in one pass
                    → probably a move. Treat as a rename,
                      not as a create plus a delete.

  · Document with no durable identifier
                    anywhere in the source
                    → CORPUS-DEPENDENT. Composite keys,
                      rename heuristics, or accepting
                      periodic reconciliation.

  · Records written without carrying the
    document identity forward
                    → FAILS SILENTLY. Updates and deletions
                      have nothing to target later.

Location becomes metadata, and there is more than one

Once identity is separate from location, the path is free to be what it always was: a useful fact about the document, stored alongside it, and allowed to change.

Plural, usually. One document is often reachable at several paths — a copy in an archive folder, a mirror on a second share, an attachment in a mail thread. Recording all known locations against one identity is what makes it possible to answer “where does this actually live”, and it is what stops a deletion at one path being mistaken for the document ceasing to exist.

Detecting a move when the source will not tell you

Where the path has to be the key, a move can still be inferred at reconciliation time rather than accepted as churn.

The signal is a coincidence: within one enumeration pass, a path that used to exist is gone, and a path that did not exist now holds content whose hash matches the record from the vanished path. That pairing is a rename with high confidence. Treat it as one — reassign the location on the existing record, keep the identity, and do no reprocessing at all.

The honest caveats are worth stating rather than hiding. The heuristic needs stored content hashes to compare against. It cannot distinguish a move from a copy-then-delete, though the outcome is the same. It fails on a document that was moved and edited in the same window, which then legitimately looks like a new document. And it needs both halves of the pairing to land in the same pass, so a slow enumeration across a long reorganisation will still produce churn.

Reconciliation, not the fast path, is where this belongs. Renames are rare and clustered; paying for the comparison on every incremental run buys nothing.

What this stage hands on

Text and metadata attached to an identifier that means “this document”, not “this file, in this folder, today” — with the locations recorded as facts that are expected to change.

Everything downstream of ingestion inherits that decision and cannot revisit it. An update needs a target, a deletion needs a target, and a duplicate check needs to know what would count as the same thing. All three are the same key, chosen once, at the beginning, usually by whoever wrote the connector and did not realise they were choosing.