The same document three times

The corpus contains policy.pdf, policy-v2.pdf, policy (1).pdf, Policy FINAL.pdf and policy-2019.pdf. Some are the same file. Some differ by a footer. Two of them are genuinely different documents that supersede each other, and one of those is the only one anybody should be answered from.

Deduplication advice treats this as one problem. It is three, they need opposite handling, and getting the distinction wrong is the most destructive mistake available at this stage.

Three things that look the same

Byte-identical copies. The same file, present more than once, because somebody copied a folder or two systems were merged. There is exactly one document here and any copy of it will do.

Near duplicates. Substantively the same content differing in something incidental — a different footer, a header with a different date, whitespace, an export from a different tool, the same page retrieved with and without a consent banner. Still one document in any meaningful sense.

Successive versions. Genuinely different documents where one replaces another. The 2019 policy and the 2024 policy are both real, they say different things, and the difference is the entire point.

The first two should be collapsed. The third must never be collapsed. Removing older versions as duplicates deletes the record; keeping them as equals means the system answers from a superseded document half the time. Neither is acceptable, and the correct operation — superseding — is not deduplication at all.

Identical copies: the easy case, with one trap

Content hashing solves this. Hash the extracted text rather than the file bytes, index the hash, and collapse matches.

Hashing the text rather than the file matters more than it sounds. The same document exported twice can differ in bytes — embedded timestamps, metadata, compression — while extracting to identical text. File-level hashing misses those and reports two documents; text-level hashing gets it right.

The trap is what you do with the collapsed copies. Collapse the content, keep the locations. Five copies of one document at five paths is one piece of content that five different people can find in five different places. Discarding four of those paths means a user who knows the document by its location cannot be told where it is, and it means deletion later has to deal with a path nobody recorded.

Near duplicates: cheap detection, real judgement

Detecting near duplicates does not require comparing every document to every other one. The standard approach is to reduce each document to a compact fingerprint that preserves similarity — shingles of consecutive words hashed down to a small signature — so that similar documents produce similar fingerprints and can be grouped without a full comparison.

The mechanism is not the interesting part. Two decisions are.

What threshold counts as “the same”. Corpus-dependent, and worth setting by looking at real pairs rather than by picking a number. A collection of contracts from one template will be mutually similar at a level that would be conclusive elsewhere.

Which copy to keep. The one with the most complete extraction, not the newest or the shortest. If one copy has a text layer and another is a scan of the same pages, keep the one that extracted cleanly — and note that the other existed.

Much of what near-duplicate detection finds is not really duplication at all: it is boilerplate making unrelated documents look alike. Stripping furniture before fingerprinting makes the results dramatically more honest, which is one reason the order of pipeline stages matters.

Versions: supersede, do not delete

This needs to be an explicit concept in the pipeline rather than a consequence of a similarity threshold.

A version relationship means: these are distinct documents, one is current, the others are historical, and all of them remain in the record. What changes is not their existence but their status.

The honest position on detecting this automatically is that it is unreliable. Filename conventions lie — FINAL is famously not final, and v2 may predate an untitled revision. Modification timestamps reflect when a file was copied as often as when it was written. Dates inside the document are the best available signal and are frequently absent.

So the useful design is not clever detection. It is:

Take version information from the source system when it exists. A document management system knows its own version history. Use it rather than inferring.

When it does not exist, keep both and mark the uncertainty rather than guessing. A document flagged as possibly superseded, with the candidate that may supersede it recorded, is far more useful than a confident wrong answer in either direction.

Never let a similarity threshold silently delete the older one. This is the specific failure that turns a records problem into a data-loss problem.

The pipeline

THE PIPELINE — duplicates

  · Byte- or text-identical copies
                    → collapse content, KEEP every
                      location. One document, many paths.

  · Near duplicates differing in furniture
    or whitespace
                    → collapse, keeping the copy that
                      extracted most completely.

  · Successive versions of one document
                    → SUPERSEDE, never collapse. Both stay
                      in the record; status differs.

  · A similarity threshold applied to versions
                    → FAILS SILENTLY. The older document
                      disappears, nothing reports a loss,
                      and the corpus quietly stops
                      containing its own history.

  · Duplicates left in place
                    → FAILS SILENTLY the other way. Repeated
                      content becomes the corpus's most
                      frequent claim.

  · Similarity threshold and what counts as
    the same document
                    → CORPUS-DEPENDENT. Templated
                      collections are mutually similar by
                      construction.

Identity is the thing being decided

Underneath all three cases is one question: what is a document, and what is its stable identity?

The default answer most pipelines adopt by accident is the file path, and it is the wrong one. Path identity breaks the first time a file moves: the same content appears at a new location and is ingested as new, while the old record persists pointing at nothing. One move produces a duplicate and an orphan simultaneously, and neither is reported.

A workable identity is either an identifier from the source system — which is why using the source’s own identifiers is worth the integration effort — or a content hash, which is stable under moves and renames but changes whenever the document is edited. Content hashing therefore gives you correct duplicate handling and no version continuity; source identifiers give you continuity. Real pipelines usually need both recorded: the source identity to track a document through its life, the content hash to notice when it changed.

Deciding this early is much cheaper than discovering it later, because the identity choice is what every subsequent operation — freshness, superseding, deletion — is expressed in terms of.

What this stage hands on

One record per distinct document, carrying every location it was found at, its status as current or superseded with a pointer to what replaced it, and a note of what was collapsed into it.

The last of those is the audit trail. Somebody will eventually ask why a document they can see on the shared drive is not in the system, and the answer — collapsed into another copy, superseded by a newer version, or genuinely never ingested — has to be recoverable. Without the trail, all three look identical from the outside.