The order the stages run in
Duplicate detection has been running for months and finds almost nothing, in a corpus where everybody knows there are four copies of everything. The hashes are being computed correctly, on text that differs between copies by a byte-order mark and a run of spaces.
Every stage in that pipeline works. They run in an order that makes one of them useless, and a stage that finds nothing looks exactly like a stage with nothing to find.
Order is a dependency graph, not a preference
An ingestion pipeline is usually assembled in the sequence the stages were written, which is the sequence the problems were noticed in. That produces something that works on the corpus it was debugged against and has no defensible reason for any of its ordering.
The stages are not independent. Several of them consume something an earlier stage either produces or destroys, and the ones that destroy are the dangerous half: structure, byte-level detail and line breaks are all thrown away by ordinary processing, and any stage needing them has to run before the throwing away.
The failures from wrong ordering share a signature. Nothing errors, because every stage receives text and text is what it expects. What changes is that a stage becomes a no-op, or draws a conclusion from evidence that was already altered. This is different from the within-stage ordering question — the sequence of steps inside normalisation — although both fail the same quiet way.
Normalise before you hash
Any comparison of text is a comparison of bytes, so anything that changes bytes without changing meaning changes the answer.
Two exports of the same document differ by a byte-order mark, a few non-breaking spaces, and composed rather than decomposed accents. To a reader they are one document. To a hash they are two, and the duplicate check reports two distinct documents with total confidence.
The same argument applies to change detection. A hash taken before normalisation moves whenever a source inserts a zero-width character, so a document that nobody edited reads as edited and gets its whole derived set replaced. The pipeline does real work, repeatedly, over nothing.
Hash the normalised text, and hash the text rather than the file. The second half matters as much as the first: the same document exported twice can differ in bytes — embedded timestamps, compression, producer metadata — while normalising to identical text. Byte-level hashing of source files is useful for one narrow purpose, which is noticing that a re-fetch returned something different, and it is the wrong instrument for every question about content.
Strip before you fingerprint
Near-duplicate detection compares documents by similarity, and boilerplate is content that every document in a source shares.
Fingerprint first and unrelated documents look similar because they share a navigation menu, a consent notice and a footer. In a source where furniture outweighs substance, similarity scores compress into a narrow band near the top and stop discriminating. Every threshold is then wrong: high enough to avoid false matches means finding nothing, low enough to find real pairs means grouping the whole source.
Removing furniture first spreads the distribution out again, and the threshold becomes a decision about content rather than a fight against a constant.
There is a dependency in the other direction, which is the awkward part. Boilerplate detection works by finding text repeated across many documents in a source — and exact duplicate documents inflate that repetition count, because a block appearing once in each of five identical copies looks like a block appearing in five documents. So exact-copy collapse belongs before boilerplate detection, and near-duplicate fingerprinting belongs after it. The three stages are not interchangeable and they are frequently implemented as one.
The pipeline
THE PIPELINE — stage order
· Normalise, then hash
→ one visible string is one byte string,
so comparisons mean something.
· Hash taken before normalisation
→ FAILS SILENTLY. Copies read as
distinct, unedited documents read as
edited, and the stage reports zero
duplicates confidently.
· Furniture stripped, then fingerprinted
→ similarity reflects content.
· Fingerprinted before stripping
→ FAILS SILENTLY. Unrelated documents in
one source score alike and no threshold
separates them.
· Structure captured before text is
flattened
→ recorded as metadata. Free now,
impossible later.
· Identity assigned before any record is
written
→ updates and deletions have a target.
· Where the seam between expensive and cheap
stages sits
→ CORPUS-DEPENDENT. Set by what the
expensive stages actually cost.Structure dies at flattening
Headings, section paths, table membership, page boundaries and the position of a passage within its document exist in the extractor’s output and not in a string of text. Every stage that flattens — and most of them flatten a little — removes some of it.
So any stage that wants structure has to be upstream of the flattening, and the practical consequence is that capturing structure as metadata belongs as early as it can possibly go, at extraction, not at the point something needs it. Recovering it afterwards means re-reading the original file, which is the expensive pass the staging seam exists to avoid.
The same is true of two smaller things that are easy to lose. Line-break structure, which the hyphenation repair depends on and whitespace collapsing destroys. And per-page attribution, which page-level recognition flags depend on and any concatenation destroys.
Identity comes before the first write
Identity is not a stage in the middle. It has to be settled before anything is persisted, because every record written without it is a record nothing can find later.
The order it forces is: enumerate, establish identity, then fetch, extract and everything else — with the identity stamped on every artefact and every record along the way. A pipeline that extracts first and works out identity when it comes time to write has already lost the ability to skip unchanged documents, because skipping requires knowing which stored record corresponds to this document before doing the expensive work.
The version of this that hurts is a pipeline that ran for a year writing records keyed on nothing durable. Adding identity later fixes new documents and leaves every existing record unreachable by update or deletion — which is one of the few situations where the only remaining option genuinely is a full rebuild.
The stages that have to come last
Two, for the same reason: they depend on the population rather than on the document.
Boilerplate detection needs many documents from one source before repetition means anything, so it cannot run as part of a single document’s first pass. Near-duplicate grouping needs the fingerprints of everything it might match against. Both are corpus-scoped stages sitting in what is otherwise a per-document flow, and a pipeline that treats them as per-document either sees no repetition at all on the first document or, worse, learns its thresholds from whatever happened to arrive first.
The consequence is that a corpus’s first ingest and its steady state are different pipelines. A document arriving today gets compared against an established distribution. The documents that built that distribution were processed without one, which means they are the documents most likely to be stuck at old stage versions — and knowing that is what makes the first backfill a planned event rather than a surprise.
What this stage hands on
The same clean text and metadata every other post here hands on, from an arrangement where each stage receives evidence that has not already been altered by a stage that did not need to run first.
None of this makes any individual stage better. It stops stages from lying — and the specific failure it prevents is the worst kind to debug, because a stage in the wrong place produces no error, no warning and a plausible result, and the only way anybody notices is by knowing what the answer should have been.