Tagged “deduplication”
-
The same document three times
Identical copies, near-copies and successive versions look alike and need opposite treatment. Deduplicating a version history destroys the record.
-
Ingesting the same document twice at once
Retries and queues deliver documents more than once. If a rerun adds records instead of replacing them, one document answers with two of everything.
-
A document ID that survives a move
Path-based identity breaks the moment a file moves. One rename produces a duplicate and an orphan at the same time, and neither of them errors.
-
Which language is this document in
Language is a fact nothing records by default. Without it, translations look like duplicates and quality checks report clean text as damaged.
-
A thread is not a document
A twenty-message email thread contains the first message twenty times. Ingest it naively and the oldest, most-quoted text becomes the loudest claim.
-
The order the stages run in
Hashing before normalising, fingerprinting before stripping, writing before deciding identity. Each is a stage in the wrong place and none of them error.