Ingesting the same document twice at once

One document is returning the same passage twice in a row, word for word, from the same file. There is only one copy of the file, deduplication is working, and the corpus contains no other document that says this.

The pipeline ingested it twice. A worker timed out after writing its records but before reporting success, the work was redelivered, and the second pass added a full set of records beside the first.

Every delivery mechanism is at-least-once

The way ingestion work reaches a worker — a queue, a batch, a scheduled scan, a retry loop — has one guarantee available in practice, and it is that work will sometimes be delivered more than once.

The mechanics are ordinary. A worker finishes its writes and dies before acknowledging, so the work is redelivered. A visibility timeout expires on a document that was slow rather than stuck, so a second worker picks it up while the first is still going. A retry fires on a request that actually succeeded and whose response was lost. A backfill and the nightly incremental pass select the same document on the same night.

None of these is a bug to be eliminated. Removing the redelivery means risking losing work instead, which is a strictly worse failure for a corpus. The pipeline has to be safe to run twice on the same document, because it will be, and that safety is a design property rather than something that can be added after the duplicates appear.

The reason it goes unnoticed for a long time is the shape of the symptom. Duplicated records are not wrong — the text is real, the citation is correct, the document exists. They simply make one document say everything it says twice, which is invisible in every count except a records-per-document ratio that nobody is watching.

Idempotency is a property of the key

A write is safe to repeat when repeating it targets the same place. So the question is entirely about what identifies a record, and the failure is what happens when nothing does.

A generated identifier per write is the default in most storage clients and it is the worst possible choice here. Every run mints new identifiers, so every run adds. Nothing collides, nothing errors, and the corpus grows by a copy of a document each time anything is retried.

A key derived from the record’s content plus the document identity is repeatable: the same input produces the same key, so a second pass writes over the first rather than beside it. This is what makes an ingest naturally idempotent, and it costs nothing at the time.

A key derived from position in a sequence is repeatable only while the sequence is stable, which it is not across an edit — a point covered in replacing what an edit invalidated. It survives a retry of an unchanged document and fails at exactly the moment the document changes.

The requirement underneath all three is that the document’s identity is settled before anything is written. A pipeline that decides identity at write time cannot have idempotent writes, because the thing the key is derived from does not exist yet.

The pipeline

THE PIPELINE — repeated delivery

  · Records keyed by document identity plus
    content
                    → a second pass overwrites the first.
                      Running twice is running once.

  · Records keyed by a freshly generated
    identifier
                    → FAILS SILENTLY. Every retry adds a
                      complete second set. The document
                      says everything twice.

  · Worker dies after writing, before
    acknowledging
                    → guaranteed to happen. The redelivery
                      is correct behaviour; the duplicate
                      is not.

  · Two workers on one document at once
                    → FAILS SILENTLY. Interleaved writes
                      leave a mixture of two passes and no
                      way to tell which records are which.

  · Backfill and incremental pass both
    selecting one document
                    → the older source state can overwrite
                      the newer. Compare source times, not
                      write times.

  · Whether to serialise per document or
    detect after the fact
                    → CORPUS-DEPENDENT. Set by how often
                      documents are redelivered and what a
                      lock costs.

Two writers, and which one should win

Concurrency inside one document is the case that does real damage, because the two passes are not necessarily producing the same thing.

Two passes over the same unchanged document produce identical output, so interleaved writes are harmless with content-derived keys — both write the same records to the same keys. The problem is two passes over a document that changed between them: one holds text extracted at nine o’clock and one at ten, and both are writing. The result is a set of records that is partly one version and partly the other, with no marker distinguishing them, and it will not self-correct because the source now looks unchanged.

The rule that resolves it is that a write must be ordered by the source’s state, not by when the write happens. Every record and every document record carries the source change time it was derived from, and a write whose source time is older than what is already stored is discarded rather than applied. That single comparison makes a slow pass harmless: it can finish whenever it likes and it cannot move the index backwards.

It also removes the need for most locking. A per-document lock is the intuitive answer and it introduces its own failure modes — a lock held by a worker that died, a lock whose lease expires mid-write, a lock that serialises a backfill behind the incremental job. Where a lock is genuinely wanted, it should be an optimisation to avoid wasted work, not the mechanism correctness depends on, because a lock that fails open silently returns the pipeline to the behaviour it was protecting against.

Two-step writes are where duplicates come from

The specific sequence that produces almost every duplicated document: write the new records, then remove the old ones. A failure between the steps leaves both. The document’s source has not changed, so no future pass will look at it, and the duplication is permanent.

Three things make it survivable, and they are the same three that make any multi-step write survivable.

Record the intent before starting. The document is mid-write, this is the source state being written, these are the records being superseded. A restart can then finish rather than discovering nothing to do.

Make the removal targetable without the intent record. Every record carries its document identity and the source state it came from, so “delete everything for this document that is not from the current state” is a query that works even if the intent record was lost. This is the property that lets a cleanup pass repair the corpus after the fact.

Count records per document, routinely. A document with an unexpected number of records is the only external symptom this failure has. It is a cheap number, it belongs in the reconciliation pass alongside the identity comparison, and it is the difference between finding duplicated documents by measurement and finding them when somebody notices an answer saying the same thing twice.

Repeated work is not the same as duplicated content

Worth separating, because the two get conflated and only one of them is a defect.

Doing the same work twice costs money and time. It is wasteful and it is fine — a document extracted twice is a document extracted correctly, twice. The overlap deliberately built into a freshness window is exactly this trade, accepting repeated work to avoid gaps.

Writing the same content twice is a corpus defect that changes what the system says. Those are different problems with different urgency, and a pipeline that is idempotent at the write layer can be generous about repeated work without any of it reaching the index. That is the whole reason to spend effort on the key rather than on preventing redelivery: the cheap and reliable defence is at the write, not at the delivery, and every attempt to guarantee exactly-once delivery upstream is an attempt to solve a harder problem in a place with less information.

What this stage hands on

The same records a single clean pass would have produced, regardless of how many times the document was delivered, with each record carrying the document identity and the source state it was derived from so that a later pass can tell current records from residue.

The limit is narrow and worth being clear about. This does not prevent a document from being processed repeatedly, and it does not make the pipeline cheaper. What it prevents is the specific outcome where retrying a failed job — the safest, most routine operation available to anybody running a pipeline — is the thing that corrupts the corpus, and where the resulting damage is a document that quietly counts twice in every answer it appears in.