Reprocessing a corpus you already ingested
The extraction fix shipped six weeks ago and it works. Sampling the corpus shows a third of the documents still carry the exact damage the fix removes, because they were ingested before it existed and nothing has touched them since.
Nothing regressed. The pipeline applies the current code to documents arriving now, and no mechanism in it has any opinion about documents that arrived earlier. The corpus is a record of every version of the pipeline that has ever run.
Reprocessing is a different job from ingesting
The incremental path answers one question: what changed in the source. That is the wrong question here, because nothing in the source changed — the pipeline changed, and the documents needing work are the ones the source will never report.
So a backfill is its own operation with its own selection rule, and it has to be built as one. Bolting it onto the incremental job by widening the time window does not work: the window would have to be the whole history of the corpus, which is a full rebuild with extra steps.
The distinction that makes this tractable is that a pipeline change invalidates a specific stage, not the whole document. A normalisation fix does not require re-fetching anything. A recognition upgrade does not require re-deciding document identity. Reprocessing is cheap or ruinous depending entirely on how precisely that scope can be stated.
Scope it with stamps, not with a date
The tempting selector is “everything ingested before the deploy”. It is available, it needs no preparation, and it is far too wide — most of those documents were untouched by the change and every one of them costs a full pass.
The selector that works is a version stamp on the artefact, written at the time it was produced, as described in separating extraction from indexing. Then the population is a query: every document whose stored text was produced by extractor version four, every document whose recognition came from the engine build that had the digit-confusion bug, every document normalised under the rules from before the hyphenation repair.
More than one stamp, because changes arrive at different layers. Useful ones to carry: which fetcher retrieved the bytes, which extractor read them, which recognition engine and build produced any recognised pages, and which revision of the post-extraction rules cleaned the text. Each of those is one number, and each of them converts a vague “probably needs redoing” into a countable set.
A stamp that was never recorded cannot be reconstructed. This is the argument for writing them from the first day of the pipeline’s life rather than adding them when the first backfill is needed, because at that point the only honest selector left is “everything”.
What a change actually invalidates
Four tiers, in decreasing cost, and putting a change in the right tier is most of the work.
The bytes are wrong. A connector that truncated large files, an encoding chosen at fetch time, a partial download that was recorded as a success. Requires re-fetching, which means the source has to still hold the document — and for a corpus assembled over years, some of it will not.
The text is wrong. An extractor upgrade, a recognition engine change, a decision to start keeping table structure that was previously discarded. Requires reading the original bytes again, which is the expensive tier and the one worth scoping hardest.
The cleaning is wrong. Normalisation, hyphenation repair, boilerplate rules. Replays from the stored text. Cheap, and the tier where most changes actually land.
Only the metadata is wrong. A field that was not captured, a flag that was recorded inconsistently, a correction to how source timestamps were interpreted. Often does not need any text at all — it is an update to records that already exist.
The failure worth naming is treating tier three as tier two out of caution. Re-extracting to fix a normalisation bug is a decision that turns a two-hour job into a two-week one, and it is usually made because nobody was sure which tier the change belonged to. Being sure is a matter of having the stamps.
The pipeline
THE PIPELINE — reprocessing
· Pipeline change with a recorded stage
version
→ select exactly the documents produced
by the superseded version.
· Selector is "ingested before the deploy"
→ correct and enormous. Reprocesses
documents the change cannot affect.
· Change shipped with no backfill at all
→ FAILS SILENTLY. Old and new output
coexist, the fix appears to work
because new documents are fine, and
the corpus stays half wrong.
· Stage version bumped without invalidating
stored artefacts
→ FAILS SILENTLY. The pass runs, reads
the cache, and writes back what was
already there.
· Source no longer holds the original bytes
→ cannot reprocess. Record the document
as stuck at an old version rather than
as done.
· How wide a scope is affordable
→ CORPUS-DEPENDENT. Set by which tier
the change lands in and what the
expensive stages cost.Running it beside a live index
A backfill competes with the incremental job for the source, for the pipeline, and for the store, and the two of them can produce each other’s bugs.
Three properties keep it survivable. The backfill must lose. If an incremental run and a reprocess both produce records for one document, the one derived from the newer source state wins, which means the write has to compare source-change times rather than write times. Without that, a slow backfill overwrites a fresh edit with text extracted from bytes fetched a week ago, and the index goes backwards.
It must be resumable at document granularity. A pass over hundreds of thousands of documents will be interrupted. If the only record of progress is “the job was running”, the restart begins again from the start, and a backfill that cannot finish inside its window never finishes at all. Progress belongs on the documents — each one stamped as it is completed — not in the job.
It must be rate-limited on purpose. The reason is not politeness to the source, though that matters: it is that a backfill saturating the pipeline delays the incremental job, and a stalled incremental job is a stale index. A slower backfill that leaves headroom is strictly better than a fast one that stops freshness.
Proving it finished
A backfill that reports success is not evidence, because the most common failure is a pass that ran over the wrong set and completed cleanly.
The measurement that actually settles it is the version distribution across the corpus: how many documents carry each stage version, reported as a routine number rather than looked up during incidents. Before the backfill it is skewed old. After it, the old versions should be at zero, or at a known non-zero count with a named reason — documents whose source is gone, documents that fail on the new code, documents deliberately excluded.
That residue is the interesting part and it needs somewhere to live. A document that could not be reprocessed is not done and must not be stamped as though it were; it is a document stuck at an old version, which is a state worth counting alongside everything else the corpus is missing.
What this stage hands on
The same output as an ordinary ingest, for documents that were ingested under an older pipeline — with the stage versions updated so the next backfill can see what this one did, and the failures recorded as failures rather than absorbed into the success count.
What it does not do is make reprocessing rare. Ingestion code changes for the whole life of a corpus, and every change divides it into documents that have seen the new behaviour and documents that have not. The purpose of the stamps is not to avoid that division; it is to make it visible and to make closing it cost something proportional to the change, so that a known bug in a pipeline stops being a permanent property of the index.