Separating extraction from indexing
A bug is found in the boilerplate stripper. The fix takes ten minutes. Applying it takes four days, because the only way to re-run the stripper is to re-run everything before it: fetch every file again, extract every PDF again, recognise every scanned page again.
None of that upstream work needed doing. It was thrown away the first time, because the pipeline was written as a single pass from file to record and never kept anything in the middle.
The shape most pipelines start with
The natural first implementation is one function. It takes a path, opens the file, extracts the text, cleans it, hands it on, and returns. It is easy to write, easy to read, and it holds nothing.
What that shape assumes is that ingestion happens once. It is true for a demo and false for every real corpus, because the pipeline itself will change — a new format appears, an extraction bug is found, the normalisation rules get stricter, someone decides scanned pages need a confidence flag. Ingestion code changes more often than the corpus does, and every change wants a re-run.
The cost of that re-run is dominated by the stages that touch the original bytes. Fetching a document over a network, parsing a large binary format, and above all recognising an image of a page are expensive. Everything after the text exists is cheap by comparison, and it is where nearly all the changes happen.
Keep the text
The intervention is unglamorous: after extraction succeeds, write the extracted text somewhere durable, keyed by document identity, and treat it as an artefact rather than as a variable.
Once that exists, the pipeline splits at the obvious seam.
fetch → extract → [ stored text ] → normalise → strip → write records
^^^^^^^^^^^^^^^^^ ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
expensive, rarely changes cheap, changes constantlyA fix to anything right of the store replays from the store. No fetch, no parse, no recognition. The four-day rebuild becomes a pass over text files, which is minutes to hours depending on corpus size, and crucially it is affordable often enough that people will actually do it.
The secondary benefit is that the stored text is inspectable. When an answer cites something odd, the question “what did the extractor actually produce for this document” has a stored answer instead of requiring a re-run to reproduce. Most extraction debugging is reading that text, and the failure signatures are obvious once you can see it.
Version-stamp the artefact
A stored artefact without provenance turns into a different problem: nobody knows whether a given document’s text was produced by the current extractor or by the one from five months ago.
So each stored text carries the identity of what produced it — a version for the extraction code and, separately, one for the post-extraction rules. Then “which documents need re-extracting” is a query rather than a guess, and a backfill can be scoped to exactly the affected subset instead of the whole corpus.
Two versions rather than one, because they change at different rates and for different reasons. An extractor upgrade invalidates the stored text. A normalisation change does not — it only invalidates what was derived from it.
The pipeline
THE PIPELINE — staging
· Extraction output written durably,
keyed by document identity
→ downstream fixes replay from text.
Reprocessing becomes affordable.
· Single pass from file to records
→ any fix downstream of extraction
requires re-reading every source
file, including every scanned page.
· Stored text with no extractor version
→ FAILS SILENTLY. Old and new extraction
output sit side by side, look
identical, and nothing can tell you
which documents are stale.
· Stored text reused after the extractor
was upgraded
→ FAILS SILENTLY. The upgrade appears to
have had no effect, because the cache
served the old result.
· Source file changed but content hash
unchanged after extraction
→ no work. The cache is what makes this
cheap to discover.
· Whether to store text, structure, or both
→ CORPUS-DEPENDENT. Decided by how much
structure downstream stages consume.What to store beside the text
The text alone is enough to make replay work, and a few extra fields make it enough to make replay correct.
A content hash of the source bytes. Lets the pipeline skip extraction when nothing changed, and lets it detect that a re-fetch produced something different. This is the field that turns a cache into a change detector.
The format and how it was read. Whether this text came from a clean text layer or from recognition changes how much to trust it, and that judgement is impossible to make later from the text alone.
Extraction diagnostics. Character count, page count, the cheap quality signals — stored, not just logged, so a query can find every document that looked suspicious without re-running anything.
Structure, if downstream uses it. Headings, tables, page or section boundaries, in whatever plain representation is convenient. This is the field worth deciding deliberately: structure discarded at extraction time cannot be recovered from flat text, and re-extracting to get it back is precisely the expensive pass this whole arrangement exists to avoid.
The failure this introduces
Adding a cache adds the possibility of a stale cache, and it will happen. The signature is distinctive enough to recognise: a change to extraction ships, the pipeline runs clean, and the output is byte-identical to before. Nothing errored. The stored artefact was served, and the new code never ran on anything.
Two habits make it survivable. Make the extractor version part of the cache key rather than a field checked by convention, so a version bump invalidates by construction rather than by remembering. And report the version distribution across the corpus as a routine number — if a deployed change did not move those counts, it did not happen.
Where the seam should not be
One caution, because this pattern is easy to over-apply. The seam that pays for itself is after extraction, because extraction is the expensive, rarely-changing stage. Staging every intermediate step gets you a pile of artefacts that all need invalidating, versioning and storing, and the cheap stages did not need it.
The test is whether the stage is expensive relative to the whole pass and stable relative to the code around it. Extraction is both. Normalisation and boilerplate removal are neither — they are fast, and they are the parts you keep changing, which is why they belong on the replay side of the seam rather than behind their own cache.
What this stage hands on
The same thing as before — text plus metadata — but from a durable artefact rather than from a transient function call, with enough provenance stamped on it to know when it needs producing again.
It does not make the corpus more correct. It makes correcting the corpus cheap enough to be a routine operation instead of a project, which over the life of a pipeline is the difference between fixing known bugs and living with them.