Finding one bad document a month later

An answer cites a document and the citation is wrong — the text is not in the file. The obvious question is what the pipeline did with that document, and there is no way to ask it. The run that ingested it happened in May, its logs rotated in June, and everything recorded about it was recorded per job rather than per document.

The pipeline is fully instrumented. It has throughput, error rates, duration per stage and a dashboard. None of that can be filtered down to one file.

Debugging runs backwards; logs run forwards

Ingestion telemetry is almost always built the way a job is built: a run starts, it emits lines as it goes, and the lines are ordered by time. That is the correct shape for watching a job and the wrong shape for every question anybody actually asks afterwards.

The questions arrive from the other end. Somebody has one document and wants its history. The unit of inquiry is the document; the unit of recording is the run — and bridging between them means grepping logs for an identifier that may not appear in them, in a retention window that has probably closed.

The consequence is not that debugging is hard. It is that a whole class of defects becomes undiagnosable, so they get attributed to whatever is easiest to blame downstream. A wrongly extracted document and a retrieval problem produce the same complaint, and without a per-document history there is nothing to distinguish them.

Keep the history on the document

The intervention is to store a small amount of provenance against the document’s own record, updated as it passes through, and to treat it as data rather than as diagnostics. It is queryable, it does not rotate away, and it can be aggregated when needed.

What earns its place:

Where it came from and when. The source, the address it was fetched from, when it was fetched, and the source’s own last-modified time. Two dates, not one, because the gap between them is drift.

How it was read. Format, which extraction path handled it, whether any of it was recognised rather than read, and the stage versions involved. This is the field that answers “was this document processed by the code that has the bug”.

The cheap quality signals, stored rather than logged. Character count, page count, the proportion of word-like tokens, the fraction of the text that was stripped as furniture. Each of these is one number and together they let a query find every document that looked like this one, which is how a single report becomes a class of defect.

What was decided about it. Collapsed into another copy, superseded, excluded by a rule, downgraded to a description, marked unreadable. Every one of these is a decision the pipeline made that a person may need to contest, and each is invisible in the output.

What was removed. Not the removed text itself, necessarily, but a count and the rule responsible — enough that a suspicious gap can be traced to a filter rather than assumed absent from the source.

The list is deliberately small. It is roughly the fields that cannot be reconstructed by re-running the pipeline, and that is the test worth applying to anything else proposed for it.

The pipeline

THE PIPELINE — provenance

  · Per-document history stored beside the
    document
                    → one report becomes a query, and the
                      query finds the whole class.

  · History only in job logs
                    → FAILS SILENTLY. Correct while the run
                      is live, gone by the time anybody
                      asks, and never keyed by document.

  · Decisions made and not recorded
                    → FAILS SILENTLY. A collapsed, excluded
                      or downgraded document is
                      indistinguishable from one never seen.

  · Stripped text with no record of the rule
                    → FAILS SILENTLY. A missing answer looks
                      like a source that never said it.

  · Quality signals computed and discarded
                    → recomputing them means re-reading the
                      corpus.

  · How much provenance is worth its storage
                    → CORPUS-DEPENDENT. Bounded by keeping
                      only what a re-run cannot recover.

Three kinds of telemetry, and they do not substitute

Worth separating, because ingestion pipelines routinely have one and are described as having all three.

Metrics are aggregates over time: documents per hour, failures per source, duration per stage. They answer whether the system is working. They cannot answer anything about a document, by construction, and no amount of labelling makes them able to — a metric with a document identifier as a label is not a metric.

Logs are a time-ordered narrative of a run. They answer what happened during an incident, while the incident is recent. They are the wrong home for anything that needs to be true in six months.

Records are state attached to a thing. They answer what is true of this document now and how it got that way. This is the category ingestion pipelines skip, and it is the one that every question in this post needs.

The practical consequence is a small rule: if a fact will be needed after the run that produced it has finished, it belongs on a record. Logging it is a decision to lose it, made on a schedule set by log retention rather than by anybody’s judgement.

Storing the text is most of the answer

The single most useful thing to have when investigating a bad document is not a field. It is the extracted text, exactly as the pipeline produced it, which the staging seam already provides for unrelated reasons.

Almost every extraction question is settled by reading it: interleaved columns, an inline running header, a document that is thirty characters of furniture, recognition confusions in a document that was supposed to be born-digital. Without it the investigation begins by re-fetching and re-extracting, which reproduces today’s pipeline output rather than the output that caused the problem — and if the pipeline has changed in the meantime, the bug is unreproducible and gets closed.

The cheaper cousin, where storing all text is not affordable, is to store the first and last few hundred characters plus the quality signals for every document, and the full text only for documents flagged as suspicious. It answers most questions and it degrades honestly: the cases it cannot answer are known in advance rather than discovered during an incident.

One report is always a class

The reason to build any of this is that individual bad documents are not worth the effort of fixing individually, and classes are.

A report arrives about one file. The trace says it was recognised rather than read, by an engine build from last spring, from a source that supplies mostly scans. The next question — how many other documents share those properties — is a query, and its answer is either “one” or “eleven thousand”. Those are entirely different pieces of work, and nothing but a per-document record distinguishes them.

That query is also what makes a backfill scopeable, and what turns the coverage ledger from a single percentage into something that can be broken down by source, format and decision. The aggregate view and the single-document view are the same data read at two granularities, and only one of them can be derived from the other.

What this stage hands on

Text and metadata, plus a durable record of where each document came from, how it was read, what was decided about it and what its quality signals were — stored beside it, not in a log.

The honest limit: none of it improves an answer. It is pure overhead on a working pipeline and it only earns anything on the day something is wrong. What it buys on that day is the ability to distinguish a bad document from a bad retrieval, and to state how many documents share the defect — which is the whole difference between fixing an ingestion bug and being told that the search results are not very good.