The documents that will not ingest

A contract cannot be found in the system. It is on the shared drive, the connector can see the folder, and the pipeline has run successfully every night for eight months. Somewhere in a log file from March there is a line saying the parser threw on it.

That line is the only record the document ever produced. It was retried the following night, failed again for the same reason, and then stopped being retried at all — because a document with no record looks to an incremental pass exactly like a document that has not changed.

Failure is a state, not an event

The default handling for an unprocessable document is to catch, log and continue. It keeps the job alive, which is correct, and it discards the only evidence that the document exists, which is not.

The mechanism of the permanent loss is worth being precise about. Incremental sync selects documents whose source timestamps have moved. A document that failed has an old timestamp and will never appear in that selection again. So the failure is not “this document was skipped tonight” — it is this document has left the corpus, and only a full pass will ever bring it back.

The fix is small and it has to be deliberate: a failed document gets a record, with a failure state on it, in the same store as the successful ones. That single change converts an invisible loss into a queryable population, which is the difference between a corpus with known gaps and a corpus with unknown ones.

Transient and permanent need different queues

Retrying everything forever wastes the pipeline on documents that will never succeed. Retrying nothing loses documents that would have worked on the second attempt. So failures have to be classified at the moment they happen, because the information needed to classify them is not recoverable afterwards.

Transient. A timeout, a rate limit, a connection reset, a dependency that was restarting, an out-of-memory kill on a machine that was busy. Nothing about the document caused it. Retry is the right answer and it will usually work.

Permanent for this pipeline. A format nothing installed can read, an encrypted file with no key, a password-protected archive. Retrying changes nothing until the pipeline changes. These belong in a list somebody looks at, not in a retry loop.

Corrupt. Truncated files, malformed structure, a document the parser accepts and then produces nonsense from. Retry is pointless and the interesting question is whether the source copy is damaged or the fetch was. Those are distinguishable — re-fetch and compare hashes — and worth distinguishing, because a damaged source file is somebody else’s problem to fix and a damaged transfer is this pipeline’s.

Rejected by policy. Too large, too old, wrong type, excluded path. Not a failure at all; an exclusion, and it belongs in the coverage ledger under its own name so nobody later reads it as breakage.

The classification will sometimes be wrong, which is fine as long as it is recorded rather than inferred. What is not fine is one undifferentiated error bucket, because the only available policies for one bucket are retry everything and retry nothing.

The pipeline

THE PIPELINE — ingestion failure

  · Transient failure, recorded and retried
    with growing delay
                    → usually succeeds. Cost is bounded.

  · Failure logged and not recorded
                    → FAILS SILENTLY. The document has an old
                      timestamp, so no incremental pass ever
                      considers it again. Gone until a full
                      rebuild.

  · Every failure retried on every run,
    forever
                    → permanent failures consume the retry
                      budget and delay real work.

  · Document that crashes the worker rather
    than raising
                    → FAILS SILENTLY, and takes the batch
                      with it. Everything after it in that
                      batch is skipped too.

  · Extraction that succeeds and returns
    almost nothing
                    → not an error anywhere. Only a content
                      check catches it.

  · Partial write, then a failure
                    → worst state available. Half a document
                      is retrievable and looks whole.

  · How many attempts are worth making
                    → CORPUS-DEPENDENT. Set by how expensive
                      the document is and how often the
                      source is flaky.

What the failure record has to hold

Enough to act on without reproducing the failure, because reproducing it is the expensive part.

The document’s identity and where it was found, so the file can be looked at by a person. Obvious, and frequently missing, because the failure often happens before identity was established — which is an argument for settling identity before the first expensive stage.

Which stage failed. Fetch, extract, recognise, normalise, write. This is the field that turns a pile of failures into a diagnosis, because failures cluster by stage far more than by document.

A classification and the raw error. The class drives the retry policy; the raw text is what somebody reads when the class turns out to be wrong.

Attempt count and first-seen time. A document failing since March and a document failing since last night are different problems presented identically by a count.

The pipeline version that failed. So that when the fix ships, the population to reprocess is a query rather than a guess. This is the field that makes fixing a class of failure a bounded piece of work.

Retrying without making it worse

Two properties, and both exist to stop a retry policy from becoming the outage.

Delay that grows between attempts, with a cap and some jitter. A source that rate-limited one request will rate-limit an immediate retry, and a batch of documents that all failed together will all retry together unless something spreads them out. The growth bounds the total cost; the jitter is what stops a synchronised second wave.

A ceiling, after which the document moves to a list rather than a loop. The number is not the interesting part — what matters is that “give up” is a state with a name, so the document stays visible instead of either retrying forever or vanishing. A document that has exhausted its attempts is still a document the corpus is missing.

The thing worth watching is a whole-source failure, because it is not a retry problem at all. When every document from one connector fails, the cause is credentials, a network path or a service — and retrying thousands of documents individually against a source that is down produces a very large amount of load and no progress. A failure rate per source, checked before the retries run, turns that into one alert instead of fifty thousand attempts.

The two failures that are not failures

Both defeat everything above, because neither one raises anything.

The document that crashes the process. A parser that segfaults or exhausts memory on a malformed file does not throw an exception that a handler can catch, so no failure record gets written — and everything else in that batch is lost alongside it, with no indication of which document was responsible. The defence is isolation: read untrusted files in a process that can die without taking the batch, with a per-document timeout and memory limit, so that a hostile document produces one failure record rather than a gap.

The extraction that succeeds and returns nothing useful. No error exists anywhere in the pipeline. The document is recorded as ingested, holds thirty characters of furniture, and is counted as coverage. Only a content-level check on the output — a length, a word-likeness ratio — will ever notice, which is why those checks belong in the pipeline as routing signals rather than in a dashboard nobody opens.

Together they are why a failure count is a much weaker indicator than it appears. The failures a system records are the ones it survived. The interesting ones are the documents with no record at all and the documents whose record looks fine, and the only way to find either is to reconcile what the index holds against what the source actually contains.

What this stage hands on

Text and metadata for the documents that worked, plus a record for every document that did not — classified, attributed to a stage, counted, and visible to a query.

None of this recovers a document that cannot be read. What it changes is that an unreadable document becomes a known, listed, countable gap rather than an absence indistinguishable from a file that was never there. That is a lower bar than it sounds, and clearing it is the difference between a corpus somebody can make statements about and one where the only honest answer to “is this document in here” is to go and look.