Counting what you did not ingest
Somebody asks how much of the shared drive is actually searchable. The honest answer available is the number of records in the store, which is not a number of documents, and a count of files in the source that was last checked when the pipeline was built.
Nobody can say which documents are missing, because a document that was never ingested leaves nothing behind. The absence has no record, and an absence with no record cannot be counted.
The number that does not exist by default
An ingestion pipeline naturally produces one statistic: how many things it wrote. That number goes up, it looks healthy, and it says nothing at all about coverage, because it has no denominator.
Coverage is a comparison between two populations — what the source holds and what the index represents — and neither side is as easy to obtain as it sounds. The default state of a pipeline is that it can report its output and not its input, which is why “what fraction of the corpus is in here” is so often unanswerable in systems that have been running for a year.
The reason to fix it is not tidiness. Every wrong answer a retrieval system gives has two possible causes: it found the wrong thing, or the right thing was never there. Those get debugged completely differently, and without a coverage number there is no way to tell which one you are looking at.
The denominator is the first lie
The obvious denominator is the count of documents in the source. The number usually available is the count of documents the pipeline enumerated, and those differ for reasons the pipeline cannot see.
A listing that paginated short, a folder the connector’s credentials could not open, a file type the enumerator filtered before anything else got a look, a share that was mounted last month and is not this month. Each of those reduces the enumerated population, and the resulting coverage figure is a ratio of two numbers that both came from the same blind spot — so it looks excellent. This is why an incomplete listing is worth treating as its own failure rather than as a detail of the fetch stage.
The workable posture is to record the enumerated count as what it is — a count of what was seen, with the enumeration’s own boundaries stated beside it — and to obtain an independent figure where one exists. Source systems frequently expose a total that does not come from the same listing call: a reported object count, a table count, a quota figure. Two numbers that disagree are informative. One number that cannot disagree with anything is not.
Give every way out its own category
The useful artefact is not a percentage. It is a ledger where every enumerated document sits in exactly one state, and the states are named.
Ingested. Text exists, records were written, and they represent the document’s content.
Ingested as a description only. The deliberate downgrade: a workbook that is a calculation tool, a deck that is a visual artefact, a file whose contents were judged not worth admitting. Title and metadata are in; content is not. This must be its own state, because formats with no reading order produce it on purpose and a naive count reads it as a full success.
Present but unreadable. A page image nothing could recognise, a handwritten form, a format with no extractor. The document is known to exist and known to be opaque. This is a good outcome, correctly recorded — and a terrible one recorded as a short document.
Failed. Something errored or timed out. Distinct from unreadable because it may succeed on a retry, and distinct from missing because the pipeline knows about it.
Collapsed. Not in the index as a separate document because it is a copy of one that is, with its location recorded against that document. The reason the collapse needs an audit trail is precisely so this state is answerable.
Superseded. In the record, not current. Present, deliberately not the thing answers come from.
Excluded by rule. A path pattern, a file type, an age cutoff, a size cap. Excluded on purpose, and worth counting because exclusion rules outlive the reasoning behind them.
Deleted. Gone from the source, tombstoned in the index.
Stuck at an old pipeline version. Ingested, but under code known to be superseded and never reprocessed.
The value of the list is not any single figure. It is that every document the source contains lands in one of these, so the sum equals the enumerated count and there is no residue. A residue means there is a way out of the pipeline that nobody has named, and that is exactly where documents disappear.
The pipeline
THE PIPELINE — coverage
· Every enumerated document assigned one
named state
→ the states sum to the enumeration.
Absences become answerable.
· Document dropped with no state recorded
→ FAILS SILENTLY. Indistinguishable from
a document that never existed. Nothing
will ever look for it.
· Deliberate downgrade counted as an ingest
→ FAILS SILENTLY. Coverage reads as
complete while the content is absent.
· Unreadable page recorded as a short
document
→ FAILS SILENTLY. Enters the corpus as a
header and a page number, and gets
answered from.
· Coverage computed against the enumerated
count only
→ flattering by construction. Both
numbers share the same blind spot.
· What fraction of a corpus is worth
ingesting at all
→ CORPUS-DEPENDENT. Exclusion is a
decision, not a shortfall.Count documents, not records
One source document becomes many downstream records, and the ratio varies with the document. So a record count moves for reasons that have nothing to do with coverage: a change upstream that produces more records per document raises it while coverage is flat, and a corpus losing large documents can lose coverage while the count holds.
Count at the level of the thing a person would name. Documents, per source, in states. Records are a capacity number, useful for sizing the store and useless for answering whether the corpus is complete.
Per source is load-bearing rather than a nicety. Coverage aggregated across a whole corpus hides the case that actually happens: overall coverage of ninety-something percent, and one connector at zero because its credentials expired in March. The aggregate barely moves. The distribution across sources is where the failure is visible, and a per-source count with a date attached is the whole mechanism.
Where the numbers have to come from
Two habits, and both are about where the count lives rather than how it is calculated.
States belong on records, not in logs. A log line saying a document was skipped is a message that a human might read once. A record whose state is “excluded by rule, pattern seven” is a row somebody can query, group and count a year later. Everything in the ledger above is worthless if it exists only as output from a run that has since rotated away — which is also what makes a per-document trace the thing the ledger is built on.
The comparison is a scheduled job, not a query somebody remembers to run. Coverage decays: connectors break, folders move, credentials expire, exclusion patterns start matching things nobody intended. A number computed once at launch describes a system that no longer exists. The reconciliation pass that proves the index and the source still agree on what exists is the natural place for it, because it already enumerates both sides.
What this stage hands on
Not text. A count, per source, of documents in each state, with a stated denominator and a stated date — and no unexplained difference between the enumeration and the sum of the states.
The point of building it is narrow and worth being honest about. It does not improve a single answer. What it does is convert the most common complaint about a retrieval system — it does not know about the document I am looking at — from an argument into a lookup, and it turns an ingestion pipeline from something that reports how much work it did into something that can say what it does not contain and why.