The listing that was not complete

The corpus settled at eleven thousand documents and stayed there. Everybody assumed that was the size of the share. It was the point at which the listing call stopped returning results, because nothing in the connector followed the second page.

Every stage after that worked correctly on the documents it was given. Coverage looked complete, because the denominator came from the same listing as the numerator.

Enumeration decides what the corpus is

The first thing an ingestion pipeline does is ask a source what it contains. Every subsequent stage operates only on that answer, which makes enumeration the single most consequential call in the pipeline and the one least likely to be verified.

The reason it goes unchecked is structural. A document that fails to extract leaves an error; a document that was never listed leaves nothing, and it is not merely absent from the index — it is absent from every count, every report and every retry queue. A missing document is invisible, and a missing half of a source is invisible in exactly the same way. The pipeline is not degraded; it is running perfectly against a smaller corpus than the one anybody thinks it has.

The failure also compounds, because the reconciliation pass that would normally find gaps consumes an enumeration too. Feed it a short listing and it does not report missing documents — it reports that everything in the index should be deleted.

The ways a listing comes back short

None of these raise an error and most of them are one missing line of code.

Pagination not followed. The call returns a page and a token. Handling the first page and not the loop is the oldest bug in this category and it caps a corpus at a round-ish number, which is the diagnostic tell.

A pagination token that expires. Long listings hold state on the source side, and the state has a lifetime. A slow consumer — one that processes each page before requesting the next — can outlive it, and what comes back is an error mid-listing or, worse, a silently truncated final page.

A hard result cap. Some sources will not return more than a fixed number of results for a query, regardless of paging, and the honest ones say so in documentation nobody read. The workaround is always to partition the query, by container or by date range, into pieces that stay under the cap — which has to be built deliberately.

A default filter. A listing that excludes hidden items, or items in the trash, or a type, or anything above a size, by default. The filter is applied server-side and the response looks like a complete answer to a different question than the one intended.

Traversal errors swallowed. Walking a tree, one folder denies access or times out. Catching that and continuing is the right behaviour for keeping the pass alive and the wrong behaviour if nothing records the subtree that was skipped — and a denied folder near the root can remove most of a corpus.

Eventual consistency. A listing that reflects a slightly old state omits recently created documents. Harmless on its own, since the next pass will see them, and dangerous if a reconciliation treats the listing as authoritative about what exists.

Rate limiting. The pass is throttled partway through, some requests fail, and the retry logic gives up on a page rather than on a document. The result is a listing with a hole in the middle rather than a truncation at the end, which is much harder to spot, because the count looks plausible.

The pipeline

THE PIPELINE — enumeration

  · Full listing, pagination followed to
    exhaustion
                    → the corpus is what the source holds.

  · Only the first page consumed
                    → FAILS SILENTLY. The corpus caps at the
                      page size and looks stable and healthy.

  · Pagination token expired mid-listing
                    → FAILS SILENTLY. A truncated listing is
                      indistinguishable from a short source.

  · Source-side result cap reached
                    → FAILS SILENTLY. Every result returned
                      is real; the ones beyond the cap do
                      not exist as far as the pipeline knows.

  · Folder denied or timed out during a
    traversal
                    → an unknown quantity of the corpus is
                      missing. Record the subtree, do not
                      just continue.

  · Short listing fed to a reconciliation pass
                    → catastrophic. Unlisted documents read
                      as deletions.

  · Whether the source can be enumerated at
    all in one pass
                    → CORPUS-DEPENDENT. Large sources
                      usually need partitioning by container
                      or by date.

The enumeration has to be checkable

The defences are all comparisons, because there is no way to inspect a listing and tell that it is complete.

Compare against an independent total. Many sources expose a count that does not come from the same call: a reported object count, a table row count, a quota or usage figure, a folder’s own item count. It does not have to be exact to be useful — an order-of-magnitude disagreement is the finding. A listing with nothing to disagree with is a number that can only be believed.

Compare consecutive runs. A source’s document count moves gradually. A run returning far fewer documents than the previous one is a broken listing until proven otherwise, and that check is the difference between a pipeline that notices and one that quietly re-scopes itself. This is the same refusal condition a reconciliation pass needs, and it belongs on the enumeration itself so that both benefit.

Count per container, not just in total. A share whose total is roughly right can still be missing a whole folder, and the total will not show it. Per-container counts also make the finding actionable, because they name where to look.

Record what the enumeration did not see. Denied folders, timed-out subtrees, pages that failed after retries, partitions that hit a cap. These are the enumeration’s own failure records, and they belong in the coverage ledger alongside the documents that failed to extract. Without them the ledger’s denominator is a guess presented as a measurement.

Assert the loop terminated for the right reason. A pagination loop should end because the source said there was no more, and that is distinguishable from ending because of an error, a page limit, or a maximum-iterations guard somebody added to prevent a runaway. Recording which of those happened converts the commonest silent truncation into a fact.

Partitioning, and the trap inside it

Sources too large to list in one pass get partitioned, usually by container or by a date range, and the partition scheme becomes part of the pipeline’s correctness.

Two ways it goes wrong. Partitions that do not cover the whole space leave a gap — a date-range partition starting at the oldest document anybody knew about misses everything older, and a container partition built from a list of containers misses containers created since. And partitions with boundaries that documents can cross leave documents in neither: a date-partitioned listing on a field that changes will move a document out of one partition into another between two passes, and if the passes run in the wrong order it appears in neither.

The property to aim for is that the partitions provably cover the space — derived from the source’s own structure rather than from a list somebody maintains — and that the boundaries are on something immutable. Where neither is achievable, overlapping the partitions slightly costs duplicate work and is much cheaper than a gap, which is the same trade as overlapping a freshness window rather than making it exactly contiguous.

What this stage hands on

A set of document identities that the source is believed to contain, with a stated method, a stated partition scheme, a count that has been compared against something independent, and an explicit record of every part of the source the pass could not see.

That last clause is the whole contribution. Enumeration cannot be made reliable — sources throttle, tokens expire, folders deny, caps apply — and pretending otherwise is what produces a pipeline that silently redefines the corpus. What it can be made is honest, so that everything downstream, including the count somebody quotes when asked how much of the drive is searchable, is scoped by a number that came with its own caveats attached.