The reconciliation pass

The nightly sync has succeeded every night for a year. Nobody can say whether the index and the source still agree about what exists, because every mechanism in the pipeline reports on documents it processed and none of them reports on documents it did not.

That is not a gap in the monitoring. An incremental pass is defined by only looking at what changed, so it is structurally unable to notice anything it failed to see.

Incremental sync cannot audit itself

Every drift mechanism described in keeping an index fresh shares one property: it starts from a set of candidates supplied by the source, and it is correct exactly to the extent that the set is complete. A missing candidate produces no work, no error and no trace.

The list of ways the set is incomplete is long and each entry is ordinary. A timestamp that did not move on edit. A cursor advanced past a batch that then failed. A listing that paginated short. A folder the credentials stopped being able to read. A replacement that stopped halfway. A document that failed and left no record. A deletion, which most mechanisms cannot represent at all.

All of these are permanent until something compares the two populations directly. That comparison is the reconciliation pass, and it is the only component in an ingestion pipeline whose output is evidence rather than work.

What it compares

Enumerate every document the source currently holds. List every document the index currently represents. The comparison is on identity, and the cheap version carries one extra field per side — a content hash, or a source-reported version, or a last-modified time.

That is deliberately less than a rebuild. A rebuild re-fetches and re-extracts every document, so its cost scales with the size of the corpus. Reconciliation reads a listing and a set of keys, so it scales with the number of documents. For most corpora that difference is the whole reason this is feasible on a schedule when a rebuild is not, and it is why “just rebuild periodically” is not the same answer.

Four disagreements come out, and they are not equally easy to act on.

In the source, not in the index. A document that was never ingested, or was ingested and lost. The most common and the most valuable finding: it is a real gap in coverage, it is fixable by processing the document, and nothing else in the pipeline would ever have reported it.

In the index, not in the source. Either a deletion that never propagated or a document whose identity changed — a move, under path-based identity. These need different handling and they are hard to tell apart, which is why an identity that survives a move matters so much here: with a stable identifier this category is deletions only.

In both, and different. The hashes disagree, so an edit was missed. This finds every timestamp that lied.

In both, and the index side is malformed. The document is present but the number of derived records it holds is not what it should be, or its records carry a stage version that is no longer current. Not a sync failure at all — a write that did not complete, or a backfill that did not finish.

The pipeline

THE PIPELINE — reconciliation

  · Full enumeration compared against the
    index on identity
                    → finds missed edits, missed documents
                      and unpropagated deletions.

  · No reconciliation at all
                    → FAILS SILENTLY. Every gap incremental
                      sync opened stays open, and the
                      nightly job keeps reporting success.

  · Partial or truncated enumeration treated
    as authoritative
                    → catastrophic. Every unlisted document
                      looks deleted. This is the failure
                      that empties an index.

  · Enumeration and index read at different
    moments
                    → normal churn appears as disagreement.
                      Expected, and must not be acted on
                      blindly.

  · Document in the index and absent from the
    source
                    → may be a deletion, may be a rename.
                      Conclude only with stable identity.

  · How often the pass can run
                    → CORPUS-DEPENDENT. Bounded by what
                      enumerating the source costs, not by
                      corpus size.

The pass must be allowed to refuse

The dangerous property of reconciliation is that it is the one component with the authority to remove things, and its input is an enumeration — which is exactly the thing most likely to be quietly incomplete.

A listing that returns half the source, because a token expired mid-pass or a service degraded or pagination stopped early, tells the pass that half the corpus has been deleted. Acting on that is the worst outcome available in this entire pipeline, and it is worse than the drift the pass exists to fix: drift degrades answers, and a mistaken bulk deletion removes them.

So the pass needs a refusal condition, checked before it acts, and the shape of it is a sanity comparison rather than a threshold anybody has to be clever about. If this enumeration is dramatically smaller than the last one, stop and report rather than proceed. If a whole source enumerated to nothing, that is a broken connector, not an emptied folder. If the number of documents it is about to delete is a large fraction of what it holds, escalate to a person.

Two habits make that easier than it sounds. Reconcile one source at a time, so a failure is contained and so a suspicious result is attributable. And separate the finding from the acting: the pass writes what it found, a second step applies it, and the gap between them is where a bulk anomaly gets caught. That separation also makes the pass safe to run in a reporting-only mode indefinitely, which is the right way to introduce it to a corpus nobody has audited before.

Churn is not drift

A reconciliation pass reads two populations at two different moments, so some disagreement is expected and means nothing.

Documents created during the enumeration appear as missing from the index. Documents deleted during it appear as orphans. Documents edited during it appear as mismatched hashes. On an active corpus this is a continuous background rate, and treating it as findings produces a pass that always reports problems — which is indistinguishable from a pass that never does.

The workable posture is to require a disagreement to persist. A document flagged as missing on one pass and still missing on the next is a real gap; a document flagged once is probably churn. That costs a cycle of latency and it removes almost all the noise, which is the right trade for an operation that runs on a slow schedule anyway.

The related caution is that a corpus in the middle of a large reorganisation will produce enormous findings that are all legitimate movement. Running reconciliation during a known migration is not useful, and knowing that is a reason to record when migrations happen.

Cadence, and what it costs to skip

The pass is the correctness floor for everything else, so the cadence question is really: how long is it acceptable for a missed edit or an unpropagated deletion to persist.

For most corpora the honest answer is much shorter than the interval anybody actually chooses, and the constraint is enumeration cost against the source rather than pipeline capacity. Two things make it cheaper than a naive reading suggests. It can be sharded — a fraction of the corpus per run, cycling through, so the worst-case age of any document’s last verification is bounded without ever enumerating everything at once. And it needs no expensive stages at all: no fetch of document bodies, no extraction, no recognition. It is a comparison of keys.

What it produces feeds directly into the coverage ledger, which is the other reason to run it on a schedule. Coverage computed from the pipeline’s own output is a statement about what the pipeline believes. Coverage computed from a reconciliation is a statement about the source, and only the second one is worth reporting to anybody.

What this stage hands on

Not text. A list of disagreements between the index and the source, each classified, with a decision recorded about whether it was acted on — and, on a clean pass, the only positive evidence an ingestion pipeline is capable of producing.

The limit is precise and worth stating. Reconciliation proves that the index and the source agree about which documents exist and, if hashes are compared, about their content. It proves nothing at all about whether that content was extracted correctly: a document that reconciles perfectly can still be two columns interleaved line by line. This pass finds what is missing. Finding what is wrong is a different job.