Keeping an index fresh

The system answers from a document that was edited three weeks ago, using the version from before the edit. Someone notices because the answer contradicts what they just wrote.

The ingestion job has been running nightly the whole time, successfully. It simply never learned that this particular document changed, and nothing in the pipeline is capable of noticing that it did not.

Rebuild against sync

Two ways to keep an index current, and they trade correctness against cost.

Full rebuild. Re-ingest everything, every time. Correct by construction — the index is a pure function of the current corpus, drift is impossible, and any bug in ingestion is fixed for the whole corpus the next time it runs. Also expensive in proportion to the corpus rather than to the change, which puts a ceiling on how often it can run. A corpus that takes nine hours to rebuild can be at most a day stale, and a corpus that takes four days cannot be rebuilt on a schedule at all.

Incremental sync. Process only what changed. Cost proportional to the change, so it can run frequently. And it can be wrong in ways a rebuild cannot, because it depends entirely on correctly detecting change — every missed change is permanent until something else forces a reprocess.

The practical answer for most corpora is both: incremental sync frequently, full rebuild on a slower cycle as a correctness floor. The rebuild is what stops drift accumulating forever, and treating it as optional is how a system reaches the state described at the top of this post.

Detecting change, in decreasing order of reliability

Everything about incremental sync reduces to this question.

A change feed from the source. The source system tells you what changed — an event stream, a webhook, a changes API, a database log. Reliable, and the only mechanism that reports deletions, which is why it matters far more than it first appears. Where a source offers this, use it and accept the integration cost.

A monotonic cursor. Query the source for everything modified since a high-water mark, then advance the mark. Cheap and widely available. Two failure modes worth knowing: it never reports deletions, and it depends on the source’s modification timestamps being trustworthy — which fails when a bulk operation touches everything, or when a copy preserves the original timestamp, or when clocks disagree across a distributed store.

Content hashing. Fetch the document, extract, hash the text, compare to the stored hash. Definitive about whether content actually changed, and immune to timestamp lies — a bulk operation that rewrites metadata without changing text produces identical hashes and no work. The cost is that you must fetch and extract everything to find out, which is most of the cost of a rebuild. Useful as a confirmation step rather than as the primary detector: let a cursor propose candidates, let the hash decide whether they are real.

Enumeration and diff. List everything currently in the source, compare to what the index holds. Does not need timestamps or feeds, and it is the only mechanism besides a change feed that can find deletions — anything in the index and absent from the listing is gone. Cost is proportional to the number of documents rather than their size, so it is often affordable when a rebuild is not, and it makes a good periodic reconciliation pass even where a cursor does the day-to-day work.

Modification timestamps alone, with no reconciliation. The common default and the weakest option. It misses deletions, misses anything whose timestamp lied, and never finds out.

The pipeline

THE PIPELINE — freshness

  · Source offers a change feed
                    → use it. The only mechanism that
                      reports deletions natively.

  · Cursor on modification time
                    → cheap, and MISSES DELETIONS entirely.
                      Needs a reconciliation pass beside it.

  · Timestamp not updated on edit, or reset by
    a copy
                    → FAILS SILENTLY. The document is never
                      reprocessed and the index keeps
                      answering from the old version
                      indefinitely.

  · Bulk operation touching every timestamp
                    → the whole corpus looks changed. Content
                      hashing turns this back into no work.

  · Cursor advanced before processing succeeded
                    → FAILS SILENTLY. The batch is skipped
                      forever. Advance only after success.

  · Rebuild cadence
                    → CORPUS-DEPENDENT. Set by how long a
                      full pass takes against how stale the
                      corpus may be.

The ordering bug that eats batches

Worth naming on its own because it is easy to write and hard to see: advance the cursor only after the batch has been processed and committed.

The natural-looking implementation reads the current time, queries for everything modified since the last mark, sets the mark to the time it read, and then processes. If processing fails partway — a crash, an unhandled document, a timeout — the mark has already moved. Those documents are now permanently outside every future query window. Nothing errors on subsequent runs, because subsequent runs are correct; the gap is in the past.

The safe shape is to advance the mark from the batch’s own contents after success, and to overlap the window slightly rather than making it exactly contiguous. Reprocessing a document that had not changed costs one extraction. Missing one costs an indefinitely wrong answer.

Partial failure needs to be a state, not a log line

A document that fails to ingest — a corrupt file, a format nothing handles, a timeout, an extraction that produced nothing — leaves the pipeline in a state that needs to be recorded rather than logged and forgotten.

The failure mode otherwise is uniquely annoying: the document has no record at all, so nothing knows it should exist, and the next incremental run does not consider it because its timestamp is old. It falls out of the corpus silently and permanently, and the only thing that will ever bring it back is a full rebuild.

Recording a failed document as a record with a failure status fixes this. It makes the gap visible, it gives retries something to iterate over, and it means somebody can ask how many documents the system cannot read — a number that is always more interesting than expected.

Freshness metadata is worth carrying

Two timestamps per document, and they answer different questions.

When the source last changed is what the reader cares about, and it is the thing worth surfacing alongside an answer: content dated two years ago is not wrong, but it is a fact about the answer.

When the pipeline last processed it is what operators care about. The gap between the two is drift, and monitoring the largest gap in the corpus is the single most useful freshness signal available — better than an average, because the problem is always specific documents rather than the corpus as a whole.

Both are captured at ingest and cost nothing. Neither can be reconstructed afterwards.

What this stage hands on

Records whose content matches their sources as of a known, recorded time — plus explicit records for the documents that failed, and a reconciliation pass that periodically proves the index and the source still agree on what exists.

That last part is where the hardest case lives. Everything above assumes the operation is change. Documents that stop existing are a different problem, and the reason most freshness mechanisms handle them badly is that most of them cannot see them at all.