Tagged “freshness”
-
A website has no file list
A share can be enumerated. A site cannot: its document set is whatever the traversal reached, and nothing on the far end will report what was missed.
-
Counting what you did not ingest
Every corpus has documents that never made it in. Without a category for each way that happens, they look identical to documents that never existed.
-
Reprocessing a corpus you already ingested
An extraction fix only runs on documents ingested after it shipped. Everything already in the index keeps the old damage until something goes back for it.
-
Deletion is the hard one
Removing a source file removes nothing downstream. Everything derived from it persists, stays retrievable, and answers questions as if it still existed.
-
Keeping an index fresh
A full rebuild is correct and unaffordable. Incremental sync is affordable and drifts. What decides it is how you detect that something changed.
-
Ingesting the same document twice at once
Retries and queues deliver documents more than once. If a rerun adds records instead of replacing them, one document answers with two of everything.
-
Capturing who may see a document
Permissions live on folders and groups, not on documents. If ingestion does not record them, nothing downstream can reconstruct who may read what.
-
The reconciliation pass
Incremental sync has no way to prove it kept up. A periodic comparison of the index against the source is the only thing that finds what it missed.
-
Finding one bad document a month later
Someone reports a wrong answer citing a specific file. Ingestion logs are keyed by job and time, so nothing can say what happened to that one document.
-
Replacing what an edit invalidated
A document was edited and reprocessed correctly. The index now holds the new text and some of the old, because the previous records were never removed.
-
Which date does a document have
A file offers half a dozen dates and none of them is when the document was written. A share migration set most of them to the day the migration ran.
-
The listing that was not complete
Everything downstream of enumeration is scoped by it. A listing that stopped early produces a smaller corpus that behaves exactly like a correct one.
-
The documents that will not ingest
Every corpus has files that error, time out, or crash the parser. Handled as log lines they leave the corpus quietly and never come back.