Tagged “pipelines”
-
Deletion is the hard one
Removing a source file removes nothing downstream. Everything derived from it persists, stays retrievable, and answers questions as though it still existed.
-
Keeping an index fresh
A full rebuild is correct and unaffordable. Incremental sync is affordable and drifts. What decides the difference is how you detect that something changed.
-
The same document three times
Identical copies, near-copies and successive versions look alike and need opposite treatment. Deduplicating a version history destroys the record.
-
Boilerplate is not content
Navigation, footers and disclaimers extract perfectly and index as fact. Repeated across a corpus, they crowd out the pages they were attached to.
-
Extracting text from real documents
A PDF that extracts as scrambled reading order does not raise an error. It produces text, reports success, and poisons everything downstream of it.