Tagged “pipelines”
-
A website has no file list
A share can be enumerated. A site cannot: its document set is whatever the traversal reached, and nothing on the far end will report what was missed.
-
The page that checks who is asking
Four hundred documents in the corpus say 'Verify you are human'. Every fetch succeeded; the source served the barrier instead of the page.
-
What the converter threw away
HTML becomes markdown or plain text and the conversion never errors. Tables, alt text, heading levels and withdrawn clauses go missing on the way.
-
Counting what you did not ingest
Every corpus has documents that never made it in. Without a category for each way that happens, they look identical to documents that never existed.
-
Reprocessing a corpus you already ingested
An extraction fix only runs on documents ingested after it shipped. Everything already in the index keeps the old damage until something goes back for it.
-
Deletion is the hard one
Removing a source file removes nothing downstream. Everything derived from it persists, stays retrievable, and answers questions as if it still existed.
-
Keeping an index fresh
A full rebuild is correct and unaffordable. Incremental sync is affordable and drifts. What decides it is how you detect that something changed.
-
The same document three times
Identical copies, near-copies and successive versions look alike and need opposite treatment. Deduplicating a version history destroys the record.
-
Boilerplate is not content
Navigation, footers and disclaimers extract perfectly and index as fact. Repeated across a corpus, they crowd out the pages they were attached to.
-
Extracting text from real documents
A PDF that extracts as scrambled reading order does not raise an error. It produces text, reports success, and poisons everything downstream of it.
-
Ingesting the same document twice at once
Retries and queues deliver documents more than once. If a rerun adds records instead of replacing them, one document answers with two of everything.
-
Capturing who may see a document
Permissions live on folders and groups, not on documents. If ingestion does not record them, nothing downstream can reconstruct who may read what.
-
The reconciliation pass
Incremental sync has no way to prove it kept up. A periodic comparison of the index against the source is the only thing that finds what it missed.
-
Finding one bad document a month later
Someone reports a wrong answer citing a specific file. Ingestion logs are keyed by job and time, so nothing can say what happened to that one document.
-
Replacing what an edit invalidated
A document was edited and reprocessed correctly. The index now holds the new text and some of the old, because the previous records were never removed.
-
Formats with no reading order
Spreadsheets and slide decks have no sequence to extract. Flattening them produces values and phrases detached from the thing that gave them meaning.
-
Deciding which documents need OCR
Recognition is the most expensive stage in ingestion, and the documents that need it do not announce themselves. Routing has to be inferred per page.
-
Normalising text before anyone reads it
Invisible characters, split accents and soft hyphens break matching without changing how text looks. The damage is silent and the fix is one pass.
-
Separating extraction from indexing
One pass that reads a file and writes records means every fix re-reads the whole corpus. Keeping the extracted text makes reprocessing affordable.
-
A document ID that survives a move
Path-based identity breaks the moment a file moves. One rename produces a duplicate and an orphan at the same time, and neither of them errors.
-
Which date does a document have
A file offers half a dozen dates and none of them is when the document was written. A share migration set most of them to the day the migration ran.
-
The listing that was not complete
Everything downstream of enumeration is scoped by it. A listing that stopped early produces a smaller corpus that behaves exactly like a correct one.
-
Documents inside documents
Archives, attachments and embedded files hold most of what people search for. A pipeline that reads only the outer file ingests the wrapper and none of it.
-
The documents that will not ingest
Every corpus has files that error, time out, or crash the parser. Handled as log lines they leave the corpus quietly and never come back.
-
Pages that are empty until something runs them
A web source ingested cleanly and every page holds the same nav and a loading message. The fetch returned a shell that assembles its content elsewhere.
-
The order the stages run in
Hashing before normalising, fingerprinting before stripping, writing before deciding identity. Each is a stage in the wrong place and none of them error.