Tagged “extraction”
-
The page that checks who is asking
Four hundred documents in the corpus say 'Verify you are human'. Every fetch succeeded; the source served the barrier instead of the page.
-
What the converter threw away
HTML becomes markdown or plain text and the conversion never errors. Tables, alt text, heading levels and withdrawn clauses go missing on the way.
-
Boilerplate is not content
Navigation, footers and disclaimers extract perfectly and index as fact. Repeated across a corpus, they crowd out the pages they were attached to.
-
Extracting text from real documents
A PDF that extracts as scrambled reading order does not raise an error. It produces text, reports success, and poisons everything downstream of it.
-
Formats with no reading order
Spreadsheets and slide decks have no sequence to extract. Flattening them produces values and phrases detached from the thing that gave them meaning.
-
Deciding which documents need OCR
Recognition is the most expensive stage in ingestion, and the documents that need it do not announce themselves. Routing has to be inferred per page.
-
Normalising text before anyone reads it
Invisible characters, split accents and soft hyphens break matching without changing how text looks. The damage is silent and the fix is one pass.
-
Separating extraction from indexing
One pass that reads a file and writes records means every fix re-reads the whole corpus. Keeping the extracted text makes reprocessing affordable.
-
Which language is this document in
Language is a fact nothing records by default. Without it, translations look like duplicates and quality checks report clean text as damaged.
-
Documents inside documents
Archives, attachments and embedded files hold most of what people search for. A pipeline that reads only the outer file ingests the wrapper and none of it.
-
Pages that are empty until something runs them
A web source ingested cleanly and every page holds the same nav and a loading message. The fetch returned a shell that assembles its content elsewhere.
-
A thread is not a document
A twenty-message email thread contains the first message twenty times. Ingest it naively and the oldest, most-quoted text becomes the loudest claim.