Tagged “metadata”
-
A website has no file list
A share can be enumerated. A site cannot: its document set is whatever the traversal reached, and nothing on the far end will report what was missed.
-
What the converter threw away
HTML becomes markdown or plain text and the conversion never errors. Tables, alt text, heading levels and withdrawn clauses go missing on the way.
-
Counting what you did not ingest
Every corpus has documents that never made it in. Without a category for each way that happens, they look identical to documents that never existed.
-
Reprocessing a corpus you already ingested
An extraction fix only runs on documents ingested after it shipped. Everything already in the index keeps the old damage until something goes back for it.
-
Deletion is the hard one
Removing a source file removes nothing downstream. Everything derived from it persists, stays retrievable, and answers questions as if it still existed.
-
Keeping an index fresh
A full rebuild is correct and unaffordable. Incremental sync is affordable and drifts. What decides it is how you detect that something changed.
-
The same document three times
Identical copies, near-copies and successive versions look alike and need opposite treatment. Deduplicating a version history destroys the record.
-
Capturing who may see a document
Permissions live on folders and groups, not on documents. If ingestion does not record them, nothing downstream can reconstruct who may read what.
-
Finding one bad document a month later
Someone reports a wrong answer citing a specific file. Ingestion logs are keyed by job and time, so nothing can say what happened to that one document.
-
Formats with no reading order
Spreadsheets and slide decks have no sequence to extract. Flattening them produces values and phrases detached from the thing that gave them meaning.
-
Normalising text before anyone reads it
Invisible characters, split accents and soft hyphens break matching without changing how text looks. The damage is silent and the fix is one pass.
-
Separating extraction from indexing
One pass that reads a file and writes records means every fix re-reads the whole corpus. Keeping the extracted text makes reprocessing affordable.
-
A document ID that survives a move
Path-based identity breaks the moment a file moves. One rename produces a duplicate and an orphan at the same time, and neither of them errors.
-
Which date does a document have
A file offers half a dozen dates and none of them is when the document was written. A share migration set most of them to the day the migration ran.
-
The listing that was not complete
Everything downstream of enumeration is scoped by it. A listing that stopped early produces a smaller corpus that behaves exactly like a correct one.
-
Which language is this document in
Language is a fact nothing records by default. Without it, translations look like duplicates and quality checks report clean text as damaged.
-
Documents inside documents
Archives, attachments and embedded files hold most of what people search for. A pipeline that reads only the outer file ingests the wrapper and none of it.
-
The documents that will not ingest
Every corpus has files that error, time out, or crash the parser. Handled as log lines they leave the corpus quietly and never come back.
-
The order the stages run in
Hashing before normalising, fingerprinting before stripping, writing before deciding identity. Each is a stage in the wrong place and none of them error.