Getting Documents In

The prototype worked. It worked against a folder of clean Markdown that somebody had already tidied, and the answers were good enough to get the project approved.

Then it was pointed at the real corpus — twenty years of PDFs, some of them scans, a wiki export with navigation on every page, four copies of the same policy document with different filenames, and a shared drive where files move — and quality fell off a cliff. Nothing errored. Every stage reported success.

That is the subject of this site: the stage between a pile of real files and clean text with metadata, ready to be chunked. It gets less attention than any other part of a retrieval system and it is where most of the quality is actually lost.

The reason it is hard is not that the work is complicated. It is that ingestion fails silently. A PDF whose columns were interleaved line by line does not raise an exception; it produces text. A page of navigation links extracts perfectly and is indexed as content. A document that was superseded two years ago is still there, still retrievable, still perfectly formatted, and still wrong. Every one of those ships as success and surfaces later as an answer nobody can explain.

So the posts here are about recognising failure signatures, not about tools. What scrambled extraction looks like. Why boilerplate outranks content when it appears on nine hundred pages. Why the same document arriving three times is three different problems. How an index goes stale, and why deletion is the hardest operation in the whole pipeline.

The scope stops deliberately at clean text plus metadata. Where to cut that text, how to embed it, how to search it and how to measure the result are all real questions, and all of them are somebody else’s. This site hands over a clean input and nothing more — which is the part everybody assumed they already had.