About Getting Documents In
Getting Documents In covers one stage of a retrieval pipeline: everything between the source files and clean text with metadata attached.
Where the scope starts and stops
Starts at whatever the corpus actually is — a shared drive, a wiki export, a document management system, an inbox, twenty years of PDFs of varying provenance.
Stops at clean text plus metadata, handed to a chunker.
That boundary is deliberate and enforced. This site does not cover where to cut text into chunks, how to embed it, which index to search it with, how to combine lexical and vector signals, how to measure whether retrieval works, or how to run the database it lives in. Those are all substantial subjects and all of them are covered elsewhere; a post here that drifts into them has failed its own scope test.
What that leaves is a stage most projects treat as a solved prerequisite: extraction, normalisation, boilerplate removal, duplicate handling, document identity, freshness, and deletion.
The thesis
Ingestion fails silently.
Every other stage of a retrieval system tells you when it breaks. An embedding call errors. A query returns nothing. A database runs out of memory. Ingestion does none of that — it produces plausible text and reports success, and the consequence appears weeks later as an answer that cites a cookie banner or quotes a policy that was replaced in 2019.
That property drives everything about how this site is written. Posts describe failure signatures — what the bad output actually looks like — rather than listing tools, because recognising the signature is the whole skill. Anyone can install an extractor. Knowing that two-column text has been interleaved line by line, from looking at it, is the part that takes a bad afternoon to learn.
No tools, no code
You will not find library names, product recommendations or code on this site. Three reasons:
Tooling in this area changes fast enough that specifics rot, and a stale recommendation is worse than none. The failure modes, by contrast, are stable — a scanned page has behaved the same way for decades.
Naming a tool invites the wrong question. “Which extractor should I use” is much less useful than “how do I detect that extraction failed”, because you will need the second one regardless of your answer to the first.
And the pipeline decisions that matter — what identity a document has, when a duplicate is actually a version, what happens when a source disappears — are design questions with no library that answers them.
Who it is for
An engineer whose retrieval prototype worked and whose production system does not, who has narrowed the problem to the data rather than the model, and who needs to know what to look for. Not a beginner introduction, and not a survey.
On numbers
There are no benchmarks, accuracy figures or corpus statistics on this site, because there are none that could be honestly quoted from here. Where arithmetic is useful, the numbers are obviously round, hypothetical, and labelled as illustrative.