Documents inside documents
The signed contract is in the corpus. What is in the corpus is an email whose body says “please see attached”, and the attachment — eleven pages of terms — was never opened by anything.
No error was raised. The message extracted fine. A container was treated as a document, its content was the covering note, and the eight hundred files inside the quarterly archive on the same share got the same treatment.
A container is not a document
Real corpora are full of files whose content is other files. Archives, mail messages with attachments, documents with embedded objects, exported bundles, folders shipped as one download. In most collections a large share of the substantive material sits one level in, and often the outer file is pure packaging.
The default pipeline behaviour is the wrong one in both possible directions. A fetcher that hands the outer bytes to an extractor either produces nothing usable — an archive has no text — or produces the wrapper’s text and stops. Either way the ingest is recorded as a success, and the number of documents in the corpus is the number of containers rather than the number of documents.
The unit of retrieval has to be the unit a person would name. Nobody looks for the archive; they look for the contract inside it. So containers need to be expanded during ingestion, and expansion is a stage with its own decisions rather than a detail of reading a file.
Deciding what to keep after expanding
Three answers, and the right one varies by container type rather than by preference.
The children only. Correct when the container is packaging with no content of its own: a plain archive, a folder bundle, an export. The container becomes metadata on each child — where it came from — and is not a document.
The children and the container. Correct when the container has content that stands alone and the children also do. An email with attachments is the clear case: the message is a real document with an author and a date, and each attachment is a real document that happens to have arrived that way.
The container only. Correct when the children are not documents. An office file’s embedded fonts, images and theme resources are not content; a document with a spreadsheet embedded as an illustration probably is not either. Expanding indiscriminately fills the corpus with fragments nobody would ever search for.
Getting this wrong in the permissive direction is its own failure and less obvious than under-expansion. Recursing into everything produces enormous document counts made of resources, and it also produces a large number of copies: an archive containing a document that is also on the share directly means the same content is now in the corpus twice, at two very different addresses, and only duplicate handling will save it.
Identity for something with no address
A child inside a container has no location of its own, which breaks the usual approaches to document identity at exactly the point where the corpus grows fastest.
The workable key is a composite: the container’s identity plus a stable path to the child within it. That survives the container moving, which is the common case, and it stays stable across re-ingests as long as the container’s internal structure does not change. It does not survive the container being rebuilt with a different internal layout, which happens when somebody re-zips a folder — and then every child looks new while every old child looks deleted.
Two mitigations, neither complete. A content hash on the child detects that the new arrival is the same content as the vanished one, which is the same rename-detection reasoning applied one level down. And where the child carries its own identifier from the system that produced it, that beats any composite — an attachment that is a document from a management system usually has one embedded in its metadata.
The case with no good answer is a container that is regenerated on a schedule: a nightly export bundle whose name includes the date. Every run is a new container by any identity rule, so every child is new, and the corpus grows by the whole export every night. Recognising that shape early is worth more than any clever key, because the fix is a decision about what the source is rather than a pipeline mechanism.
The pipeline
THE PIPELINE — containers
· Archive or bundle with no content of
its own
→ expand. The children are the
documents; the container is metadata.
· Container read as a document
→ FAILS SILENTLY. The covering note or an
empty extraction is recorded as a
successful ingest of everything inside.
· Everything recursed into, without limits
→ embedded resources become documents and
the corpus count stops meaning
anything.
· Encrypted or password-protected container
→ cannot expand. Record as present and
unreadable, not as an empty archive.
· Child whose content also exists on the
share directly
→ a copy at a second location, not a
second document.
· Nested container that expands without
bound
→ depth and size limits are a
requirement, not a precaution.
· How deep to go, and into which types
→ CORPUS-DEPENDENT. Decided by where the
substantive content actually sits.Recursion needs limits it does not get by default
Expansion is the one ingestion stage where the input controls how much work the pipeline does, and that has to be bounded explicitly.
Depth. An archive inside an archive inside an archive is ordinary in backup material. Unbounded depth means the pipeline’s behaviour on one file is unpredictable.
Expanded size and child count. A small container can expand to an enormous volume, sometimes by accident and occasionally on purpose. The limits worth enforcing are on the output, checked as it is produced rather than predicted in advance, because a container does not have to declare what it holds.
Cycles and self-reference. Some container formats permit structures that expand forever. A depth limit catches most of it; tracking what has already been expanded within one document catches the rest.
Time per container. Expansion happening inside a per-document timeout means one hostile file cannot stall a batch — which pairs with reading untrusted files in a process that can die safely, since expansion is where malformed input most often takes a worker down rather than raising an error.
A container that hits a limit is not a failure to hide. It is a document partially ingested, which is its own state: some children in, a known number not, recorded as such. The alternative — expanding as much as fits and recording success — puts an arbitrary subset of a container’s contents in the corpus and no note of what was left out.
What children inherit, and what they must not
Expansion is where metadata gets fabricated, because the child has less of it than the container does.
Inherit the source and location. The child came from this container, at this address, on this share. That is the only way anybody finds the file again.
Inherit access, carefully. A child’s effective permissions are usually the container’s — nobody who cannot open the archive can read what is inside it. That inheritance has to be explicit rather than assumed, and it is one of the places where capturing access at ingest needs a rule rather than a copied field.
Do not inherit dates blindly. An archive’s modification time is when it was assembled; the child’s own recorded time is usually when it was written, which may be years earlier. Taking the container’s date makes every document in an old archive look new, which is a specific and consequential way of getting a document’s date wrong.
Do not inherit the container’s author. The person who assembled a bundle is not the author of what is in it, and treating them as such attributes hundreds of documents to whoever ran an export.
What this stage hands on
One document per thing a person would name, extracted from wherever it was actually sitting, keyed by an identity that includes its container, carrying the container’s location and access and its own dates — plus a record of every container that could not be fully expanded and how much of it was left.
The residual limit is that expansion multiplies every other problem in this pipeline rather than introducing new ones. Each child needs its own format handling, its own duplicate check, its own identity, its own failure record. The value of doing it well is not that it is interesting; it is that the alternative is a corpus whose document count looks reasonable and which is missing the specific files that people ask about most, because those are the ones somebody bothered to attach.