Pages that are empty until something runs them
Four thousand pages were ingested from an internal site without a single error. Every one of them extracted to the same eighty words: a navigation menu, a footer, and the words “Loading content”.
The fetch succeeded, the response was well-formed, and the extractor read everything that was in it. The document was never in the response — it arrives afterwards, assembled by something that a pipeline fetching bytes does not run.
A successful response is not a document
The assumption underneath every simple web fetcher is that a document’s text is in the bytes the server returned. For a large and growing share of the web, including most internal tooling, it is not. The response is a container, and the content is fetched separately and inserted once something executes.
Nothing about that produces an error. The status is a success, the body is valid markup, the extractor finds text and returns it. The failure is measured in what is absent, and absence is the one thing no stage in an ingestion pipeline checks by default.
Worse, the pages are not empty. They contain furniture, and the furniture is identical across the source, so the corpus gains thousands of near-identical documents whose only content is a menu. Left alone they become the most repeated text in the collection, and they are retrievable, well-formed and completely useless.
Four shapes of the same problem
They arrive differently and they all look like a successful ingest.
The shell. Markup with no body content, waiting to be populated. Signature: every page in the source extracts to the same short text, and the extracted length has almost no variance.
The consent or region interstitial. The response is a notice asking permission before content is served. Signature: every page extracts to the same paragraph about cookies or availability, and it reads as a real document because it is one.
The login wall. A request without a session gets a sign-in page with a success status. Signature: the whole source extracts to a form. This one is especially deceptive when a subset of the corpus is public — some pages come back real, some come back as the login page, and coverage looks partial rather than broken.
The soft not-found. A URL that no longer resolves returns a helpful page with a success status instead of an error. Signature: identical text under many distinct URLs, and a corpus where deleted pages are still present as documents that say the page could not be found. The consequence is that deletion never propagates, because nothing in the fetch reports absence.
The common property is that each of these substitutes a plausible document for the requested one, which is the same failure class as a scanned page that extracts to a running header — a real result standing in for the missing one.
Running the page is a routing decision
A pipeline can execute a page rather than merely fetching it, and doing so is expensive: it needs a browser engine, it is slow per page, it consumes far more memory, and it introduces timing questions that a byte fetch does not have. So it cannot be applied to everything and it cannot be applied to nothing, which makes it the same shape of decision as deciding which documents need recognition.
The routing signal is available and cheap: fetch the bytes first, and check whether the result looks like a document. Very little text, text that matches other pages from the same source almost exactly, or a ratio of markup to prose that is wildly high — any of those routes the URL to the expensive path. Most sources are uniform, so in practice the decision resolves per source after a handful of samples rather than per page.
Two properties of the expensive path are worth knowing before relying on it. It has no completion signal. There is no moment at which a page is definitively finished, so the pipeline waits for a condition or a timeout, and a page that was slow rather than empty comes back empty. That produces an intermittent, unevenly distributed gap — the hardest kind to notice, because re-running one document by hand usually succeeds.
And it changes what the document is. The executed page may include content that was assembled from elsewhere, personalised to whatever identity the fetcher used, or ordered differently between runs. Two ingests of one URL can legitimately differ, which means a content hash over an executed page is a less reliable change detector than one over static bytes, and freshness will report edits that nobody made.
The pipeline
THE PIPELINE — web sources
· Response whose bytes contain the document
→ extract directly. No execution needed.
· Shell response, content assembled after
the fetch
→ FAILS SILENTLY. Every page in the
source extracts to the same navigation
and a loading message.
· Consent notice or login page returned with
a success status
→ FAILS SILENTLY. A real document about
cookies or sign-in stands in for the
whole source.
· Not-found page returned with a success
status
→ FAILS SILENTLY. Deleted pages persist
as documents, and deletion never
propagates.
· Executed page that timed out before
content arrived
→ FAILS SILENTLY. Intermittent, per-page,
and succeeds when retried by hand.
· Whether a source needs executing at all
→ CORPUS-DEPENDENT. Decided by sampling a
source, not by inspecting each page.One page, many URLs
The other half of ingesting a web source is that a URL is a poor document identity, for reasons that have nothing to do with rendering.
The same content is commonly reachable at several addresses: with and without a trailing slash, with tracking parameters appended, at a printable variant, at a mobile variant, under two paths that both resolve. Each address is a distinct key, so each becomes a distinct document, and the corpus fills with copies of one page that differ only in the address they were fetched from.
Pages usually declare which address is the real one, and taking that declaration as the identity rather than using the URL that happened to be crawled removes most of the problem. Where no declaration exists, the workable fallback is a normalised form of the URL — a stated rule about parameters, casing, fragments and trailing slashes — recorded as a rule rather than applied ad hoc, plus a text-level hash to catch what the rule misses.
Detecting all of it with two numbers
Neither of these needs a crawl to diagnose, and both belong in routine reporting rather than in an incident.
Extracted length, distributed across a source. A healthy web source has enormous variance in page length. A source whose pages are all within a few words of each other is returning the same thing every time, and the distribution says so without anybody reading a page.
The proportion of text a source shares with itself. The repetition count that boilerplate detection already computes answers this for free. When stripping furniture from a source leaves almost nothing, the source was almost entirely furniture — and that is the clearest possible statement that the fetch is not returning documents.
Both numbers matter more than they look, because this failure is uniform. A broken extractor damages some documents; a source that does not return its content damages all of them equally, which means sampling one page tells you about the whole source, and nobody ever samples one page.
What this stage hands on
Text that was actually in the document rather than in its container, under an identity that does not multiply with every equivalent address, and a recorded note of which pages had to be executed to get anything at all.
The limit is worth naming, because it is the reason to prefer a data source over a web source wherever one exists. A page assembled at request time has no canonical bytes, so nothing about it is exactly reproducible: two fetches may legitimately differ, a hash cannot cleanly distinguish an edit from a re-render, and the question “what did this document say when we ingested it” only has an answer if the extracted text was stored at the time.