The page that checks who is asking

Somebody searches for a support article and gets back a document whose entire text is “Verify you are human. Checking your browser before you continue.” There are four hundred of them. They came from the source that was supposed to be the most authoritative in the corpus, they were fetched over two hours on a Tuesday, and not one of them recorded an error.

Nothing here is a parsing problem. The bytes were served, the markup was valid, and the extractor read them correctly. The source simply decided not to give the pipeline the page.

A fetch is a request, not a read

Reading a file from a share is a read: the same bytes come back whoever asks, and if they do not, something failed. A web fetch is not that. It is a request evaluated by a program at the far end, which chooses what to return based on properties of the requester — and every one of those choices produces a valid response that an ingestion pipeline will happily turn into a document.

This is a different failure from a page whose content arrives after the fetch rather than in it. That one is about when the document is assembled. This one is about whether the far end decided you should have it, and it can happen to a page that is completely static.

Five requester properties routinely change what comes back, none of which the pipeline usually records.

The address it came from. Ranges belonging to hosting providers are treated differently from ranges belonging to consumer connections. A pipeline moved from a laptop to a server can start receiving barriers for the same source that worked in testing, with no change to the code.

The client it declares itself to be. Some sources serve a reduced page, or a redirect to a landing page, to a client string they do not recognise.

Whether it carries a session. A source that is public to a logged-in browser may not be public at all; the tester who checked it was signed in without noticing.

How fast it is asking. This is the one that produces half-broken corpora, described below.

Where it appears to be. Region-specific consent screens, regional content variants and jurisdictional blocks all return a real document that is not the one that was requested.

Then there is the sixth case, which is not a property of the requester so much as a verdict on it: the challenge page served when a source concludes that it is not talking to a person.

The barrier as a purchased capability

It is worth being precise about what “handling” a challenge means, because it is now a line item rather than something pipelines do themselves. Serply’s description of its own fetching infrastructure states that requests go out from that infrastructure and that most challenges are cleared before a response is returned, with proxy rotation and retries happening behind the endpoint. That is a fair summary of what the category sells: the requester-identity problem, moved to somebody else’s side of an API call.

Which is the practical point for an ingestion pipeline, whether or not it buys that. The far end’s decision about the requester is a real stage of your pipeline. It has inputs, it has outcomes, and if nobody owns it then it is being decided by whatever defaults a fetching library ships with.

There is also a fork here that ingestion work should be explicit about rather than clever about. Either the source is yours, or you have an agreement with it, or you are working around its refusal. The first two are solved with credentials and an agreed request rate, which are also the versions that keep working next quarter. The third is a decision somebody should have made deliberately and written down — and for an internal corpus it is almost never the right one, because the source will grant access if asked.

The rate limit poisons half a corpus

The uniform failures are the kind people eventually notice: every page from a source is the same short notice, and the source stands out immediately. The rate limit is worse, because it produces a corpus that is partly correct.

A run starts, the first several hundred pages come back real, the source’s threshold trips, and everything after that is a throttle notice returned with a success status. The result is a source that is genuinely ingested for its first slice and barrier text for the rest, split at an arbitrary point that corresponds to nothing in the content. Coverage looks partial rather than broken. And the failure does not reproduce: the next morning, fetching any one of the bad URLs by hand works perfectly, so the report reads as unreproducible and gets closed.

The tell is that the boundary is temporal, not structural. Sort a source’s documents by fetch time and the bad ones are contiguous. Nothing about their content or their place in the site explains the split.

The pipeline

THE PIPELINE — what the far end served

  · Response that is the requested document
                    → extract and continue.

  · Challenge, consent wall or sign-in page
    returned with a success status
                    → FAILS SILENTLY. The barrier becomes a
                      document, repeats across the source,
                      and reads as the corpus's most
                      confident statement.

  · Throttle notice after a rate threshold
                    → FAILS SILENTLY, and only for part of
                      the source. Contiguous in fetch time,
                      succeeds on manual retry, closed as
                      unreproducible.

  · Which properties a source reacts to
                    → CORPUS-DEPENDENT. Established by
                      sampling the source, not by reading
                      its documentation.

Asserting on the fetch instead of the parse

The check that catches all of this belongs at the fetch, before extraction, and it is cheap because barrier pages have properties that real pages do not.

A per-source expectation. For each web source, one recorded fact that a real page from it satisfies — a marker that appears in the site’s own furniture, or a minimum plausible length. A response that fails the expectation is not a document; it is a fetch failure that happened to return text, and it should be recorded as a document that did not ingest rather than stored.

Identity across responses. Barrier pages are byte-identical to each other far more often than real pages are. A count of exactly repeated whole responses within one run finds them without knowing what they say, which matters because the wording varies by source and by year.

Fetch-time clustering. For each source, the distribution of failures over the run. Real gaps are scattered; served barriers arrive in a block.

None of these need a human to look at a page, and all of them are computable during the run rather than weeks later.

Record the fetch, not only the text

The metadata that makes this diagnosable is thrown away by almost every pipeline, and it costs nothing to keep: the status, the final address after redirects, the identity the fetcher presented, the timestamp, and the verdict of the expectation check. Without those, a suspect document is a piece of text with no history, and tracing it back a month later is guesswork.

What this stage hands on

Text that the source actually served in response to a request it accepted, plus a record of the request that produced it and an explicit outcome for every fetch that produced something else.

The limit is that no amount of reading the text can settle this. A challenge page is a genuine, well-formed, grammatical document; it fails no extraction check and no encoding check, and the only thing that distinguishes it from a real page is a statement, made in advance and per source, about what a real page from there looks like. Without that statement, the pipeline has no way to tell a refusal from an answer.