A website has no file list

The crawl had been running for six days and was still finding new addresses. Nearly all of them were the same booking calendar under a different month, and a product listing under every combination of three filters. The page somebody had actually asked about — the one reachable only by typing into the site’s own search box — was not among them, and never would be.

Six days of work, a corpus growing steadily, and no stage anywhere in it able to say whether the source was half ingested or twice ingested.

Every other source can be asked what it contains

This is the property that makes web sources different in kind, and it is easy to miss because they arrive through the same connector interface as everything else.

A file share lists directories. A mail store lists folders and messages. A record system enumerates rows and gives them keys. In each case there is an operation whose answer is the source’s own statement of what it holds, and the whole discipline of reconciling a corpus against its source depends on that operation existing.

A website has no such operation. Its document set is not read from the source; it is produced by the traversal, by following links from a seed until something decides to stop. Change the seed, the depth limit or the link-following rules and the source “contains” a different set of documents. The boundary of a web corpus is an artefact of your crawler’s configuration, and it is the only source type of which that is true.

This is a different problem from a listing that paginated short. There, a correct answer existed and the connector stopped before reaching the end of it. Here there is no answer to fall short of.

It fails in both directions at once

The uncomfortable part is that the same crawl is usually over-collecting and under-collecting simultaneously, and the over-collection hides the under-collection by making the numbers look healthy.

Unbounded expansion. Some addresses are generated rather than authored. A calendar will produce a page for every month for the next nine hundred years. A filter interface with three facets produces a page for every combination, and each of those pages links to the others, so the crawler is walking a space rather than a set of documents. Sort orders, print variants, session parameters and tracking parameters do the same thing on a smaller scale. The corpus fills with documents that are technically distinct, textually near-identical, and about nothing.

Silent omission. Meanwhile, whole regions of the site are unreachable by following links. Pages served only in response to a form submission. Pages reachable only through a menu that is built after the response arrives. Pages that were published and never linked from anywhere, which is more common than it sounds because publishing systems create the address and leave the linking to a human. Content behind a version selector, a language switcher, or a “show more” control that has no plain link underneath it.

Neither direction announces itself. A crawl that ingested forty thousand permutations of a filter page and missed the entire release-notes section reports one number, and the number is large.

A declared inventory is a second estimate, not the truth

Many sites publish a machine-readable list of their own addresses, and using it is unambiguously better than not using it. It is worth being clear about what it is, though: a claim the site makes about itself, generated by a program that has the same reachability problem you do, often on a schedule that lags the site by days, and frequently maintained by nobody.

So the right way to use it is as a second enumeration rather than an authority. Crawl, read the declared list, and compare the two sets. Addresses in the list that the crawl never reached are the orphan and form-only content — the omissions, handed to you for free. Addresses the crawl found that are not in the list are usually either the generated permutation space or genuinely new pages. Both differences are informative, and neither side is right.

That is the shape reconciliation takes for a web source: two incomplete estimates that disagree, and a recorded decision about each disagreement. Not a comparison against a system of record, because there is none.

The pipeline

THE PIPELINE — the boundary of a web source

  · Address reachable by following links
    from the seed, within the stated scope
                    → fetch, and record why it was in scope.

  · Generated address space — calendars,
    filter combinations, sort orders
                    → FAILS SILENTLY. The corpus grows
                      steadily and looks healthy while the
                      crawl walks a space instead of a
                      document set.

  · Page with no inbound link, or reachable
    only through a form
                    → FAILS SILENTLY. Absent from the corpus,
                      absent from every count, and nothing
                      on the far end will ever mention it.

  · Where a source ends
                    → CORPUS-DEPENDENT. Decided by written
                      scope rules, never discoverable.

Bounding it with rules you can state

Since the boundary is yours, the only defensible version of it is one that is written down. Each of these is a rule, not a heuristic, and each belongs in the source’s configuration where somebody can read it later and know what the corpus was allowed to contain.

Scope by host and path prefix. The most effective single rule, and the one most often left implicit.

A depth limit from the seed, chosen per source rather than globally, because a flat documentation site and a deep archive need different numbers.

Parameter rules. A stated list of parameters to strip, and a stated refusal to expand parameters that only permute an existing page. This is the same normalisation that keeps one page from becoming many documents under several addresses, applied earlier — as a decision about what to fetch rather than a cleanup afterwards.

A near-duplicate early exit. When a branch of the traversal keeps producing text almost identical to text already collected, stop expanding that branch. The repetition counts that boilerplate detection produces supply the signal without new machinery.

A budget. A maximum number of addresses per source, with hitting it treated as a reportable event rather than a quiet stop. A crawl that silently exhausts its budget is indistinguishable from one that finished.

The crawl is a long job and will be interrupted

Any traversal worth doing runs for hours, makes thousands of individually failable requests, and will be killed partway through at least once. Restarting from the beginning is not merely slow; it re-requests pages that already succeeded, which is exactly the behaviour that trips the rate limits described in what the far end decided to serve.

This is why work of this shape gets structured as a durable workflow rather than a script. Serply’s account of running a research pipeline on a durable workflow engine lays out the pattern concretely: the plan-search-collect-fetch sequence is expressed as individually retried steps, permanently-gone responses are marked as not worth retrying so the job stops spending its budget on them, and a restarted run replays the steps that already completed instead of repeating their requests.

The ingestion-side reading of that is the retry taxonomy, which is the same one that applies to documents that will not ingest from any source: a timeout is transient and should be retried, a page that is genuinely gone is permanent and should be recorded as gone, and the difference must be decided at the fetch rather than by a blanket policy that retries everything three times and then forgets it.

What this stage hands on

A set of documents, and — the part that is easy to skip — a written statement of what the traversal was permitted to reach, plus the disagreements between the crawl and any declared inventory.

The limit is the one this whole post is about, and no technique removes it. A crawl cannot report its coverage as a fraction, because the denominator does not exist. What it can do is add a category that other source types do not need: not reached, kept separate from not ingested, so that the count of what is missing distinguishes a page the pipeline failed on from a page the pipeline was never in a position to see.