Boilerplate is not content

Someone asks the system a question and gets back a cookie consent notice. Or a navigation menu. Or the same legal disclaimer that appears at the bottom of nine hundred pages, quoted confidently as though it answered something.

The extraction was perfect. The disclaimer really is on the page. That is the whole problem: boilerplate is not malformed content, it is well-formed content that happens not to be what the document is about.

Why repetition is the damaging part

A single footer on a single page is harmless noise. The failure comes from scale, and it has a specific mechanism worth understanding.

Boilerplate appears on every page of a corpus. Whatever it says therefore becomes the most frequently stated thing in the entire collection, by a wide margin. A phrase in a footer on nine hundred pages is present nine hundred times; the actual subject of any one of those pages is present once.

That asymmetry hurts in two independent ways.

It crowds out real content. If a page’s extracted text is two hundred words of navigation and seventy words of substance, most of what represents that page is furniture. Pages become similar to each other — they share their furniture and differ only in the small part that matters.

It becomes the highest-confidence answer to vague questions. Ask something general and the most repeated text in the corpus is a plausible match. The system is not malfunctioning; it is returning the thing the corpus says most often.

What counts as boilerplate

Broader than most people first assume, and the categories behave differently.

Structural furniture. Navigation, breadcrumbs, sidebars, “related articles”, pagination, search boxes, social buttons, cookie and consent notices, skip-to-content links. Never content, always removable.

Legal and administrative text. Confidentiality footers, copyright lines, retention notices, regulatory boilerplate, email signature blocks and their disclaimers. Almost never what anyone is searching for — but occasionally the corpus is about these, so removal should be a corpus decision rather than a universal rule.

Print furniture. Running headers and footers, page numbers, “continued overleaf”, watermarks like DRAFT or CONFIDENTIAL rendered into the text layer.

Template scaffolding. In a document management system, forms with fixed field labels. Every instance carries identical labels; only the filled values differ. Here the labels are not exactly boilerplate — they are the only thing giving the values meaning — so they need keeping with their values rather than stripping or indexing alone.

Repeated preamble. Standard introductions, “about this document” sections, and the same three paragraphs of context pasted at the top of every report in a series.

Detecting it without hand-written rules

Hand-written selectors work for one source and break on the next. The general technique is cheaper and does not need maintaining: find the text that repeats across documents, and treat repetition itself as the signal.

Concretely, look at lines or short blocks and count how many distinct documents each appears in. A block appearing in one document is content. A block appearing in most documents is furniture. There is a threshold in between that depends on the corpus, and the useful part is that you do not have to know what the boilerplate says in advance — it falls out of the frequency distribution.

Two refinements make this reliable.

Compare within a source, not across sources. Two different systems have different furniture. A phrase that repeats constantly in one wiki may be genuine content in a document set from elsewhere.

Watch for the legitimate high-frequency line. Some real content repeats — a standard clause that is actually the subject of a compliance question, a product name in every heading. This is why the threshold is a corpus decision and why removals are worth logging so a bad rule can be found later.

For print documents there is a stronger and simpler signal: a short line appearing at the same position on many pages of the same document is a running header or footer, essentially always.

The pipeline

THE PIPELINE — boilerplate removal

  · Text block appearing in one document
                    → content. Keep.

  · Text block appearing across most documents
    in a source
                    → furniture. Remove, and log what was
                      removed.

  · Furniture left in place
                    → FAILS SILENTLY. Extracts perfectly,
                      indexes as fact, and becomes the most
                      frequently stated thing in the corpus.

  · Over-aggressive removal
                    → FAILS SILENTLY too. Real repeated
                      content disappears and nothing reports
                      a missing answer — only its absence.

  · Form labels around filled values
                    → keep WITH the values. The label is
                      what gives the value meaning.

  · Where the repetition threshold sits
                    → CORPUS-DEPENDENT. Decided by looking
                      at the frequency distribution, not by
                      a default.

Over-removal is the mirror failure

Worth stating as plainly as the first one, because the fix for boilerplate creates it.

An aggressive filter deletes real content, and that failure is even quieter than the original. Missing boilerplate shows up as bad answers you can inspect. Missing content shows up as an answer the system cannot give, and there is nothing in the output pointing at the reason. A question simply gets a worse answer than it should have, forever, and no metric flags it.

Two habits make this recoverable. Log what was removed, per document, so a suspicious gap can be traced to a filter rather than assumed to be absent from the source. And keep the raw extracted text as well as the cleaned text, so a rule can be re-run rather than needing the whole corpus re-fetched.

Structure is worth keeping while you strip

Boilerplate removal is usually the last point in the pipeline where structural information still exists — which makes it the right moment to capture it rather than discard it.

Headings, section paths, list membership, table association, the position of a passage within its document: all of that is visible now and gone once text is flattened. Recording it as metadata costs nothing and is the difference between a passage that knows it came from a section called Termination and a passage that is just words.

What that metadata is later used for is not this stage’s business. Capturing it is.

What this stage hands on

Text with the furniture gone, structure recorded as metadata, and a log of what was stripped from each document.

The log is the part people skip and the part that pays. Boilerplate rules are wrong at first, on some sources, in ways that only show up as slightly worse answers weeks later — and the difference between diagnosing that and shrugging at it is whether anything wrote down what was thrown away.