What the converter threw away

A question about which support tier includes phone contact comes back with a passage that reads “Standard Professional Enterprise 12 24 unlimited yes no yes”. Every one of those tokens is really on the page. It was a table, and the numbers were under headings, and now they are in a row with nothing left to say which number belongs to which tier.

The fetch worked. The page was complete. The damage happened in the two lines of code between the response and the text, in a stage that most pipelines do not think of as a stage at all.

The conversion nobody owns

Between “we have the bytes of a web page” and “we have the text of a document” there is a transform. It takes markup and produces something flatter — markdown, or plain text, or markdown-flavoured text — and in most pipelines it is a default nobody chose, called with no arguments, reviewed by nobody.

It is also the last moment at which the document’s structure exists. After it runs, headings, tables, figures, emphasis and citation targets are either represented in the output or they are gone, and nothing downstream can recover what was not written down.

The conversion cannot fail loudly, because it has no notion of a correct result. Given markup it produces text. Given markup whose meaning lives somewhere it does not look, it produces less text with the same confidence. There is no exception, no warning, and no count of what was dropped.

Read the promise, not the format name

“Converted to markdown” is not a specification. Converters differ in what they carry across, and the practical way to find out what one does is to read what it claims to preserve — the claim is usually accurate, and what is missing from the claim is what you are about to lose.

That is a real exercise, not a rhetorical one. Serply’s page-fetch endpoint, which returns either markdown or the unmodified HTML, describes its markdown output as one in which headings, prose, lists and links survive. Take that at face value and it is a genuinely useful statement, because it is also a list of four things and the web page in front of you contains rather more than four kinds of structure. Every converter has such a list, explicit or not. The ones that publish it are easier to plan around than the ones that do not.

The general shape holds across tools: the well-served core is headings, paragraphs, lists and links, and everything below is where the differences live.

Six things that leave without a trace

Each of these has a signature — a way the damaged text reads — because that is how you recognise it in your own corpus rather than in the abstract.

Tables reduced to a stream of cells. The value keeps its characters and loses its coordinates. Signature: a passage with no grammar, short labels and bare numbers alternating. This is the same loss that formats with no reading order produce from a spreadsheet, arriving one stage earlier and from a source everyone assumed was text.

Heading hierarchy that was never in the markup. A page whose section titles are styled to look like headings without being marked as headings converts to an unbroken run of paragraphs. Signature: a long document recorded with zero headings. The inverse is just as common and harder to spot: heading markup used for visual size, so a sidebar label and a photo caption arrive as top-level sections of the document.

Alt text and captions. For a diagram, the alt text is frequently the only sentence in existence that says what the diagram shows. Drop it and the document keeps the paragraph that says “as shown below” and loses the below. Signature: dangling references to figures that are not in the text.

Footnote and reference anchors. The marker survives as a digit, the target lands somewhere else or nowhere, and the connection between them is not represented. Signature: numerals fused into words — “the revenue3 grew” — and a block of unattached citation text at the end of the document.

Withdrawn text promoted to current text. Struck-through and marked-as-removed passages are ordinary characters once the markup around them is gone. Signature: an answer that quotes a clause which the page itself visibly shows as deleted. This one is worse than a missing passage, because the corpus now asserts something the source explicitly retracted.

Frames that were only visual. Callouts, warnings, admonitions and quoted material merge into the prose around them. Signature: “Do not run this against a production database” sitting mid-paragraph as though it were the instruction rather than the warning about it. Preformatted regions lose their line structure the same way, and a listing that depended on its line breaks arrives as one long line for normalisation to guess at.

The pipeline

THE PIPELINE — markup to text

  · Headings, prose, lists, links
                    → carried across by essentially any
                      converter. Safe to assume.

  · Tables, alt text, footnote targets,
    struck-through passages
                    → FAILS SILENTLY. Text arrives shorter,
                      well-formed and confident; a retracted
                      clause reads as current, and a value
                      loses the label that meant something.

  · Structure carried by styling rather
    than by markup
                    → FAILS SILENTLY. A structured document
                      is recorded as one flat run with no
                      sections at all.

  · How much fidelity a source needs
                    → CORPUS-DEPENDENT. Decided by reading
                      converted output beside the original
                      for a sample of each source.

Fidelity is a per-source decision

There is no setting that is right for the whole web. A documentation site is mostly prose with a few tables that carry the actual answers; a specification is mostly tables; a knowledge base is mostly short articles with warnings that must not be flattened into their surroundings. The cost of preserving more is that the output is noisier, and noise is not free either — a table rendered as pipes and dashes is structure to a reader and clutter to anything counting words.

So the decision is made per source, and the way to make it is unglamorous: convert ten pages, open them beside the originals, and read for the thing that is missing rather than the thing that is wrong. Ten pages is enough because the loss is systematic. A converter that drops alt text drops all of it.

The one thing worth doing globally is recording what survived: whether the document has headings at all, whether any table structure was represented, how many images had descriptive text. Those counts per source are how a converter change is noticed later, and a document with a strange profile is findable only if somebody stored the profile.

Keep the input while the choice is still open

The reason to store the fetched bytes alongside the converted text is precisely this stage. A conversion decision made once, wrongly, and applied to a hundred thousand pages is a re-fetch of a hundred thousand pages to undo — unless the input is still on disk, in which case it is a re-run of one cheap stage. That is the same argument for keeping extraction separate from indexing, and web sources make it sharper, because a page fetched today may not exist to fetch again.

What this stage hands on

Text that still carries the parts of the document that meant something, plus a recorded note of which structural features the source had and which the conversion represented.

The limit is honest and worth stating: no conversion is lossless, and the goal is not fidelity but a known loss. A pipeline that can say “tables from this source are flattened, and here is how many documents that affects” is in a completely different position from one that produced clean-looking text and never asked what it cost.