Normalising text before anyone reads it

A search for a term that is definitely in a document returns nothing. You open the extracted text, find the word, copy it, paste it into the query, and now it matches. The word you typed and the word in the document are different strings that render identically.

Nothing was corrupt and nothing errored. Extraction produced valid text, and valid text includes several ways to write the same thing.

Why identical-looking text is not identical

Text has more degrees of freedom than a reader can see. The same visible word can differ in the bytes underneath it for at least four independent reasons, and document formats exercise all four.

Composed and decomposed accents. An accented letter can be one character, or a plain letter followed by a combining mark. Both render the same. Neither is wrong. They are different strings, and one will not match the other.

Compatibility variants. Full-width forms, ligature characters, superscript digits, the several distinct characters that all look like a hyphen or an apostrophe. Word processors and typesetting engines insert these deliberately, and they survive extraction intact.

Invisible characters. Zero-width spaces and joiners, byte-order marks landing mid-document, soft hyphens, non-breaking spaces, directional marks. They occupy no visual width and they are present in the string.

Whitespace and line structure. Where the format cared about line width rather than about sentences, you get hard line breaks mid-sentence, runs of spaces used as layout, and tabs standing in for table columns.

These differences do not degrade matching, they eliminate it. A term that differs by one invisible character is not a near miss; it is a different token entirely.

The hyphen case, because it is the worst one

Print-typeset documents break words across lines. Depending on the source and the extractor, the break survives extraction as a real hyphen, a soft hyphen, or a hyphen plus a newline.

So information arrives as infor- mation, and it is now two fragments that are not words. Neither half matches anything. The word has vanished from the document while remaining perfectly legible to a human reading the extracted text, which is why this survives review.

Repairing it means joining a line ending in a hyphen with the line that follows. The obvious caution is that some hyphens are real: a genuinely hyphenated compound that happens to fall at a line end should keep its hyphen. There is no rule that gets this right in every case, and the useful posture is to prefer the repair — a wrongly joined compound is still a searchable word, while a split word is nothing at all.

Order matters

Normalisation steps interact, and running them in the wrong order quietly undoes work.

A workable sequence: decode to a single encoding first, then remove invisible characters, then apply a compatibility and composition normalisation, then repair hyphenation, then collapse whitespace, and only then consider case.

The reason for that order is mundane. Removing soft hyphens after collapsing whitespace works; doing it after a compatibility mapping may not, because the mapping can have already turned the character into something else. Collapsing whitespace before repairing hyphenation destroys the line structure the repair depends on. Each of these is a ten-minute bug that presents as “normalisation seems not to have run”.

The pipeline

THE PIPELINE — normalisation

  · Bytes with a declared, correct encoding
                    → decode. Record what it was.

  · Bytes with a wrong or absent declared
    encoding
                    → FAILS SILENTLY. Decodes to plausible
                      wrong characters. Accented words
                      become mojibake and stop matching,
                      and the document still looks fine
                      in aggregate.

  · Composed versus decomposed accents
                    → one normalisation form, applied
                      everywhere, including to queries.

  · Zero-width and directional marks
                    → FAILS SILENTLY. Invisible in every
                      review, and the token containing one
                      matches nothing.

  · Word split across a line by hyphenation
                    → FAILS SILENTLY. Both halves are
                      non-words. Repair by joining.

  · Whether to fold case and strip accents
                    → CORPUS-DEPENDENT. Depends on the
                      languages present and on what
                      distinctions the corpus needs kept.

Case and accent folding are not free

The temptation is to lowercase everything and strip diacritics, on the grounds that it makes matching more forgiving. Sometimes right, sometimes destructive, and the difference depends on the corpus.

Folding case loses the distinction between an acronym and an ordinary word, which matters in corpora where short capitalised strings are identifiers. Stripping accents merges words that are genuinely different in several languages. In a technical corpus, case is often meaningful in part numbers and symbols.

The safer general posture is to normalise the things that are unambiguously representational — encoding, invisible characters, composition form, whitespace, hyphenation — and to leave case and diacritics alone at the ingestion stage, because they carry meaning that the pipeline cannot recover once folded. Folding is cheap to apply later to a copy; it is impossible to undo.

Normalise the query the same way, or none of it works

The part that catches people: normalisation is only useful if both sides of a comparison get it.

Text normalised at ingest and a query that was not, or was normalised with different rules, produces exactly the original symptom in the other direction — a document that contains the term, a query that looks like the term, and no match.

The ingestion side of this is to record which normalisation was applied, as metadata, in the same way an extractor version is recorded. What the query path then does with that is not this stage’s business, but it cannot do the right thing if the information was never written down.

Detecting the damage without reading everything

Cheap corpus-wide counts find most of it, and each one is a routing signal rather than a verdict.

Replacement-character frequency. Any occurrence means a decode went wrong somewhere. A per-document count sorts the corpus by how badly.

Invisible-character counts by class. Should be near zero after normalisation. If they are not, a source is inserting something the pass does not handle.

Rate of lines ending in a hyphen. High rates identify print-typeset sources that need the hyphenation repair, and they cluster by source rather than scattering.

Proportion of tokens that are not words. Catches mojibake, split words and encoding damage in one number. Comparing that proportion across sources is more informative than its absolute value, because what counts as normal depends entirely on the corpus.

Distinct normalisation forms present. If both composed and decomposed accents appear in the stored text, the pass did not run or ran inconsistently.

Where normalisation stops

It cleans representation. It does not clean content — a document full of navigation furniture is a different problem, and one that no amount of character handling touches.

It also cannot repair a decode that already went wrong. Once bytes have been interpreted with the wrong encoding and written down, the original characters are usually gone; what remains is a plausible-looking string that no normalisation form will turn back. That is the argument for getting encoding right at the moment of reading rather than treating it as something to clean up afterwards, and for keeping the original bytes available when the guess turns out wrong.

What this stage hands on

Text where one visible string is one byte string, whitespace means what it says, words are not split by layout, and the transformation applied is recorded rather than assumed.

It is the least interesting stage in the pipeline and one of the two or three most consequential, because every comparison made anywhere downstream — matching, hashing, duplicate detection, anything at all — compares strings, and strings that differ invisibly will fail every one of those comparisons without producing a single error along the way.