Which language is this document in
The extraction quality report flags nine hundred documents as badly damaged: word-like token rates near zero, dictionary hit rates near zero, the signature of severe encoding corruption. Every one of them is perfectly clean text in a language the check was not built for.
Nothing in the pipeline was wrong except an assumption nobody wrote down. Language is a property of every document in the corpus, it is not recorded anywhere, and half the ingestion stages quietly assume they know what it is.
Nothing records it unless something is asked to
Document formats can declare a language and the declaration is unreliable in a specific way: it reflects the authoring environment rather than the content. A template created in one locale produces documents whose declared language is that locale forever, regardless of what anybody types into them. A scanned page has no declaration at all. A web page’s declared language is a property of the site, so a site with content in three languages declares one.
So language has to be inferred from the text, which makes it an ingestion decision — it is derived from the extracted content, it is cheap at that moment, and it is a fact that every later stage would like to have. It is also one of the few fields that can be recomputed later from stored text, which makes it lower-stakes than permissions or recognition flags, but only if the text was stored.
The cost of not recording it is not that queries in other languages fail. It is that several ingestion stages behave as though the corpus were monolingual, and each of them fails differently and silently.
Detection is easy on prose and unreliable on everything else
Identifying a language from a reasonable quantity of running text is close to a solved problem, and the reason it disappoints in practice is that a lot of a corpus is not a reasonable quantity of running text.
Short documents. A title, a slide fragment, a form with six filled fields. Detection on a handful of words is a guess, and it returns a confident-looking answer.
Text that is not prose. Tables of numbers, part codes, addresses, logs, identifiers. There is no language here and a detector will name one.
Mixed documents. The common case in any organisation that operates across regions: a message in one language quoting a document in another, a report with a translated summary, a contract with parallel columns. A document-level answer is wrong by construction, because the document does not have one language.
Recognised text. Character-level confusions from a recognition pass push text away from every language model at once, so detection quality degrades exactly where the other quality signals are already unreliable.
The practical shape is to detect at a finer granularity than the document — per section, per paragraph, per message — and to record both a dominant language and the fact that others are present. A single label plus a confidence is enough for most decisions; a single label with no confidence is what turns a guess into a recorded fact.
The pipeline
THE PIPELINE — language
· Substantial prose, one language
→ detect and record it, with a
confidence.
· Language taken from the format's own
declaration
→ FAILS SILENTLY. Reflects the authoring
template, not the content, and looks
authoritative.
· Mixed-language document given one label
→ FAILS SILENTLY. Every stage that acts
on the label acts wrongly on part of
the document.
· Word-likeness checks run against one
language's expectations
→ FAILS SILENTLY. Clean text is reported
as corrupt, and real corruption hides
in the noise.
· Translation of a document already ingested
→ not a duplicate and not a version. A
third relationship.
· Short text, tables, codes and identifiers
→ no language to detect. Record that,
rather than a confident guess.
· Whether language-specific handling is
worth building
→ CORPUS-DEPENDENT. Set by the
distribution, which is usually
unmeasured until somebody counts.Translations are a third relationship
The duplicate taxonomy — identical copies, near duplicates, successive versions — needs a fourth entry as soon as a corpus is multilingual, and it does not fit any of the three.
A translated document is not a copy: the text is entirely different. It is not a near duplicate: no similarity measure over text will group them. And it is not a version: neither supersedes the other, both are current, and both are correct.
What it is, is the same content with two expressions, and the useful handling is to record the relationship rather than to collapse or rank. Both documents stay, each labelled with its language, each pointing at the other as a translation. That is a metadata fact, it usually comes from the source system when it comes from anywhere, and inferring it from text is not realistic at ingest.
Two failures worth anticipating. Where a source stores translations as versions of one record — which some do — the pipeline will see a series of edits and replace the derived set each time, so the corpus holds whichever language was written last. And where a translation is partial, a document can be one language’s content with another language’s headings and boilerplate, which reads as a mixed document and is really an incomplete one.
Language interacts with the stages before it
The reason to record language during ingestion rather than treating it as an annotation is that earlier stages have already made assumptions about it.
Normalisation. Folding case and stripping accents is destructive in different amounts depending on the language: harmless in some, meaning-changing in others. A single global rule is a decision to damage part of the corpus, and the language label is what makes a per-document rule possible at all.
Hyphenation repair. Joining a line that ends in a hyphen is a rule about print typesetting conventions, and those conventions differ. In a language that hyphenates compounds routinely, aggressive joining produces more false repairs.
Quality signals. This is the one that produces the report at the top of this post. Word-like token rates and dictionary hit rates are the strongest extraction-failure detectors available, and both are defined against a language. Run against the wrong one, every document reads as damaged — and because the false positives are overwhelming, the real damage becomes invisible inside them. Worse for languages that do not put spaces between words, where a check counting space-delimited tokens reports one enormous token per line and concludes the document is a single unbroken string.
Boilerplate detection. Repetition is measured within a source, and a source serving several languages has several sets of furniture. Grouping by language before counting repetition makes each set findable; not doing so dilutes every count and pushes real furniture below the threshold.
The pattern in all four is the same. None of them errors on unexpected input. Each of them produces a plausible result computed against the wrong expectation, which is the site’s usual story arriving through an unusual door.
What this stage hands on
Text with a recorded language per document and, where the document has more than one, per section — with a confidence, with “no detectable language” as a legitimate value, and with translation relationships recorded where the source knows about them.
The narrow point of the field is worth being clear about, since it would be easy to overclaim. It does not make anything multilingual and it does not decide how anything downstream should treat the corpus. What it does is stop four ingestion stages from silently applying one language’s assumptions to a corpus that has several, and give the coverage ledger a dimension to break down by — which is usually how anybody discovers what the distribution actually is.