A thread is not a document

An answer quotes a decision that was reversed. The text it cites is real and it appears in the corpus more than a dozen times, because every reply in the thread quoted the message before it, and the original message was carried along in all of them.

Nothing was extracted wrongly. Each message really does contain that text. Email is the one format where the source deliberately duplicates its own content, and a pipeline that treats each message as a document inherits the duplication at full strength.

The reply chain is the loudest text in the corpus

A thread of twenty messages, each quoting its predecessor, contains the first message roughly twenty times, the second nineteen, and the last once. The repetition is inverted relative to usefulness: the oldest text is the most present, the most recent — usually the decision, the answer, the correction — is the least.

That has the same mechanism as boilerplate and a worse shape. Furniture is at least recognisably not content, so a repetition-based filter finds it. A quoted paragraph is content; it was content when it was written; it is just content that has already been counted.

The damage is that frequency and currency point in opposite directions. Whatever mechanism a retrieval system uses, a corpus in which superseded statements are the most repeated ones is a corpus that will keep surfacing them, and no downstream tuning fixes an input that says the wrong thing most often.

Forwarding makes it worse in a way worth anticipating. A forwarded thread re-quotes everything already quoted, sometimes with different quoting conventions applied by a different client, so the same paragraph appears with two or three distinct prefixes and does not look identical to a naive comparison.

Deciding what one document is

Three defensible answers, and the decision has to be explicit because the default falls out of whatever the connector returns.

One message is one document. Matches the source’s own unit, gives every document a real sender, recipient list and timestamp, and gives it a genuine source-native identifier. The cost is the duplication above, which has to be handled separately.

One thread is one document. Reads naturally — the conversation is the thing that has a subject and an outcome — and eliminates the quoting duplication if the thread is assembled from the newest message plus the distinct earlier bodies. The costs are real: a thread has no single author, no single date, and it keeps growing, so it is a document that changes every time somebody replies and needs its derived records replaced each time.

One message per document, with the quoted portion removed. In practice the most common workable answer. Each message carries only what its sender added, so the thread’s content exists exactly once across the set, distributed over the messages that contributed it.

None of these is right in the abstract. What is not defensible is having no answer, because the consequence is that document identity gets decided by the connector, and identity is the one decision nothing downstream can revisit.

Stripping the quoted part, and what it costs

Removing quoted text is a boilerplate problem with an unusually good signal and no clean solution.

The signals are the quote prefix, the attribution line a client inserts above the quoted block, and position — quoted material is nearly always contiguous and at the end. Any one of them is unreliable alone; a client that quotes without prefixes, a language whose attribution line does not match the pattern, or a reply typed underneath instead of above and each signal fails separately.

Two failure modes, and they are not symmetrical.

Under-stripping leaves the duplication in place. Noisy, and recoverable later by another pass over stored text.

Over-stripping removes text the sender wrote. The specific case is interleaved replies — somebody answering point by point inside the quoted block — where the answers are the only new content in the message and are structurally indistinguishable from what they are answering. Strip the quoted block and the message becomes empty; keep it and the message is mostly duplication. There is no rule that gets both.

The honest posture is to detect the interleaved case rather than solve it: a message whose quoted block contains lines that are not prefixed as quoted is probably an interleaved reply, and it is worth keeping whole and marking as such. A short list of exceptions is better than a rule that silently deletes replies.

The pipeline

THE PIPELINE — email

  · Message with a clean trailing quoted block
                    → strip it. The thread's content stays
                      present exactly once.

  · Thread ingested message by message with
    quotes intact
                    → FAILS SILENTLY. The oldest message
                      becomes the most repeated text in the
                      corpus and outweighs the outcome.

  · Interleaved reply inside the quoted block
                    → FAILS SILENTLY if stripped. The only
                      new content in the message is the part
                      that looks quoted.

  · Message with plain-text and rich
    alternatives
                    → pick one deliberately. Ingesting both
                      duplicates every message.

  · Attachment carried on the message
                    → a document in its own right, related
                      to the message, not the message body.

  · Whether a message or a thread is one
    document
                    → CORPUS-DEPENDENT. Decided by whether
                      answers are about conversations or
                      about statements.

Two bodies, and choosing one

Most messages carry the same content twice, in a plain-text part and a rich part, as alternatives meant for a client to choose between. A pipeline that walks every part and concatenates gets both, which duplicates every message in the corpus and produces a document where the same sentences appear consecutively in two formattings.

Pick one, per message, by a stated rule. The rich part usually has better structure and brings markup furniture with it; the plain part is cleaner and sometimes an auto-generated summary of the rich one rather than an equal. Neither is universally better, and the decision matters far less than making it once instead of taking whatever the parser yields.

The related trap is that walking parts recursively will also pull in attachments, inline images and nested forwarded messages as though they were body text. An attached document is a document inside a document and belongs to that problem, not to this one — but a parser that does not distinguish them produces a message body with a spreadsheet spliced into the middle of it.

Headers are the best metadata any source offers

The compensation for all of the above is that email is unusually rich in exactly the metadata other formats make you guess at.

A stable identifier per message, assigned by the source, unique, and durable across folders and moves. Most corpora have nothing this good.

An unambiguous author and audience. Sender and recipients, as identities rather than as strings in a document. This is also where access is captured: the participants are the natural permission set, recorded at ingest for enforcement elsewhere.

A real date, and a reliable one. Unlike a file whose modification time reflects when it was copied, a message’s send time is a fact about the content. Worth taking seriously when deciding which date a document has.

Thread membership. The reply relationships are recorded in the message itself, so the thread does not have to be inferred from subject lines — which is fortunate, because subject-line matching merges unrelated conversations that share a generic subject.

Two things to plan for. Signature blocks and legal disclaimers repeat across every message from an organisation and are ordinary furniture, findable by repetition within a source. And mailboxes contain a large volume of automated mail — notifications, receipts, alerts, calendar traffic — which is templated, enormously repetitive, and usually not worth ingesting at all. That is an exclusion decision, and one that belongs in the coverage ledger rather than in a filter nobody recorded.

What this stage hands on

One record per message or per thread, according to a decision somebody made on purpose, containing the text its sender actually contributed, with sender, recipients, date and thread membership as metadata, and attachments handed on separately as documents of their own.

What it does not resolve is the thing that makes email hard in the first place. A message is a fragment of a conversation, and its meaning frequently sits in the message before it — which means even a perfectly de-quoted mailbox contains text whose context has been deliberately removed, and it is the one corpus where stripping duplication and preserving meaning are genuinely in tension rather than merely difficult.