Which date does a document have

The most recent version of a procedure, according to the system, is a document from 2019. Two people independently check and find a 2023 revision sitting in the same folder, ingested, present, and stamped with an earlier date than the document it replaced.

Nothing was computed wrongly. The pipeline recorded the file’s modification time, the 2019 file was touched by a migration last spring, and the 2023 file was not.

A document has several dates and needs one

Ask a source when a document is from and it will offer some subset of the following, all real, all different, and none of them labelled as the answer:

Filesystem created time. When this copy of the bytes came into existence at this location. Not when the document was written, unless the document has never been copied.

Filesystem modified time. When these bytes last changed at this location. Moves it for edits, and also for copies, restores, permission tools, virus scanners and bulk operations.

The source system’s own modified time. Usually better, because a document management system tracks its records rather than its files. Still a fact about the record, so it moves when metadata changes.

A date inside the document’s own metadata. The authoring application’s created and modified fields. Genuinely about the document — and inherited by anything made from a template, so a whole series of documents can carry the creation date of the template.

A date in the content. A cover page, a revision table, a header, an effective date. Almost always the date a reader means.

Ingest and extraction time. When the pipeline saw it. A fact about the pipeline, not the document, and the only one of these that is reliably correct about what it claims.

The document’s date is a decision, and the default is whatever the connector put in the obvious field. That default is a filesystem timestamp, which is the weakest candidate on the list.

Copies and migrations reset the ones people use

The failure at the top of this post is not exotic; it is the normal history of any share that has been in use for a decade.

A share migration copies files to new storage, and depending on how it was done every file’s created and modified times become the date of the migration. A restore from backup does the same to whatever it restored. A sync tool that rewrites files rather than moving them does it continuously. A user opening and saving a document without changing anything moves its modified time by a year.

Two consequences, and they point in opposite directions, which is why neither is self-correcting.

Timestamps that moved make old documents look new. Every date-based judgement is then wrong in favour of stale content, including the superseding decision about which of two versions is current — and that is a decision the pipeline makes and records, not something a reader can override later.

Timestamps that did not move make edited documents look unchanged, which is the drift problem that freshness mechanisms exist for. A cursor over modification times inherits every lie in the field it reads.

The tell for a corpus with damaged timestamps is a spike: an implausible number of documents sharing one date, or sharing a date that is a weekend, or a whole source whose dates begin abruptly at a point in the past with nothing older. That is a cheap thing to plot and it is usually never plotted.

The pipeline

THE PIPELINE — dates

  · Source system reports its own document
    date
                    → capture it, and record which field
                      it came from.

  · Filesystem modified time used as the
    document's date
                    → FAILS SILENTLY. A migration or a
                      restore stamps the whole corpus with
                      one day and old documents read as new.

  · Timestamp not moved on edit
                    → FAILS SILENTLY. The document is never
                      reprocessed and the index answers
                      from the old text.

  · Date inherited from a template
                    → FAILS SILENTLY. A whole document series
                      shares one creation date that belongs
                      to none of them.

  · Date parsed from content in an ambiguous
    numeric format
                    → FAILS SILENTLY. Day and month swap for
                      twelve days in every month and produce
                      a valid, wrong date.

  · Timestamp with no offset recorded
                    → the day is uncertain by one either way.

  · Which date a corpus's readers mean
                    → CORPUS-DEPENDENT. Contracts, news and
                      procedures mean different fields.

The date in the content is the best one and the hardest

Where a document states its own date, that is the date a reader means, and extracting it is worth real effort. It is also the least tractable field in this post.

Locating it. A cover page, a revision history table, a header on every page, an “effective from” clause, or nowhere. In a corpus assembled from many sources the location differs per source, and per-source rules are the only thing that works. They are also stable, which makes them worth writing once.

Ambiguous numeric formats. A date written as three numbers is two different dates depending on convention, and both parse. This produces a valid wrong date for roughly twelve days in every month and is undetectable per document — you can only find it by noticing that a source’s date distribution is strange.

Multiple dates that mean different things. Written, reviewed, approved, effective, expires, superseded. A revision table has several rows. Picking the largest is not right; picking the first found is not right either. The field being captured has to be named, so that whatever consumes it knows what it got.

Dates that are about something else. A document referring to a meeting in 2011 is not from 2011. Any extraction that scans for date-shaped text and takes the first hit gets this wrong constantly.

The honest posture is a hierarchy with the source recorded: use the source system’s date where one exists, prefer a content date where a per-source rule can find one reliably, fall back to a file timestamp, and record which of those was used on every document. That last part is what makes the whole field trustworthy — a date with a known provenance can be reasoned about, and a date in a column called date cannot.

Timezones, and the day that moves

Small, unglamorous, and it produces a class of confusion that costs people afternoons.

Timestamps arrive with an offset, in a stated zone, or with nothing. Where nothing is stated, an assumed zone is being applied by whatever parsed it, and for a timestamp near midnight the assumption changes the date. Documents then appear on the wrong day, and a date-partitioned enumeration can put a document in a partition that another pass does not read — which is one of the ways a partitioned listing leaves gaps.

Two rules avoid all of it. Store the instant with its offset rather than a local date, and convert only when displaying. And where a source supplies a date with no time and no zone at all — which is normal for a content date — record it as a date rather than converting it into an instant at midnight in some zone, because that conversion invents precision the source did not have.

What this stage hands on

For each document: a document date, the field it was taken from, and separately the pipeline’s own view of when the source last changed and when the document was last processed — the two timestamps whose gap is drift.

Where this stops is worth stating, because dates are unusually tempting to over-interpret. Ingestion captures which date a document has and where that date came from. What anything downstream does with recency is not this stage’s concern, and the reason to be careful anyway is that a wrong date does not produce a visibly wrong answer. It produces a confidently current answer from a superseded document, which is the same failure as an unpropagated deletion arriving through a field nobody thought was load-bearing.