Capturing who may see a document

Somebody searches the system and gets an answer drawn from a salary review they have never had access to. The file is correctly locked down on the shared drive. The record derived from it carries a title, a path, a date and no permission information at all.

The pipeline read the document with credentials that could see everything, which is what a service account is for, and then wrote text with no memory of the fact that most people cannot read it.

Permissions are the one thing that cannot be added later

Ingestion loses information at several stages, and most of it is recoverable by re-reading the document. Permissions are not: they are a property of the source at the moment of reading, they change without touching the document, and there is no way to work out from text who was allowed to see it.

That makes access capture a now-or-never field, in the same category as whether a page was recognised rather than read. It costs almost nothing at ingest. Reconstructing it afterwards means enumerating the source again and matching every document back up, which is a project.

The division of labour is worth stating once and then respecting. Capturing access is an ingestion concern. Enforcing it is not. A pipeline that records permissions has done its part; what the query path does with them is somebody else’s design. But the query path cannot invent what was never written down, so a missing capture is an unfixable defect wearing the costume of a downstream bug.

Permissions do not live on documents

The reason this is harder than it first looks is that almost no source stores a self-contained answer to “who may read this”.

They are inherited. A document’s effective access is usually computed from the folder, site, library or workspace containing it, possibly several levels up, possibly with overrides at any level. The document itself frequently carries nothing.

They are mediated by groups. The entry says a group may read it. Resolving that to people means expanding the group, which may nest other groups, and the expansion is a different system’s data with its own freshness.

They include denials and exceptions. Some models allow explicit exclusion that overrides an inherited grant, so the effective set is not a union of what appears in the entries.

They include mechanisms that are not lists at all. A document shared by link, accessible to anyone in an organisation, or public. Those are not sets of people and a capture that only understands lists of principals will represent them wrongly or drop them.

None of that is exotic; it is how ordinary document systems work. The consequence for ingestion is that the permission field is not a value to copy but something to resolve, and resolution has a cost per document that grows with how many levels have to be walked.

Two ways to record it, and both go stale

Snapshot the resolved set. At ingest, compute the effective principals and store them against the document. Self-contained and cheap to consult later. It is a copy of a mutable fact, so it is wrong from the moment anything changes, and the drift is invisible — the record looks as authoritative on the day the group changed as it did the day it was written.

Record a reference instead. Store what the source said: this document inherits from that container, these groups are granted, this is the entry as it stood. Resolution happens elsewhere, against current data, so group membership changes are picked up without reprocessing. The cost is that something outside ingestion has to be able to resolve it, which is a dependency at query time rather than a lookup.

The pattern that works in practice is both: the reference as the durable truth, plus a resolved snapshot and the time it was resolved. The snapshot is usable on its own if the resolver is unavailable, the reference makes it repairable, and the timestamp is what turns “this may be stale” from an anxiety into a number.

What is not viable is a snapshot with no timestamp and no reference, because nothing about it can be verified or refreshed. That is the common case, and it is how a permission capture becomes worse than none — an authoritative-looking field that was accurate once.

The pipeline

THE PIPELINE — access capture

  · Source exposes an effective permission set
    per document
                    → capture it, with the time it was read.

  · Access not captured at all
                    → FAILS SILENTLY. Records are readable by
                      everyone, look exactly like correct
                      records, and cite documents nobody
                      was cleared for.

  · Resolved snapshot with no timestamp and no
    reference
                    → FAILS SILENTLY. Stale from the first
                      group change and impossible to verify
                      or refresh.

  · Permission changed on a parent container
                    → thousands of documents change access
                      while every timestamp stays put.

  · Cursor-based sync only
                    → FAILS SILENTLY. Permission changes do
                      not move modification times, so no
                      incremental pass sees them.

  · Share-by-link and organisation-wide access
                    → not a set of people. Must be
                      representable, not flattened.

  · Whether to snapshot, reference, or both
                    → CORPUS-DEPENDENT. Decided by whether a
                      resolver is available at query time.

Access changes are invisible to change detection

The property that makes this operationally hard, and it is worth stating separately from the capture problem: a permission change usually touches no content and no modification timestamp.

So the document does not appear in any cursor query, its hash is unchanged, and nothing in the freshness mechanism has any reason to look at it. The captured access is now wrong and the pipeline is running perfectly.

Two aggravating factors. Changes are made at the container level far more often than at the document level, so one action alters the effective access of thousands of untouched documents — meaning the population needing re-capture is defined by a change to something that is not a document. And group membership changes are external to the source entirely: somebody leaves a team, and every snapshot that expanded that group is now over-permissive with no event anywhere near the corpus.

The mechanisms that find these are the same ones that find deletions, which is not a coincidence — both are changes that are not edits. A source that emits permission events is the good case. Failing that, re-capturing access during the reconciliation pass works, because that pass is already enumerating the source and can carry a permission field beside the identity at little extra cost. Where containers are the unit of change, watching the containers rather than the documents is much cheaper than re-reading everything under them.

Handling a revocation as an ordinary update is the whole mechanism, and it is why a removal of access resembles a deletion closely enough to be worth treating with the same seriousness.

Every derived record has to carry it

The last mile is the one that gets dropped. Permissions captured onto a document record and not propagated onto the records derived from it are permissions that exist in the pipeline and not in the index.

The requirement is the same as for identity: whatever a retrieval path can reach must carry, or be able to reach, the access information. Which means access is one of the fields that has to survive the whole flow — including surviving a replacement after an edit, where a rebuild of the derived set from text alone will quietly drop it.

One consequence worth anticipating: documents that were collapsed as copies may have had different permissions at their different locations. Collapsing content while keeping locations means the surviving record needs the access information for each location, not the access of whichever copy happened to be kept. Taking the most permissive by accident is how a restricted copy of a public document becomes readable.

What this stage hands on

Text and metadata plus, for every document, a record of what the source said about who may read it — as a reference where one exists, with a resolved set and the time it was resolved, propagated onto every derived record.

Where the stage stops is exact. It does not filter anything, it does not decide what any user sees, and it has no opinion about how a query should apply what it captured. It writes down a fact that is only available at the moment of reading, and the reason to be careful about it is that this is the one ingestion field whose absence does not produce a worse answer — it produces a correct, well-cited answer delivered to somebody who should never have seen the document.