Your RAG system's biggest risk isn't hallucination. It's permissions.

Ask a team how they are testing their retrieval system and you will hear about hallucination. They have a set of questions, they check whether the answers are grounded in the source documents, and they track a faithfulness score. That work is worth doing. It is also aimed at the second-most-important failure mode.

The first is quieter. The system retrieves a real passage, from a real document, cites it correctly, and shows it to somebody who was never allowed to open that file. Every quality metric passes. The answer is faithful, grounded, and accurate. It is also a data incident.

How it happens

Almost always the same way. The indexing job runs as a service account, because that is the straightforward way to read every document in the corpus. The account needs broad access to do its job, so it gets broad access. The embeddings that come out the other side carry no memory of who could read the source.

At query time the system searches that flattened index. The retrieval layer knows about semantic similarity. It does not know that the HR investigation file is restricted, because by the time the text became a vector, that fact was gone.

The permissions were not bypassed. They were discarded at ingestion, and nothing downstream ever knew they existed.

Why the usual mitigations don't hold

Three fixes come up repeatedly, and each fails in a way worth understanding before you rely on it.

Telling the model not to

A system prompt instructing the model to withhold restricted material is a request, not a control. The passage is already in the context window. You are asking the model to be discreet about information you handed it, and you cannot audit that decision after the fact.

Filtering after retrieval

Better, and still leaky. If you retrieve ten chunks and drop three the user cannot see, the seven that remain were selected in the context of the three. Worse, the shape of what is missing is often informative on its own — a user learning that four documents about their own department exist but cannot be shown has learned something.

One index per team

This works until permissions stop being neatly hierarchical, which is immediately. Matter-level access in a law firm, per-patient access in a clinic, and per-deal access in finance are all overlapping rather than nested. You end up with an index per user, and then you have a rebuild problem instead of a security problem.

What actually works

Carry the permission with the chunk, and filter inside the search rather than around it.

1. 1At ingestion, capture the ACL of the source document and store it as metadata on every chunk derived from it.
2. 2At query time, resolve the caller's identity into the set of principals they hold — user, groups, roles, and any matter or case scoping.
3. 3Pass those principals into the vector search as a pre-filter, so restricted chunks are never candidates for retrieval in the first place.
4. 4Re-check at generation time against the live source of truth, because group membership changes between indexing and query.

# The filter belongs INSIDE the search, not after it.
results = index.query(
    vector=embed(question),
    top_k=8,
    filter={"acl_principals": {"$in": caller.principals()}},
)

The difference between a pre-filter and a post-filter looks like an implementation detail and is not. A pre-filter changes what is retrievable. A post-filter changes what is displayed, after the restricted content has already influenced ranking and already entered your logs.

The retention problem nobody plans for

One more consequence, because it arrives about a year in. If your retention policy says a document class is deleted after seven years, that policy now applies to the embeddings and to any cached generation derived from them. Most teams discover this when someone asks whether a deletion request actually deleted anything.

Plan for it at design time. Keep a durable link from every chunk back to its source document so deletion can cascade. Retrofitting that link into a live index is unpleasant work, and it is always scheduled at the least convenient moment.

None of this is difficult engineering. It is just work that has to happen at ingestion, which is the point in the project where everyone is most eager to see the demo work and least interested in access control.