raggate tests retrieval behaviour as a black box. It only needs one thing from your pipeline: the ability to ask it for results as a given identity. Everything else — the verdicts, the evidence, the remedies — is built on top of that single seam.
You bring three things: a set of scenarios (an identity plus a query), an identity roster, and a permission snapshot describing who is allowed to read what.
| Class | Meaning |
|---|---|
| PASS | The chunk came back and the asking identity is authorized to read it. |
| LEAK | The chunk came back, reached the context window, and the identity is not authorized. This is what fails the build. |
| NEAR_MISS | A forbidden chunk was recalled by the retriever, but at a rank beyond the final cutoff or below the relevance floor — so it would have been filtered before reaching a model. It is recorded as a measurement, not treated as a pass: the same chunk one rank higher is a leak. |
The bundled two-tenant corpus contains three deliberately planted leak classes, so the gate's ability to catch them is observable rather than asserted:
An identity loses access to a project between snapshot versions. The chunks remain indexed and still recall for them.
One tenant's identity recalls another tenant's near-identical document. Tenant separation is absolute in the oracle: no grant can override it.
A document that was broadly readable is tightened to a smaller audience. The index does not know, and keeps serving it.
Alongside them sit 10 negative-control scenarios that must produce zero leaks. A gate that only ever says FAIL is not a gate — the controls are how you know it can still say PASS.
Finding a leak is only useful if something can be done about it. raggate carries a remedy catalog ordered safest-first, and deliberately prefers the rungs that change the least.
| Rung | Remedy | What it does |
|---|---|---|
| 1 | pre-filter | Narrow retrieval so the forbidden chunk is never a candidate. Deletes nothing. On Elasticsearch this is a filtered alias per tenant. |
| 2 | alias-rollback | Point retrieval back at the previous index or embedding version — the natural answer when an upgrade widened the access surface. |
| 4 | re-ingest | Rebuild affected documents with corrected metadata. |
| 6 | tombstone | Make a chunk unretrievable but recoverable. |
| 8 | hard-delete | Remove it. Last resort, and guarded behind an explicit flag. |
The catalog holds eight rungs; these are the load-bearing ones. Which remedies can actually be applied depends on the backend — a remedy needs a mechanism, and raggate refuses the apply rather than pretending when one is missing.
When one tenant's query recalls another tenant's document, the document is not the problem — it is somebody's legitimate file, correctly indexed. Deleting it would be destroying valid data to fix a retrieval-scope bug. That is why the pre-filter sits at rung 1: it changes who can reach the document rather than whether it exists.
Embedding upgrades are sold as recall improvements. They are also access-surface changes, and nothing in a relevance benchmark will tell you so.
The access-surface diff holds permissions constant and compares the reachable set under two configurations — the current embedding and the candidate. Anything that becomes reachable and is forbidden under the snapshot is reported.
$ raggate diff --corpus corpus --snapshot corpus/acl_snapshots/v2.yaml \ --scenarios corpus/scenarios --config-a baseline --config-b broad newly reachable: 17 docs; of which FORBIDDEN: 10 HEADLINE: the embedding change made 10 forbidden documents reachable with permissions unchanged scope: probe-scope — compared 26 probe queries, not a corpus-wide scan; a clean result covers these queries only
Verbatim from bash scripts/demo.sh on the sample corpus. Note the tool
reports its own scope limit in the output — a clean diff over 26 probes is evidence about those 26
probes, not a proof about your corpus.
Everything above sits behind a single seam: retrieve(query, identity, top_k).
If your pipeline can answer that, raggate can gate it.
Expose your pipeline — filters, reranking, query rewriting and all — behind one HTTP endpoint and point the gate at it. The contract is a documented request and response shape; a shim is typically a few dozen lines. What gets tested is your real pipeline, not a reimplementation of it.
If your endpoint omits a relevance score, the gate treats the chunk as fully relevant — a forbidden chunk cannot dodge the threshold by declining to report one.
Ten vector-store adapters ship in the repository, including sqlite-vec for local runs, Elasticsearch and OpenSearch, pgvector, Qdrant, Milvus, Weaviate, Redis and Azure AI Search. Scores are recomputed locally as exact cosine so a threshold means the same thing on every backend.
Adapters vary in what they can apply: some support the full remedy loop, others are detection-and-plan only. The CLI tells you which up front.