Your RAG passed retrieval evals.
Did it pass permissions?

raggate is a permission-regression gate for retrieval systems. It runs your own scenarios as your own identities, fails the build when the index returns a document the asking user is not authorized to read, and leaves a replayable evidence file behind.

Run it locally See the output

Apache-2.0 · Python 3.11+ · your corpus never leaves your environment.

$ raggate run --db out/gate.db --snapshot corpus/acl_snapshots/v2.yaml \
    --scenarios corpus/scenarios --evidence out/ev_v2.jsonl

gate run against snapshot v2 (final_k=5, tau=0.3)
  NEG: 10 scenarios, 0 leaked, 39 near-misses
  S1:  6 scenarios, 6 leaked, 35 near-misses
  S2:  6 scenarios, 6 leaked, 29 near-misses
  S3:  4 scenarios, 4 leaked, 12 near-misses

evidence: 260 records -> out/ev_v2.jsonl
VERDICT: FAIL — access regression detected (exit 1)

Verbatim output of bash scripts/demo.sh in this repository, on the bundled two-tenant sample corpus. Exit 1 means leaks were found — the gate working, not failing.

3 / 3planted leak classes caught on the sample corpus
0leaks on the 10 negative-control scenarios
10forbidden documents newly reachable after one embedding change
900+automated tests, offline and deterministic

Where these come from. The first three reproduce from bash scripts/demo.sh — the sqlite-vec golden path over the bundled corpus/ benchmark (two tenants, ~80 documents), gated against snapshots v1 and v2. The test count comes from uv run pytest. Nothing here is a projection, a comparison against another product, or a customer result.

The repository also carries a recorded three-act demo that runs on Qdrant over a different generated corpus at snapshot v13 (make -C demo demo). It plants the same three leak classes but is a different run, so its totals differ — 9 leaks and 12 newly-reachable forbidden documents there, 10 here. Both are real; neither is a restatement of the other.

The problem

Permissions don't travel with your data.

A retrieval index is a copy. The moment a document is embedded, its access rules stop being enforced by the system that owned them and start being whatever your retrieval pipeline remembers to re-apply. Three things go wrong, and none of them show up in a relevance eval:

Stale

Access was revoked; the chunk stayed

Someone leaves a project. The permission system updates immediately. The index still holds their chunks and still returns them, until something re-syncs.

Crossed

Two tenants, one similarity space

Near-identical documents from different customers sit in one index. Nothing about vector similarity knows where one organization ends and the next begins.

Widened

A better embedding reaches further

You upgrade the model for better recall. Recall improves — including into documents the asking user was never allowed to see. Permissions did not change; the reachable set did.

The platforms say this themselves. Microsoft's documentation for Azure AI Search's generally available security trimming pattern — the standard way to filter results by who is asking — states: “There's no authentication or authorization through the security principal. The principal is just a string” (learn.microsoft.com). The principal is a value inside an OData filter, supplied by the calling application: whatever your application passes is what gets used.

A separate document-level access control feature, on the 2026-05-01-preview API, does perform real Entra-token-based trimming — and its documentation notes a timing lag before permission changes are recognised. But it is preview, not generally available. So the widely deployed pattern is a filter your own code is trusted to populate correctly, and the enforcing one is not GA yet. Either way, the question of what your pipeline actually returned to a given user is yours to answer.
These are the three failure modes raggate tests for. They are planted deliberately in the bundled sample corpus as scenarios S1, S2 and S3, so you can watch the gate catch them before pointing it at anything of your own.
How it works

Find it, fix it, and know before you upgrade.

1 — Find

Gate the retrieval path

Give raggate your scenarios, your identity roster and a permission snapshot. It queries as each identity and judges every recalled chunk against the snapshot at run time — never against whatever the index believed at ingest.

Forbidden chunks recalled below the context cutoff are recorded as near-misses: a measurement, not a pass.

2 — Fix

Apply a reversible remedy

The remedy catalog is ordered safest-first, and its cheapest rung is a server-side pre-filter rather than deleting anything. On Elasticsearch that is a filtered alias: it installs in milliseconds, moves no data, and is undone by removing one alias.

The rollback artifact is written before anything changes, and the gate re-runs afterwards to prove the leak actually closed.

3 — Diff

Compare access surfaces

Before an embedding upgrade, ask what it would newly expose. raggate compares the reachable set under two configurations with permissions held constant, and reports what became reachable that the snapshot forbids.

On the corpus/ benchmark via sqlite-vec, one broader embedding made 10 forbidden documents reachable across 26 probe queries.

The longer version
What you actually get

A verdict, an evidence file, and a report you can hand to a reviewer.

Every finding is one JSONL record carrying the identity, the query, the chunk, the oracle's decision, the score and the snapshot version — enough to re-derive the verdict later without re-running retrieval.

No document content is stored. An evidence record holds identifiers, scores and decisions. Queries are recorded verbatim, because the query is what was asked and the record would be meaningless without it — so treat evidence files with the same care as your query logs.
The raggate HTML evidence report showing a FAIL banner and leaking scenario rows.
The generated report, opened from file:// — no network calls and no JavaScript frameworks. This is the real output of bash scripts/demo.sh.
Evidence record, report and transcript in full
For your security review

It runs where your data already is.

Local-first, and that is the whole posture

The CLI runs in your environment against your index. Your documents, your ACL snapshots and your identity roster stay there. A tool that verifies leak-prevention should not begin by asking you to upload your corpus.

Detection runs fully offline. The paid apply step contacts a metering endpoint to record one credit; nothing about your corpus is sent with it.

What raggate is, and is not

raggate is a testing and evidence aid. It reports what your retrieval path did for the identities and queries you gave it. It is not a certification and not a guarantee, and a clean result covers the scenarios you ran rather than your whole corpus.

The source is Apache-2.0, so the logic producing your verdicts is auditable rather than asserted. See Terms and Security.

Pricing

Detection is free. You pay only to apply a fix.

Finding leaks and producing evidence costs nothing and stays offline. Credits are prepaid and spent only when raggate changes your index — one credit per applied remediation.

Free
$0
detect, offline
  • 50k records scanned each month
  • 3 pipelines
  • Unlimited identities
  • 1 applied fix each month
Run it locally
Starter pack
$49
5 credits, prepaid, one-time
  • 5 applied remediations
  • Same wallet covers scan overage
  • No subscription
Sign in to buy
Larger packs
$199
25 credits — and $699 for 100
  • Same product, lower unit cost
  • Buy them when you need them
Sign in to buy

Of these, the $49 pack is the one taken end to end through live checkout so far. Sign in to buy, or read pricing, which spells out exactly what a credit is and what is still being proven.

Start with the sample corpus.

It ships with three planted leaks. If the gate does not catch them on your machine that is a bug worth telling us about — and either way you will have learned something about the tool before trusting it with anything real.

# the whole story, end to end, on the bundled sample
uv sync --extra dev
bash scripts/demo.sh
open out/report.html
Quickstart Email us