Everything below was generated by running bash scripts/demo.sh in this
repository: the sqlite-vec golden path over the bundled corpus/ benchmark (two
tenants, ~80 documents), gated against snapshots v1 and v2. Nothing here is a
mock-up or an illustration — it is the output, copied out.
A second, recorded demo in the repository runs the same three leak classes on
Qdrant over a different generated corpus at snapshot v13
(make -C demo demo) and therefore reports different totals. Counts on this page belong to
the run named above and should not be compared line-for-line with that one.
One self-contained file you can send to a reviewer or attach to a change ticket. It
opens from file://, makes no network calls, and uses no JavaScript framework — the same
discipline the rest of the tool follows.
That link is the real 55 KB file produced by the demo run, committed as-is. The corpus behind it is synthetic and ships with the repository, so there is nothing confidential in it.
The evidence file is JSONL: one line per recalled chunk. This is a real line from the run, reformatted onto multiple lines for readability and otherwise untouched.
{
"scenario_tag": "S1",
"leak_id": "5e58fa3557365502",
"identity": "bob@acme",
"identity_tenant": "acme",
"query": "Project Atlas architecture and design decisions",
"chunk_id": "acme-atlas-01#0",
"doc_id": "acme-atlas-01",
"chunk_tenant": "acme",
"oracle_verdict": "DENY",
"classification": "LEAK",
"score": 0.764057447960416,
"rank": 0,
"snapshot_version": "v2",
"final_k": 5,
"similarity_threshold": 0.3,
"retrieval_k": 10,
"pipeline_config_hash": "d4a5ea72f1c1be54",
"timestamp": "2026-07-27T07:41:09.352823+00:00",
"severity": 5.4
}| Field | What it is for |
|---|---|
identity, query | Who asked, and what they asked. Without both, the record cannot be reasoned about later. |
doc_id, chunk_id, chunk_tenant | Which chunk came back, and which tenant owns it — identifiers, not content. |
oracle_verdict, classification | What the permission snapshot said (ALLOW/DENY) and what the gate concluded (PASS/LEAK/NEAR_MISS). |
score, rank, final_k, similarity_threshold |
Enough to re-derive the classification: whether the chunk was inside the context window and above the relevance floor. |
snapshot_version, pipeline_config_hash | Which policy and which pipeline configuration produced this. Two runs with the same knobs share a hash; change one and the hash changes. |
severity | A ranking aid for triage, present on findings. |
The gate is a command that exits non-zero when it finds something, which is all CI needs from it. This is the demo run, verbatim.
== 2/5 Run the gate against v1 (baseline, before the permission change) == gate run against snapshot v1 (final_k=5, tau=0.3) NEG: 10 scenarios, 0 leaked, 39 near-misses S1: 6 scenarios, 0 leaked, 26 near-misses S2: 6 scenarios, 6 leaked, 28 near-misses S3: 4 scenarios, 1 leaked, 11 near-misses evidence: 260 records -> out/ev_v1.jsonl VERDICT: FAIL — access regression detected (exit 1) (v1 gate exit code: 1 — pre-existing cross-tenant S2 leaks) == 3/5 Run the gate against v2 (after bob loses proj-atlas + roadmap goes VP-only) == gate run against snapshot v2 (final_k=5, tau=0.3) NEG: 10 scenarios, 0 leaked, 39 near-misses S1: 6 scenarios, 6 leaked, 35 near-misses S2: 6 scenarios, 6 leaked, 29 near-misses S3: 4 scenarios, 4 leaked, 12 near-misses evidence: 260 records -> out/ev_v2.jsonl VERDICT: FAIL — access regression detected (exit 1) (v2 gate exit code: 1 — the change INTRODUCED S1 + S3 regressions) == 4/5 Access-surface diff (same permissions, a broader embedding) == newly reachable: 17 docs; of which FORBIDDEN: 10 HEADLINE: the embedding change made 10 forbidden documents reachable with permissions unchanged scope: probe-scope — compared 26 probe queries, not a corpus-wide scan; a clean result covers these queries only == 5/5 Render the HTML evidence report == report: 260 evidence records -> out/report.html