From d351bca4079f83828973766d8de87df030823fac Mon Sep 17 00:00:00 2001 From: eeee2345 Date: Tue, 11 Aug 2026 23:34:24 +0800 Subject: [PATCH] docs(benchmarks): ATR-only baseline on Garak and a PINT-format corpus MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit Rebuilt on current main. Two corrections to the framing rather than the measurements. The corpus described as 'Lakera's public Prompt Injection Test set' is not one. It is a self-built 850-sample corpus in PINT's format, assembled from deepset/prompt-injections and Lakera/gandalf_ignore_instructions; Lakera's actual benchmark is private and roughly five times larger. The numbers stand, the label did not, and the distinction matters to anyone comparing engines against it. Made the version pinning explicit. Every figure is measured on commit 1831d0d5 (v3.5.7, 714 rules), and the upstream table has moved since — the all-families garak figure in particular. Anyone adding a second engine column should pin to the same commit or re-run both on a newer one. --- benchmarks/atr-baseline.md | 88 ++++++++++++++++++++++++++++++++++++++ 1 file changed, 88 insertions(+) create mode 100644 benchmarks/atr-baseline.md diff --git a/benchmarks/atr-baseline.md b/benchmarks/atr-baseline.md new file mode 100644 index 0000000..4a8a6ae --- /dev/null +++ b/benchmarks/atr-baseline.md @@ -0,0 +1,88 @@ +# ATR-only baseline (Garak + PINT) + +A reproducible, ATR-only baseline on Garak (adversarial recall) and the Lakera +PINT set (precision / recall / F1), contributed per the discussion in +[issue #66](https://github.com/StackOneHQ/defender/issues/66) as a reference +point before deciding whether to bundle ATR rules into Defender. + +Defender-side numbers on the same corpora are intentionally left blank for the +Defender team to fill in — this file is the ATR half of a like-for-like table. + +## What was measured + +- **Engine**: [`agent-threat-rules`](https://github.com/Agent-Threat-Rule/agent-threat-rules) + v3.5.7, 714 rules, commit [`1831d0d5`](https://github.com/Agent-Threat-Rule/agent-threat-rules/commit/1831d0d5) on `main`. + + Every figure below is that commit, not the current release. The upstream + benchmark table moves — on 3.5.11 the all-families garak number is + materially higher than the 38.2% recorded here — so compare against this + pinned commit or re-run both engines on the same newer one, but do not mix + the two. + Numbers below are pinned to that commit; rerun the scripts against a newer + commit for a current figure — the ruleset grows over time. +- **Garak**: the public in-the-wild jailbreak corpus plus a 23-family sweep + (3,475 samples). Two families are excluded from the headline recall number: + `snowball` (1,500 reasoning-error probes, not agent-attack payloads) and + `harmbench` (200 raw harmful-intent completions, out of ATR's agent-attack + scope). In-scope = 1,775 samples across 21 families. +- **PINT-format**: a self-built 850-sample corpus (451 attack / 399 benign) in + the format Lakera's PINT benchmark uses, assembled from + `deepset/prompt-injections` and `Lakera/gandalf_ignore_instructions`. It is + **not** a run of Lakera's PINT benchmark, which is private and roughly five + times larger. Its value here is that it is the only corpus in this document + with a real, measured precision number rather than `precision = 1` by + construction on an all-adversarial set. Read it as a prompt-injection-family + score: only a minority of the ruleset fires on it at all, and one rule + accounts for most of the detections. + +## Reproduce + +```bash +git clone https://github.com/Agent-Threat-Rule/agent-threat-rules +cd agent-threat-rules && git checkout 1831d0d5 +npm ci && npm run build +npx tsx scripts/run-garak-full-benchmark.ts # -> data/garak-benchmark/garak-full-report.json +npx tsx src/eval/run-pint-benchmark.ts # -> data/pint-benchmark/pint-eval-report.json +``` + +The eval harness is not shipped in the published npm package (`dist` / `spec` / +`rules` only), so reproducing requires the repo checkout above rather than +`npm install agent-threat-rules`. + +## Results (ATR-only) + +### Garak — recall + +| Scope | Recall | +| :-- | :-- | +| In-scope (21 families, 1,775 samples) | **74.4%** (1,321 / 1,775) | +| Overall (all 23 families, 3,475 samples) | 38.2% (1,329 / 3,475) — dragged down by the two out-of-scope families above | + +Strongest families: `autodan` 100%, `sysprompt_extraction` 100%, `dan` 87.7%, +`inthewild` 87.5%, `agent_breaker` 85.7%. Weakest in-scope: `latentinjection` +34.4%; `badchars` / `malwaregen` / `sata` / `smuggling` ~12.5% each (small +families, 16 samples each). + +### PINT — precision / recall / F1 + +| Metric | Value | +| :-- | :-- | +| Precision | 99.7% | +| Recall | 63.6% | +| F1 | 77.7% | +| Confusion | TP=287, FP=1, TN=398, FN=164 | + +The single false positive is a documented, known case (`pint-0121`) tracked +against the rule that fires on it. + +## Reading the numbers honestly + +- Garak measures **recall** on adversarial-only corpora, so it says nothing + about false positives; PINT is the only line here with a real precision + number because it carries benign samples. +- The overall-Garak 38.2% is not a like-for-like figure — it includes two + families outside ATR's scope by design. The in-scope 74.4% is the number to + compare against a detector run on the same 21 families. +- If a tool-result-focused subset is the right starting point for a bundled + pack (per the Tier-1 surface-fit point in #66), the per-family breakdown + above shows which ATR families are already strong candidates.