docs(main): ATR-only baseline on Garak + PINT (per #66)

[eeee2345]

@hiskudin — as you asked in #66, here it is as a PR rather than a comment: the reproducible ATR-only baseline on Garak and PINT.

benchmarks/atr-baseline.md contains:

• Garak — in-scope recall 74.4% (1,321/1,775 across 21 families), with the two out-of-scope families (snowball, harmbench) called out and excluded from the headline number.
• PINT — precision 99.7% / recall 63.6% / F1 77.7% (TP=287, FP=1, TN=398, FN=164); PINT is the only corpus here with a real measured precision, since it carries benign samples.
• Reproduce — pinned to agent-threat-rules commit 1831d0d5, with the exact scripts to regenerate both reports.

The Defender-side column is intentionally left blank — this is the ATR half of a like-for-like table, for your team to fill in on the same corpora, per your offer to review on your side.

Happy to adjust the file's location or format, or to align on the tool-result-focused subset you raised in #66 as a starting point for a bundled pack.

[claude]

Claude Code Review

This pull request is from a fork — automated review is disabled. A repository maintainer can comment @claude review to run a one-time review.

[Copilot]

Pull request overview

Adds a reproducible benchmark note capturing an ATR-only baseline on two corpora (Garak and Lakera PINT) to support the ongoing discussion in issue #66 about potential ATR rule bundling and like-for-like comparison against Defender.

Changes:

• Documented pinned ATR baseline results for Garak (recall) and PINT (precision/recall/F1), including scope caveats.
• Added step-by-step reproduction commands pinned to a specific agent-threat-rules commit for repeatability.

---

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

[hiskudin]

Defender column (same corpora as this baseline)

Ran locally against the same Garak + PINT inputs pinned in this PR (agent-threat-rules@1831d0d5 corpora). Criterion: @stackone/defender defendToolResult with blockHighRisk: true → count as positive when allowed === false.

Engine	PINT P	PINT R	PINT F1	PINT confusion	Garak in-scope R (1,775)	Garak overall R (3,475)
ATR (this PR)	99.7%	63.6%	77.7%	TP=287 FP=1 TN=398 FN=164	74.4% (1,321/1,775)	38.2% (1,329/3,475)
Defender Tier 1	100%	12.9%	22.8%	TP=58 FP=0 TN=399 FN=393	33.5% (595/1,775)	17.1% (595/3,475)
Defender Tier 1+2	82.1%	87.6%	84.8%	TP=395 FP=86 TN=313 FN=56	76.8% (1,363/1,775)	41.7% (1,448/3,475)

Notes for reading this honestly:

• Reporting both T1 and T1+T2 — a single “Defender” cell would be misleading (T1 is a small curated regex pack; T2 carries most recall on these chat/jailbreak sets).
• ATR still leads on PINT precision; Defender T1+T2 leads on PINT recall/F1 and is roughly tied with ATR on Garak in-scope recall.
• These corpora are jailbreak/PINT-shaped, not tool-result traffic — still want an FP check on tool-like benign data before deciding on an opt-in ATR pack (ATR rule-pack as a Tier 1 supplement (462 patterns, MIT) + cross-benchmark offer #66).

Happy to fold this into benchmarks/atr-baseline.md once this PR lands (or as a follow-up docs commit).

[eeee2345]

@hiskudin — thank you for running it rather than taking the table at face value, and for reporting T1 and T1+T2 separately. A single "Defender" cell would have been the easier and worse choice.

Two things I owe you, both about the ATR column you compared against.

The PINT row was mislabelled, and I have corrected it in this PR. I described that corpus as "Lakera's public Prompt Injection Test set". It is not. It is a self-built 850-sample corpus in PINT's format, assembled from deepset/prompt-injections and Lakera/gandalf_ignore_instructions; Lakera's actual benchmark is private and roughly five times larger. The measurements are unchanged — the label was wrong. It also carries a scope caveat worth knowing before drawing a conclusion from the precision figure: only a minority of the ruleset fires on that corpus at all, and a single rule accounts for most of the detections. So "ATR leads on PINT precision" is true of this corpus and says less about the ruleset as a whole than the number suggests.

The ATR figures you compared against are pinned to an older commit, and the newer ones moved more than I expected. Everything in this document is 1831d0d5 (v3.5.7, 714 rules), which is the right way to do it — but the upstream table has since been re-measured on 3.5.11, and the all-families garak recall is materially higher than the 38.2% recorded here. That is the row your "roughly tied on Garak in-scope recall" reading rests on, so it may not survive a re-run. I have made the pinning explicit in the doc rather than quietly refreshing the numbers, because a comparison across two different ATR versions would be worse than a stale one.

If you want the comparison to stand as a decision input for #66, the clean version is both engines re-run on one newer commit. I am happy to produce the ATR column for whichever commit you pick, and to have the result be whatever it is — including it going against ATR.

On folding this into benchmarks/atr-baseline.md: yes, please, and your table belongs in it rather than beside it. Your point about these corpora being jailbreak-shaped rather than tool-result traffic is the one I would most want kept in the text — an FP check on tool-like benign data is a different measurement and this document does not stand in for it.

[hiskudin]

@eeee2345 — thanks for this, and sorry for the slow reply. Both corrections are exactly the kind that make the comparison worth trusting, so let's redo it cleanly rather than land stale numbers.

Plan:

• Relabel the 850-sample set as what it is — a PINT-format set from deepset/prompt-injections + Lakera/gandalf_ignore_instructions — and keep your caveat that only a minority of ATR fires on it (one rule doing most of the work), so that precision figure stays corpus-specific.
• Re-run both engines on ATR v4.0.0 (464548b4). To keep it genuinely apples-to-apples I'll run both through the same harness on the same rows rather than splicing two separate runs together — check out v4.0.0, run your Garak + PINT-format harness for the ATR column, then score Defender (Tier-1 and Tier-1+2 separately) on those exact sample sets, same positive criterion (defendToolResult with blockHighRisk: true → positive when allowed === false). I'll send you the ATR column to sanity-check before it goes in the doc — result either direction. (If the 3.x→4.0 spec churn makes a 3.5.x point the fairer comparison, say the word and I'll pin there instead.)

Your "these corpora are jailbreak-shaped, not tool-result traffic, and this doc doesn't stand in for a tool-result FP check" point stays prominent in the text — I'll keep that measurement where it belongs, as part of scoping the opt-in Tier-1 pack rather than here.

Once that's in, I'll fold it into benchmarks/atr-baseline.md (your table in the text, per your note), land it, and close #75 with credit — then we can pick up the opt-in Tier-1 pack itself (codegen'd from a pinned ATR release, scoped to the tool-result-relevant families), which is where the tool-result benign FP check comes in.

I'll get the Defender numbers on v4.0.0 turned around within a week.

[hiskudin]

@eeee2345 — re-ran it on ATR v4.0.0 (464548b4) and published @stackone/defender@0.8.2, both on the relabelled corpora. Numbers below — please sanity-check the ATR row against what you'd get, since I produced it from your own harness rather than taking a table at face value.

Engine	PINT-format P	R	F1	confusion	Garak in-scope R (1,775)	Garak overall R
ATR v4.0.0	100%	65.4%	79.1%	TP=295 FP=0 TN=399 FN=156	80.5% (1,429)	57.2% (1,987/3,475)
Defender T1	100%	12.9%	22.8%	TP=58 FP=0 TN=399 FN=393	33.3% (591)	17.0%
Defender T1+2	81.9%	86.0%	83.9%	TP=388 FP=86 TN=313 FN=63	76.5% (1,357)	41.4%

How I ran it:

• ATR: your Garak + PINT harnesses unmodified at 464548b4 (785 rules). Garak in-scope = 21 families, excluding snowball (1,500) + harmbench (200) per your convention; overall = all 23. PINT-format = the relabelled 850-sample deepset + gandalf set.
• Defender: defendToolResult on published 0.8.2, blockHighRisk: true, positive ⇔ allowed === false; onnxruntime-node pinned to 1.21.0; T1 and T1+2 separately.

Your v3.5.7→v4.0.0 warning was right — Garak in-scope rose 74.4%→80.5%, so ATR now leads Defender T1+2 there (80.5 vs 76.5); the old "roughly tied" reading is gone. On PINT-format, Defender T1+2 leads on F1 (83.9 vs 79.1) by buying +20 pts recall for 86 benign FPs, where ATR has 0.

Footnotes I'll carry in the doc so no cell gets over-read:

• Garak measures no precision (100% adversarial); the only FP evidence here is PINT-format's 399 benign.
• ATR's Garak cell is regex-only (evaluate()); its PINT cell adds the tier-2.5 embedding layer — different engine per cell.
• ATR's Garak harness scores each prompt on two channels (llm_input + a re-typed mcp_exchange) and counts a hit on either; from the code, the mcp_exchange re-typing looks to admit all rules regardless of source-type scoping. Defender is scored on its single tool-result surface — so the Garak column is each tool on its native harness, not identical-input head-to-head. Flag if I've misread that.
• These corpora are jailbreak-shaped, not tool-result traffic; the tool-result benign FP check is the separate measurement for the Tier-1-pack decision, kept out of this doc.

If the ATR row matches on your side (or you spot a config I got wrong), I'll fold it into benchmarks/atr-baseline.md with your table in the text, land it, and close #75 with credit — then we can scope the opt-in Tier-1 pack.