ATR rule-pack as a Tier 1 supplement (462 patterns, MIT) + cross-benchmark offer

[eeee2345]

Defender's two-tier shape is clean and the in-package ONNX bundle is a sharp call (no model-download surprise on first call). I maintain Agent Threat Rules (ATR), an MIT-licensed YAML rule library with 462 rules across 10 attack categories. Microsoft Agent Governance Toolkit (PR #1277, 287 rules, weekly auto-sync), Cisco AI Defense skill-scanner (PR #99, 314 rules), MISP via CIRCL Luxembourg (PR #1207, 336 rules), and Microsoft PyRIT (PR #1715 merged 2026-05-27 by Roman Lutz) all run ATR in production.

Two questions:

1. Tier 1 supply. Your src/classifiers/patterns.ts and src/sanitizers/ look like the natural place ATR's pattern-method rules would live. Of ATR's 462 rules, ~200 are pure regex (method: pattern in our SPEC v1.1) with explicit true_positive / true_negative fixtures and a 0% FP gate on a 432 benign-skill corpus. Would you take a src/classifiers/atr-patterns.ts that imports the JSON-bundled rule subset? Stays compatible with your existing pattern format.

2. Cross-benchmark. On the public corpora we've used (NVIDIA Garak, plus a self-built 850-sample PINT-format set from deepset + Lakera Gandalf), we'd be happy to run ATR-only and post comparable F1 numbers next to Defender's, with code in benchmarks/. Useful for both sides — your ML obviously catches what regex can't, and vice versa.

If this isn't the right shape, no worries.

ATR repo: https://github.com/Agent-Threat-Rule/agent-threat-rules

[hiskudin]

Thanks for the detailed proposal! I had a chance to review the ATR repository and we believe it will be a useful addition to Defender.

Re: cross-benchmark3: Yes, please go ahead and open a PR with reproducible scripts and F1 tables. Comparing ATR-only against Defender on Garak and your PINT set would be very useful as a baseline before we commit to bundling rules. We'll review on our side once it's up.

Re: Tier 1 supply (point 1): The PatternDefinition shape and patterns.ts location are the right integration point. We'd prefer an opt-in ATR supplement (e.g. enableAtrPatterns or a companion package) rather than merging ~200 rules into the default pattern set out of the box. We're however open to promoting a high-confidence subset into defaults eventually if benchmark results are strong.

A few things we'd want to align on for the subset:

Surface fit: Defender Tier 1 scans tool-result payloads (emails, API JSON, documents). ATR rules with scan_target: skill or mcp may not map cleanly; a tool-result-focused pack would be the right starting point.
Codegen over hand-port: JSON-bundled ATR rules → generated PatternDefinition[], version-pinned to an ATR release, rather than a manually maintained atr-patterns.ts.
Happy to collaborate on category mapping and which rules belong in a tool-result pack. Looking forward to the benchmark PR.

[eeee2345]

ATR baseline: reproducible benchmark on Garak + PINT

Follow-up to #66. As discussed, this is the ATR-only baseline: reproducible
scripts + F1 tables on Garak (adversarial recall) and the Lakera PINT
benchmark (precision/recall/F1). Defender-side numbers left for your team to
fill in on the same corpora, per your offer to review on your side.

What was measured

• Engine: agent-threat-rules (npm) — v3.5.7, 714 rules, commit
  1831d0d5
  on main.
• Garak: the public in-the-wild jailbreak corpus + a 23-family sweep
  (3,475 samples total). Two families are intentionally excluded from the
  headline recall number: snowball (1,500 reasoning-error probes, not
  agent-attack payloads) and harmbench (200 raw harmful-intent completions,
  out of ATR's agent-attack scope). In-scope = 1,775 samples across 21
  families.
• PINT: Lakera's public Prompt Injection Test set (850 samples: 451
  attack / 399 benign) — the only corpus here with a real, measured
  precision number (not "precision=1 by construction" like an all-adversarial
  set).

Reproduce

git clone https://github.com/Agent-Threat-Rule/agent-threat-rules
cd agent-threat-rules && git checkout 1831d0d5
npm ci && npm run build
npx tsx scripts/run-garak-full-benchmark.ts   # -> data/garak-benchmark/garak-full-report.json
npx tsx src/eval/run-pint-benchmark.ts        # -> data/pint-benchmark/pint-eval-report.json

(The eval harness isn't shipped in the published npm package — files:
dist/spec/rules only — so reproducing requires the repo checkout above,
not just npm install agent-threat-rules.)

ATR-only results

Garak (recall; PINT below has the only real precision number)

Scope	Recall
In-scope (21 families, 1,775 samples)	74.4% (1,321/1,775)
Overall (all 23 families, 3,475 samples)	38.2% (1,329/3,475) — dragged down by the two out-of-scope families above

Strongest families: autodan 100%, sysprompt_extraction 100%, dan 87.7%,
inthewild 87.5%, agent_breaker 85.7%. Weakest in-scope: latentinjection
34.4%, badchars/malwaregen/sata/smuggling ~12.5% each (small families,
16 samples each).

PINT

Metric	Value
Precision	99.7%
Recall	63.6%
F1	77.7%
Confusion	TP=287 FP=1 TN=398 FN=164

The single FP is a documented, known case (pint-0121) already tracked
against the rule that fires on it.

Over to you

Per your note in #66 — happy to align on category mapping once you've run
Defender against the same corpora. If a tool-result-focused subset makes
sense as the starting point (per your Tier 1 surface-fit point), the
category breakdown above should make it easy to see which ATR families are
already strong candidates for that pack.