feat(ENG-1084): detect-and-gate + sentence-level cleaning (v0.8.2)

[hiskudin]

Summary

Ports the TS 0.8.2 detect-and-gate + sentence-level cleaning architecture (TS #85) to Python. Defender no longer mutates content in place; it returns sanitized — a sentence-cleaned copy of the flagged fields (restores the 0.7.4 cleaning) — plus the allowed verdict. sanitize_content=False → sanitized is the input verbatim.

Blocking is allowed. The composite Sanitizer and pattern_remover are deleted; role_stripper is kept for the marker strip.

Default-behavior note: sanitize_content defaults to True, so sanitized is a sentence-cleaned copy by default — a Tier-2 false positive can drop legit sentences from field content. Callers that must not mutate payloads should pass sanitize_content=False (detect-and-gate — content verbatim) and gate on allowed.

Behavior changes (not breaking) / Migration

Feature + bug-fix release — no released (0.7.4) consumer changes code:

• sanitized is a sentence-cleaned copy (its 0.7.4 meaning), now at sentence granularity; fires only at aggregate high/critical. sanitize_content=False → input verbatim.
• default_risk_level now low (was medium) — reported risk tracks the model; blocking still keys on allowed.
• Removed: block_high_risk option; pattern_remover / sanitizer modules.

Changes

• Detect-and-gate core (tool_result_sanitizer.py) — detect-only; evidence-based fields_sanitized/methods_by_field/patterns_by_field; top-level string + non-plain-object handling.
• Sentence-level cleaning (sentence_cleaner.py) — gated high/critical; drops offending sentences and leaves a [CONTENT SANITISED] marker where each run was cut (mid-content drops stay visible), keeps the rest; single-sentence → unchanged, all-high → just the marker; verbatim when nothing dropped.
• fields_sanitized = fields the cleaner changed (was Tier-1 detect-only). API-visible semantic change.
• Object-KEY scanning + non-destructive wide-container cap (coverage_degraded) + per-field cap.
• Tier 2 availability — require_tier2, tier2_available, warn-once.
• Evidence-driven encoding — decode, then run the real detector; escalate only on a genuine pattern.
• Hardening — ReDoS bounds; non-finite model output → skip.

Post-review fixes (StuBehan, ported from #85)

• Strings inside a risky field's array now Tier-1 scanned ({"name": [INJ]}) — previously fell through _sanitize_value to the pass-through. Risky-field allowlist preserved.
• New detected_field_count (keys of patterns_by_field) — first-class threat-count signal, since fields_sanitized now means cleaner-changed fields.
• Size-metrics double-count fixed + iterate obj.items() directly instead of list(obj.items()) on wide payloads.
• Tier 1 detection bounded by the call-scoped byte budget (max_size), replacing the per-container 100-item cap (per-container, bypassable, blinded normal >1000-item payloads). skip_large_arrays/large_array_threshold kept as deprecated, off-by-default opt-ins (non-breaking).

Validation

• 292 tests pass, ruff clean.
• Connector FPR (940 payloads, TS parity): content changed 3/940 (0.32%) SFE off / 2/940 (0.21%) SFE on (those few now carry a [CONTENT SANITISED] marker at the cut); verdict FPR byte-identical to detect-only.
• SFE FPR (800 payloads): 79.9% low / 15.9% med / 4.25% high — matches TS ~81/14/5 (was 0% low / 95% med under the old floor).

AgentShield 537 (vs published v0.7.4, both onnxruntime@1.21.0 — by parity with TS #85)

Metric	v0.7.4	This PR	Δ
Final	87.5	87.8	+0.3
Composite	89.0	89.2	+0.2
Penalty	1.43	1.43	—
Prompt Injection	188/205	190/205	+2

All other categories identical — verdict-neutral. Real gates: Claude Code plugin repro 0.951 → 0.199 + connector FPR above.

Version

→ 0.8.2, pinned via Release-As (below) to match the TS package (which landed on 0.8.2 after two publish-retry patch bumps). The org release-please config bumps feat to a patch pre-1.0, so the version is pinned explicitly. Squash-merge so the clean feat(ENG-1084): title is the release commit and the old ! footer is dropped.

Release-As: 0.8.2

🤖 Generated with Claude Code

[Copilot]

Pull request overview

Ports the “detect-and-gate” architecture to the Python Defender SDK so tool-result processing becomes detection-only: the returned sanitized payload preserves the original content, while threat signals are recorded as evidence and gating is expressed via allowed.

Changes:

• Reworks Tier 1 tool-result traversal to detect threats without mutating content; adds key scanning and coverage/analysis caps.
• Updates the risk model (default risk floor to low) and surfaces Tier 2 availability + coverage degradation in DefenseResult.
• Removes mutation-based sanitizer modules and updates tests/docs; includes ReDoS hardening and Tier 2 robustness improvements.

Reviewed changes

Copilot reviewed 15 out of 15 changed files in this pull request and generated 2 comments.

Show a summary per file

File	Description
tests/test_sanitizers.py	Removes tests tied to deleted mutation sanitizers; retains unit coverage for remaining detection helpers.
tests/test_integration.py	Updates integration assertions to “detect-and-gate” semantics (content preserved; evidence + allowed gate).
src/stackone_defender/types.py	Adjusts defaults/metadata to reflect detection evidence; adds tier2_available and coverage_degraded.
src/stackone_defender/sanitizers/sanitizer.py	Deleted: composite mutation sanitizer implementation removed.
src/stackone_defender/sanitizers/role_stripper.py	Deleted: role-marker stripping module removed.
src/stackone_defender/sanitizers/pattern_remover.py	Deleted: pattern redaction/removal module removed.
src/stackone_defender/sanitizers/encoding_detector.py	ReDoS hardening for Morse detection gate.
src/stackone_defender/sanitizers/init.py	Re-exports shift toward detection-only helpers (no mutation/redaction exports).
src/stackone_defender/core/tool_result_sanitizer.py	Major rewrite to detect-only traversal, evidence recording, key scanning, and capped analysis without dropping data.
src/stackone_defender/core/prompt_defense.py	Adds Tier 2 “require/fail-closed”, warn-once degraded behavior, and coverage_degraded plumbing.
src/stackone_defender/config.py	Introduces default per-field analysis length cap for Tier 1 (ReDoS/cost guard).
src/stackone_defender/classifiers/tier2_classifier.py	Treats non-finite model outputs as a classifier skip (not a benign score).
src/stackone_defender/classifiers/patterns.py	ReDoS-hardens Morse + markdown hidden-instruction regexes.
src/stackone_defender/classifiers/onnx_classifier.py	Moves repeated ONNX load-failure warning responsibility to the caller.
README.md	Documents breaking “detect-and-gate” semantics and updated defaults/behavior.

Suppressed comments (3)

src/stackone_defender/core/tool_result_sanitizer.py:212

• SizeMetrics.object_count is incremented twice for dicts: update_size_metrics() already increments object_count in _sanitize_value(), and _sanitize_object() increments it again. This doubles object_count in metadata.size_metrics.

        metadata.size_metrics.object_count += 1

src/stackone_defender/core/tool_result_sanitizer.py:252

• entries = list(obj.items()) allocates an extra list of all key/value pairs; for large paginated responses this can be a significant and unnecessary memory hit. Use len(obj) for the scan limit and iterate obj.items() directly.

        entries = list(obj.items())
        scan_limit = self._detection_scan_limit(len(entries), metadata)
        for i, (key, val) in enumerate(entries):
            entry_detect = detect and i < scan_limit

src/stackone_defender/core/tool_result_sanitizer.py:278

• entries = list(obj.items()) duplicates all key/value pairs in memory before iterating. For wide wrapped payloads this adds avoidable overhead; you can derive the scan limit from len(obj) and iterate obj.items() directly.

        entries = list(obj.items())
        scan_limit = self._detection_scan_limit(len(entries), metadata)
        for i, (key, val) in enumerate(entries):
            entry_detect = detect and i < scan_limit

---

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.