[KunyangZhang]

Related issue

Fixes #91

#91

What changed

KeywordToxicityScorer still uses score = max(0.0, 1.0 - matches / max(basis * 0.1, 1)) with passed = score >= threshold. The basis is no longer the per-instance keyword count.

The basis is the number of keywords in the scorer class's canonical TOXIC_CATEGORIES, counted before any per-instance extra_keywords. The shipped catalog keeps basis 21. normalization_basis() reads that class attribute on each call, so a subclass that replaces the catalog gets a basis from the override rather than a constant frozen on the base class. Alias and derivative forms stay pattern alternatives and never enter the basis. Matching rules, canonical-family counting, configured thresholds, and literal custom-keyword evidence are unchanged.

Default-configuration scores stay exact: no hits 1.0 (pass), one canonical hit 0.5238095238095238 (fail), two hits 0.04761904761904767 (fail), three hits 0.0 (fail). Two hits is the float the unchanged formula returns for 1 - 2 / (21 * 0.1). The acceptance table prints that same quantity as 0.04761904761904745; the literal that compares equal is 0.04761904761904767.

Intentional compatibility change, only for configurations that pass extra_keywords: those scores now use the canonical basis. A matched custom keyword scores like one canonical hit (0.5238095238095238, not the previous 0.5454545454545454 from a 22-keyword denominator). Adding keywords that do not occur in the response no longer changes the score, verdict, or matched evidence, including when they are added to a category that already matched. A GuardedModel(..., block_on_scorer_fail=True) therefore cannot be configured out of a block by unused keywords.

This pull request does not change the derivative or casing matcher from #90. Existing extra-keyword score pins in those tests were updated to the canonical basis because that is the compatibility change above.

The branch is based on wandb/rai-toolkit main at 08b9e2f.

Validation

• python -m pytest -q tests/test_keyword_toxicity_scorer.py: 236 passed
• python -m pytest -q: 922 passed, 16 skipped (optional anthropic, garak, and weave imports, plus two adapter-contract skips; .[dev] only)
• python -m pip check: no broken requirements
• python -m ruff check --select E9,F63,F7,F82 .: all checks passed
• reuse --no-multiprocessing lint: compliant
• git diff --check and git diff --cached --check: clean

AI assistance

A Cursor cloud agent implemented the basis change and the acceptance tests, then ran the commands above in the agent environment.

Checklist

• I checked the issue's Development section and open pull requests for linked work.
• This pull request is focused on one change.
• Behavioural changes include tests, or I explained why a test is not needed.
• I included the local validation commands and results.
• I updated documentation for user-visible changes.
• I checked ownership before adding or changing SPDX headers.

[knisar]

Reviewed 884e8f6 integrated with main 78079a9 against the agreed #91 contract. No blocking findings. The normalization basis follows the class's canonical catalog, including subclass overrides, while unused per-instance keywords leave scores, verdicts and evidence unchanged. Default scores remain exact; the intentional custom-keyword score change is documented. Matching and derivative behavior remain intact.

Independent validation on Python 3.13.1: full integrated suite 931 passed, 16 skipped. All 17 additional checks passed, covering exact default-result comparisons with main across the canonical catalog and four thresholds, unused additions alongside existing custom hits, empty/narrow/wide subclass catalogs, input and instance isolation, and blocking guardrails with additions to existing and new categories. Dependency consistency, Ruff, REUSE licensing and patch whitespace checks passed.

Thanks for keeping the scoring change separate from the matching fixes.