[Jah-yee]

Related issue

Closes #108

This is a replacement for the closed #110, rebuilt from current wandb/rai-toolkit main (ea99e45) per the recovery instructions knisar left on #110. The previous branch had no shared history with upstream main, so GitHub showed the whole repository as newly added; this branch is a normal two-file diff on top of current main.

What changed

Only two files are touched, exactly as requested on #110:

• rai_toolkit/evaluation/report.py
• tests/test_compliance_report_unassessed.py

rai_toolkit/evaluation/report.py

ScorerResult.assessed == False marks a row the scorer could not evaluate (missing context, parser failure, prediction error). Those rows are coverage gaps, not measured failures, but ComplianceReport was still counting them as policy violations:

• to_dict() filtered failed_scores on not result.passed and included a row in failed_items if any scorer had passed=False. Both now require result.assessed and not result.passed, so an unassessed result never appears in failed_scores and never puts its row on the failure list.
• to_summary() counted failures with the same not result.passed check. It now uses the same assessed guard, so the Items with failures: N/M line is not inflated by unassessed rows.
• The category line now surfaces the coverage gap instead of hiding it: when a category has unassessed results the summary reads MIT-1.1: 0.0% (0/1 passed, 1 un-assessed) [FAIL]; categories with no unassessed results keep the existing MIT-1.1: 100.0% (1/1 passed) [PASS] format.

No production behaviour changes beyond the unassessed handling were made.

tests/test_compliance_report_unassessed.py

18 focused tests covering the all-unassessed case, mixed pass/unassessed, assessed failure plus unassessed on the same row, fully assessed rows (unchanged output), two assessed failures on one row, JSON serialization, input preservation, and failure-count text.

The three corrections knisar listed on #110 are included:

1. Two assessed failures on one row — the fixture now uses the aggregator's category totals total_items=2 / failed_items=2, while the report still counts one failed row, and the test asserts both the 0/2 passed category line and Items with failures: 1/1.
2. Input preservation — the report methods are now compared against a copy.deepcopy snapshot of the entire EvaluationResults object after to_summary(), to_dict() and to_json() run, instead of only checking the emitted input string.
3. Assessed failure plus unassessed on one row — the test now asserts that the dictionary output and the serialized JSON output retain only the assessed failure (failed_scores keys are exactly {"genuine"}).

Validation

• python -m pytest tests/test_compliance_report_unassessed.py -q: 18 passed
• python -m pytest tests/test_pipeline_prediction_errors.py tests/test_redteam_report_surfaces.py -q: 18 passed, 8 skipped
• python -m pytest -q (full suite, this branch): 738 passed, 15 skipped, 211 failed, 64 errors
• python -m pytest -q (full suite, unmodified upstream/main at ea99e45, same environment): 720 passed, 15 skipped, 211 failed, 64 errors
• Diffing the sorted FAILED/ERROR lists from the two runs: identical — the 211 failures and 64 errors are pre-existing in this checkout (missing optional SDKs such as anthropic, weave, and other optional extras) and are not caused by this change. The branch adds 18 passing tests and no new failures.

The GitHub Actions run for this PR is currently at action_required — it is waiting on the upstream maintainer to approve the workflow run for this fork branch, as expected for a first-time external contribution. No test job has run yet; the numbers above come from the local runs.

AI assistance

AI assistance was used to carry the two-file change over from #110, apply the three test corrections listed in knisar's #110 review, and run the test suites above. I verified the diff scope by hand (only the two requested files), confirmed the failure sets are identical to unmodified main, and personally reviewed every changed line.

Checklist

• PR targets main
• Single commit, GPG-signed with a verified key
• Only the two requested files are modified
• Focused tests and full suite run, results listed above
• Closes #108 linked
• SPDX headers present on the new test file

[knisar]

The rebuilt branch has the correct history and a focused two-file diff. All three corrections requested on #110 are present. Missing measurements stay visible in category lines, and only assessed failures enter the row count and nested failure payload.

Independent validation on Python 3.13.1: python -m pytest -q -p no:cacheprovider passed 949 tests with 16 skipped. A further 24 checks using the real evaluation pipeline passed, covering shared and separate categories, prediction errors, multiple scorer failures on one row, JSON file output, complete input preservation, and exact compatibility with main for fully assessed output. The full-suite failures reported from your environment do not reproduce here.

Dependency consistency, Ruff (--select E9,F63,F7,F82), REUSE lint, and the committed patch whitespace check all passed. This resolves #108.