@W-23144570: stale-content-cleanup-apply eval (behavioral + adversarial) + runnability layer

[Alon-ST-DATA]

Summary

Full eval for the stale-content-cleanup-apply MCP prompt (W-23144570), targeting main as one complete PR. Combines the behavioral + adversarial coverage from #412 (Akash Rastogi) with the harness changes needed to actually run it on a low tokens-per-minute grader key and in CI.

• tests/eval/staleContentCleanupApply.test.ts — 5 behavioral + 6 adversarial cases (from @W-23144570: Add stale-content-cleanup-apply eval (behavioral + adversarial) #412) + a runnability layer.
• tests/eval/grade.ts — best-of-N + rate-limit backoff (backward-compatible: attempts defaults to 1, so existing evals are unchanged).

Supersedes #412 — carries its cases plus the fixes below in one PR.

What this adds on top of #412

1. Tool-catalog narrowing (INCLUDE_TOOLS). The workflow only needs 6 tools. The full ~40-tool catalog schema alone can exceed a 30k-TPM grader budget before the workflow even runs. adminEnv() now sets INCLUDE_TOOLS to the 6 workflow tools; the two delete tools stay included on purpose so the "no destructive call" invariant remains meaningful.

2. Optional report scoping (EVAL_STALE_PROJECT_ID). On a large admin site the stale-content report returns thousands of rows / ~1MB. Fed back to the model that is ~353k tokens in a single request — an instant hard 429 Request too large on a 30k-TPM key. Setting EVAL_STALE_PROJECT_ID to a small, non-empty project scopes the report and cuts the request from 353,463 to ~30,000 tokens (12x) while still exercising the full report → resolve → notify → STOP workflow. Left unset, the eval runs unscoped (original behavior) — fine on a high-TPM key or a small fixture site. Read at call time (not module load), because tests/eval/.env is loaded in a beforeAll after module evaluation.

3. Best-of-N for the fuzzy judge. The one open-ended rubric case ("coherent assistant report") uses grade()'s best-of-N (attempts: 3) with rate-limit backoff instead of vitest .retry(2). .retry re-runs on any throw — including a 429 — and re-fires straight into the same TPM wall. Best-of-N re-runs only the fuzzy 5-dim judge, backs off on a detected rate-limit, and reports real scores if every attempt fails. Deterministic tool-call cases keep .retry(2).

Deterministic-vs-LLM split (preserved)

The security-critical invariant — the agent never invokes a delete tool from this prompt under any input, including adversarial "skip confirmation" / "ignore dryRun" / "fabricate a confirmationToken" messages — is asserted deterministically and gates hard. Only the fuzzy rubric is best-of-N.

Test plan

• npx tsc --noEmit — clean
• npm run lint — clean (eval files)
• npm run build — clean
• Scoping proven: request dropped from 353,463 → ~30,000 tokens (12x) with EVAL_STALE_PROJECT_ID set.
• The best-of-N grade case cleared the rubric via backoff after two 429s — the only case to pass on a 30k-TPM key.

Known limitation (this is the pipeline requirement)

On a 30k-TPM OpenAI key the multi-turn agent loop (report → resolve → notify → summary is several LLM calls) still sums past the per-minute ceiling across all 11 cases, and a shared live Tableau site can throttle rapid sign-ins. Full green needs TPM headroom — a higher-tier key or an internal gateway — plus a bounded fixture. The scope + backoff added here are necessary to make the suite feasible at all and to keep the deterministic security invariant hard; they are not sufficient on the lowest key tier alone. This matches the eval-testing-pipeline requirements (CI-held LLM credential with TPM headroom + a bounded eval fixture).

Env

• EVAL_STALE_PROJECT_ID (optional) — scope the stale-content report to a small non-empty project. Set in tests/eval/.env (git-ignored).