eval: prove complex VRS and provider receipt boundary

CATALOG.md

@@ -1,8 +1,8 @@
 <!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
 # Current eval catalog and evidence
 
-This is the sole current corpus surface: **47 included cells** (16 model-free,
-31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
+This is the sole current corpus surface: **42 included cells** (16 model-free,
+26 model-backed) and **12 retired exclusions**. Inventory fields are derived from the active
 KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
 history: a failed last run is visible without being advertised as accepted evidence, and cells with no
 structured run row say so explicitly.
@@ -54,14 +54,9 @@ harness-native loader plus canonical hook file.
 | `targeted-reconcile-isolation` | model-free | — | 0 | none | `60s` | 7 | — | **NO STRUCTURED RUN** |
 | `test-writing` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
 | `two-networks-coexist` | model-free | — | 0 | none | `120s` | 6 | — | **NO STRUCTURED RUN** |
-| `vrs-cross-file-absent` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 6 | — | **NO STRUCTURED RUN** |
-| `vrs-cross-file-present` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 6 | — | **NO STRUCTURED RUN** |
-| `vrs-definition-of-done-absent` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 7 | — | **NO STRUCTURED RUN** |
-| `vrs-definition-of-done-present` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 7 | — | **NO STRUCTURED RUN** |
-| `vrs-scope-drift-absent` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 7 | — | **NO STRUCTURED RUN** |
-| `vrs-scope-drift-present` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 7 | — | **NO STRUCTURED RUN** |
-| `vrs-scope-pressure-absent` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 6 | — | **NO STRUCTURED RUN** |
-| `vrs-scope-pressure-present` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 6 | — | **NO STRUCTURED RUN** |
+| `vrs-catalog-activation-absent` | Claude | `claude-sonnet-5` / medium | 1 | low | `1200s` | 8 | — | **NO STRUCTURED RUN** |
+| `vrs-catalog-activation-present` | Claude | `claude-sonnet-5` / medium | 1 | low | `1200s` | 8 | — | **NO STRUCTURED RUN** |
+| `vrs-command-policy-demo` | Claude | `claude-sonnet-5` / medium | 1 | low | `1200s` | 8 | — | **NO STRUCTURED RUN** |
 | `weird-git-setup` | Claude | `claude-sonnet-5` / medium | 1 | low | `600s` | 5 | **PASS** 2026-07-28T20:10:20Z, 6/6, source [`28d7fb6`](https://github.com/compoundingtech/evals/commit/28d7fb6e1235dc3680081bb3ee4e67109dc10c89), [receipt](evidence/stable-main-phase4-runs-20260728.json) | **PASS** 2026-07-28T20:10:20Z, 6/6, source [`28d7fb6`](https://github.com/compoundingtech/evals/commit/28d7fb6e1235dc3680081bb3ee4e67109dc10c89), [receipt](evidence/stable-main-phase4-runs-20260728.json) |
 
 ## Append-only run history
@@ -77,6 +72,14 @@ while the last-run column makes a recorded failure distinct from a cell with no
 - `compose-config-load` — retired composition/config-loader discriminator
 - `compose-global-skill` — retired composition/global-skill discriminator
 - `team-standup` — retired runtime seat-generation discriminator
+- `vrs-cross-file-absent` — superseded checklist-completion control; historical VRS evidence retained
+- `vrs-cross-file-present` — superseded checklist-completion treatment; historical VRS evidence retained
+- `vrs-definition-of-done-absent` — superseded packaging-compliance control; historical VRS evidence retained
+- `vrs-definition-of-done-present` — superseded packaging-compliance treatment; historical VRS evidence retained
+- `vrs-scope-drift-absent` — superseded leading scope-escalation control; historical VRS evidence retained
+- `vrs-scope-drift-present` — superseded leading scope-escalation treatment; historical VRS evidence retained
+- `vrs-scope-pressure-absent` — superseded leading refusal-escalation control; historical VRS evidence retained
+- `vrs-scope-pressure-present` — superseded leading refusal-escalation treatment; historical VRS evidence retained
 
 `hook-integrity` is not excluded: it is now a model-free, current-native
 `st2 up --materialize-only` hook installation and idempotency test.
@@ -124,21 +127,29 @@ The conservative future paid run requires an exact repeated `--cell` queue and s
 either a hard usage error or the informational Codex reset-available banner:
 
 ```sh
-bin/overnight.sh --run --cell ghost-bug --cell ghost-bug-codex --state-dir .eval-runs/overnight
+env -u ANTHROPIC_API_KEY -u CLAUDE_CODE_OAUTH_TOKEN \
+  bin/probe-claude-auth.sh --run \
+  --receipt .eval-runs/overnight/claude-real-auth.env
+bin/overnight.sh --run \
+  --claude-auth-receipt .eval-runs/overnight/claude-real-auth.env \
+  --cell ghost-bug --cell ghost-bug-codex --state-dir .eval-runs/overnight
 ```
 
 Because that common informational banner would prevent a full unattended sweep, the exact future full-run
 command makes both its complete-inventory selection and separately reviewed higher-spend choice visible:
 
 ```sh
-bin/overnight.sh --run --all --allow-informational-reset-banner --state-dir .eval-runs/overnight
+bin/overnight.sh --run --all \
+  --claude-auth-receipt .eval-runs/overnight/claude-real-auth.env \
+  --allow-informational-reset-banner --state-dir .eval-runs/overnight
 ```
 
 With no selector, dry-run remains inventory-only; paid `--run` rejects omitted, duplicate, unknown, and
 retired selections. Explicitly selected maintained model-free cells execute through the same st2 lifecycle
-and receipts without provider checks; Claude/Codex binary and authentication checks apply only when the
-selected subset needs that provider. Neither `--all` nor the informational-banner opt-in is approved merely
-because it is documented or shown by `--dry-run`.
+and receipts without provider checks. A Claude-selected run additionally requires the separately authorized
+bounded real-provider probe's fresh, sanitized receipt; auth metadata alone cannot authorize paid execution.
+Codex authentication checks apply only when the selected subset needs Codex. Neither the probe, `--all`, nor
+the informational-banner opt-in is approved merely because it is documented or shown by `--dry-run`.
 
 The runner executes one cell at a time, applies each cell's declared timeout plus a watchdog grace period,
 keeps durable logs and atomic PASS receipts, and skips matching completed receipts on resume. Hard quota/rate

README.md

@@ -48,26 +48,43 @@ bin/overnight.sh --dry-run --cell ghost-bug --cell ghost-bug-codex
 ```
 
 Paid execution requires `--run`, a clean `main` worktree, and either one or more explicit `--cell` selections
-or separately reviewed `--all`. Repeated selectors preserve CLI order. With no selector, the default dry run
-is inventory-only and can never imply a 28-cell paid launch. The conservative default stops before the next
-cell on either a hard usage error or an informational reset-available banner:
+or separately reviewed `--all`. Repeated selectors preserve CLI order. A Claude-selected run also requires a
+fresh, sanitized receipt from one exact real-provider turn: `claude auth status` metadata alone is not bearer
+proof. After an explicitly authorized human OAuth refresh, create the short-lived proof:
+
+```sh
+env -u ANTHROPIC_API_KEY -u CLAUDE_CODE_OAUTH_TOKEN \
+  bin/probe-claude-auth.sh --run \
+  --receipt .eval-runs/overnight/claude-real-auth.env
+```
+
+The probe is dry-run-only unless `--run` is explicit, pins Claude Sonnet 5 at medium effort, disables tools and
+session persistence, requires the exact response `AUTH_OK`, caps the turn at USD 0.05, and never records secret
+values. The resulting receipt is bound to the CLI version, source commit, non-secret state/config context, and
+a maximum age of ten minutes.
+
+With no selector, the default dry run is inventory-only and can never imply a full paid launch. The
+conservative default stops before the next cell on either a hard usage error or an informational
+reset-available banner:
 
 ```sh
 bin/overnight.sh --run \
+  --claude-auth-receipt .eval-runs/overnight/claude-real-auth.env \
   --cell ghost-bug \
   --cell ghost-bug-codex \
   --state-dir .eval-runs/overnight
 ```
 
 An explicit selection may include maintained model-free cells; they still execute through st2 and receive the
-same cleanup and durable receipts. Claude/Codex binary and authentication checks run only when the selected
-subset actually contains work for that provider.
+same cleanup and durable receipts. Claude/Codex checks run only when the selected subset actually contains
+work for that provider. Claude additionally fails closed without the fresh real-provider proof above.
 
 The exact future full-run command must be separately reviewed because it may spend more model quota after an
 informational Codex banner:
 
 ```sh
 bin/overnight.sh --run --all \
+  --claude-auth-receipt .eval-runs/overnight/claude-real-auth.env \
   --allow-informational-reset-banner \
   --state-dir .eval-runs/overnight
 ```
@@ -78,6 +95,17 @@ receipt. A resumed run skips only matching completed receipts. Hard quota/rate-l
 and record its result first, then stops by default; only the explicit informational-banner opt-in permits the
 next paid cell to start.
 
+Provider execution additionally fails closed unless every declared seat has an exact USD 0.05 hard ceiling and
+JSON-output contract. After every provider termination—including a failed or watchdog-timed-out cell—the runner
+parses and atomically persists one normalized usage/cost receipt before classifying the product verdict. A valid
+under-budget product failure or timeout leaves the paired control eligible; a missing, malformed, contradictory,
+or over-budget receipt writes `STOPPED` before another paid cell can start. Exercise that boundary without a
+provider call with:
+
+```sh
+bin/check-overnight-receipt-boundary.sh
+```
+
 ## Run one cell
 
 `st2 eval` creates a hermetic temporary catalog, copies the fixture, boots declared agents and model judges,

SKILL.md

@@ -35,20 +35,38 @@ st2 eval ./cells/<cell>/ --keep
 When an exact conservative paid queue is explicitly authorized, repeat `--cell` in the approved order:
 
 ```sh
-bin/overnight.sh --run --cell <cell> --cell <cell> --state-dir .eval-runs/overnight
+env -u ANTHROPIC_API_KEY -u CLAUDE_CODE_OAUTH_TOKEN \
+  bin/probe-claude-auth.sh --run \
+  --receipt .eval-runs/overnight/claude-real-auth.env
+bin/overnight.sh --run \
+  --claude-auth-receipt .eval-runs/overnight/claude-real-auth.env \
+  --cell <cell> --cell <cell> --state-dir .eval-runs/overnight
 ```
 
 The common Codex reset-available banner stops the default run before the next cell. A future full unattended
 run therefore requires separate human review of both `--all` and the higher-spend informational-banner opt-in:
 
 ```sh
-bin/overnight.sh --run --all --allow-informational-reset-banner --state-dir .eval-runs/overnight
+bin/overnight.sh --run --all \
+  --claude-auth-receipt .eval-runs/overnight/claude-real-auth.env \
+  --allow-informational-reset-banner --state-dir .eval-runs/overnight
 ```
 
 Paid `--run` rejects an omitted selector, duplicate, unknown, or retired cell. Explicitly selected maintained
-model-free cells execute through the same st2 lifecycle and receipts without provider checks; Claude/Codex
-binary and authentication checks apply only when the selected subset needs that provider. Hard quota/rate-limit
-errors stop unconditionally in both modes. Never infer approval for either paid command.
+model-free cells execute through the same st2 lifecycle and receipts without provider checks. A
+Claude-selected run requires a receipt from the separately authorized bounded real-provider probe; metadata
+from `claude auth status` is supplemental and cannot authorize execution. Codex checks apply only when the
+selected subset needs Codex. Hard quota/rate-limit errors stop unconditionally in both modes. Never infer
+approval for either paid command or for the auth probe.
+
+Before any provider run, verify that the selected KDL exposes the runner's exact USD 0.05 hard-budget and
+structured JSON contract. The runner persists one normalized usage receipt after every provider termination
+and before verdict classification. Missing, malformed, contradictory, or over-budget receipts stop before the
+next cell. The hermetic public integration proof is:
+
+```sh
+bin/check-overnight-receipt-boundary.sh
+```
 
 ## Cell rules
 

bin/check-claude-auth-proof.sh

@@ -0,0 +1,145 @@
+#!/usr/bin/env bash
+# Model-free mutation gate for real-provider freshness receipts. Never invokes a
+# provider: the probe is exercised only in its default dry-run path.
+set -euo pipefail
+
+repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+cd "$repo_root"
+
+mkdir -p .eval-runs
+scratch="$(mktemp -d .eval-runs/claude-auth-check.XXXXXX)"
+fake_bin="$(mktemp -d)"
+marker="$fake_bin/provider-reached"
+cleanup() {
+  rm -rf -- "$scratch"
+  rm -rf -- "$fake_bin"
+}
+trap cleanup EXIT
+
+printf '%s\n' \
+  '#!/usr/bin/env bash' \
+  ': > "${CLAUDE_PROBE_MARKER:?}"' \
+  'exit 99' >"$fake_bin/claude"
+chmod +x "$fake_bin/claude"
+
+PATH="$fake_bin:$PATH" CLAUDE_PROBE_MARKER="$marker" \
+  bin/probe-claude-auth.sh --dry-run --receipt .eval-runs/proof.env \
+  >"$scratch/dry-run.out"
+[ ! -e "$marker" ] || {
+  echo "FAIL: Claude auth probe dry-run reached the provider binary" >&2
+  exit 1
+}
+grep -Fq 'DRY RUN ONLY' "$scratch/dry-run.out"
+grep -Fq 'claude-sonnet-5, medium effort' "$scratch/dry-run.out"
+grep -Fq 'USD 0.05 maximum' "$scratch/dry-run.out"
+
+auth_context_sha256="$(
+  printf '%s\0' \
+    "${XDG_STATE_HOME-__UNSET__}" \
+    "${XDG_CONFIG_HOME-__UNSET__}" \
+    "${CLAUDE_CONFIG_DIR-__UNSET__}" |
+    sha256sum | awk '{ print $1 }'
+)"
+now="$(date -u +%s)"
+source_commit="$(git rev-parse HEAD)"
+cli_version="$(claude --version | awk 'NR == 1 { print $1 }')"
+
+write_receipt() {
+  local destination="$1" completed="$2" context="$3" source="$4" response="$5"
+  umask 077
+  printf '%s\n' \
+    "receipt_version=1" \
+    "result=PASS" \
+    "probe=real-provider-one-turn" \
+    "model=claude-sonnet-5" \
+    "effort=medium" \
+    "cli_version=$cli_version" \
+    "source_commit=$source" \
+    "completed_epoch=$completed" \
+    "auth_context_sha256=$context" \
+    "response=$response" \
+    "input_tokens=2" \
+    "output_tokens=3" \
+    "cache_creation_input_tokens=0" \
+    "cache_read_input_tokens=0" \
+    "total_cost_usd=0.001" \
+    >"$destination"
+}
+
+valid="$scratch/valid.env"
+write_receipt "$valid" "$now" "$auth_context_sha256" "$source_commit" AUTH_OK
+bin/validate-claude-auth-proof.sh "$valid" >/dev/null
+
+expect_rejected() {
+  local file="$1" expected="$2"
+  if bin/validate-claude-auth-proof.sh "$file" >"$scratch/rejected.out" 2>&1; then
+    echo "FAIL: invalid Claude auth proof was accepted: $file" >&2
+    exit 1
+  fi
+  grep -Fq "$expected" "$scratch/rejected.out" || {
+    sed -n '1,80p' "$scratch/rejected.out" >&2
+    echo "FAIL: invalid proof rejection did not explain: $expected" >&2
+    exit 1
+  }
+}
+
+metadata_only="$scratch/metadata-only.env"
+printf 'loggedIn=true\n' >"$metadata_only"
+chmod 600 "$metadata_only"
+expect_rejected "$metadata_only" 'missing, duplicate, or unknown fields'
+
+stale="$scratch/stale.env"
+write_receipt "$stale" "$((now - 601))" "$auth_context_sha256" "$source_commit" AUTH_OK
+expect_rejected "$stale" 'stale or future-dated'
+
+wrong_context="$scratch/wrong-context.env"
+write_receipt "$wrong_context" "$now" "$(printf wrong | sha256sum | awk '{ print $1 }')" "$source_commit" AUTH_OK
+expect_rejected "$wrong_context" 'state/config context changed'
+
+wrong_source="$scratch/wrong-source.env"
+write_receipt "$wrong_source" "$now" "$auth_context_sha256" "$(printf '0%.0s' {1..40})" AUTH_OK
+expect_rejected "$wrong_source" 'source commit changed'
+
+wrong_response="$scratch/wrong-response.env"
+write_receipt "$wrong_response" "$now" "$auth_context_sha256" "$source_commit" 'AUTH OK'
+expect_rejected "$wrong_response" 'exact AUTH_OK'
+
+duplicate="$scratch/duplicate.env"
+cp "$valid" "$duplicate"
+printf 'result=PASS\n' >>"$duplicate"
+expect_rejected "$duplicate" 'missing, duplicate, or unknown fields'
+
+public="$scratch/public.env"
+write_receipt "$public" "$now" "$auth_context_sha256" "$source_commit" AUTH_OK
+chmod 644 "$public"
+expect_rejected "$public" 'must not be group/world accessible'
+
+over_budget="$scratch/over-budget.env"
+write_receipt "$over_budget" "$now" "$auth_context_sha256" "$source_commit" AUTH_OK
+sed -i 's/^total_cost_usd=.*/total_cost_usd=0.051/' "$over_budget"
+expect_rejected "$over_budget" 'exceeded the USD 0.05 bound'
+
+if ANTHROPIC_API_KEY=synthetic-override \
+  bin/validate-claude-auth-proof.sh "$valid" >"$scratch/rejected.out" 2>&1; then
+  echo "FAIL: API-key override did not invalidate the OAuth proof" >&2
+  exit 1
+fi
+grep -Fq 'ANTHROPIC_API_KEY is present' "$scratch/rejected.out"
+if CLAUDE_CODE_OAUTH_TOKEN=synthetic-override \
+  bin/validate-claude-auth-proof.sh "$valid" >"$scratch/rejected.out" 2>&1; then
+  echo "FAIL: OAuth-token override did not invalidate the stored OAuth proof" >&2
+  exit 1
+fi
+grep -Fq 'CLAUDE_CODE_OAUTH_TOKEN is present' "$scratch/rejected.out"
+
+grep -Fq \
+  'claude --model claude-sonnet-5 --effort medium --permission-mode dontAsk --safe-mode --tools "" --no-session-persistence --max-budget-usd 0.05 --output-format json --print "Return exactly AUTH_OK and no other text."' \
+  bin/probe-claude-auth.sh || {
+    echo "FAIL: real Claude auth probe is not the reviewed exact bounded command" >&2
+    exit 1
+  }
+
+printf '%s\n' \
+  "PASS: Claude auth probe dry-run cannot reach the provider and exposes its exact bounded shape" \
+  "PASS: fresh exact one-turn receipt passes without reading secret values" \
+  "PASS: metadata-only, stale, context/source-drifted, override-tainted, public, non-exact, duplicate, and over-budget receipts fail closed"

bin/check-corpus.sh

@@ -33,6 +33,7 @@ bash -n "${shell_files[@]}"
 echo "PASS: ${#shell_files[@]} shell files parse"
 
 bin/check-preflight-safety.sh
+bin/check-claude-auth-proof.sh
 bin/check-model-policy.sh
 bin/check-model-policy-mutations.sh
 bin/check-run-history.sh
@@ -43,8 +44,7 @@ bin/check-kdl-parse.sh
 bin/check-st2-semantic.sh
 bin/check-fixture-reset-terminal.sh
 bin/check-harness-contract.sh
-bin/check-vrs-scope-drift.sh
-bin/check-vrs-variations.sh
+bin/check-vrs-complex.sh
 bin/check-weird-git-setup.sh
 bin/check-overnight-policy.sh
 bin/check-no-pii.sh