Skip to content

feat(hal): HAL_SCORE_GATE_MODE shadow quorum verdict per score event + per-verdict receipt id (default off) - #1570

Draft
DealAppSeo wants to merge 2 commits into
mainfrom
claude/hal-score-gate-shadow-2fa469
Draft

DealAppSeo wants to merge 2 commits into
mainfrom
claude/hal-score-gate-shadow-2fa469

Conversation

@DealAppSeo

Copy link
Copy Markdown
Owner

Owner action

You decide (MAJOR: governs how RepID could move). Shipped shadow-only. Draft. Do not auto-merge.

With both new flags unset, nothing changes: no provider call, no new row, no new response field. tests/hal-score-gate-shadow.test.ts pins that (fetch is never called with the mode off).

To use it after merge, two steps, each optional and reversible:

  1. Create the shadow table (prod DDL is yours). Paste into the SQL editor of the Trinity production Supabase project (New query) the contents of supabase/migrations/20261116180000_hal_score_gate_shadow.sql. It creates one empty table, takes no lock on any existing table (no foreign key, on purpose), and turns RLS on with no grants. Expected result: Success. No rows returned. Rollback: drop table if exists public.hal_score_gate_shadow;
  2. Turn on shadow: Railway → project repid-engine → service repid-engine → Variables → HAL_SCORE_GATE_MODE = shadow. Boot log line: [hal-score-gate] HAL_SCORE_GATE_MODE resolved to 'shadow'. Off again: delete the variable.

Optional third: HAL_VERDICT_RECEIPT_ID_ENABLED = true on the same service adds receipt_id to POST /api/v1/hal/evaluate responses.

Re-verification overturned half of the defect [V, read-only Supabase, 2026-11-16]

The eval (E2E_EVAL_REPORT.md §7 gap 2) says the score pipeline defaults to the extractor-only HAL path. That is true of the code default and false of production.

  • repid_config.HAL_STRICTNESS = 2, and getHalConfig() reads repid_config first.
  • Every HAL_SCORE_EVENT since 2026-06-04 records hal_mode = 'fact-check', or 'extractor-fallback' when no provider answered (those are all neutralized to flagged, delta 0).
  • In the last 30 days, quorum vetoes and quorum cleans both moved RepID. Counts are in the private report, not here.

The extractor default applies to a fresh, local or self-hosted deploy, or to HAL_CONFIG_FROM_DB=false with no env. That is what the eval ran (Eiffel-in-Berlin: flagged, delta 0, the same as a true claim). The pipeline comment that implied production used the extractor is corrected in this PR.

So I built the switch so that it costs nothing in production today and still answers the eval's question wherever the extractor does score.

What it does

HAL_SCORE_GATE_MODE = off (default) | shadow | enforce.

  • shadow: after each score event is written, a detached job records in hal_score_gate_shadow, beside the live verdict:

    • the quorum's verdict on the same answer,
    • the HAL delta each verdict would apply (before decay and the clamp),
    • a receipt id for each.

    The live row, the delta and repid_agents never change. The job runs after the write, so it adds no latency to it.

    • Live path is the extractor (strictness 1): the quorum judges the answer with strictness 2, the same providers the live quorum would use (providersFn hoisted, no behaviour change), the two-family early stop, and the HAL cache under its own key namespace.
    • Live path is already the quorum (strictness 2, production today): no second call. The live verdict is recorded as both. The quorum's delta is recomputed independently by quorumWouldBeDelta, which copies the live gate chain. Any row where quorum_source = 'live' and the two deltas differ means that copy has drifted.
  • enforce: refused (r23: original + live = shadow only until your GO). It runs as shadow and logs at boot: HAL_SCORE_GATE_MODE=enforce REFUSED ... Running as 'shadow'. There is no override variable on purpose: lifting the refusal is a code change and a review, not an env edit.

  • An unknown value resolves to off, loudly.

Cost bound (quorum calls per score event, in shadow)

case provider calls
live path is the quorum (production today) 0
score-gate cache hit (same answer + context, 1 h TTL) 0
cache miss 1 quorum evaluation: cost-ordered waves (free first), stopping when two families on two hosts agree, with in-flight siblings aborted. At most one request per enabled provider; usually only the free wave.
flag off 0, and none of this code runs

A degraded answer (no quorum) is not cached, so an outage cannot pin "no quorum" on a text for the TTL. Measured in the test: the first Eiffel event made ≤ 2 provider requests; a second event on the same answer was a cache hit with 0.

Per-verdict receipt id

halv1_ + sha256 over the verdict record:

  • the sha256 of the judged text (never the text),
  • the decision, hal_score to 4 places, mode, strictness and quorum_met,
  • the sorted families, and each provider's (provider, model, verdict).

It needs no table, and anyone holding the record recomputes it. Latency, confidence and cache markers are excluded, so a cached replay keeps the id. Two identical judgments of the same text share an id by design; link a verdict to an event beside the id, never inside it.

  • In shadow: live_receipt_id and quorum_receipt_id are on every row.
  • For TrustShell to cite: POST /api/v1/hal/evaluate adds receipt_id when HAL_VERDICT_RECEIPT_ID_ENABLED=true (exact string; anything else returns the same body object unchanged). trustshell verify currently reads its receipt id from POST /hal/receipt (the vote-row path, which says columns-missing). Printing this id instead is a TrustShell change, left for the agent working there (hand-over note written).

Tests

tests/hal-score-gate-shadow.test.ts (6). Only the providers' HTTP answers, the provider list, the DB and the cache store are fakes. halService, factCheck, the early stop and the gate chain are the real code.

  • Eiffel-in-Berlin.
    • Mode off: live flagged, delta 0, and no provider is called.
    • Mode shadow: the live outcome fields are identical and current_repid is identical, while the shadow row says quorum vetoed, quorum_met, families openai + qwen, and a negative would-be delta.
  • A true claim: quorum clean, positive would-be delta, where the extractor gave flagged / 0.
  • Cache hit makes no provider call.
  • enforce behaves as shadow.
  • A missing table is loud and leaves the live result alone.
  • Strictness 2: no second quorum call, and quorum_hal_delta === live_hal_delta.

tests/hal-verdict-receipt-id.test.ts (24): id stability, what changes it, no text in the record, flag behaviour, mode parsing, and the boot log.

Fails on main, passes on the branch. Outputs are saved under E:\dev\reports\2026-11-16\hal-score-gate\ (main.txt/.json, branch.txt/.json).

  • On origin/main: 0 of 6 pass. The shadow file runs, because it imports only runScoreEvent. Its live assertions pass (flagged, delta 0, no provider called), and it fails for the real reason: no quorum verdict is recorded. The receipt-id suite cannot load.
  • On the branch: 30 of 30 pass.

Non-vacuity: forcing the shadow to copy the live verdict on the extractor path turned 4 of 6 red; reverted.

Full suite (Windows checkout, dummy Supabase URL): 21 suites red. I re-ran those 21 on origin/main src and on the branch with the same command.

  • 16 fail identically on main. These are pre-existing on this checkout (ENV).
  • 1 was real and is fixed in 831297c. hal-unearned-reward checks by source order that the live reward floor is read before the first computeDelta({; the shadow helper had been placed above runScoreEvent.
  • 4 pass when re-run. They fail only under full parallel load and touch no file here.

CI on Linux is the reference.

tsc --noEmit clean.

Not done

  • Enforce. Production already enforces the quorum through HAL_STRICTNESS=2, so enforce here would only matter for deploys on the code default. Changing the code default to 2 is a separate MAJOR call (it adds provider calls to every self-hosted score event).
  • Grounding abstain (proof-carrying answers under HAL_GROUNDING_MODE=enforce) is not copied into the would-be delta; documented on quorumWouldBeDelta.
  • The eval's other asks for this gap (a decontaminated private holdout, and P/R/F1/ECE with the ruler hash) are a measurement task, not this PR.

🤖 Generated with Claude Code

DealAppSeo and others added 2 commits November 16, 2026 12:41
…+ per-verdict receipt id (default off)

Gap 2 of the 2026-11-16 E2E eval. HAL_SCORE_GATE_MODE=off|shadow|enforce, default off.
In shadow, each score event also gets the cross-provider quorum's verdict (strictness 2,
HAL cache in its own namespace, two-family early stop), recorded with the HAL delta each
verdict would apply and a receipt id each, in hal_score_gate_shadow (draft migration).
The live row, delta and score never change. enforce is refused at boot (r23) and runs
as shadow. When the live path is already the quorum (prod: repid_config
HAL_STRICTNESS=2) no second call is made.

Receipt id: halv1_ + sha256 over the verdict record (text hash, decision, score, mode,
families, per-provider verdicts). Also returned by POST /api/v1/hal/evaluate when
HAL_VERDICT_RECEIPT_ID_ENABLED=true (default off).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pers moved below runScoreEvent)

tests/hal-unearned-reward.test.ts checks by source order that reward_suppressed is read
before computeDelta. The shadow helper's computeDelta sat above runScoreEvent and tripped it.
Moved, no behaviour change (function declarations hoist).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DealAppSeo DealAppSeo added the grok-review Ask the grok-review loop to review this PR (also drafts) label Nov 16, 2026

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

grok-review Ask the grok-review loop to review this PR (also drafts)

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant