Repository navigation
feat(hal): HAL_SCORE_GATE_MODE shadow quorum verdict per score event + per-verdict receipt id (default off) - #1570
Draft
DealAppSeo wants to merge 2 commits into
Draft
DealAppSeo wants to merge 2 commits into
DealAppSeo wants to merge 2 commits into
Conversation
…+ per-verdict receipt id (default off) Gap 2 of the 2026-11-16 E2E eval. HAL_SCORE_GATE_MODE=off|shadow|enforce, default off. In shadow, each score event also gets the cross-provider quorum's verdict (strictness 2, HAL cache in its own namespace, two-family early stop), recorded with the HAL delta each verdict would apply and a receipt id each, in hal_score_gate_shadow (draft migration). The live row, delta and score never change. enforce is refused at boot (r23) and runs as shadow. When the live path is already the quorum (prod: repid_config HAL_STRICTNESS=2) no second call is made. Receipt id: halv1_ + sha256 over the verdict record (text hash, decision, score, mode, families, per-provider verdicts). Also returned by POST /api/v1/hal/evaluate when HAL_VERDICT_RECEIPT_ID_ENABLED=true (default off). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…pers moved below runScoreEvent) tests/hal-unearned-reward.test.ts checks by source order that reward_suppressed is read before computeDelta. The shadow helper's computeDelta sat above runScoreEvent and tripped it. Moved, no behaviour change (function declarations hoist). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
This branch has not been deployed
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Owner action
You decide (MAJOR: governs how RepID could move). Shipped shadow-only. Draft. Do not auto-merge.
With both new flags unset, nothing changes: no provider call, no new row, no new response field.
tests/hal-score-gate-shadow.test.tspins that (fetchis never called with the mode off).To use it after merge, two steps, each optional and reversible:
supabase/migrations/20261116180000_hal_score_gate_shadow.sql. It creates one empty table, takes no lock on any existing table (no foreign key, on purpose), and turns RLS on with no grants. Expected result:Success. No rows returned. Rollback:drop table if exists public.hal_score_gate_shadow;repid-engine→ servicerepid-engine→ Variables →HAL_SCORE_GATE_MODE=shadow. Boot log line:[hal-score-gate] HAL_SCORE_GATE_MODE resolved to 'shadow'. Off again: delete the variable.Optional third:
HAL_VERDICT_RECEIPT_ID_ENABLED=trueon the same service addsreceipt_idtoPOST /api/v1/hal/evaluateresponses.Re-verification overturned half of the defect [V, read-only Supabase, 2026-11-16]
The eval (
E2E_EVAL_REPORT.md§7 gap 2) says the score pipeline defaults to the extractor-only HAL path. That is true of the code default and false of production.repid_config.HAL_STRICTNESS=2, andgetHalConfig()reads repid_config first.HAL_SCORE_EVENTsince 2026-06-04 recordshal_mode = 'fact-check', or'extractor-fallback'when no provider answered (those are all neutralized toflagged, delta 0).The extractor default applies to a fresh, local or self-hosted deploy, or to
HAL_CONFIG_FROM_DB=falsewith no env. That is what the eval ran (Eiffel-in-Berlin:flagged, delta 0, the same as a true claim). The pipeline comment that implied production used the extractor is corrected in this PR.So I built the switch so that it costs nothing in production today and still answers the eval's question wherever the extractor does score.
What it does
HAL_SCORE_GATE_MODE=off(default) |shadow|enforce.shadow: after each score event is written, a detached job records in
hal_score_gate_shadow, beside the live verdict:The live row, the delta and
repid_agentsnever change. The job runs after the write, so it adds no latency to it.providersFnhoisted, no behaviour change), the two-family early stop, and the HAL cache under its own key namespace.quorumWouldBeDelta, which copies the live gate chain. Any row wherequorum_source = 'live'and the two deltas differ means that copy has drifted.enforce: refused (r23: original + live = shadow only until your GO). It runs as shadow and logs at boot:
HAL_SCORE_GATE_MODE=enforce REFUSED ... Running as 'shadow'. There is no override variable on purpose: lifting the refusal is a code change and a review, not an env edit.An unknown value resolves to
off, loudly.Cost bound (quorum calls per score event, in shadow)
A degraded answer (no quorum) is not cached, so an outage cannot pin "no quorum" on a text for the TTL. Measured in the test: the first Eiffel event made ≤ 2 provider requests; a second event on the same answer was a cache hit with 0.
Per-verdict receipt id
halv1_+ sha256 over the verdict record:hal_scoreto 4 places, mode, strictness andquorum_met,It needs no table, and anyone holding the record recomputes it. Latency, confidence and cache markers are excluded, so a cached replay keeps the id. Two identical judgments of the same text share an id by design; link a verdict to an event beside the id, never inside it.
live_receipt_idandquorum_receipt_idare on every row.POST /api/v1/hal/evaluateaddsreceipt_idwhenHAL_VERDICT_RECEIPT_ID_ENABLED=true(exact string; anything else returns the same body object unchanged).trustshell verifycurrently reads its receipt id fromPOST /hal/receipt(the vote-row path, which sayscolumns-missing). Printing this id instead is a TrustShell change, left for the agent working there (hand-over note written).Tests
tests/hal-score-gate-shadow.test.ts(6). Only the providers' HTTP answers, the provider list, the DB and the cache store are fakes.halService,factCheck, the early stop and the gate chain are the real code.flagged, delta 0, and no provider is called.current_repidis identical, while the shadow row says quorumvetoed,quorum_met, familiesopenai+qwen, and a negative would-be delta.clean, positive would-be delta, where the extractor gaveflagged/ 0.enforcebehaves as shadow.quorum_hal_delta === live_hal_delta.tests/hal-verdict-receipt-id.test.ts(24): id stability, what changes it, no text in the record, flag behaviour, mode parsing, and the boot log.Fails on main, passes on the branch. Outputs are saved under
E:\dev\reports\2026-11-16\hal-score-gate\(main.txt/.json,branch.txt/.json).origin/main: 0 of 6 pass. The shadow file runs, because it imports onlyrunScoreEvent. Its live assertions pass (flagged, delta 0, no provider called), and it fails for the real reason: no quorum verdict is recorded. The receipt-id suite cannot load.Non-vacuity: forcing the shadow to copy the live verdict on the extractor path turned 4 of 6 red; reverted.
Full suite (Windows checkout, dummy Supabase URL): 21 suites red. I re-ran those 21 on
origin/mainsrc and on the branch with the same command.hal-unearned-rewardchecks by source order that the live reward floor is read before the firstcomputeDelta({; the shadow helper had been placed aboverunScoreEvent.CI on Linux is the reference.
tsc --noEmitclean.Not done
HAL_STRICTNESS=2, soenforcehere would only matter for deploys on the code default. Changing the code default to 2 is a separate MAJOR call (it adds provider calls to every self-hosted score event).HAL_GROUNDING_MODE=enforce) is not copied into the would-be delta; documented onquorumWouldBeDelta.🤖 Generated with Claude Code