Repository navigation
fix(settlement): one SERVICE_FULFILLED round per contract, however many runs hold it (R1b) - #1560
Conversation
…ny runs hold it The handler's claim writes metadata and leaves status 'escrowed', so two concurrent runs both hold one contract (its comment said "lost race => 0 rows", which is true only for a row that moved on). The escrowed -> fulfilled update had no status guard, so both runs applied the deltas. Measured: contract 7ca47a1d (2026-07-08) got two rounds 3.5 s apart: two outbox rows, three on-chain writes, the buyer +5 twice (1030->1035->1040), and the provider ledger shows two +10 rows over one 1595->1605 move. 1 of 724 contracts with fulfil events shows this double-run. Fix: the transition is conditional on status 'escrowed' and returns the row; only the run that makes it applies the deltas. POST /contracts/:id/fulfill already did this. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ransition's row processOne now reads the escrowed -> fulfilled update's row (maybeSingle); the queue-based double in tests/services/service-handler-base.test.ts had no entry for it, so every fulfil read as a lost race. Assertions unchanged. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Strix Security Review✅ No open security findings. PR summary: Settlement fix: `ServiceHandlerBase.processOne` now only applies the SERVICE_FULFILLED payout when it is the run that actually transitions a contract escrowed → fulfilled, and the dispute branch was reordered so the disputed transition is guarded the same way before a dispute queue row is filed. Adds a test with a PostgREST-like fake table proving two concurrent runs settle once. Review summaryThe handler-side settlement race this PR targets is genuinely closed: the escrowed → fulfilled transition is now a single conditional UPDATE whose loser returns Ruled out (3)
Review history (2)
Review 2 on this PR · Reviewed by Strix |
The fulfil transition was guarded, but the dispute branch still inserted a dispute_validation_queue row first and then set status 'disputed' with no status guard. Two runs routing one contract to dispute each filed a row, and the dispute worker applies deltas per row, so the contract was charged twice; a run could also flip a contract another run had fulfilled. Now: the disputed transition is conditional on status 'escrowed' and returns the row; only the run that makes it files the dispute. If the queue insert then fails, the contract is put back to escrowed (guarded on 'disputed') so a later cycle retries; if that also fails it is visibly stuck, never paid twice. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Grok review (fenced lane) round 2 on
|
Owner action
You decide (MAJOR: RepID). Draft. Do not auto-merge. No flag: from merge on, a contract held by two runs pays out once.
What happened on 2026-07-08 [V SQL]
One settlement, contract
7ca47a1d, went throughapplyServiceFulfilledDeltastwice, 3.5 s apart:SERVICE_FULFILLEDSERVICE_FULFILLEDservice_fulfilled_settledoutbox rowSo the buyer took +5 twice. The provider's score moved once, but its ledger holds two +10 rows over a single move, so the ledger overstates. The two outbox rows produced three
giveFeedbacktxs (two for 1564 come from the inline/worker double that #1556 fixes).Scope: of 724 contracts with fulfil events, this is the only double-run [V SQL]. None in the last 30 days.
One more contract (
16a6eb16, 2026-06-27) has two buyer rows with different deltas (+5, +20) and a single outbox row. That is not this pattern. Not investigated here.It was test traffic. The contract came from an E2E buy-loop run (payload "Eligible buy-loop 3", mock validators, 16:xx and 20:xx UTC, not the 12:00Z cron). The ledger effects were real, though: the buyer's row took +5 twice.
Client retry, or server? [V SQL + code]
POST /contracts/:id/fulfill. That route is guarded, and it runs no validation. Here, validation ran twice: threepcp_validationcalls at 20:17:32.17–32.50 and three more at 32.78–33.09, followed by the two delta rounds at 32.66 and 36.28.llm_call_lognames validators, not contracts, so this is strong timing evidence, not proof.processOne: the cascade worker's tick, andPOST /api/v1/agent/process-contracts, which the runner (scripts/demo/run-e2e-transactions.ts) calls once per contract with no retry in the committed code.Why [V code]
ServiceHandlerBase.claimNextContract"claims" by writingmetadata.claimed_atand leavesstatus = 'escrowed'. A second claimer racing it also matches.eq('status','escrowed')and also gets the row. The comment said "lost race ⇒ 0 rows", which is true only for a row that had already moved on. ThenprocessOne'sescrowed → fulfilledupdate had no status guard, so both runs transitioned and both applied the deltas. That contract's metadata saysclaimed_by_handler: verification, so it took this path.POST /contracts/:id/fulfillwas already right: it transitions with.eq('status','escrowed')and applies deltas only when a row comes back.Fix
The handler's fulfil transition is now
.eq('status','escrowed').select('id').maybeSingle(). Only the run that makes the transition applies the deltas. The losing run logs and returnsprocessed: false. The claim comment is corrected.Not changed: the claim still is not exclusive, so a second run still does the work (LLM validation) and then stands down. The disputed branch inserts its queue row before transitioning, so a double run could still file two dispute rows; it moves no RepID. Making the claim exclusive is the follow-up that removes both.
Why not dedupe on
(contract_id, x402_payment_id)? That would catch the outbox rows but not the score rows. The transition guard covers both, because one state change means one round. A unique index would also be a good second guard; it is DDL, so it is your call.Tests
tests/service-handler-fulfil-once.test.ts: two concurrentprocessOneruns on one contract (both claim before either transitions), with a fake table that honours the update's filters the way PostgREST does.origin/main's handler: 2 of 3 fail (deltas applied twice; a contract that moved on still earns). The single-run control passes on both.service-handler-claim-orderandcascade-settlement-worker, 20/20.tsc --noEmitclean.CI and review
testwas red ona9495997, and REAL, from this change:tests/services/service-handler-base.test.tsservesmaybeSingle()from a queue and had no entry for the transition's new row read, so every fulfil read as a lost race. Fixed ind6d324cbby queueing that row. The assertions are unchanged, and that suite plus the three above pass (27/27).Grok (fenced lane, v4) r1 on
a9495997: NO_FINDINGS. The source is unchanged since then; only the test double moved.Strix (LOW): confirmed, fixed in
1d70cc21. The dispute branch still inserted adispute_validation_queuerow, then setdisputedwith no status guard. Two racing runs could each file a dispute row, and the dispute worker applies deltas per row, so the contract was charged twice. A run could also flip a contract another run had fulfilled. Now the disputed transition is guarded and runs before the insert, so only the winner files a dispute. If the insert fails, the contract is put back toescrowedso a later cycle retries. Two new tests fail on the previous source; the handler suites pass 29/29,zkp-audit-service43/43. Strix's threads are replied to and resolved.No Grok round covers
1d70cc21yet.Related: #1556 fixes the inline/worker double write of one outbox row. This PR fixes the double settlement that produced the second outbox row.
🤖 Generated with Claude Code