Skip to content

fix(settlement): one SERVICE_FULFILLED round per contract, however many runs hold it (R1b) - #1560

Merged
DealAppSeo merged 3 commits into
mainfrom
claude/r1b-fulfil-once
Nov 16, 2026
Merged

DealAppSeo merged 3 commits into
mainfrom
claude/r1b-fulfil-once

Conversation

@DealAppSeo

@DealAppSeo DealAppSeo commented Nov 16, 2026 •

Copy link
Copy Markdown
Owner

Owner action

You decide (MAJOR: RepID). Draft. Do not auto-merge. No flag: from merge on, a contract held by two runs pays out once.

What happened on 2026-07-08 [V SQL]

One settlement, contract 7ca47a1d, went through applyServiceFulfilledDeltas twice, 3.5 s apart:

run 1 run 2
provider SERVICE_FULFILLED +10, 1595→1605 +10, 1595→1605 (read before run 1's write landed)
buyer SERVICE_FULFILLED +5, 1030→1035 +5, 1035→1040
service_fulfilled_settled outbox row 1563 1564

So the buyer took +5 twice. The provider's score moved once, but its ledger holds two +10 rows over a single move, so the ledger overstates. The two outbox rows produced three giveFeedback txs (two for 1564 come from the inline/worker double that #1556 fixes).

Scope: of 724 contracts with fulfil events, this is the only double-run [V SQL]. None in the last 30 days.

One more contract (16a6eb16, 2026-06-27) has two buyer rows with different deltas (+5, +20) and a single outbox row. That is not this pattern. Not investigated here.

It was test traffic. The contract came from an E2E buy-loop run (payload "Eligible buy-loop 3", mock validators, 16:xx and 20:xx UTC, not the 12:00Z cron). The ledger effects were real, though: the buyer's row took +5 twice.

Client retry, or server? [V SQL + code]

  • It was not a re-sent POST /contracts/:id/fulfill. That route is guarded, and it runs no validation. Here, validation ran twice: three pcp_validation calls at 20:17:32.17–32.50 and three more at 32.78–33.09, followed by the two delta rounds at 32.66 and 36.28. llm_call_log names validators, not contracts, so this is strong timing evidence, not proof.
  • So it was two handler runs. Two server entry points call processOne: the cascade worker's tick, and POST /api/v1/agent/process-contracts, which the runner (scripts/demo/run-e2e-transactions.ts) calls once per contract with no retry in the committed code.
  • Cannot tell whether the second run was a script retry (the exact "Eligible buy-loop" variant is in no commit) or the script's call racing a worker tick. Nothing records the request.
  • Either way, two runs became two payouts only because of the missing guard. A server must not depend on its clients calling once.

Why [V code]

ServiceHandlerBase.claimNextContract "claims" by writing metadata.claimed_at and leaves status = 'escrowed'. A second claimer racing it also matches .eq('status','escrowed') and also gets the row. The comment said "lost race ⇒ 0 rows", which is true only for a row that had already moved on. Then processOne's escrowed → fulfilled update had no status guard, so both runs transitioned and both applied the deltas. That contract's metadata says claimed_by_handler: verification, so it took this path.

POST /contracts/:id/fulfill was already right: it transitions with .eq('status','escrowed') and applies deltas only when a row comes back.

Fix

The handler's fulfil transition is now .eq('status','escrowed').select('id').maybeSingle(). Only the run that makes the transition applies the deltas. The losing run logs and returns processed: false. The claim comment is corrected.

Not changed: the claim still is not exclusive, so a second run still does the work (LLM validation) and then stands down. The disputed branch inserts its queue row before transitioning, so a double run could still file two dispute rows; it moves no RepID. Making the claim exclusive is the follow-up that removes both.

Why not dedupe on (contract_id, x402_payment_id)? That would catch the outbox rows but not the score rows. The transition guard covers both, because one state change means one round. A unique index would also be a good second guard; it is DDL, so it is your call.

Tests

tests/service-handler-fulfil-once.test.ts: two concurrent processOne runs on one contract (both claim before either transitions), with a fake table that honours the update's filters the way PostgREST does.

  • on origin/main's handler: 2 of 3 fail (deltas applied twice; a contract that moved on still earns). The single-run control passes on both.
  • with the fix: 3/3, plus service-handler-claim-order and cascade-settlement-worker, 20/20. tsc --noEmit clean.

CI and review

  • test was red on a9495997, and REAL, from this change: tests/services/service-handler-base.test.ts serves maybeSingle() from a queue and had no entry for the transition's new row read, so every fulfil read as a lost race. Fixed in d6d324cb by queueing that row. The assertions are unchanged, and that suite plus the three above pass (27/27).

  • Grok (fenced lane, v4) r1 on a9495997: NO_FINDINGS. The source is unchanged since then; only the test double moved.

  • Strix (LOW): confirmed, fixed in 1d70cc21. The dispute branch still inserted a dispute_validation_queue row, then set disputed with no status guard. Two racing runs could each file a dispute row, and the dispute worker applies deltas per row, so the contract was charged twice. A run could also flip a contract another run had fulfilled. Now the disputed transition is guarded and runs before the insert, so only the winner files a dispute. If the insert fails, the contract is put back to escrowed so a later cycle retries. Two new tests fail on the previous source; the handler suites pass 29/29, zkp-audit-service 43/43. Strix's threads are replied to and resolved.

  • No Grok round covers 1d70cc21 yet.

Related: #1556 fixes the inline/worker double write of one outbox row. This PR fixes the double settlement that produced the second outbox row.

🤖 Generated with Claude Code

…ny runs hold it

The handler's claim writes metadata and leaves status 'escrowed', so two concurrent runs both hold
one contract (its comment said "lost race => 0 rows", which is true only for a row that moved on).
The escrowed -> fulfilled update had no status guard, so both runs applied the deltas.

Measured: contract 7ca47a1d (2026-07-08) got two rounds 3.5 s apart: two outbox rows, three
on-chain writes, the buyer +5 twice (1030->1035->1040), and the provider ledger shows two +10 rows
over one 1595->1605 move. 1 of 724 contracts with fulfil events shows this double-run.

Fix: the transition is conditional on status 'escrowed' and returns the row; only the run that
makes it applies the deltas. POST /contracts/:id/fulfill already did this.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
…ransition's row

processOne now reads the escrowed -> fulfilled update's row (maybeSingle); the queue-based double in
tests/services/service-handler-base.test.ts had no entry for it, so every fulfil read as a lost race.
Assertions unchanged.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@strix-security

strix-security Bot commented Nov 16, 2026 •

Copy link
Copy Markdown

Strix Security Review

✅ No open security findings.

PR summary: Settlement fix: `ServiceHandlerBase.processOne` now only applies the SERVICE_FULFILLED payout when it is the run that actually transitions a contract escrowed → fulfilled, and the dispute branch was reordered so the disputed transition is guarded the same way before a dispute queue row is filed. Adds a test with a PostgREST-like fake table proving two concurrent runs settle once.

Review summary

The handler-side settlement race this PR targets is genuinely closed: the escrowed → fulfilled transition is now a single conditional UPDATE whose loser returns processed:false before any delta, so two runs holding one contract can no longer apply SERVICE_FULFILLED twice, and the dispute branch added in the last commit is mutually exclusive with it. Reading through the rest of the change reached path turned up one remaining hole: the same once-per-contract invariant is not enforced off the handler. "Dispute route enqueues one validation row per request and the dispute worker settles each row, so a party can re-settle the same contract repeatedly" — POST /contracts/:id/dispute re-opens a fulfilled or already-disputed contract with no status precondition and files a new queue row on every call, and the dispute worker settles each row independently, so one contract can be made to record many provider/buyer RepID rounds against a counterparty that never acted. I also ruled out three suspicions after tracing them: the DB status trigger forbids returning a paid contract to escrowed, the new dispute put-back cannot touch a fulfilled row (and is in fact a dead disputed → escrowed write the trigger rejects, a liveness defect rather than a payout one), and the non-exclusive claim's duplicate LLM work is a pre-existing cost the new stand-down strictly reduces.

Ruled out (3)
  • Can a contract be moved back into `escrowed` after it was fulfilled, so a second `processOne` run pays the same contract again?
    The service_contracts status trigger admits `escrowed` only from `pending` and makes settled/resolved/expired/cancelled terminal, so the unguarded status writes in the escrow, cancel and resolve routes cannot re-arm a paid contract.
  • Does the new dispute put-back on an insert failure resurrect a contract or allow a second settlement?
    It is gated on `status='disputed'` so it cannot touch a fulfilled row, and that transition is rejected by the status trigger anyway — a dead recovery path that applies no deltas.
  • Can an outside party exploit the non-exclusive claim to exhaust the shared LLM validation quota?
    The duplicate work is pre-existing and costs the same per request either way, the sustained-burn path does not depend on claim exclusivity, and the new stand-down reduces a duplicate run's impact from double payout to discarded work.
Review history (2)
  • Review 2 · 1d70cc2 · Nov 16 05:37 UTC · Nothing new
  • Review 1 · d6d324c · Nov 16 05:10 UTC · Found 1 Low Vulnerability

Review 2 on this PR · 1d70cc2 · requested Nov 16 05:32 UTC on a new commit


Reviewed by Strix
Re-run review · Configure security review settings · Try a deeper review with @strix-security /deep

@strix-security strix-security Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Strix flagged a new security finding below. See the pinned summary comment for the full PR status.

Comment thread src/services/service-handler-base.ts Resolved
Comment thread src/services/service-handler-base.ts Resolved
The fulfil transition was guarded, but the dispute branch still inserted a dispute_validation_queue
row first and then set status 'disputed' with no status guard. Two runs routing one contract to
dispute each filed a row, and the dispute worker applies deltas per row, so the contract was charged
twice; a run could also flip a contract another run had fulfilled.

Now: the disputed transition is conditional on status 'escrowed' and returns the row; only the run
that makes it files the dispute. If the queue insert then fails, the contract is put back to
escrowed (guarded on 'disputed') so a later cycle retries; if that also fails it is visibly stuck,
never paid twice.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@DealAppSeo
DealAppSeo merged commit 5e3cb78 into main Nov 16, 2026
19 checks passed
@DealAppSeo
DealAppSeo deleted the claude/r1b-fulfil-once branch November 16, 2026 08:52
@DealAppSeo

Copy link
Copy Markdown
Owner Author

Grok review (fenced lane) round 2 on 1d70cc21: NO_FINDINGS

I ran it through the fenced wrapper (grok-review.v4.mjs, reviewer profile: read-only, MCP off). The wrapper accepted the answer: the nonce matched, the checkout was on 1d70cc21030de3c02a0bf6814793c42b9f59ccba, and Grok opened every file it cites. Round 1 was on a9495997. This round covers the two commits since then: the guarded dispute branch (Strix finding) and the test-double fix.

The spec Grok checked:

  • Fulfilled path (from round 1): the escrowed → fulfilled update requires the contract to still be status = 'escrowed'. Deltas apply only when that update returned a row, so they apply once per contract.
  • Dispute path (new): the escrowed → disputed update has the same status guard, and it now runs before the dispute_validation_queue insert. If it returns no row, no dispute row is filed. If the insert fails, the contract is put back to escrowed (guarded on status = 'disputed') and the run throws, so a later cycle retries.
  • Tests: two concurrent runs that both route to dispute file one row. A contract that another run has already fulfilled is not flipped to disputed.

Evidence Grok read:

  • r1b.r2.test-output-head.txt: 12/12 pass on head.
  • r1b.r2.test-output-old-src.txt: the same tests against the pre-fix processOne. The concurrent-dispute test fails with two dispute rows for one contract.

Result: NO_FINDINGS, nothing listed as not checked.

My adjudication: I agree with the spec result, and found one remaining risk outside it. It is not blocking, and this PR does not make it worse:

  • The risk: a dispute-row insert that commits on the database but whose response is lost reads as an error. The contract is then put back to escrowed, and a retry would file a second row.
  • This existed before the PR. The old order (insert, then transition) had the same exposure. It also had a second double-row path, a failed transition after a good insert, which this PR removes.
  • dispute_validation_queue has no unique index on contract_id [V: only the primary key, plus plain indexes on status and contract_id].
  • It has never happened. 0 of the 34 contracts with a dispute row have more than one [V, read-only SQL].
  • Two possible closes, neither in this PR: check for an existing row before the put-back, or add a partial unique index (DDL, so Sean's).

Not merged, and auto-merge is off.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant