perf(consensus): batch block UTXOs under a shared read budget

[evan-forbes]

Motivation

Independent architectural comparison with #918, based on the same main commit (0f7b9e836). This draft does not include or change #918.

The serial path sends one state request and starts one blocking read per external input. #918 overlaps those requests but retains per-transaction scheduling and does not bound aggregate read work across blocks. This draft moves lookup ownership to the block and batches work through the database boundary.

Keep this as a comparison draft. The tests pass, but this design introduces a block-wide availability barrier and the development-profile probe shows small-set overhead. Production sync performance has not been established.

Solution

• A lazy, block-owned resolver deduplicates external outpoints. Transactions run their existing quick checks before awaiting it. Block-local outputs retain priority.
• The resolver sends batches of at most 64 outpoints, with at most four outstanding batches per block. Cloned transaction requests share the same resolution and result.
• StateService subscribes to missing-output notifications before checking queued blocks or reading the database. This prevents an output arriving during read admission from being lost.
• One read budget permits at most four blocking batch-read tasks across the state's clones. A running task owns its permit until it finishes, even if verification is cancelled. Missing-output waits hold no read permit.
• Each database batch uses native RocksDB MultiGet for transaction locations and outputs. Both reads share one snapshot. The database format stays unchanged.
• The transaction verifier fills spent outputs in input order. It keeps resolved external outputs separate from the block's new outputs. Contextual validation remains authoritative for chain membership, spend order, double spends, coinbase maturity, and value balance.

The shared budget covers the new batched block-lookup path, not unrelated RPC or mempool database work. Mempool lookup behavior and standalone transaction requests without a resolver retain their existing paths.

Comparison with #918

Concern	#918	This draft
Lookup ownership	Each transaction	One resolver per block attempt
State requests	One per external input	One per group of up to 64 unique external inputs
Running read work	No shared lookup budget	At most four batch-read tasks per state instance
Missing outputs	Per-input waits	State notifications, outside the read budget
Verification progress	A transaction waits for its own inputs	A transaction with external inputs waits for the block's external-input set
Timeout scope	Six minutes per admitted input	One six-minute deadline, including admission, for the block resolver
Integration cost	Small internal change	New state request/response variants and transaction-request context

The deadline starts when a transaction first polls the resolver. Later batches do not receive an additional six minutes. Errors and timeouts still fail the verification attempt, but concurrent failures can change which error arrives first. Dropping all resolver owners cancels pending requests. Already-running disk work finishes under its retained permit.

The API changes require zakura-state and zakura-consensus 8.0.0-rc0. The RPC crate moves to 10.0.0-rc0 because its public service bounds expose these dependency types. The utility crate takes a patch bump so its rewritten dependency requirement can publish. After merging main, the chain dependency is 7.0.0-rc0. Header-chain, node-services, script, and network take patch bumps so their updated chain requirements publish together. Without those bumps, the registry copies pull in chain 6 alongside chain 7. The node's release version remains under the release workflow's control. No crates are published by this PR.

Testing

• All 767 consensus/state library tests passed; four ignored tests were skipped.
• New tests cover shared and deduplicated requests, both batch boundaries, reversed responses and v5 sighashes, sliding-window refill, quick rejection, script rejection through the semantic block verifier, incomplete responses, state errors, timeout mapping, cancellation, read-permit ownership, lost-notification races, and finalized/queued output metadata.
• Snapshot tests cover unsorted and duplicate keys, missing keys, mutation after snapshot creation, and input-order preservation.
• cargo +1.97.0 check --workspace --all-targets --offline passed after the version updates.
• cargo +1.97.0 clippy -p zakura-consensus -p zakura-state --locked --all-targets -- -D warnings, formatting, and git diff --check passed.

Audit fixes

The resolver cache now has an 8 MiB limit with fallback lookups. Quick checks reject duplicate spends and oversized scripts before shared resolution. Batch hits notify other waiters and remove pending entries. Notifications can complete requests while disk admission is blocked. Commit and block-restoration paths recheck pending outputs. Fork-cache removal preserves outputs owned by surviving blocks. The RPC regression, Clippy, formatting, and changelog checks passed.

Warm-database diagnostic

The ignored utxo_state_path_timing test compares three scheduling modes against the same seeded, ephemeral RocksDB instance. The modes share the same stream orchestration. The database contains synthetic transaction-location and output indexes, not a valid chain. Each mode warms its keys before measurement. These are local means from the development profile with Rust 1.97.0 and four Tokio worker threads.

Inputs	Serial single requests	64 concurrent single requests	64-output batches, four concurrent
1	93.5 µs	94.8 µs	116.6 µs
8	465.8 µs	131.5 µs	175.3 µs
64	3.355 ms	0.805 ms	1.025 ms
1,001	39.68 ms	10.55 ms	4.27 ms

This probe measures one input set at a time. It does not answer the multi-transaction contention question raised on #918. It isolates the state path. It does not measure whole PRs, resolver assembly, cryptographic verification, cold-cache I/O, or concurrent-block sync. It shows a large-set benefit and small-set overhead in this build, not a production throughput guarantee.

cargo +1.97.0 test -p zakura-consensus -p zakura-state --locked --lib \
  utxo_state_path_timing -- --ignored --nocapture

Changelog

Added docs/changelog/unreleased/919.md and four entries in docs/changelog/params.md. The changelog check passed.

Specifications & References

• Bounded alternative: fix(consensus): prevent serial UTXO lookup delays #918.
• Existing AwaitUtxo informational semantics remain the basis for lookup results.

Follow-up Work

• Measure optimized cold/warm database performance and concurrent-block sync before selecting limits or promoting this draft.
• Measure the block-wide availability barrier during out-of-order sync. Consider transaction-specific readiness if that barrier loses useful verification overlap.
• Resolve the small-set overhead before claiming this architecture improves normal operation.

[v12-auditor]

Note

Complete: Audit complete. V12 found six issues worth reviewing.

Open the full results here.

Finding	Severity	Details
F-269562	🟠 High	Block UTXO bytes exhaust memory The new block resolver bounds outpoints per request and request concurrency, but it does not bound the total bytes returned for those outpoints. A candidate block can reference tens of thousands of historical UTXOs whose heap-backed locking scripts were paid for in earlier blocks, and BlockUtxos retains every full Utxo in one block-wide map until verification ends. Each consuming transaction then clones its selected Utxo and clones the contained Output again before transparent script verification. Cross-transaction references to the same outpoint are deduplicated only for lookup, so many distinct transactions can also fan one large historical output into many transaction-local copies before the later block-wide duplicate-spend check. A patient attacker can pre-position large unspent outputs and submit a bounded 2 MB block that drives aggregate allocations far beyond the candidate block size.
F-269563	🟠 High	Proposal floods starve UTXO admission An unauthenticated getblocktemplate proposal request can submit attacker-controlled block bytes to semantic verification without proof of work. Each proposal owns its complete deduplicated external-outpoint vector, can keep four 64-outpoint AwaitUtxos batches outstanding, and registers pending subscriptions before waiting for the four-permit read semaphore. These limits apply independently per proposal and there is no aggregate bound on pending-map entries, semaphore admission waiters, or proposal requests. The TLS RPC accept loop spawns a task per connection without a connection semaphore, and proposal validation directly awaits CheckProposal without an application-level deadline. Repeated proposals with distinct missing prevouts can therefore retain blocks, resolver state, pending channels, and read-admission futures for the six-minute resolver lifetime while competing with real block verification for the same budget.
F-269564	🟡 Medium	Batch hits neither broadcast nor clean up await_utxos subscribes every requested outpoint in the global PendingUtxos map before reading state. When a cloned-state or RocksDB read finds an output, the function adds it only to its request-local available map and skips the associated response future; it never calls PendingUtxos::respond. If no other receiver exists, the sender becomes a dead map entry until the synchronous 30-second prune. If another request is already waiting for the same outpoint, that receiver remains blocked even though the later read has proved the UTXO exists. The block-wide resolver makes both effects reachable in large bursts before invalid scripts are checked.
F-269565	🟡 Medium	Delivered outputs wait behind disk admission After registering notification receivers, every non-empty batch unconditionally acquires the shared read semaphore and completes a blocking state/database read before polling any receiver. If all requested outputs arrive while the request is queued for admission, PendingUtxos::respond buffers the values successfully, but those values cannot complete the batch until a permit is obtained and the redundant read finishes. The included test demonstrates this ordering by notifying every output while all permits are held and observing that the lookup remains pending until the permits are released. Because canceled running reads retain their permits, timed-out callers do not immediately restore capacity. A backlog of proposal reads can therefore consume the block resolver's single six-minute deadline even after all dependencies have arrived.
F-269566	🟡 Medium	Later state transitions strand UTXO waits AwaitUtxos performs one cache/state read and then relies exclusively on the notification receivers registered at request start. Block output notifications are emitted at semantic or checkpoint admission, not when a later write actually becomes visible, and successful writer commits only publish updated state snapshots. A request that registers after the admission notification but takes its read before commit therefore remains blocked after the output becomes readable. The same defect applies when ReconsiderBlock restores an invalidated output: the state snapshot is republished but existing UTXO waiters are not notified or rechecked. The batch resolver converts the stranded dependency into TransparentInputNotFound when its shared deadline expires.
F-269567	🟡 Medium	Cache removal hides surviving fork outputs Both queued and sent block caches flatten all available outputs into a single OutPoint -> Utxo map while separately tracking which outpoints belong to each block. If two different fork blocks contain the same transaction, they legitimately share the same output key in these availability caches. Removing, failure-pruning, or height-pruning either block unconditionally deletes that key without checking whether another queued or sent block still owns it. A later AwaitUtxos call can therefore miss an output that remains in the pipeline, take a pre-commit state read, and enter notification-only waiting. The surviving block's eventual queued-to-sent transition or successful commit does not emit a new notification, so the dependency can remain stranded until timeout.

And one more auto-invalidated finding.

Analyzed 11 files, diff 1697ad0...76f2198.

[evan-forbes]

closing for now as we merged #918