fix(state): preserve shared cached UTXOs

[jonwalch]

Motivation

Queued and sent block caches stored one UTXO value per outpoint but removed entries per block. When competing blocks contained the same non-coinbase transaction, removing either provider deleted the shared outpoint even while another live block still provided it. AwaitUtxo could then wait and retry unnecessarily.

Restored sent-block forks were also flattened into one pruning batch, so height pruning could stop at an ineligible block from one fork and leave eligible blocks from another fork behind.

Solution

Track each cached UTXO by outpoint and provider block hash. Storage preserves every live provider value so removing one provider exposes the exact metadata supplied by a survivor. Lookup intentionally returns an arbitrary live provider while a collision remains; it does not select a branch.

Use an inline single-provider representation for the common case, promote to a provider map only on collisions, and collapse it after removals. Missing ownership is asserted in debug builds and remains a no-op in release builds.

Make sent-block registration idempotent and reconstruct one ordered pruning batch per non-finalized chain so every restored fork is pruned independently.

Integrate queued-body replacement with provider ownership: replacing one block removes its old outputs without deleting outputs still supplied by a competing block. Preserve the admission handling and queue bound introduced by #748.

Performance tradeoffs

The common case keeps one provider hash and UTXO inline, avoiding a separate allocation but adding one 32-byte block hash per distinct cached outpoint. Shared outpoints allocate a provider map and retain each provider-specific UTXO until the collision is resolved; the entry collapses back to inline storage when only one provider remains. A reference count would use less memory, but it could not recover the surviving provider's metadata when the currently exposed provider is removed.

Testing

Verified at 0af90da75, rebased onto origin/main at 9b83b94fd.

• Added regression coverage for queued dequeue, queued height pruning, sent height pruning, rejected sent-block removal, and both supported provider-removal orders.
• Added duplicate queued and sent block consistency tests.
• Verified exact surviving provider metadata and final cache cleanup.
• Added fork reconstruction, checkpoint-verified provider, and three-provider promotion/collapse coverage.
• Added unknown-provider and unknown-outpoint invariant tests in debug builds. Release no-op coverage checks both Single and Multiple, preserves both provider-specific values after an unknown-provider removal, and verifies subsequent cleanup.
• Retained the recursive failed-subtree cleanup test from fix(sync): retain full-state header forks #831 and added coverage for shared-provider cleanup across a failed subtree and a live competing branch.
• Retained feat(mining): advertise admitted blocks optimistically #748's body-replacement and queue-bound tests. Added replacement coverage for both providers, exact surviving metadata, stale-output removal, and final cache/index cleanup.
• cargo test -p zakura-state service::queued_blocks --lib --locked (20 passed)
• cargo test -p zakura-state service::queued_blocks --lib --release --locked (19 passed)
• cargo test -p zakura-state --lib --locked --quiet -- --test-threads=2 (574 passed, 4 ignored)
• cargo clippy -p zakura-state --all-targets --release --locked -- -D warnings
• cargo fmt --all -- --check
• git diff --check
• ./scripts/changelog.py check
• npx --yes markdownlint-cli@0.41.0 docs/changelog/unreleased/797.md --config .github/.markdownlint.yaml

Changelog

Added docs/changelog/unreleased/797.md for shared-provider retention and complete height pruning across restored competing chains. Validated it with the changelog and Markdown checks above.

Specifications & References

Closes #493.

[evan-forbes]

The provider-aware cache design looks correct. It preserves provider-specific UTXO data, removes one provider at a time, and keeps duplicate sent-block registration idempotent.

This PR now conflicts with main because #831 changed the same queued-block test file. Please restack the PR and preserve both sets of tests. The production changes from #831 merge automatically, but the test conflict needs a deliberate resolution. The retained #831 test also calls queue.known_utxos.is_empty(), while UtxoProviderCache currently exposes len() but not is_empty().

Please also add one regression that exercises both changes together. The test should remove a failed descendant subtree that contains one provider for an outpoint while a live block outside that subtree provides the same outpoint. It should verify the following behavior:

• Cleanup removes the providers from the failed subtree.
• Cleanup preserves the live provider's exact UTXO metadata.
• Cleanup removes the outpoint after the final provider leaves.

[jonwalch]

@evan-forbes, the requested changes are addressed in the restacked branch through 77bff4b7e:

• Rebasing onto current main preserved fix(sync): retain full-state header forks #831's recursive failed-subtree production changes and its dequeue_descendants_removes_the_complete_failed_subtree regression.
• UtxoProviderCache now exposes the test-only is_empty() required by that retained test.
• fail_descendants_preserves_shared_utxo_from_live_branch_until_removed removes the failed subtree, checks both failure responses and queue indexes, verifies the live competing provider's exact UTXO metadata, and verifies final-provider cleanup.

Issue #493 and the earlier review findings are also covered:

• Queued dequeue, queued height pruning, sent height pruning, and sent rejection each retain shared UTXOs until the final provider leaves.
• Duplicate queued and sent registrations remain idempotent, and rejection cleanup removes stale batch indexes.
• Restored competing chains use separate ordered sent batches; the regression and changelog cover pruning every fork.
• Three-provider promotion/removal/collapse, missing provider/outpoint assertions, and release-mode no-op behavior have direct cache tests.
• The module and PR explain why provider-specific values are retained instead of a reference count, disclose the 32-byte common-case ownership cost, and document arbitrary live-provider lookup.
• Duplicate queued requests now document that the ignored request's response channel closes. Entry-based removal and outer-only shrinking remain deliberate: the former permits final outer removal through one lookup, and collision maps are freed when they collapse to Single.

Verification on the current branch:

• Focused queued-block tests: 17 passed in debug and 16 passed in release.
• Full zakura-state library tests: 552 passed, 3 ignored.
• Clippy, rustfmt, diff checks, changelog validation, and Markdown lint all pass.

Could you please re-review? A maintainer also needs to add the C-bug label. I don't have triage or write permission to apply it.