fix(consensus/XDPoS,core,eth/downloader): rebuild missing V2 gap snapshots

[gzliudan]

Proposed changes

A node can lose its persisted V2 gap snapshot when the process exits between
writeHeadBlock and StoreSnapshot in writeBlockWithState: the head markers are
already on disk, the snapshot is not, and nothing recreates it. getSnapshot then
fails with a leveldb "not found" error for the whole affected epoch and the node
drops out of consensus participation. Making that write ordering atomic prevents
new holes but cannot repair a database that already has one.

Add a guarded self-healing path in engine_v2.getSnapshot: when no snapshot is
stored for the gap block, rebuild it from the committed state trie at
gapHeader.Root, then persist and cache it.

The derivation itself moves into engine_v2.BuildSnapshotFromState, which
eth/downloader.generateSnapshot now also uses. The downloader already carried
its own copy of this derivation, and the unstable xdc_sort ordering it shares
with core.BlockChain.UpdateM1 must not drift: a different equal-stake order
yields a different masternode set. The shared builder also refuses to produce an
empty snapshot, which the downloader previously persisted and which would then
permanently mask the missing masternode list. A gap pivot state sync that finds
no candidates therefore now fails loudly instead of storing that empty snapshot.

The rebuild is only attempted when it is both safe and useful:

• skip when a snapshot is already stored: only a genuinely missing key is
  healed, so a decode or I/O error never overwrites a stored masternode set;
• skip at or before V2 SwitchBlock, where the initial snapshot still comes from
  Initial() rather than the state trie;
• reject numbers that are not real gap blocks, since they can originate from
  unauthenticated vote/timeout messages;
• skip gap blocks more than two epochs behind the chain head, so a peer cannot
  force trie reads and database writes for arbitrary historic gap numbers;
• skip when the chain reader does not implement the new GapStateReader
  interface, which core.BlockChain satisfies via a compile-time assertion;
• skip while the gap block itself is not imported yet, where the missing state
  is transient and retrying later succeeds.

Past that point the gap block is imported, so a rebuild that failed on its state
cannot start working later: the trie is pruned and can never come back, so the
gap block is recorded in an LRU and never retried, instead of re-reading the
trie on every vote verification. Concurrent callers for the same gap block are
collapsed with singleflight. Pruned state is logged at warn rather than error,
and everything degrades to debug while syncing, where a missing gap snapshot is
expected and not actionable.

The rebuild needs the gap block's state root to still be readable, which the
targeted case implies: a node only keeps the gap block as its head across a
restart when that root was committed. Otherwise loadLastState finds no head
state, repair() rewinds to an ancestor that has one, and re-importing the gap
block runs UpdateM1 and writes the snapshot again. What is left are roots that
are gone for good, such as offline pruning, gap blocks below a fast sync pivot,
or a side chain whose trie was collected. No later attempt can recover those,
which is what the permanent give-up above encodes.

A rebuilt snapshot is derived from the gap block state, while UpdateM1 takes
candidates from the head state and stakes from the validator contract at
"latest". Those agree on the canonical import path, where the head is the gap
block itself, but not necessarily on the reorg path, so a rebuilt masternode set
can differ from what a peer persisted through a reorg. That is accepted here:
the alternative is no snapshot at all.

Add coverage for the state-derived builder and for every getSnapshot guard, and
seed the downloader test genesis with the minimum masternode voting contract
storage the builder reads.

Types of changes

What types of changes does your code introduce to XDC network?
Put an ✅ in the boxes that apply

• build: Changes that affect the build system or external dependencies
• ci: Changes to CI configuration files and scripts
• chore: Changes that don't change source code or tests
• docs: Documentation only changes
• feat: A new feature
• fix: A bug fix
• perf: A code change that improves performance
• refactor: A code change that neither fixes a bug nor adds a feature
• revert: Revert something
• style: Changes that do not affect the meaning of the code
• test: Adding missing tests or correcting existing tests

Impacted Components

Which parts of the codebase does this PR touch?
Put an ✅ in the boxes that apply

• Consensus
• Account
• Network
• Geth
• Smart Contract
• External components
• Not sure (Please specify below)

Checklist

Put an ✅ in the boxes once you have confirmed below actions (or provide reasons on not doing so) that

• This PR has sufficient test coverage (unit/integration test) OR I have provided reason in the PR description for not having test coverage
• Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs.
• Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet.
• Tested the backwards compatibility.
• Tested with XDC nodes running this version co-exist with those running the previous version.
• Relevant documentation has been updated as part of this PR
• N/A

[Copilot]

Pull request overview

Fixes a consensus liveness failure in XDPoS v2 where a missing on-disk “gap snapshot” (e.g., due to an unclean shutdown between head write and snapshot persistence) causes getSnapshot to error repeatedly and drop a node out of consensus for up to an epoch window.

Changes:

• Adds an optional chainStateReader extension interface to open committed state by root without changing consensus.ChainReader.
• Implements rebuildSnapshotFromState to reconstruct a missing gap snapshot directly from the committed state trie and persist it back to LevelDB + in-memory LRU.
• Extends getSnapshot to attempt this self-healing only for V2-era gap blocks (gapBlockNum > SwitchBlock), preserving the pre-Initial() behavior around the V1→V2 boundary.

[coderabbitai]

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 3c1473d9-549f-41f6-9b92-fa293f84220a

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

• 🔍 Trigger review

✨ Finishing Touches

🧪 Generate unit tests (beta)

• Create PR with unit tests

---

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

• X
• Mastodon
• Reddit
• LinkedIn

_{Comment @coderabbitai help to get the list of available commands.}

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 3 comments.

Comments suppressed due to low confidence (2)

consensus/XDPoS/engines/engine_v2/snapshot.go:94

• The sort comparator must be a strict ordering. Using >= 0 means equal stakes will return true for both (i,j) and (j,i), which violates the sort.Interface contract and can lead to undefined ordering.

Use a strict > 0 comparison, and add a deterministic tie-breaker (e.g., address) so rebuilt snapshots are stable across nodes when stakes are equal.

	xdc_sort.Slice(pairs, func(i, j int) bool {
		return pairs[i].Stake.Cmp(pairs[j].Stake) >= 0
	})

consensus/XDPoS/engines/engine_v2/snapshot.go:76

• The added tests do not exercise rebuildSnapshotFromState or the new self-healing path in getSnapshot (they only validate mock helper methods / singleflight behavior). This risks giving a false sense of coverage for consensus-critical recovery logic.

Consider adding a unit test that builds a real state.StateDB (in-memory), populates the validator contract storage slots (candidates + caps), and calls rebuildSnapshotFromState, asserting the persisted snapshot order and DB write.

// rebuildSnapshotFromState reconstructs a missing gap-block snapshot by reading
// candidate addresses and stakes directly from the committed state trie at
// gapRoot. It uses the same StateDB slot-read path as core.BlockChain. UpdateM1
// (when it can read candidates from state) and eth/downloader.generateSnapshot,
// so no EVM client is required. On success the snapshot is persisted to the
// database and added to the in-memory LRU cache.
func (x *XDPoS_v2) rebuildSnapshotFromState(sr chainStateReader, number uint64, hash common.Hash, gapRoot common.Hash) (*SnapshotV2, error) {

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated 2 comments.

Comments suppressed due to low confidence (1)

consensus/XDPoS/engines/engine_v2/snapshot.go:97

• The sort comparator used for rebuilding snapshots is not a strict ordering: it returns true even when stakes are equal (Cmp == 0). Go's sort requires a strict weak ordering; violating it can produce non-deterministic ordering across Go versions/architectures, which is risky for consensus because candidate ordering can change when stakes tie. Add a deterministic tie-breaker (e.g., by address) and use > 0 for the stake comparison.

	xdc_sort.Slice(pairs, func(i, j int) bool {
		return pairs[i].Stake.Cmp(pairs[j].Stake) >= 0
	})

[Copilot]

Pull request overview

Copilot reviewed 3 out of 3 changed files in this pull request and generated no new comments.

Comments suppressed due to low confidence (3)

consensus/XDPoS/engines/engine_v2/snapshot.go:97

• The sort comparator used to order candidates is not a strict ordering (>= 0). For equal stakes it returns true in both directions, which violates sort.Less requirements and can yield non-deterministic ordering across nodes (consensus-critical). Use > 0 for stake ordering and add a deterministic tie-breaker (e.g., address comparison).

	xdc_sort.Slice(pairs, func(i, j int) bool {
		return pairs[i].Stake.Cmp(pairs[j].Stake) >= 0
	})

consensus/XDPoS/engines/engine_v2/snapshot_test.go:56

• The new self-healing logic (rebuildSnapshot + singleflight path in getSnapshot) is not covered by tests. The added tests here only re-test snapshot (de)serialization and candidate membership, and the mock types below are currently unused, so regressions in the rebuild path (e.g., candidate ordering, persistence, singleflight reuse) would go unnoticed.

// ============================================================================
// Mock implementations for self-healing tests
// ============================================================================

// mockStateDB is a minimal mock implementation of state.StateDB
// for testing snapshot reconstruction from state trie
type mockStateDB struct {
	candidates  []common.Address
	stakes      map[common.Address]*big.Int

consensus/XDPoS/engines/engine_v2/snapshot_test.go:86

• The mock mockChainStateReader currently cannot return a successful StateAt result (it always returns an error), and mockStateDB can't actually be used where *state.StateDB is required. As written, this mock section is dead code/misleading; either remove it or replace it with a test setup that can construct a real *state.StateDB for the rebuild success path.

// mockChainStateReader is a mock implementation of chainStateReader
// that returns a mock StateDB
type mockChainStateReader struct {
	stateDB      *mockStateDB
	shouldError  bool
	errorMessage string
}

func (m *mockChainStateReader) StateAt(root common.Hash) (*state.StateDB, error) {
	if m.shouldError {
		return nil, errors.New(m.errorMessage)
	}
	return nil, errors.New("mockChainStateReader: no StateDB configured")
}

[gzliudan]

replaced by #2507