Skip to content

fix(consensus/XDPoS,core,eth): repair missing V2 gap snapshots at startup - #2507

Open
gzliudan wants to merge 1 commit into
XinFinOrg:dev-upgradefrom
gzliudan:repair-snapshot-at-startup
Open

fix(consensus/XDPoS,core,eth): repair missing V2 gap snapshots at startup#2507
gzliudan wants to merge 1 commit into
XinFinOrg:dev-upgradefrom
gzliudan:repair-snapshot-at-startup

Conversation

@gzliudan

@gzliudan gzliudan commented Aug 9, 2026

Copy link
Copy Markdown
Collaborator

Proposed changes

A node can lose its persisted V2 gap snapshot when the process exits between writeHeadBlock and UpdateM1 in writeBlockWithState: the head markers are already on disk, the snapshot is not, and nothing recreates it. Every consumer of that snapshot then fails for a whole epoch. A node that keeps up with the chain head still imports blocks, because the fetcher path verifies headers with fullVerify disabled and trusts header.Validators, but it can no longer mine or vote. A node that falls behind takes the downloader path, where fullVerify is enabled, and stalls on the epoch switch block while reporting it as a bad block.

Repair the hole once at startup, before the chain is used: for each gap block whose snapshot is missing, rebuild it from the state committed at that block and store it.

Only the last two gap blocks at or below the head are considered. A gap block G is among them exactly while the head is in [G, G+2*Epoch), and G's snapshot is consulted exactly while the head is in [G+Gap, G+Gap+Epoch), so every hole that can still affect the running chain is covered. Checking a single gap block is not enough: a node killed at G restarts with the head on G itself, while a node that stalled on the following epoch switch restarts several hundred blocks later, and no single derivation yields both.

The derivation is shared with core.BlockChain.UpdateM1 and the downloader through the unstable xdc_sort ordering, which must not drift, since a different equal-stake order yields a different masternode set. It also refuses to produce an empty snapshot, which would load back fine and permanently mask the missing masternode list.

The rebuild is skipped when a snapshot is already stored, so a decode or I/O error never overwrites a masternode set persisted through a reorg, and at or below the V2 switch block, where the snapshot still comes from Initial(). It needs the gap block's state root to still be readable, which the targeted case implies: a node only keeps that root when it was committed, and this trie database never prunes what it has written. Otherwise loadLastState finds no head state, repair() rewinds to an ancestor that has one, and re-importing the gap block runs UpdateM1 and writes the snapshot again. Roots that are gone for good, such as after offline pruning or below a fast sync pivot, cannot be recovered, so every failure is only logged and startup continues.

Types of changes

What types of changes does your code introduce to XDC network?
Put an in the boxes that apply

  • build: Changes that affect the build system or external dependencies
  • ci: Changes to CI configuration files and scripts
  • chore: Changes that don't change source code or tests
  • docs: Documentation only changes
  • feat: A new feature
  • fix: A bug fix
  • perf: A code change that improves performance
  • refactor: A code change that neither fixes a bug nor adds a feature
  • revert: Revert something
  • style: Changes that do not affect the meaning of the code
  • test: Adding missing tests or correcting existing tests

Impacted Components

Which parts of the codebase does this PR touch?
Put an in the boxes that apply

  • Consensus
  • Account
  • Network
  • Geth
  • Smart Contract
  • External components
  • Not sure (Please specify below)

Checklist

Put an in the boxes once you have confirmed below actions (or provide reasons on not doing so) that

  • This PR has sufficient test coverage (unit/integration test) OR I have provided reason in the PR description for not having test coverage
  • Tested on a private network from the genesis block and monitored the chain operating correctly for multiple epochs.
  • Provide an end-to-end test plan in the PR description on how to manually test it on the devnet/testnet.
  • Tested the backwards compatibility.
  • Tested with XDC nodes running this version co-exist with those running the previous version.
  • Relevant documentation has been updated as part of this PR
  • N/A

@coderabbitai

coderabbitai Bot commented Aug 9, 2026

Copy link
Copy Markdown

Important

Review skipped

Auto reviews are disabled on base/target branches other than the default branch.

Please check the settings in the CodeRabbit UI or the .coderabbit.yaml file in this repository. To trigger a single review, invoke the @coderabbitai review command.

⚙️ Run configuration

Configuration used: defaults

Review profile: CHILL

Plan: Pro Plus

Run ID: 314cef8e-9bf5-4a2c-9cbf-42411142f214

You can disable this status message by setting the reviews.review_status to false in the CodeRabbit configuration file.

Use the checkbox below for a quick retry:

  • 🔍 Trigger review

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Pull request overview

This PR adds a startup-time repair step for XDPoS v2 “gap block” snapshots to recover from a crash window where the chain head markers are persisted but the corresponding V2 snapshot key is missing, which can otherwise break voting/mining (and stall syncing on epoch transitions).

Changes:

  • Invoke EngineV2.RepairGapSnapshots during eth backend startup after engine initialization.
  • Add a DB existence probe (rawdb.HasXdposV2Snapshot) and implement snapshot rebuilding from historical state (BuildSnapshotFromState) with logic to consider the last two relevant gap blocks.
  • Add focused unit tests covering snapshot derivation and the repair candidate selection/behavior.

Reviewed changes

Copilot reviewed 5 out of 5 changed files in this pull request and generated 2 comments.

Show a summary per file
File Description
eth/backend.go Calls the new startup repair routine after consensus engine initialization.
core/rawdb/accessors_xdc.go Adds HasXdposV2Snapshot to probe snapshot existence without loading/decoding it.
core/blockchain_reader.go Declares BlockChain implements engine_v2.GapStateReader (supports StateAt).
consensus/XDPoS/engines/engine_v2/snapshot.go Implements snapshot derivation from state and startup repair of missing gap snapshots.
consensus/XDPoS/engines/engine_v2/snapshot_test.go Adds tests for deterministic snapshot building and repair behavior across candidate gap blocks.

💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.

Comment thread consensus/XDPoS/engines/engine_v2/snapshot.go
Comment thread consensus/XDPoS/engines/engine_v2/snapshot.go Outdated
Comment thread consensus/XDPoS/engines/engine_v2/snapshot.go
Comment thread eth/backend.go Outdated
@gzliudan
gzliudan force-pushed the repair-snapshot-at-startup branch 2 times, most recently from c7d7679 to b6451ef Compare August 11, 2026 08:58
…rtup

A node can lose its persisted V2 gap snapshot when the process exits between
writeHeadBlock and UpdateM1 in writeBlockWithState: the head markers are
already on disk, the snapshot is not, and nothing recreates it. Every consumer
of that snapshot then fails for a whole epoch. A node that keeps up with the
chain head still imports blocks, because the fetcher path verifies headers with
fullVerify disabled and trusts header.Validators, but it can no longer mine or
vote. A node that falls behind takes the downloader path, where fullVerify is
enabled, and stalls on the epoch switch block while reporting it as a bad block.

Repair the hole once at startup, before the chain is used: for each gap block
whose snapshot is missing, rebuild it from the state committed at that block
and store it.

Only the last two gap blocks at or below the head are considered. A gap block G
is among them exactly while the head is in [G, G+2*Epoch), and G's snapshot is
consulted exactly while the head is in [G+Gap, G+Gap+Epoch), so every hole that
can still affect the running chain is covered. Checking a single gap block is
not enough: a node killed at G restarts with the head on G itself, while a node
that stalled on the following epoch switch restarts several hundred blocks
later, and no single derivation yields both.

The derivation is shared with core.BlockChain.UpdateM1 and the downloader
through the unstable xdc_sort ordering, which must not drift, since a different
equal-stake order yields a different masternode set. It also refuses to produce
an empty snapshot, which would load back fine and permanently mask the missing
masternode list.

The rebuild is skipped when a snapshot is already stored, so a decode or I/O
error never overwrites a masternode set persisted through a reorg, and at or
below the V2 switch block, where the snapshot still comes from Initial(). It
needs the gap block's state root to still be readable, which the targeted case
implies: a node only keeps that root when it was committed, and this trie
database never prunes what it has written. Otherwise loadLastState finds no head
state, repair() rewinds to an ancestor that has one, and re-importing the gap
block runs UpdateM1 and writes the snapshot again. Roots that are gone for good,
such as after offline pruning or below a fast sync pivot, cannot be recovered,
so every failure is only logged and startup continues.
@gzliudan
gzliudan force-pushed the repair-snapshot-at-startup branch from b6451ef to cc9c8a2 Compare August 11, 2026 09:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants