Reject abandoned chain anchor hashes

[Frozen]

Summary

• set the mainnet shard 0 emergency-recovery retained block to 92,730,034
• document the shard 1 recovery boundary as 94,978,278
• set EmergencyRecoveryViewIDFloor to 1,000,000,000
• reject the first abandoned shard 0 child (92,730,035) by exact hash: 0x5de06979a333f20afb8b245a8cf44472dc5bfc7383a57ddee48e1809bcee7c5d
• reject the first abandoned shard 1 child (94,978,279) by exact hash: 0xc936581d391b74a620bf6636519834b14a9a2d4e9a5154867c8407f219d8a878
• keep exact-hash rejection for the known abandoned branch anchors as defense in depth
• apply rejection before consensus/header/signature caches and before core, receipt-chain, epoch-chain, head-write, and staged-sync shortcuts
• reject authenticated references to denied hashes in beacon crosslinks and incoming CX receipt source headers before off-chain metadata is written
• freeze CommitOffChainData latest-crosslink rollup on post-boundary mainnet shard 0 blocks so abandoned crosslink records cannot advance local last-continuous markers during recovery
• preserve normal crosslink rollup for retained mainnet history, testnet, and non-recovery execution
• allow replacement history because rejection is limited to exact abandoned hashes

Rejecting an abandoned branch anchor prevents its descendants from attaching through normal parent validation. The embedded-reference checks also prevent a valid outer block from reintroducing crosslink or CX metadata tied to a rejected source block.

The off-chain rollup freeze prevents empty recovery blocks from repeatedly scanning stale crosslink records and advancing the local last-crosslink pointer, which could otherwise emit stale heartbeats and impair crosslink liveness when the feature is re-enabled.

This PR prevents new acceptance on a clean or recovered database. It intentionally does not repair a database that already has a rejected anchor or descendant as its persisted head. Deployment over affected data still requires offline rollback/purge and verification of the retained head before starting network services.

[Test]

Previous CI on commit 458591a38 passed:

• build harmony/bootnode binary (amd64 and arm64)
• RPC integration tests (amd64 and arm64)
• pyhmy tests (amd64 and arm64)
• Greptile review
• changed packages in both unit-test jobs: consensus, consensus/engine, and internal/params
• core in both unit-test jobs

For commit e1d43c235:

• gofmt
• git diff --check
• added TestEmergencyRecoverySkipsLatestCrossLinkRollup covering post-boundary mainnet freeze, retained mainnet behavior, testnet behavior, and non-beacon behavior
• local targeted go test ./core -run '^TestEmergencyRecoverySkipsLatestCrossLinkRollup$' -count=1 is blocked by missing host BLS headers; CI runs with the repository BLS setup

The previous overall unit-test jobs failed later in the unrelated core/state/snapshot test TestDiffLayerExternalInvalidationPartialFlatten. Local make test cannot reach repository tests because the legacy harmonyone/localnet-test:latest runner requires a TTY, ships Go 1.16 while the repository requires Go 1.24, and cannot resolve the host worktree .git pointer inside the container.

[greptile-apps]

Greptile Summary

This PR implements a prevention-focused emergency recovery boundary that rejects exact abandoned-chain anchors and validates recovered shard-0 state before networked startup.

• Pins the retained mainnet shard-0 checkpoint, validator-list manifest, and recovery ViewID floor.
• Applies rejected-hash and embedded-reference checks across consensus, chain import, and metadata-writing paths.
• Adds offline rollback verification and repair, network isolation, recovery payload freezes, and focused recovery tests.

Confidence Score: 5/5

The PR appears safe to merge because no blocking failure remains after the checkpoint and validator-manifest startup issues were fixed and the persisted-head limitation was explicitly accepted with an offline recovery requirement.

No blocking failure remains.

Important Files Changed

Filename	Overview
consensus/engine/rejected_block.go	Defines exact abandoned-anchor hashes and a shared fail-closed hash validator used by the recovery checks.
consensus/recovery_checkpoint.go	Pins and validates the retained checkpoint tuple, canonical ancestry, persisted heads, state availability, and commit certificate.
consensus/recovery_validator_list.go	Supplies the release validator-list count and digest and validates the exact ordered persisted list.
cmd/harmony/main.go	Adds verified offline rollback and repair, networked startup validation, recovery isolation, and safe consensus initialization.
core/recovery_freeze.go	Centralizes the post-boundary recovery feature policy used to suppress unsafe payloads and metadata advancement.
core/rejected_block.go	Enforces exact-hash and embedded-reference rejection in core block-processing paths.
internal/params/emergency_recovery.go	Defines the retained recovery block and ViewID floor used across recovery enforcement.

Flowchart

%%{init: {'theme': 'neutral'}}%%
flowchart TD
    A[Start Harmony node] --> B{Offline maintenance?}
    B -->|Yes| C[Open database for verified rollback]
    C --> D[Verify retained checkpoint state and certificate]
    D --> E[Repair validator list and roll back canonical head]
    E --> F[Close database successfully]
    B -->|No| G{Mainnet shard 0?}
    G -->|Yes| H[Force network isolation]
    H --> I[Validate checkpoint ancestry, persisted heads, certificate, and validator manifest]
    I --> J[Initialize recovery ViewID floor and deterministic leader]
    G -->|No| J
    J --> K[Process consensus and block imports]
    K --> L{Rejected exact hash or embedded reference?}
    L -->|Yes| M[Reject before caches or metadata writes]
    L -->|No| N[Continue normal validation]
    N --> O{Recovery feature freeze active?}
    O -->|Yes| P[Suppress frozen payloads and crosslink rollup]
    O -->|No| Q[Normal processing]

Loading

_{Reviews (12): Last reviewed commit: "fix: make recovery rollback restartable" | Re-trigger Greptile}

[Frozen]

Shard-aware rollback is still required before this recovery binary can be used for both shards.

The node already opens its local shard chain through currentNode.Blockchain(), so the recovery policy must be selected from chain.ShardID():

Shard	Retained checkpoint	First removed/rejected child	Rejected child hash
0	92,730,034	92,730,035	0x5de06979a333f20afb8b245a8cf44472dc5bfc7383a57ddee48e1809bcee7c5d
1	94,978,278	94,978,279	0xc936581d391b74a620bf6636519834b14a9a2d4e9a5154867c8407f219d8a878

At the moment, verifyRevertTargetState compares every target against the shard-0-only EmergencyRecoveryRetainedBlock = 92,730,034. Consequently, the correct shard-1 command (--revert.to 94978279, retaining 94,978,278) is rejected before rollback.

The fix should use one fail-closed recovery manifest selected by (network, shardID) and containing the retained height, expected retained hash/state root, and any shard-specific postcondition material. Reading a hash/root from the same local DB and comparing it back to itself would not independently attest the retained branch.

Release-material status currently visible in this PR:

• shard 0 retained hash is pinned, but its retained state root is still a placeholder;
• shard 1 rejected-child hash is pinned, but its retained hash/state root are not yet pinned;
• the validator-list count/SHA-256 manifest is also still unset.

Until those values are supplied and the target selection is made shard-aware, shard 0 and shard 1 must not be documented as equally supported by the recovery binary.

[Frozen]

Blocking recovery issues found. This PR is authored by the current GitHub account, so this is submitted as a COMMENT review rather than REQUEST_CHANGES; treat the inline findings as release blockers.

[Frozen]

Confirmed retained state root for the shard-0 recovery checkpoint. The value to use for block 92,730,034 (0x30c35d2f2291e4b27debe7862956cf7a0cc7abefc044273d6823567335086d8d) is:

0x39e72dc20835abe61f69966bec2cc4766bb9e893c4168e117154dd539f2fc728

This exact (height, block hash, state root) tuple was verified against three local chain sources. This is the shard-0 state root, not the shard-1 checkpoint.

[Frozen]

Release constants are now pinned on PR head fac6b1cdf5cc30be9293d904200fabfcc483ee5e.

Shard-0 retained checkpoint:

• height: 92,730,034
• block hash: 0x30c35d2f2291e4b27debe7862956cf7a0cc7abefc044273d6823567335086d8d
• state root: 0x39e72dc20835abe61f69966bec2cc4766bb9e893c4168e117154dd539f2fc728

Validator-list manifest at that exact retained state:

• ordered address count: 771
• RLP byte length: 16,194
• SHA-256: 0xf5dc6b4879ed956818c19d7e68b41044be251284d37b09735d896cc3d657050d

The manifest was independently reproduced from two read-only shard-0 database copies. Both copies matched the pinned block hash/state root and produced stored=771, filtered=771, removed=0, and the same RLP digest.

The prior state-root commit passed all PR checks. The new manifest head is awaiting its own CI run. The staged-sync stale-body shortcut and rollback no-progress loop remain separate release blockers and are not claimed fixed by these constant updates.

[greptile-apps]

Want your agent to iterate on Greptile's feedback? Try greploops.

[Frozen]

Update: fixed the recovery rollback failure mode.

• Rollback progress / missing parent (b3e5c0b50): recovery rollback now preflights the complete ancestry before the first mutation, then verifies after every chain.Rollback() that the head moved exactly one block to the expected parent. Missing parents, no-progress rollbacks, nil heads, and unexpected head transitions fail closed instead of looping. Commit certificates are written only after the verified transition.

The staged-sync changes were removed from this PR. Recovery will start from the prepared snapshot with sync disabled, so api/service/synchronize/stagedstreamsync/stage_states.go is outside this rollout's scope and has no net diff in PR #13.

Regression coverage includes incomplete ancestry, no-progress rollback, and successful multi-block rollback.

[Test]

go test -count=1 -mod=readonly \
  ./api/service/synchronize/stagedstreamsync \
  ./cmd/harmony \
  ./consensus

All three packages pass at the rewritten PR head.

[Frozen]

Follow-up high-assurance review found two additional rollback durability gaps. They are fixed in 524017888dbff0cfbe0d45b432cc0b27bc6c3ba6:

• ancestry preflight now must terminate at the exact release-pinned target hash, not merely the target height, before any mutation;
• the target commit certificate is cryptographically verified, written, and read back before the first head rewind;
• intermediate per-height certificate writes were removed from the rollback loop;
• BlockChainImpl.Rollback now propagates fast/full persisted-head write errors and changes each corresponding in-memory head only after its persistent write succeeds, making interrupted retries restartable;
• the startup checkpoint gate now reads back LastHeader, LastFast, and LastBlock and requires all three persisted hashes to match the in-memory current head.

Added regressions cover wrong-target ancestry, missing ancestry, rollback no-progress, certificate-write failure before head movement, successful exact-target rollback, and stale persisted-head keys.

Fresh local validation on this exact commit:

• go test -count=1 -mod=readonly ./consensus ./cmd/harmony ./core — pass
• go vet -mod=readonly ./consensus ./cmd/harmony ./core — pass
• git diff --check — pass

Repository-wide native verification also completed successfully on this exact commit: make go-test (go test -vet=all -race ./...) — pass. Same-head GitHub CI is now fully green: amd64/arm64 builds, unit tests, RPC integration tests, pyhmy tests, and Greptile all passed for 524017888dbff0cfbe0d45b432cc0b27bc6c3ba6.

The staged-sync stale-body issue remains intentionally outside the currently isolated shard-0 rollout and is still a blocker before staged sync is re-enabled on an unpurged database.