Skip to content

fix(continuous-sync): require evidence before halting stalled nodes - #846

Open
evan-forbes wants to merge 6 commits into
mainfrom
fix/evidence-based-sync-stall
Open

fix(continuous-sync): require evidence before halting stalled nodes#846
evan-forbes wants to merge 6 commits into
mainfrom
fix/evidence-based-sync-stall

Conversation

@evan-forbes

@evan-forbes evan-forbes commented Aug 30, 2026

Copy link
Copy Markdown
Contributor

Motivation

The continuous-sync controller treated every unchanged block height as a
stalled node. That rule stopped the v2-only canary during a valid 667-second
Mainnet block interval. The controller sent SIGTERM five seconds before the
next block existed.

The wall-clock sync estimate did not prove that the network had advanced. Five
healthy peers advertised the same exact tip as the canary.

The controller also lacked a distinct policy for active legacy fallback. Once
fallback owns block sync, the Zakura header height stops tracking the committed
chain, so neither height is valid backlog evidence and the run reported missing
status instead of naming the handoff.

Solution

  • Read the exact best committed block height and exact local header-chain height
    from existing metrics.
  • Exclude finalized-only, verifier-only, and wall-clock height gauges from stall
    evidence.
  • Classify a header height at or below the committed height as no local header
    backlog.
  • Start the stall deadline only while the local header height remains ahead of
    the committed height.
  • Restart the backlog deadline when committed blocks advance.
  • Pause an active backlog deadline during a metrics error instead of erasing
    the prior evidence.
  • Give metrics errors and missing exact heights a separate
    status-unavailable deadline and failure reason.
  • Stop a dual-stack run on the first sample where the existing
    sync.zakura.legacy_fallback.active gauge reports active fallback, naming
    the handoff as the failure reason.
  • Keep the legacy-only canary on its existing 1800-second height-only policy.
  • Keep the dual-stack stall deadline at the fleet default of 600 seconds.
  • Require observed exact committed progress from both nodes before the cluster
    monitor opens or clears a sync-stall alert.

The local getblockchaininfo RPC exposes the same committed and header heights.
Enabling it would add a listener and authentication configuration without
adding independent evidence. This change uses the metrics that the controller
already queries.

A local query cannot prove a header-sync failure when the node never learns a
newer header. The cluster monitor retains that responsibility. It reports a
local stall only after another healthy canary shows an exact committed-height
increase while the local canary remains stationary.

Failure Semantics

The controller records one of these evidence states in each sample:

  • no_local_header_backlog
  • local_header_backlog
  • legacy_fallback
  • unknown
  • legacy_height_only

An individual query error or timeout records unknown evidence and does not stop
the node. The controller pauses any prior backlog deadline during that
unavailable interval. Continuous unknown evidence stops the node after
status_unavailable_seconds with an observability failure instead of a
sync-stall failure.

Testing

  • python3 -m unittest discover -s deploy/continuous-sync/tests -p 'test_*.py'
  • python3 -m py_compile deploy/continuous-sync/continuous-sync.py deploy/continuous-sync/alert-status.py deploy/continuous-sync/alert-monitor.py deploy/continuous-sync/deploy.py
  • git diff --check
  • ./scripts/changelog.py check-pr --base origin/main --head HEAD --pr 846

The 66 tests cover natural block gaps, persistent local backlog, active legacy
fallback, a committed height above the selected header height, backlog after a
long idle period, committed progress, unavailable-status deadlines, backlog
pause and resume, non-authoritative height rejection, exact cluster peer
evidence, exact stall recovery, legacy behavior, and rendered per-node policy.

Rollout

  1. Archive the existing failed run artifacts before any restart.
  2. Deploy with --no-start.
  3. Resume the v2-only canary and inspect samples.jsonl through a long tip
    interval.
  4. Deploy the auxiliary-admission build from fix(header-chain): preserve auxiliary capacity and diversity #845 to the dual-stack canary.
  5. Resume the dual-stack canary and verify that a fallback handoff stops the
    run and names itself as the reason.

Why fallback halts rather than waits

#732 proposed the opposite policy and was rejected on review: catching a v2
stall before legacy takes over is the purpose of this canary, and allowing the
handoff to complete masks the failure it exists to find. The 600-second
controller deadline therefore matches the node's own fallback threshold by
design, and this PR no longer overrides it.

PR #732 proposed giving the dual-stack canary room to complete its legacy
fallback handoff. The review rejected it: catching a v2 stall before legacy
takes over is the point of this canary, and waiting out the handoff masks the
failure it exists to find.

This branch had reintroduced that allowance twice: a 1800-second stall_seconds
override on temp-zakura-sync-test-1, and a legacy_fallback evidence state that
tracked committed progress once fallback owned block sync. Legacy advances after
a handoff, so the second one reports a healthy node while the v2 stack stays
stalled.

Both are removed. The stall deadline returns to the fleet default of 600
seconds, deliberately matching the node's own fallback threshold, and an active
fallback now stops the run on the sample that observes it. The evidence state
survives as the failure reason, so the run says the handoff happened rather
than reporting missing header evidence.

The evidence-based classification that motivated this PR is unchanged: a
natural Mainnet block gap still does not start the stall deadline.
@evan-forbes
evan-forbes marked this pull request as ready for review August 31, 2026 13:29
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant