fix(continuous-sync): require evidence before halting stalled nodes - #846
Open
evan-forbes wants to merge 6 commits into
Open
fix(continuous-sync): require evidence before halting stalled nodes#846evan-forbes wants to merge 6 commits into
evan-forbes wants to merge 6 commits into
Conversation
PR #732 proposed giving the dual-stack canary room to complete its legacy fallback handoff. The review rejected it: catching a v2 stall before legacy takes over is the point of this canary, and waiting out the handoff masks the failure it exists to find. This branch had reintroduced that allowance twice: a 1800-second stall_seconds override on temp-zakura-sync-test-1, and a legacy_fallback evidence state that tracked committed progress once fallback owned block sync. Legacy advances after a handoff, so the second one reports a healthy node while the v2 stack stays stalled. Both are removed. The stall deadline returns to the fleet default of 600 seconds, deliberately matching the node's own fallback threshold, and an active fallback now stops the run on the sample that observes it. The evidence state survives as the failure reason, so the run says the handoff happened rather than reporting missing header evidence. The evidence-based classification that motivated this PR is unchanged: a natural Mainnet block gap still does not start the stall deadline.
evan-forbes
marked this pull request as ready for review
August 31, 2026 13:29
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Motivation
The continuous-sync controller treated every unchanged block height as a
stalled node. That rule stopped the v2-only canary during a valid 667-second
Mainnet block interval. The controller sent SIGTERM five seconds before the
next block existed.
The wall-clock sync estimate did not prove that the network had advanced. Five
healthy peers advertised the same exact tip as the canary.
The controller also lacked a distinct policy for active legacy fallback. Once
fallback owns block sync, the Zakura header height stops tracking the committed
chain, so neither height is valid backlog evidence and the run reported missing
status instead of naming the handoff.
Solution
from existing metrics.
evidence.
backlog.
the committed height.
the prior evidence.
status-unavailable deadline and failure reason.
sync.zakura.legacy_fallback.activegauge reports active fallback, namingthe handoff as the failure reason.
monitor opens or clears a sync-stall alert.
The local
getblockchaininfoRPC exposes the same committed and header heights.Enabling it would add a listener and authentication configuration without
adding independent evidence. This change uses the metrics that the controller
already queries.
A local query cannot prove a header-sync failure when the node never learns a
newer header. The cluster monitor retains that responsibility. It reports a
local stall only after another healthy canary shows an exact committed-height
increase while the local canary remains stationary.
Failure Semantics
The controller records one of these evidence states in each sample:
no_local_header_backloglocal_header_backloglegacy_fallbackunknownlegacy_height_onlyAn individual query error or timeout records unknown evidence and does not stop
the node. The controller pauses any prior backlog deadline during that
unavailable interval. Continuous unknown evidence stops the node after
status_unavailable_secondswith an observability failure instead of async-stall failure.
Testing
python3 -m unittest discover -s deploy/continuous-sync/tests -p 'test_*.py'python3 -m py_compile deploy/continuous-sync/continuous-sync.py deploy/continuous-sync/alert-status.py deploy/continuous-sync/alert-monitor.py deploy/continuous-sync/deploy.pygit diff --check./scripts/changelog.py check-pr --base origin/main --head HEAD --pr 846The 66 tests cover natural block gaps, persistent local backlog, active legacy
fallback, a committed height above the selected header height, backlog after a
long idle period, committed progress, unavailable-status deadlines, backlog
pause and resume, non-authoritative height rejection, exact cluster peer
evidence, exact stall recovery, legacy behavior, and rendered per-node policy.
Rollout
--no-start.samples.jsonlthrough a long tipinterval.
run and names itself as the reason.
Why fallback halts rather than waits
#732 proposed the opposite policy and was rejected on review: catching a v2
stall before legacy takes over is the purpose of this canary, and allowing the
handoff to complete masks the failure it exists to find. The 600-second
controller deadline therefore matches the node's own fallback threshold by
design, and this PR no longer overrides it.