[kyoto] expose chain_tip and peer_info, add peer-flag metrics#490
Closed
0xsiddharthks wants to merge 1 commit intosiddharth/kyoto-supervisor-calibrationfrom
Closed
[kyoto] expose chain_tip and peer_info, add peer-flag metrics#4900xsiddharthks wants to merge 1 commit intosiddharth/kyoto-supervisor-calibrationfrom
0xsiddharthks wants to merge 1 commit intosiddharth/kyoto-supervisor-calibrationfrom
Conversation
Add three diagnostic surfaces using APIs that landed in bip157 v0.5.0:
- MonitorClient::chain_tip() -> synchronous query of kyoto's tip,
independent of the cached watch channel. Useful for healthz
endpoints that want to confirm the kyoto node is alive (a stale
cache cannot tell the difference between "no new blocks for a
while" and "kyoto is wedged").
- MonitorClient::peer_info() -> live snapshot of (AddrV2,
ServiceFlags) for every connected peer. Exposes which peers are
advertising NODE_COMPACT_FILTERS and NODE_P2P_V2 — needed to
diagnose "we're connected but filter sync isn't progressing"
cases.
- Two prometheus gauges populated by a 30s background poller:
* hashi_kyoto_peers_with_compact_filters
* hashi_kyoto_peers_v2
Both new client calls and the poller go through the existing
rpc_workers JoinSet and use a 5s timeout so a wedged kyoto node
cannot stall a worker forever; on timeout / rebuild-in-progress the
metrics keep their previous value until the next tick.
The poller arm uses MissedTickBehavior::Skip so a long-running select
arm doesn't trigger a burst of catch-up polls.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
MonitorClient::chain_tip()— synchronous query into kyoto's tip, independent of the cached watch channel. Useful for healthz endpoints that need to distinguish "no new blocks for a while" from "kyoto is wedged".MonitorClient::peer_info()— live snapshot of(AddrV2, ServiceFlags)for every connected peer. Lets us diagnose "we're connected but filter sync isn't progressing" cases without scraping logs.hashi_kyoto_peers_with_compact_filters— peers advertisingNODE_COMPACT_FILTERS. If this drops to 0 we cannot make filter-sync progress even when overall connection counts look healthy.hashi_kyoto_peers_v2— peers using BIP-324 (P2P V2) transport.rpc_workersJoinSet so a wedged kyoto node cannot stall a worker. On timeout / rebuild-in-progress the metrics keep their previous value until the next tick. The poller's interval usesMissedTickBehavior::Skipto avoid catch-up bursts.Test
cargo nextest run -p hashi(274 passed)make fmt && make clippyFollow-up
siddharth/kyoto-stale-tip-rebuild(next): useWarning::PotentialStaleTipas a third trigger for the connectivity-supervisor rebuild (today it only bumps a metric). Catches the "connected but stuck" case thatKYOTO_MAX_CONSECUTIVE_FAILURESdoes not.