Skip to content

feat(node): expose foreign-chain RPC provider health metric - #3849

Draft
haiyuechen-nearone wants to merge 1 commit into
3764-node-startup-health-checkfrom
3764-foreign-chain-health-metric
Draft

feat(node): expose foreign-chain RPC provider health metric#3849
haiyuechen-nearone wants to merge 1 commit into
3764-node-startup-health-checkfrom
3764-foreign-chain-health-metric

Conversation

@haiyuechen-nearone

@haiyuechen-nearone haiyuechen-nearone commented Jul 17, 2026

Copy link
Copy Markdown
Contributor

Part of #3764. Stacked on #3848, merge bottom-up:

  1. feat(node): probe foreign-chain RPC providers on startup #3848 — startup probe + per-provider logs
  2. feat(node): expose foreign-chain RPC provider health metric #3849 (this PR) — per-chain configured / healthy Prometheus gauges
  3. feat(node): surface healthy provider counts on /debug/node_config #3850 — healthy counts on /debug/node_config

Publishes two per-chain gauges from the startup probe: mpc_foreign_chain_rpc_providers_configured (set up front for every configured chain) and mpc_foreign_chain_rpc_providers_healthy (providers that passed the identity probe). A configured chain with no identity reads as 0/N.

Adds ForeignChain::label() as the single source of the snake_case chain label shared by config keys, metric labels, and health-check result keys.

@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from c2645f8 to 420b59d Compare July 17, 2026 08:03
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch 2 times, most recently from e9e4b81 to 4eeed22 Compare July 17, 2026 08:23
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch 2 times, most recently from 56ea9d2 to 0e54756 Compare July 17, 2026 08:35
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 4eeed22 to 8cdfbfa Compare July 17, 2026 08:35
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from 0e54756 to e1eb9d4 Compare July 20, 2026 09:57
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 36e439e to 7df70d0 Compare July 20, 2026 11:02
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from e1eb9d4 to d1cf5a8 Compare July 20, 2026 11:05
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 7df70d0 to f2aadc2 Compare July 20, 2026 11:19
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from d1cf5a8 to 94b2bab Compare July 20, 2026 11:19
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from f2aadc2 to a560733 Compare July 20, 2026 16:28
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch 2 times, most recently from 318ecc3 to 1b3e1e6 Compare July 20, 2026 16:47
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch 2 times, most recently from 03ef050 to c42c075 Compare July 20, 2026 16:55
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from 1b3e1e6 to 214d78d Compare July 20, 2026 16:55
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from c42c075 to a2805eb Compare July 20, 2026 17:09
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from 214d78d to a30098d Compare July 20, 2026 17:09
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch 2 times, most recently from 30c77ea to 208c837 Compare July 21, 2026 09:24
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from a30098d to e3328e2 Compare July 21, 2026 09:27
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 208c837 to 7958f4b Compare July 21, 2026 11:30
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch 3 times, most recently from 287686c to 938f549 Compare July 22, 2026 13:18
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch 4 times, most recently from a0bb5de to a327d7a Compare July 22, 2026 14:46
@haiyuechen-nearone
haiyuechen-nearone marked this pull request as ready for review July 22, 2026 14:51
@claude

claude Bot commented Jul 22, 2026

Copy link
Copy Markdown

Pull request overview

Publishes two per-chain Prometheus gauges from the foreign-chain startup health check — mpc_foreign_chain_rpc_providers_configured (N) and mpc_foreign_chain_rpc_providers_healthy (n) — so operators can dashboard "n/N providers healthy". Introduces ForeignChain::label() as a single source of truth for the snake_case chain identifier used by both metric labels and result keys, replacing the duplicated string literals in the health-check crate. The configured gauge is set unconditionally before probing so it survives skipped/undetermined-network paths; healthy is only written when the probe actually runs.

Changes:

  • Add ForeignChain::label() (exhaustive match) and share it across metric labels and health-check result keys.
  • Refactor check_all_providers_with_golden and every run_* helper to take ForeignChain and derive the label internally, removing the parallel "…" string literals.
  • Publish FOREIGN_CHAIN_RPC_PROVIDERS_CONFIGURED before the probe route is decided and FOREIGN_CHAIN_RPC_PROVIDERS_HEALTHY after log_results.
  • New tests cover: every configured chain keyed by its label, configured populated on the undetermined-network path, and per-chain healthy counting across mixed pass/fail/skip outcomes.
  • Extend docs/design/node-operator-metrics.md with the two new gauges.

Reviewed changes

Per-file summary
File Description
crates/near-mpc-contract-interface/src/types/foreign_chain.rs Adds ForeignChain::label() with an exhaustive match covering all 13 variants.
crates/foreign-chain-health-check/src/lib.rs Switches run_evm/run_bitcoin/… to accept ForeignChain and call .label() internally; adds a test asserting every configured chain shows up in results keyed by its label.
crates/foreign-chain-health-check/Cargo.toml Adds near-mpc-contract-interface dependency (also reflected in Cargo.lock).
crates/node/src/foreign_chain_health.rs Sets CONFIGURED gauge upfront and adds set_healthy_metric after log_results; two new tests.
crates/node/src/metrics.rs Registers FOREIGN_CHAIN_RPC_PROVIDERS_CONFIGURED and FOREIGN_CHAIN_RPC_PROVIDERS_HEALTHY IntGaugeVecs labeled by chain.
docs/design/node-operator-metrics.md New "Foreign-chain RPC provider health" table describing the two gauges and how to interpret n < N.

Findings

No blocking issues.

Non-blocking (suggestions/follow-ups):

  • crates/node/src/foreign_chain_health.rs:75set_healthy_metric is only reached when the probe actually ran, so on HealthCheckRoute::Skip(...) paths (local chain without golden, undetermined network) the configured gauge is published but healthy is not. A dashboard computing healthy / configured will show "no data" for healthy on those startups even though the intent is clearly "0 confirmed healthy". Consider seeding HEALTHY to 0 per configured chain in the same upfront loop that sets CONFIGURED (line 26), so both series always exist together. The current asymmetry is intentional per the PR description, but it makes alerting slightly awkward.
  • crates/near-mpc-contract-interface/src/types/foreign_chain.rs:1292ForeignChain::Ton is included in label() but there is no corresponding field in ForeignChainsConfig, so it never surfaces via iter_chains() and the ton metric label is currently unreachable. Not wrong — it's future-proofing so a Ton config field cannot land without a label — just worth mentioning so the exhaustive match doesn't rot silently.
  • crates/foreign-chain-health-check/src/lib.rs:398 — the new test iterates via fc.iter_chains() and checks results.iter().any(|r| r.chain == label); because check_all_providers already emits a placeholder row for every known chain regardless of configuration, this assertion passes even if a configured chain is silently dropped from the fan-out (its label would still appear via the not_configured/all_skipped placeholder). If the intent is "each configured chain contributes a probed row", asserting on results.iter().any(|r| r.chain == label && r.provider == "only") would catch a regression more precisely.
  • crates/near-mpc-contract-interface/src/types/foreign_chain.rs:1277-1279 — the doc comment on label() is slightly explanatory beyond what a caller needs ("Exhaustive on purpose to avoid drift" is an implementation-detail rationale). Not a bug, and public-API doc comments have latitude per engineering-standards.md, but the second sentence could be trimmed.

✅ Approved

@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch 4 times, most recently from 996daa8 to e7a8755 Compare July 22, 2026 17:47
@haiyuechen-nearone

Copy link
Copy Markdown
Contributor Author

@claude

Publish two per-chain Prometheus gauges from the startup probe:
`mpc_foreign_chain_rpc_providers_configured` (set up front for every configured
chain) and `mpc_foreign_chain_rpc_providers_healthy` (providers that passed the
identity probe). A configured chain with no identity reads as 0/N.

Adds `ForeignChain::label()` as the single source of the snake_case chain label
shared by config keys, metric labels, and health-check result keys.
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-node-startup-health-check branch from 315514c to 748639d Compare July 30, 2026 19:23
@haiyuechen-nearone
haiyuechen-nearone force-pushed the 3764-foreign-chain-health-metric branch from 3798e3d to 61a7d6f Compare July 30, 2026 19:23
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant