[Brindrajsinh-Chauhan]

This adds a metrics to measure the missed rounds with labels including the proposer height and the round missed. This is helpful to create alerting rules for missed rounds and identify validators that might be having issues.

[ZhiyuCircle]

@Brindrajsinh-Chauhan thanks for contribution, could you fix the rust formatting/lint

[Brindrajsinh-Chauhan]

@ZhiyuCircle Would be grateful to know the next step to get this in or will this be first merged in internal repos and then reverse synced to upstream

[romac]

Given that height grows forever and round can spike during outages, this creates one time series per missed height/round/proposer for the process lifetime, which can grow unbounded very quickly if eg. a validator is down for an extended period of time.

We have actually seen such an issue on testnet with another metric which was recording outbound IP address with their ephemeral ports, causing problems when ingesting those metrics into Prometheus/Datadog.

I am not sure what the best solution here is, since I agree that having the height and round is useful information.

---

Curious to hear what you think!

In the meantime, I'll see if I can find a good solution for this.

[Brindrajsinh-Chauhan]

Thank you for the feedback, I will also think on this to find a solution for this

[Brindrajsinh-Chauhan]

I think at the moment, I should drop the height and round from the metric and just have missed proposer since that can be a good alerting metrics if there is a increase in missed round by a proposer. The warn log you suggested below could then be used to investigate the height and rounds the proposer missed. So alerting like

increase(consensus_rounds_missed_total{proposer="0xABC"}[5m]) > 0

could work. Would be grateful to get your thoughts on this. I know removing the labels is not the best possible options but gets us the important thing of being able alert / detect issues causing the proposer to miss multiple rounds

[romac]

Agreed, sounds like a reasonable middle ground to me.

[ZhiyuCircle]

@ZhiyuCircle Would be grateful to know the next step to get this in or will this be first merged in internal repos and then reverse synced to upstream

@Brindrajsinh-Chauhan after second pass review, we have few concerns like @romac mention above

[Brindrajsinh-Chauhan]

@romac @ZhiyuCircle Looking forward on getting this merged

[osr21]

Chiming in as an external builder on Arc Testnet — this metric would have been directly useful for a problem we hit recently.

Operational context: early signal before a full sync stall

In #214 we documented a sync wedge where a node stopped advancing entirely. Looking back at that incident, the pattern was a rapid spike in missed rounds at a specific height before the node fully stalled. A consensus_round_missed alert on the affected proposer would have fired well before the node fell out of sync — potentially early enough to restart before missing blocks rather than after. This is a more valuable alert than monitoring fell_behind_count alone, which fires late (after the stall is complete).

On the alerting rule threshold

The increase(consensus_round_missed_total{proposer="0xABC"}[5m]) > 0 rule will be noisy in practice. Tendermint-based consensus has normal single-round misses — a proposer offline for one slot, a brief network partition, a slow block. Alerting on any miss at all will produce false positives on a healthy testnet. A more actionable rule:

# Alert if a proposer misses more than 3 rounds in any 5-minute window
increase(arc_malachite_app_consensus_round_missed_total{proposer="0xABC"}[5m]) > 3

or, if you want to catch a sustained degradation rather than a spike:

# Rolling miss rate > 10% of expected rounds
rate(arc_malachite_app_consensus_round_missed_total[10m])
  / rate(arc_malachite_app_height_restart_count[10m] + ...)  > 0.1

The right threshold depends on your expected block time and validator count, but > 0 is too sensitive for production alerting.

Potential follow-up: paired blocks_proposed_total counter

The proposer-only label is the right call for the cardinality reason @romac identified. One thing this metric can't do on its own is express a validator efficiency ratio (proposed / (proposed + missed)), which is the most actionable per-validator health signal. A paired arc_malachite_app_blocks_proposed_total{proposer} counter — incremented when a validator's proposal is committed — would enable that without changing this metric's label set. As far as I can tell from the METRICS.md, that counter doesn't exist yet. Happy to send a follow-up PR if that's useful.

The metric + warn log split is the right pattern: alerting without cardinality explosion, full detail available for investigation via logs.

[romac]

This is now available in Arc.

[osr21]

@romac good call, will inform youe of any changes