Add a metric for settlement fee and surplus calculation failures

[squadgazzz]

Description

Fee and surplus math failures in the settlement observer only show up as warn logs. During the xdai incident (#4669) the observer persisted zeroed fees for two days before the solvers team spotted them in the DB, nobody was alerted.

We want an alert. Alerting off the warn line means matching its exact text, either through a Vector log_to_metric transform or a vmalert LogsQL rule. That coupling is fragile: reword the message and the metric silently stops counting while the alert keeps evaluating a flat zero, exactly the failure mode an alert must not have. A counter incremented next to the calculation cannot drift out of sync with the code.

The counter is also published at zero when the observer starts. A counter series does not exist in Prometheus until its first increment, so a series born with the first failure already reads one, increase() has nothing to subtract from, and the alert would sit quiet until a second failure arrived.

Changes

• settlement_math_errors counter, labeled by calculation kind (surplus, fee, fee_breakdown), incremented whenever summarize() falls back to zeroed values
• all three series are published at zero when the settlement observer is constructed

How to test

Existing tests. The alert rule on the new metric follows in the infrastructure repo:

sum by (network, kind) (increase(gp_v2_autopilot_domain_settlement_math_errors{cow_fi_environment="prod"}[6h])) > 0

[claude]

Claude finished @squadgazzz's task in 1m 11s —— View job

---

PR Review: Settlement math error metric

• Read the diff and surrounding context
• Verify metric registration & label conventions
• Check for correctness/cardinality issues
• Post feedback

Verdict: Looks good. Clean, minimal change that follows the existing metric conventions in this crate. The counter is correctly placed inside each unwrap_or_else fallback, labeled by kind (surplus / fee / fee_breakdown), and label cardinality is bounded to 3. The reasoning in the PR description — a counter next to the calc can't drift out of sync with the code the way a log-text-matching transform can — is sound.

One optional, non-blocking suggestion left inline on domain/mod.rs:28: consider pre-initializing the three label series to 0 at startup so the time series exist before the first failure — useful for verifying the alert wiring and getting a clean dashboard baseline. Existing counters here don't do this, so it's purely a nice-to-have given the alerting motivation.