feat: sozu top btop-style operator TUI + cardinality-lease verb

.github/workflows/ci.yml

@@ -230,6 +230,51 @@ jobs:
           path: fuzz/artifacts
           retention-days: 14
 
+  # ┌──────────────────────────────────────────────────────────────┐
+  # │ Group C′ — cargo audit: surface RUSTSEC advisories on the    │
+  # │   whole workspace including the `tui` feature surface, which │
+  # │   pulls 12 optional crates (color-eyre, crossterm, ratatui,  │
+  # │   tui-input, tui-big-text, …) outside the default build's    │
+  # │   coverage. Treats every advisory as a warning; the job      │
+  # │   fails only on `--deny warnings` so a maintainer can react  │
+  # │   before the next release lands.                              │
+  # └──────────────────────────────────────────────────────────────┘
+  cargo-audit:
+    name: cargo audit
+    runs-on: ubuntu-latest
+    timeout-minutes: 10
+    steps:
+      - name: Install system dependencies
+        run: sudo apt-get update && sudo apt-get install -y protobuf-compiler
+
+      - name: Checkout sources
+        uses: actions/checkout@de0fac2e4500dabe0009e67214ff5f5447ce83dd  # v6
+
+      - name: Install rust 1.88.0
+        uses: dtolnay/rust-toolchain@1.88.0
+
+      - name: Cache cargo target
+        uses: Swatinem/rust-cache@e18b497796c12c097a38f9edb9d0641fb99eee32  # v2
+        with:
+          cache-all-crates: true
+          prefix-key: ci-audit
+          workspaces: . -> target/
+
+      - name: Install cargo-audit
+        run: cargo install cargo-audit --locked
+
+      - name: Run cargo audit (default features)
+        run: cargo audit --deny warnings
+
+      - name: Run cargo audit (all features incl. tui)
+        run: |
+          # `cargo audit` reads `Cargo.lock`, which already includes the
+          # `tui`-feature transitive deps since this PR landed them as
+          # workspace-locked entries. Re-running with the same lockfile
+          # produces a consistent advisory set; the duplicate invocation
+          # is kept symmetrical with the per-feature build matrix above.
+          cargo audit --deny warnings
+
   # ┌──────────────────────────────────────────────────────────────┐
   # │ Group D — Bombardier load benches (folded from benchmark.yml) │
   # └──────────────────────────────────────────────────────────────┘

CHANGELOG.md

@@ -69,6 +69,134 @@ upgrade. See `doc/upgrade/1.x-to-2.0.md` for the full migration guide.
   `*.com`) is intentionally not implemented — that is a browser-policy
   concern, not a reverse-proxy responsibility.
 
+- **`QueryMetrics` now folds single-worker fleets correctly**: the master's
+  `QueryMetricsTask::on_finish` (`bin/src/command/requests.rs`) used to gate
+  `AggregatedMetrics::merge_metrics()` on `server.workers.len() > 1`, leaving
+  one-worker setups with empty `clusters` / `proxying` maps and stranding all
+  per-worker data in `workers`. Every CLI/TUI consumer that reads the merged
+  shape (`sozu metrics get` without `--workers`, every `sozu top` pane) silently
+  showed zero data. The fix drops the `> 1` guard — `merge_metrics()` correctly
+  handles one worker (the relocation loop runs once) and zero workers (no-op).
+  Regression test `merge_relocates_single_worker_to_top_level` in
+  `command/src/proto/mod.rs` pins the contract.
+- **`sozu top` BACKENDS bandwidth column shows a per-second rate in Mbps**:
+  the column was previously a cumulative byte count since worker start —
+  `1.4G/5.6G` after a few hours, monotonically increasing, almost never
+  what an operator needs while monitoring live traffic. Mirror the
+  cluster-RPS pattern: the shared `RateCalculator` derives per-second
+  deltas from the cumulative `BYTES_IN`/`OUT` counters under
+  `__backend.<cluster>.<backend>.<bytes_in|bytes_out>` keys, the App
+  caches the rates on `backend_rate_in_bps` / `backend_rate_out_bps`,
+  and the renderer formats `bytes/sec × 8` as Kbps / Mbps / Gbps
+  base-1000 to match the networking convention (`nload`, `iftop`,
+  Grafana panels). Stale per-backend keys are pruned on every snapshot
+  alongside the cluster-RPS sweep so `RateCalculator.history` cannot
+  grow unbounded as a fleet churns. Column header changes from
+  `bw down/up B` to `bw down/up Mbps`; the BACKENDS-sort `Bandwidth`
+  comparator now uses the rate.
+- **`sozu top` CLUSTERS rps column auto-scales to K/M/G**: dashboards
+  with traffic over 1k req/s were truncating the `rps` cell at the
+  8-character width. The column is now 14 wide and the value renders
+  with a base-1000 suffix (`94.8K req/s`, `1.20M req/s`) so the cell
+  stays readable for any cluster scale.
+- **`sozu top` BACKENDS pane "bw down/up B" now populates**: the renderer
+  was reading `names::backend::BACK_BYTES_IN` / `BACK_BYTES_OUT` from the
+  per-backend filing, but those keys are emitted as no-label proxy
+  counters via `count!(key, value)` at every byte-flow site in
+  `lib/src/protocol/{pipe,kawa_h1/mod,mux/mod,proxy_protocol/{send,relay}}.rs`
+  — they land in `worker.proxy[…]` and are never tagged
+  `(cluster_id, backend_id)`. The per-backend cumulative bytes flow
+  through `record_backend_metrics!` (`lib/src/metrics/mod.rs`) at request
+  end under the keys `BYTES_IN` / `BYTES_OUT` with `$bin` / `$bout`
+  sourced from `SessionMetrics.backend_bin` / `backend_bout` — i.e.
+  the same backend-socket accumulators. The TUI now reads those.
+- **`sozu top` overview sparklines fill the full pane width**: the
+  default `RenderDirection::LeftToRight` left-anchored samples, which
+  made the graph appear to use only the left half of its cell until
+  the ring filled out to column-width. Switch to `RightToLeft` so the
+  newest sample sits on the right edge and history scrolls leftward
+  as new ticks arrive — the familiar `vmstat` / `htop` direction.
+- **`sozu top` F-key bar actually does things**: F1 Help and F10 Quit
+  were wired but F2-F9 were placeholders. F2 cycles the resolved
+  glyph mode (Block → Braille → Tty); F3 and F4 open the colon
+  palette so operators can type `:cluster <id>` etc.; F5 (also `p`)
+  pauses snapshot ingest without dropping the transport lease so the
+  TUI freezes on the last frame for closer inspection, and the label
+  flips to "Resume" while paused; F6 cycles the active pane's sort
+  column. F7/F8/F9 remain reserved slots (`·`) so the bar width
+  stays stable across builds.
+- **`sozu top` CLUSTERS column now shows req/s, not cumulative counter**:
+  the column header was `rps` but the cell rendered the cumulative
+  `names::backend::REQUESTS` count, which only changes when a backend
+  responds. Per-cluster `RateCalculator` keyed by `__cluster.<id>.requests`
+  derives the per-second delta once per ingest and the renderer prints
+  `<rps> req/s`. Stale cluster keys are pruned on each snapshot so the
+  rate history cannot grow unbounded as clusters churn.
+- **`sozu top` BACKENDS up/total now reads the cluster rollup gauge**:
+  the renderer was reading `names::backend::AVAILABLE` (a per-backend
+  gauge set on backend up/down transitions) at the cluster level, which
+  was always `0` because the gauge is keyed `(cluster, backend)` and
+  only fires on transitions. A new `names::cluster::AVAILABLE_BACKENDS`
+  constant captures the canonical per-cluster rollup (`available`
+  argument of `gauge!` in `lib/src/backends.rs::notify_availability`),
+  which is the authoritative aggregate and refreshes every health-check
+  tick. Per-backend `backend.available` (under backend-detail filing)
+  remains a fallback for the very first snapshot.
+- **`sozu top` OVERVIEW replaces 5xx panel with sozu service-time
+  p99**: the 5xx cell duplicated the LATENCY p99 cell's intent (both
+  read worst-case quality). Operators want sozu's own request-processing
+  time distinct from backend latency, so the bottom-left OVERVIEW cell
+  now plots p99 of `names::event_loop::SERVICE_TIME` from the proxy
+  map. The critical-banner threshold reuses `latency_p99_critical_ms`
+  with the `SOZU SLOW` headline so a stalled event loop still trips
+  the alert overlay.
+- **`sozu top` CERTS pane renders addresses as `ip:port`**: the address
+  column was printing `{:?}` of the proto `SocketAddress` (`SocketAddress
+  { ip: IpAddress { … } }`), which is unreadable. The pane now goes
+  through the existing `From<SocketAddress> for SocketAddr` conversion
+  so v4 prints as `1.2.3.4:443` and v6 as `[::1]:443`.
+- **`sozu top` H2 pane trend column actually plots a trend**: the
+  trend cells rendered as `"—"` placeholders. Each H2-pane metric now
+  has a 60-sample `SparkRing` populated by `App::fold_h2_trends` per
+  ingest; the renderer prints Unicode bars (`▁▂▃▄▅▆▇█`) scaled to the
+  ring's max sample so even zero-flat series stay distinguishable from
+  the cold-start placeholder.
+- **`sozu top` OVERVIEW and CLUSTERS read counters under backend-detail
+  filing**: the TUI auto-leases `MetricDetail::Backend` on startup so the
+  BACKENDS pane has per-row data, which routes every per-cluster counter
+  (`requests`, `http.status.5xx`, `backend_response_time`) into the worker's
+  `clusters[<id>].backends[<id>].metrics` map rather than `clusters[<id>].cluster`.
+  The OVERVIEW and CLUSTERS panes were reading the cluster-level map
+  exclusively, so the REQUESTS / SEC big-numeral, 5xx ratio, latency p99, and
+  every per-cluster row read 0 the moment the lease elevated detail. New
+  helpers in `bin/src/ctl/top/app.rs` (`cluster_count_total`, `cluster_p99_max`,
+  `cluster_p50_max`) sum across both filings; magic-string keys are replaced
+  with `sozu_lib::metrics::names::*` constants so a future name rename cannot
+  silently zero the TUI again. The `proxying` fallback now uses
+  `names::http::REQUESTS` (`http.requests`, incremented at request-receive
+  time) so traffic surfaces even before the first backend round-trip
+  completes.
+- **`SetMetricDetail` no longer flaps systemd `RELOADING=1` every TTL/2**: the
+  cardinality-lease verb was in the `is_mutating_verb` allowlist
+  (`bin/src/command/requests.rs`), which brackets each call with
+  `sd_notify(STATE_RELOADING)` / `READY=1`. The TUI auto-renews its lease every
+  `ttl/2` seconds (≈ 30 s by default) to keep the lease alive, so a long-lived
+  `sozu top` session caused the unit to flap `reloading`/`active` every 30 s in
+  `systemctl status` and external monitors. `SetMetricDetail` is now excluded:
+  it does not change cluster/listener/cert state or fleet topology, so it does
+  not belong in the systemd bracket set. The audit-log trail
+  (`EventKind::METRIC_DETAIL_CHANGED`, proto tag 30) still records every apply,
+  clear, and TTL expiry, so SOC visibility is unaffected. Regression test
+  `set_metric_detail_is_not_mutating` in `bin/src/command/requests.rs` pins
+  membership.
+- **Cross-worker `Percentiles` aggregation no longer silently drops on
+  `merge_metrics()`**: `command/src/proto/mod.rs::is_mergeable` returned `false`
+  for `Inner::Percentiles`, so any cluster metric emitted in percentile shape
+  (notably `backend_response_time`) was discarded when more than one worker
+  contributed. The merge now propagates the element-wise max per quantile and
+  accumulates `samples` + `sum`, preserving the "worst observed quantile"
+  upper bound across workers. The companion `<name>_histogram` value remains
+  the source of truth for statistically accurate aggregation.
 - **Response-side header edits no longer re-injected on every prepare
   cycle (`H2BlockConverter::finalize` "out buffer not empty" leak)**:
   `apply_response_header_edits` was called inconditionally before
@@ -346,6 +474,53 @@ upgrade. See `doc/upgrade/1.x-to-2.0.md` for the full migration guide.
 
 ### 🌟 Added
 
+- **`sozu top` btop/htop-style live operator TUI**: a new clap subcommand
+  gated by the optional `tui` Cargo feature on `bin/`
+  (default builds remain lean — `sozu --version` reports `+tui` when the
+  subcommand is linked in). Surfaces metrics that Sōzu already emits in
+  six panes — OVERVIEW (sparklines + big numerals), CLUSTERS / BACKENDS
+  (sortable tables), LISTENERS (5 s ListListeners poll), H2 (streams +
+  flow control + CVE flood mitigations), CERTS (15 s certificates
+  poll), EVENTS (colour-coded SubscribeEvents tail). Built on
+  `ratatui = "0.30"` + `crossterm = "0.29"` over four synchronous
+  transport threads (snapshots, listeners, certs, events — no `tokio`
+  runtime in v1 by design); render loop is capped at 30 fps with
+  DEC-mode-2026 synchronized output for tmux-free flicker. The TUI auto-elevates
+  metric cardinality via a `SetMetricDetail` TTL lease (`client_id`-
+  keyed; default 60 s TTL, server clamp 300 s; auto-renewed at half-TTL;
+  self-expires server-side on crash). A 4-tick pulse tint surfaces
+  cluster appearance / disappearance / backend-went-down transitions
+  visually. A `tui-big-text` alert banner overlays the active pane when
+  the threshold table fires (p99 > 500 ms / 5xx > 1 % / saturation >
+  80 %). `--skin <name>` (with `SOZU_TOP_SKIN` env override) resolves
+  TOML skins under `$XDG_CONFIG_HOME/sozu/skins/`. Default Okabe-Ito
+  colour-blind safe categorical + Viridis-shaped continuous ramp;
+  trend glyphs `▲ ▼ ●` back up the colour cue. Three glyph modes
+  (Braille / Block / TTY-ASCII) auto-detect against `TERM` / `LANG`.
+  Full operator guide: [`doc/sozu-top.md`](doc/sozu-top.md).
+
+- **`SetMetricDetail` runtime cardinality verb**
+  (`command/src/command.proto:55`, `ResponseContent::MetricDetailStatus =
+  16`, `EventKind::METRIC_DETAIL_CHANGED = 30`): operators (and any TUI
+  client) can elevate the worker's `MetricDetail` floor at runtime via
+  a TTL-bounded lease. Multiple clients lease independently — the
+  effective level is `max(configured, max(active leases))`. Leases
+  self-expire server-side after `ttl_seconds` so a crashed client
+  cannot permanently elevate cardinality. Workers that pre-date the
+  verb return the standard `unknown request type` error, which surfaces
+  in the normal fan-out error tally on `MetricDetailStatus`; production
+  deployments keep master + workers in sync via the `UpgradeMain`
+  hot-upgrade flow, so this mixed-version state is transient. Full
+  semantics in `command.proto`'s `SetMetricDetail` doc comment. Audit-scope: every
+  cardinality transition emits `EventKind::METRIC_DETAIL_CHANGED` —
+  operator-initiated transitions are logged by the master at the
+  dispatch site, AND the worker now emits the same event for janitor-
+  driven lease expiries and post-fan-out worker-arm apply/clear via
+  the new worker→master audit IPC, so the audit log carries the full
+  history regardless of origin. Lease ownership is bound to the
+  connecting peer's PID + session ULID (`SO_PEERCRED`-derived);
+  cross-operator `clear` requests are refused at the worker.
+
 - **Pre-built binaries for tagged releases
   ([#1089](https://github.com/sozu-proxy/sozu/issues/1089))**: a new
   `.github/workflows/release.yml` triggers on tag push (`X.Y.Z` and