feat: sozu top btop-style operator TUI + cardinality-lease verb

[FlorentinDUBOIS]

Summary

Adds a btop/htop-style live operator TUI behind a new tui Cargo feature on bin/. Seven panes (OVERVIEW, CLUSTERS, BACKENDS, LISTENERS, CERTS, H2, EVENTS) surface metrics Sōzu already emits over four synchronous transport threads, plus a colon command palette, F-key bar, TOML skin loader, and big-text alert banner. No tokio runtime in v1 by design.

Also introduces SetMetricDetail, a runtime proto verb that lets clients lease elevated metric cardinality. The lease is TTL-bounded (default 60 s, hard cap 300 s), client_id-keyed, bound to the connecting peer's SO_PEERCRED PID + session ULID so cross-operator clears are refused, length-capped on the wire (≤ 64-byte client_id, ≤ 256-byte reason), and self-expires server-side so a crashed sozu top cannot permanently elevate cardinality. The TUI auto-leases Backend detail on startup, then clears on exit. Every transition — operator-initiated AND worker-local janitor-driven — is audited via EventKind::METRIC_DETAIL_CHANGED. Workers that pre-date the verb return the standard unknown request type error which surfaces in the normal fan-out error tally; production keeps master + workers in sync via UpgradeMain so the mixed-version state is transient.

Review-pass fixes (2026-05-12)

Fourteen follow-up commits address findings from a multi-agent review pass (Claude + Codex + Lisa security threat-hunt + plan-vs-PR diff + guidelines + simplify).

Metrics-wiring root cause the operator reported (OVERVIEW pane stuck at RPS=0 and CLUSTERS pane at backends_up/total=0/0) traced to:

• bin/src/command/requests.rs::query_metrics skipped merge_metrics() whenever server.workers.len() <= 1. Single-worker fleets had every per-worker metric stranded in AggregatedMetrics.workers while the merged clusters / proxying maps stayed empty. Three companion fixes also landed: per-cluster counter reading under backend-detail filing (004a1965), the cluster.available_backends rollup gauge name constant (b9625b42), and the cluster RPS rate / backends rollup / service-time / cert addr / H2 trend wiring (f5e90156).

Ship-blockers (P1) closed:

• SIEM column-smuggling in the SetMetricDetail failure-reason aggregation. Worker error templates at lib/src/server.rs previously echoed the operator-supplied client_id verbatim into WorkerResponse::error strings, which the master joined into extras.reason and rendered through the weak sanitize_for_audit (control bytes only). A client_id like "x,actor_user=mallory" (legal under the 64-byte cap, unrestricted alphabet) could forge sibling KV pairs in the audit row. The fix drops the raw client_id echo from worker error templates and tightens both sanitize_for_audit and sanitize_for_audit_kv to also reject the full Unicode control category (NEL U+0085, CSI U+009B, all C1 controls), BOM (U+FEFF), and Unicode line/paragraph separators (U+2028, U+2029) — closing C1 control + Unicode line-break injection paths the prior C0-only predicate missed.

• Lease-renewal binding overwrite. The previous DetailGuard design opened a separate Channel for the renewer thread; the master stamps a fresh session_ulid on each ClientSession, so renewer writes overwrote the apply-time PeerBinding in the worker's lease table. After the first renewal (~30 s into a session), the apply channel's own clear-on-Drop failed as Unauthorized because the binding no longer matched; same-UID lease takeover became possible from another session with the same client_id. The refactor moves to a single-owner-thread topology: one thread owns the Channel and serialises Apply / Renew / Clear writes via crossbeam_channel::Sender<DetailRequest>. Worker always sees the same session_ulid for every write from a given DetailGuard; binding overwrite is structurally impossible from the legitimate path.

Other should-fix items (P2) closed:

• Worker lease_clear now enforces LEASE_CLIENT_ID_MAX_BYTES (defence-in-depth against direct-worker-IPC bypass).
• Events transport thread observes an Arc<AtomicBool> shutdown flag and a 1 s read timeout — channel leak closed; run_top now joins all four transport handles on exit.
• RawModeGuard::install inverted so a partial-install failure (e.g. EnterAlternateScreen errors after enable_raw_mode succeeded) cannot leak raw mode to the operator's shell.
• ctrlc/termination feature enabled so kill -TERM (and SIGHUP) restores the terminal cleanly; the previous default-feature setup only caught SIGINT despite the docs claiming SIGTERM coverage.
• Seven handle_key arms (Tab, BackTab, sort cycle / reverse, palette open, palette typing, help toggle, direct tab jump) now call app.mark_dirty() so the dirty-gated render loop redraws on input within one frame instead of waiting up to one snapshot tick (1 s default).
• Skin loader uses O_NOFOLLOW on the leaf open — closes the TOCTOU window between canonicalize() + starts_with(anchor) and File::open(&resolved) that a writable-skins-dir scenario could exploit via symlink swap.
• Background-thread spawn failures map to a new CtlError::SpawnFailed { label, source } and propagate cleanly; previously .expect("spawn …") panicked under RLIMIT_NPROC / transient OOM.
• Three eprintln! sites in transport.rs (write to a wiped alt-screen, operator never sees them) reuse the existing StatusSlot pattern so transport errors surface in the status bar.

Simplification (P2.4) accepted by the operator:

• The proto_version capability handshake is deleted entirely (WorkerInfo.proto_version, MetricDetailStatus.unsupported_workers[], sozu_command_lib::SOZU_PROTO_VERSION, WorkerSession.proto_version, MIN_PROTO_VERSION_FOR_SET_METRIC_DETAIL, and the capability-partition logic in set_metric_detail_request). Production deployments keep master + workers in sync via the existing UpgradeMain flow, so the mixed-version-fleet state the field was designed for does not occur in practice. The implementation was also structurally broken — the master stamped every worker's proto_version from its own compile-time constant rather than the worker's own. Workers without tag 55 surface as WorkerResponse::error("unknown request type") which folds into the standard fan-out error tally (extras.fanout.workers_err). The two proto tag slots are reserved 4; / reserved 5; per project convention.

Polish (P3):

• record_backend_metrics! macro emits via names::backend::* constants instead of raw literals (adds names::backend::CONNECTION_TIME for completeness; on-wire metric names unchanged).
• cluster_rows backends_available flattens the dead .max(0) arm into a single if/else.
• PanicHookGuard restores the prior panic hook on Drop; repeated render::run calls (tests, embedded callers) no longer stack hook layers indefinitely.
• SOZU_TOP_SYNC=0 env opt-out now actually wired at the BeginSynchronizedUpdate / EndSynchronizedUpdate call sites.
• bin/README.md gains a sozu top highlight; doc/configure_cli.md and doc/sozu-top.md scrubbed of stale --no-color / --log-file references; bin/tests/sozu_top_e2e.rs ignored test stripped of the same removed flag (it would have failed when run with --ignored); two doc/code-mismatch comments fixed (transport.rs "drop oldest" → "drop newest"; metrics/mod.rs "TTL clamped" → "TTL rejected with TtlOutOfRange").

Behaviour and surface

sozu top TUI (opt-in tui Cargo feature on bin/)

• Seven panes: OVERVIEW (sparklines + active sessions + RPS + backend-latency p99 + sozu service-time p99 + saturation), CLUSTERS (sortable table — default sort 5xx-rate desc; rps column auto-scales K/M/G), BACKENDS (flat per-backend rows; bw down/up rendered as Mbps with Kbps/Gbps auto-scale), LISTENERS (HTTP/HTTPS/TCP listener config), CERTS (per-address certificate summaries with ip:port rendering), H2 (active streams + flow-control gauges + CVE flood-mitigation counters with per-row 60-sample Unicode-bar trend), EVENTS (colour-coded SubscribeEvents tail).
• Colon command palette (:overview, :clusters, :b, :help, :quit, …) via tui-input, plus a numbered F-key bar: F1 Help, F2 Glyphs (cycle Block/Braille/Tty), F3/F4 open the palette, F5/p Pause (freezes the visible state without dropping the transport lease), F6 Sort (cycles the active pane's sort column), F10 Quit.
• --glyphs resolves into GlyphMode → fed into ratatui's Sparkline::bar_set AND the inline H2 trend renderer so Block / Braille / Tty alphabets land in both surfaces; resize events redraw via App::mark_dirty so the dirty-gate cannot swallow a post-resize repaint; OVERVIEW sparklines render RightToLeft so the newest sample anchors on the right edge and history scrolls leftward like vmstat / htop.
• TOML skin loader at $XDG_CONFIG_HOME/sozu/skins/<name>.toml with SOZU_TOP_SKIN env override (k9s parity). Loader is fail-closed: anchor canonicalize failure → default skin + status; outside-anchor escape → default + status; parse error → default + status. Leaf File::open uses O_NOFOLLOW so a symlink swap on the resolved path returns ELOOP instead of opening the wrong file.
• Render loop dirty-gates on take_dirty() || pulse.has_active() at a 30 fps cap; quiet-system CPU drops from ratatui buffer-diff cost to near zero. Every keypress that mutates visible state marks the app dirty so input redraws within one frame.
• Big-text alert banner at the top when the threshold table flags HIGH LATENCY / ERROR SURGE / SATURATION; pulse tint on transitions (cluster disappeared, backend down, new cluster).
• ctrlc::set_handler with the termination feature catches SIGINT, SIGTERM, and SIGHUP — kill -TERM <pid> no longer leaves the terminal in alt-screen + raw mode. Graceful degrade when a prior handler exists in the same process address space (multi-run_top callers). The crossterm event loop also observes Ctrl-C as a keypress, so the dedicated signal handler is belt-and-braces; failure surfaces on the status bar instead of aborting the binary.
• RawModeGuard::install constructs the guard immediately after enable_raw_mode() succeeds and progressively flips alt_entered / mouse_enabled flags as the follow-on execute! calls succeed, so a partial-install failure cleanly tears down whatever did succeed instead of leaking raw mode.
• PanicHookGuard restores the prior panic hook on Drop so repeated render::run invocations in the same process don't stack hook layers.
• SOZU_TOP_SYNC=0 opts the synchronised-output (DEC mode 2026) frame wrap out at the operator's discretion.
• Renewer thread errors land on a shared status slot the render loop drains once per tick, instead of eprintln! to the wiped alt-screen. The four transport threads (collector, listeners, certs, events) use the same status surface — no more eprintln! invisible to the operator.
• Events transport thread observes an Arc<AtomicBool> shutdown flag and a 1 s read timeout, so run_top can drop receivers and join all four transport handles cleanly (no leaked SubscribeEvents subscriptions; tests can re-enter run_top without leaking a Channel per call).
• Certs poll accepts both CertificatesByAddress and CertificatesWithFingerprints response variants (master returns the latter today for QueryCertificatesFromTheState); the conversion drops PEM + private-key fields immediately so key material never reaches the renderer or scrollback. Unexpected-variant errors print a discriminator name only — never the payload.

SetMetricDetail proto verb

• Tag 55 on Request. TTL-bounded lease keyed by client_id (the TUI uses top:<pid>:<8-hex>). ttl_seconds is capped at 300 s server-side; the master rejects out-of-range TTLs BEFORE fan-out so a buggy or malicious request can't N×amplify worker-side rejections + audit lines. The worker also enforces the cap on both apply AND clear branches (defence-in-depth).
• Lease ownership is bound at apply-time to the connecting peer's SO_PEERCRED PID + session ULID (peer_pid / peer_session_ulid fields, master-populated; clients leave them empty). Subsequent clear requests are authorised against the binding — cross-operator clears return FAILURE. The DetailGuard now uses a single owner thread that holds the one Channel for apply, renew, and clear; the worker always observes the same session_ulid for every write, so the binding-overwrite class of bug (which broke clear-on-Drop AND enabled same-UID takeover) is structurally impossible from the legitimate path.
• Server-side caps: LEASE_TABLE_CAP = 64 rejects new inserts (renewals of existing entries still succeed); LEASE_CLIENT_ID_MAX_BYTES = 64 rejects oversized lease keys on both apply AND clear. Master enforces the same caps before fan-out.
• Random suffix in client_id uses libc::getrandom(GRND_NONBLOCK) on Linux (no fs dependency); non-Linux Unix targets fall back to /dev/urandom; u32::from_le_bytes for cross-arch reproducibility.
• MetricDetailStatus reply (new ResponseContent variant tag 16) carries the master's (configured, effective, previous_effective) + per-worker WorkerMetricDetailStatus (tag 17 — each worker's own quartet, returned via ContentType::WorkerMetricDetailStatus in the worker's response payload). Workers that pre-date the verb return the standard WorkerResponse::error("unknown request type") which folds into the fan-out's existing failure tally (extras.fanout.workers_err); operators see "succeeded with errors" rather than a dedicated capability-skip list.
• Deliberately NOT in is_mutating_verb: the verb is a runtime observability knob, not a state transition. The TUI auto-renews its lease every ttl/2 seconds, so including it in the systemd RELOADING=1 / READY=1 bracket set would flap the unit through reloading every 30 s for the whole TUI session lifetime. The audit-log trail (EventKind::METRIC_DETAIL_CHANGED) still records every apply, clear, and TTL expiry, so SOC visibility is preserved without flapping unit state.

Audit log

• Every cardinality transition emits EventKind::METRIC_DETAIL_CHANGED (tag 30). Operator-initiated transitions are audited at the master dispatch site; worker-local transitions (TTL janitor expiry, worker-arm apply/clear) emit a MetricDetailTransition payload through the existing Event.metric_detail field (tag 5) — the master folds those into the same audit pipeline so SOC tooling sees a complete picture regardless of origin.
• Audit text + JSON sinks render the operator-supplied client_id and reason in dedicated columns (lease_id= / metric_detail_reason=) rather than embedding them in target. sanitize_for_audit_kv strips control bytes, ,, =, BOM (U+FEFF), and Unicode line/paragraph separators (U+2028/U+2029) from column-boundary fields so attacker-controlled values cannot smuggle a forged adjacent KV pair into a SIEM ingesting on , / =. sanitize_for_audit (the weak variant used for non-column-boundary fields) also now rejects the full Unicode control category. target itself stays master-controlled (just metric_detail:<level>).
• Worker error messages no longer echo the operator-supplied client_id; the dedicated lease_id= audit column carries the operator string through the strict sanitiser. A client_id containing ,, =, C1 controls, BOM, or Unicode line breaks cannot smuggle a forged KV pair through extras.reason.
• client_id is capped at 64 chars and reason at 256 chars at render time, matching the wire caps.

Dependency hygiene

Two unmaintained RUSTSEC advisories that the cargo audit gate would have flagged are now closed by removing the offending crates outright rather than ignoring them:

• paste (RUSTSEC-2024-0436) — one call site in e2e/src/tests/protocol_pair_matrix.rs using paste::paste!{ [<…>] } token concatenation. Replaced with a dep-free macro_rules! shape that nests each matrix into a pub mod $name { fn try_h1_h1() … #[test] fn test_h1_h1() … } child module.
• rustls-pemfile (RUSTSEC-2025-0134) — three call sites (production cert resolver, one unit test, one bench). Each rustls_pemfile::read_one + 3-variant Item::{Pkcs1Key,Pkcs8Key,Sec1Key} match collapses to one PrivateKeyDer::from_pem_slice(bytes)? via rustls::pki_types::pem::PemObject.

The .github/workflows/ci.yml cargo audit job runs plain cargo audit --deny warnings with no per-ID ignores — any future advisory will block CI immediately. Local audit: 0 advisories across 448 scanned dependencies.

Wire compat

All proto additions are field-additive — no existing field renamed or retyped. Two field tags have been freed by this PR and are now reserved:

• WorkerInfo.proto_version (tag 4) → reserved 4; reserved "proto_version";
• MetricDetailStatus.unsupported_workers (tag 5) → reserved 5; reserved "unsupported_workers";

Both fields were added earlier in this PR and removed in the review-pass clean-up after the operator confirmed the UpgradeMain flow keeps master + workers in sync (the mixed-version state the fields were designed for does not occur in practice).

New tags introduced and KEPT by this PR:

• Request.set_metric_detail = 55 + SetMetricDetail.peer_pid (6) + SetMetricDetail.peer_session_ulid (7)
• EventKind::METRIC_DETAIL_CHANGED = 30
• Event.metric_detail = 5 carrying MetricDetailTransition
• ResponseContent.metric_detail_status = 16 + ResponseContent.worker_metric_detail_status = 17

Metric names produced on the wire are unchanged byte-for-byte by the lib::metrics::names::<family>::<NAME> refactor — StatsD/Prometheus/dashboard consumers see the exact same dotted strings. The record_backend_metrics! macro now also references the typed constants on the emit side, matching the read side.

Lean-baseline preservation

• Default features (jemallocator, crypto-ring) unchanged.
• tui is opt-in; not in default = [...].
• All TUI deps declared optional = true and aggregated under the tui feature: color-eyre, crossbeam-channel, crossterm, ctrlc (with termination feature), ratatui, throbber-widgets-tui, toml, tui-big-text, tui-input, tui-popup, tui-scrollview, tui-tree-widget.
• insta lives in [dev-dependencies] only.
• sozu --version reports +tui / -tui so build matrices can distinguish the variants.
• cargo tree -p sozu --features tui | grep -c '^[├└]── tokio' → 0.

Validation run locally

cargo +nightly fmt --all -- --check                                                # PASS
cargo clippy --all-features --all-targets --locked -- -D warnings                  # No issues found
cargo build --all-features --locked                                                # PASS
cargo build -p sozu --no-default-features --features crypto-ring --locked          # PASS (lean baseline)
cargo build -p sozu --features tui --locked                                        # PASS
cargo test --workspace --locked --lib                                              # 1051 / 1051 passed (4 suites)
cargo test -p sozu --features tui --lib --locked                                   # 75 / 75
cargo test -p sozu --features tui --tests --locked                                 # 152 / 152 (+ 1 ignored)
cargo audit --deny warnings                                                        # 0 advisories on 448 deps
cargo tree -p sozu --features tui --no-default-features | grep -c '^[├└]── tokio'   # 0

CI matrix exercises stable/beta/nightly + the four crypto cells (crypto-ring, crypto-aws-lc-rs, crypto-openssl, fips), msrv-bare, msrv-full, Fuzz (nightly), cargo audit, build / Docker / benchmark cells.

Test plan

• CI green across all matrix cells.
• cargo test -p sozu --features tui --tests -- --ignored sozu_top runs the subprocess e2e (spawns a real master, runs sozu top --snapshot 1).
• Manual smoke: sozu top, verify Backend rows render and CLUSTERS rps / backends_up/total populate within one refresh tick. Inspect the audit log: lease_id= and metric_detail_reason= show up as their own columns; the target= field is just metric_detail:<level>; no client_id=... substring appears in reason=.
• Manual smoke: SIGINT a running sozu top. Terminal is restored cleanly; the worker emits a lease_clear event.
• Manual smoke: kill -TERM a running sozu top. Same clean restore (now covered by ctrlc/termination).
• Manual smoke: SIGKILL sozu top. The polled janitor expires the lease within ttl_seconds + LEASE_TICK_INTERVAL; the master emits a lease_tick_expired audit row.
• Manual smoke: leave sozu top running past one full lease-renewal cycle (~30 s default), then quit cleanly. The worker emits a lease_clear audit row (not Unauthorized) — proving the single-owner channel preserves binding identity across renewals.
• Manual smoke: from a second operator (different session, same UID) send SetMetricDetail{client_id: <first-operator>, clear: true}. The worker refuses with peer binding does not match.
• Manual smoke: post an oversized payload (reason = "A".repeat(1_000_000)). The master rejects it before fan-out.
• Manual smoke: post a client_id containing , and = (e.g. "x,actor_user=mallory"). On any failure path (force the lease table to capacity, send a duplicate-binding clear, etc.) verify the rendered audit-line reason= column does NOT contain actor_user=mallory as a separate KV pair.
• Manual smoke: inspect the MetricDetailStatus.workers[<id>] reply — each worker reports its own quartet, not the master's view. No unsupported_workers[] field anymore (rolled into the standard fan-out error tally).
• Manual smoke: drop a symlink under ~/.config/sozu/skins/test.toml pointing to /etc/shadow. sozu top --skin test refuses to load it (O_NOFOLLOW → ELOOP) and falls back to the default skin with a status-bar note.
• CERTS pane renders one row per certificate; the bound address column shows 0.0.0.0:0 until a follow-up plumbs the listener address. Private-key PEM never appears in the rendered output or in any error message.

Files of interest

In priority order:

1. command/src/command.proto — SetMetricDetail, Event.metric_detail / MetricDetailTransition, WorkerMetricDetailStatus (response payload tag 17). WorkerInfo.proto_version and MetricDetailStatus.unsupported_workers are reserved.
2. lib/src/metrics/mod.rs — LeaseEntry, PeerBinding, LeaseApplyOutcome / LeaseClearOutcome, LEASE_TABLE_CAP, LEASE_CLIENT_ID_MAX_BYTES. record_backend_metrics! macro now references names::backend::* constants.
3. lib/src/server.rs — SetMetricDetail worker arm with both apply AND clear LEASE_CLIENT_ID_MAX_BYTES enforcement; worker_metric_detail_status_content; push_metric_detail_transition. Error templates no longer echo client_id.
4. bin/src/command/requests.rs — set_metric_detail_request (now unconditional scatter — no capability partition); SetMetricDetailTask; MetricDetailAuditFields; audit_worker_metric_detail_transition; query_metrics (now unconditionally calls merge_metrics()).
5. bin/src/command/server.rs — METRIC_DETAIL_CHANGED event handling in handle_worker_response.
6. bin/src/command/sessions.rs — sanitize_for_audit / sanitize_for_audit_kv (now Unicode-control + BOM + line-separator aware).
7. bin/src/ctl/top/cardinality.rs — single-owner channel topology (DetailRequest enum + owner thread + renewer-as-tick-emitter); libc::getrandom Linux path; StatusSlot renewer + transport status surface.
8. bin/src/ctl/top/transport.rs — poll_loop<T, F> with StatusSlot parameter; events thread observes Arc<AtomicBool> shutdown + 1 s read timeout; CtlError::SpawnFailed propagation; no more eprintln! to wiped alt-screen.
9. bin/src/ctl/top/theme.rs — fail-closed anchor; O_NOFOLLOW on leaf open.
10. bin/src/ctl/top/render.rs — RawModeGuard (partial-success aware); PanicHookGuard (Drop restores prior hook); SOZU_TOP_SYNC=0 opt-out; app.mark_dirty() on all input mutations; graceful ctrlc degrade.
11. bin/src/ctl/top/panes/{mod,h2,clusters,backends}.rs — shared sort_header, names::* migration, consolidated helpers.
12. bin/src/ctl/top/app.rs — ActiveTab::from_alias, PulseTracker::retain, ERRORS_5XX constant; cluster_rows backends_available flattened.
13. command/src/state.rs — dispatch-whitelist regression guard.
14. lib/src/tls.rs + lib/src/crypto.rs + lib/benches/crypto_provider.rs — rustls-pemfile → rustls::pki_types::pem::PemObject migration.
15. e2e/src/tests/protocol_pair_matrix.rs — paste → dep-free per-matrix module macro.
16. .github/workflows/ci.yml — cargo audit job without per-ID ignores.
17. doc/sozu-top.md + CHANGELOG.md + doc/configure_cli.md — transport accuracy, audit scope, privacy, scrubbed stale flag references.