[operations] docs: add operational alert runbooks for spindle metrics

runbooks/README.md

@@ -0,0 +1,15 @@
+# Spindle Alert Runbooks
+
+This directory contains operational runbooks for alert rules defined in `deploy/prometheus/spindle-alerts.yaml`.
+
+## Alerts Overview
+
+| Alert Name | Severity | Description | Runbook |
+|---|---|---|---|
+| `SpindleForkCase3Elevated` | info | Case 3 state resolutions exceed SPEC §18.3 target (<0.1%) | [SpindleForkCase3Elevated.md](SpindleForkCase3Elevated.md) |
+| `SpindleAppendP99AboveTarget` | warning | p99 commit duration under group durability > 10ms | [SpindleAppendLatencyHigh.md](SpindleAppendLatencyHigh.md) |
+| `SpindleAppendP50AboveTarget` | warning | p50 commit duration under group durability > 2ms | [SpindleAppendLatencyHigh.md](SpindleAppendLatencyHigh.md) |
+| `SpindleFederationBacklog` | warning | Federation outbox queue > 500 events for 30m | [SpindleFederationBacklog.md](SpindleFederationBacklog.md) |
+| `SpindleSyncLagHigh` | warning | p99 sync delivery lag > 30s | [SpindleSyncLagHigh.md](SpindleSyncLagHigh.md) |
+| `SpindleDown` | critical | Metrics listener unresponsive for 5m | [SpindleDown.md](SpindleDown.md) |
+| `SpindleServerErrors` | warning | HTTP 5xx error rate > 1% over 5m | [SpindleServerErrors.md](SpindleServerErrors.md) |

runbooks/SpindleAppendLatencyHigh.md

@@ -0,0 +1,26 @@
+# Runbook: SpindleAppendLatencyHigh
+
+## Overview
+- **Alert Names**: `SpindleAppendP99AboveTarget`, `SpindleAppendP50AboveTarget`
+- **Severity**: `warning`
+- **Metric**: `spindle_append_duration_seconds_bucket{durability="group"}`
+- **Condition**: p99 > 10ms or p50 > 2ms over 5 minutes, sustained for 15 minutes.
+
+## Context
+SPEC §18.3 targets local send commit latencies of p50 < 2ms and p99 < 10ms for `group` durability. `spindle_append_duration_seconds` measures the exact storage commit phase (including I/O and fsync).
+
+## Impact
+- Higher append latency directly slows down event submission and increases HTTP response times for sending messages.
+
+## Diagnostic Steps
+1. Check disk I/O metrics and storage backend performance (fjall store disk write latency, IOPS, and fsync queue time).
+2. Query PromQL for append duration quantiles by durability mode:
+   ```promql
+   histogram_quantile(0.99, sum by (le) (rate(spindle_append_duration_seconds_bucket[5m])))
+   histogram_quantile(0.50, sum by (le) (rate(spindle_append_duration_seconds_bucket[5m])))
+   ```
+3. Check system resource utilization (disk throughput, CPU usage, memory pressure).
+
+## Mitigation
+- Verify storage device health and ensure adequate I/O throughput for disk flushes.
+- Inspect room lock contention metrics (`spindle_room_lock_acquisitions_total`) to rule out concurrency bottlenecks.

runbooks/SpindleDown.md

@@ -0,0 +1,26 @@
+# Runbook: SpindleDown
+
+## Overview
+- **Alert Name**: `SpindleDown`
+- **Severity**: `critical`
+- **Metric**: `up{job="spindle"}`
+- **Condition**: `up == 0` for 5 minutes.
+
+## Context
+Prometheus failed to scrape the Spindle metrics endpoint (`GET /metrics`) for 5 minutes.
+
+## Impact
+- Telemetry and operational monitoring for Spindle are offline.
+- If the entire Spindle process crashed, client and federation traffic is unserved.
+
+## Diagnostic Steps
+1. Check if the Spindle process is running on the host/container.
+2. Check metrics endpoint responsiveness directly:
+   ```bash
+   curl -I http://127.0.0.1:9090/metrics
+   ```
+3. Inspect Spindle stderr/stdout system logs for panics, OOM kills, or crash tracebacks.
+
+## Mitigation
+- If the process crashed, restart the Spindle service.
+- If the process is alive but metrics port is unresponsive, verify network binding configuration in `spindle.toml` under `[metrics]`.

runbooks/SpindleFederationBacklog.md

@@ -0,0 +1,22 @@
+# Runbook: SpindleFederationBacklog
+
+## Overview
+- **Alert Name**: `SpindleFederationBacklog`
+- **Severity**: `warning`
+- **Metric**: `spindle_federation_queue_depth`
+- **Condition**: Outbound federation queue > 500 events for destination (excluding `other`), sustained for 30 minutes.
+
+## Context
+Spindle tracks outbound federation event delivery queues per destination server. A growing backlog indicates outbound transactions are failing, timing out, or being rate-limited by the target remote Matrix homeserver.
+
+## Impact
+- Delivery delays to remote Matrix homeservers; users on target homeservers will receive events late.
+
+## Diagnostic Steps
+1. Identify the affected destination server from the `destination` label.
+2. Check network connectivity and DNS resolution for the remote server domain.
+3. Inspect Spindle server federation logs for outbound connection errors, TLS handshakes, or HTTP response errors (e.g. 502, 504, 429) from the target peer.
+
+## Mitigation
+- If the remote server is down, wait for remote recovery; queue depth will drain once connectivity is restored.
+- If the remote server is unreachable due to routing/DNS issues, resolve host connectivity.

runbooks/SpindleForkCase3Elevated.md

@@ -0,0 +1,27 @@
+# Runbook: SpindleForkCase3Elevated
+
+## Overview
+- **Alert Name**: `SpindleForkCase3Elevated`
+- **Severity**: `info`
+- **Metric**: `spindle_fork_resolutions_total{case="3"}` / `spindle_events_appended_total{origin="federated"}`
+- **Condition**: Ratio > `0.001` (0.1%) over a 1-hour rate, sustained for 6 hours.
+
+## Context
+SPEC §18.3 establishes that Case-3 fork resolutions (contested state events within the fork window) should constitute less than 0.1% of federated events. A rise above this threshold indicates that state resolution is running on the append path more frequently than expected, or that inbound federated events with state conflicts are being rejected/resolved at an elevated rate.
+
+## Impact
+- Elevated Case-3 occurrences increase CPU overhead per append.
+- Currently, Spindle's ingest path rejects Case-3 appends pending full wiring of state resolution (#16). Thus, elevated Case-3 rate indicates incoming federated events are being rejected.
+
+## Diagnostic Steps
+1. Verify the current ratio using Prometheus:
+   ```promql
+   rate(spindle_fork_resolutions_total{case="3"}[1h]) / ignoring(case) rate(spindle_events_appended_total{origin="federated"}[1h])
+   ```
+2. Check logs for incoming federation state conflict errors or event rejections.
+3. Identify participating federated servers and room IDs experiencing high branch creation / concurrent state updates.
+
+## Mitigation
+- If a specific remote server is generating high volumes of conflicting state events, inspect room state and federation status.
+- Monitor overall server performance and CPU consumption.
+- Report persistent high Case-3 ratios upstream with trace data as a benchmark finding.

runbooks/SpindleServerErrors.md

@@ -0,0 +1,24 @@
+# Runbook: SpindleServerErrors
+
+## Overview
+- **Alert Name**: `SpindleServerErrors`
+- **Severity**: `warning`
+- **Metric**: `spindle_http_requests_total{status=~"5.."}` / `spindle_http_requests_total`
+- **Condition**: HTTP 5xx error rate > 1% (0.01) over 5 minutes, sustained for 10 minutes.
+
+## Context
+Measures the proportion of HTTP requests resulting in internal server errors (5xx status codes).
+
+## Impact
+- Failures for client API calls or inbound federation endpoints.
+
+## Diagnostic Steps
+1. Identify affected routes and methods in Prometheus:
+   ```promql
+   sum by (route, method, status) (rate(spindle_http_requests_total{status=~"5.."}[5m]))
+   ```
+2. Search server logs for 500 error stack traces and internal panic messages.
+3. Check database/store write permissions and storage state.
+
+## Mitigation
+- Address root cause identified in application error logs (e.g. database locks, unhandled error conditions).

runbooks/SpindleSyncLagHigh.md

@@ -0,0 +1,26 @@
+# Runbook: SpindleSyncLagHigh
+
+## Overview
+- **Alert Name**: `SpindleSyncLagHigh`
+- **Severity**: `warning`
+- **Metric**: `spindle_sync_lag_seconds`
+- **Condition**: p99 lag of delivered events > 30 seconds over 5 minutes, sustained for 15 minutes.
+
+## Context
+`spindle_sync_lag_seconds` measures the age of the newest event delivered to a client in a `/sync` response at the moment of delivery. Elevated sync lag means connected clients are receiving timeline events significantly after they were committed.
+
+## Impact
+- Degraded user experience: message delivery and notification lag on Matrix clients (Element, Cinny, etc.).
+
+## Diagnostic Steps
+1. Query p99 sync lag in Prometheus:
+   ```promql
+   histogram_quantile(0.99, sum by (le) (rate(spindle_sync_lag_seconds_bucket[5m])))
+   ```
+2. Check active subscriber count via `spindle_sync_subscribers`.
+3. Check HTTP request latency for `/sync` routes using `spindle_http_request_duration_seconds`.
+4. Inspect room lock and store read performance to determine if event fetch or notification dispatch is delayed.
+
+## Mitigation
+- Ensure server resources (CPU/memory) are not exhausted.
+- Check storage read latency to ensure historical event retrieval is not stalling sync workers.