bench(#874): idle-budget fixture + CI gate; delete the O(N) metrics gauge loop

.github/workflows/ci.yml

@@ -841,3 +841,50 @@ jobs:
         env:
           XERJ_BIN: engine/target/ci-test/xerj
         run: bash .github/scripts/autoindex-fd-smoke.sh
+
+  # Issue #874's verification fixture, kept runnable: 15 repos x ~30 datasets
+  # (~450 tiny indices), boot, settle 120 s, measure idle CPU / wakeups / RSS
+  # via /proc, gate coarsely. The unit-level guards (merge-evaluation counter,
+  # idle-flush, memtable recount tests in xerj-engine) pin the mechanisms; this
+  # job pins the PROCESS-LEVEL budget they exist to protect: an idle node must
+  # cost < 0.5 % of one core, O(1) wakeups/s, <= 0.2 MB per idle index, and
+  # boot without replaying WAL on a cleanly-flushed corpus — independent of N.
+  # ubuntu only on purpose: the sampler walks /proc. Thresholds are coarse
+  # (~2x the measured healthy values, see benchmarks/idle-budget/README.md):
+  # a 2-4 vCPU shared runner cannot support ±1 % claims, and the regressions
+  # this catches (per-index timers, per-index polling, per-index retention)
+  # arrive as multiples, not percents.
+  idle-budget:
+    name: Idle budget (many-index at-rest CPU/RSS)
+    runs-on: ubuntu-latest
+    # One tier up from the no-compile tier: compiles one crate (xerj-server)
+    # on the shared ci-test cache, then boots and measures. The fixture itself
+    # is ~8 min measured (two 120 s settle windows + ~450 PUT/_bulk rounds);
+    # 30 min leaves 3x+ headroom for a cold cache and a slow runner.
+    timeout-minutes: 30
+    steps:
+      - uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
+
+      - name: Install Rust
+        uses: dtolnay/rust-toolchain@032958afbdc797a9164d3bc0b56325c1308924a5 # 1.97.1
+
+      - name: Cache
+        uses: Swatinem/rust-cache@6323deb102c322ba6fcbdcafc7e3dddab59af2b6 # v2.9.2
+        with:
+          workspaces: engine
+          # Read-only consumer of build-test's ci-test dependency cache; that
+          # job owns the key (same arrangement as api-checks / conformance).
+          shared-key: ci-test-linux
+          save-if: false
+
+      # ci-test, not release: the thresholds are coarse (O(N) blowups, not
+      # ±1 %), so fat LTO would buy measurement precision this gate does not
+      # claim to have. Same opt-level and features as every other booting job.
+      - name: Build xerj (ci-test profile)
+        working-directory: engine
+        run: cargo build --profile ci-test -p xerj-server
+
+      - name: Idle budget fixture (450 indices, 120 s settle, coarse thresholds)
+        env:
+          XERJ_BIN: ${{ github.workspace }}/engine/target/ci-test/xerj
+        run: bash benchmarks/idle-budget/run.sh

CHANGELOG.md

@@ -7,6 +7,26 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Fixed
+
+- **An idle node no longer spends CPU per index on Prometheus gauges (#874).**
+  The 10 s background loop that fed `xerj_doc_count` / `xerj_segment_count` /
+  `xerj_wal_size_bytes` / `xerj_memory_usage_bytes` walked every index's WAL
+  subtree on an async runtime worker on every tick — one `read_dir` plus one
+  `stat()` per WAL shard (~16 per index at default sharding) whether anyone
+  read the values or not. The new at-rest fixture
+  (`benchmarks/idle-budget/`) measured that at ~0.7–0.9 % of one core for 450
+  idle indices (against #874's < 0.5 % budget), entirely in the `xerj-rt`
+  workers; deleting the loop and refreshing the gauges at scrape time in the
+  `/v1/metrics` handler (the same way the query-cache gauges already
+  reconcile) drops idle CPU to 0.15–0.18 % and wakeups from ~46/s to ~26/s on
+  the same corpus, with the WAL walk moved to the blocking pool. A scrape is
+  the only moment these gauges are observable, so nothing is lost between
+  scrapes. The fixture and its coarse CI gate (`idle-budget` job) keep the
+  whole budget honest: CPU < 0.5 % of one core, O(1) wakeups/s, ≤ 0.2 MB RSS
+  per idle index, boot-to-green with zero WAL replay on a cleanly-flushed
+  corpus.
+
 ## [1.0.0-rc.77] - 2026-09-21
 
 The stateless-index and reader-fairness release. Two headline changes:

benchmarks/idle-budget/gen_corpus.sh

@@ -0,0 +1,34 @@
+#!/usr/bin/env bash
+# Generate the issue #874 idle-budget corpus: 15 "repos" x ~30 "datasets" of
+# a few small docs each — the synthetic equivalent of the corpus the at-rest
+# budgets were measured on (autoindex over 15 reference repos => ~464 small
+# indices + system indices; see docs/measurements/autoindex-watch-2026-09-19.md).
+#
+# Shape: one directory per repo, one CSV per dataset, UNIQUE column names per
+# dataset — the same dataset-inference key .github/scripts/autoindex-fd-smoke.sh
+# uses (a distinct schema => a distinct inferred dataset). run.sh turns each
+# CSV into one index over the ES wire.
+#
+#   benchmarks/idle-budget/gen_corpus.sh <out-dir> [repos] [datasets]
+#   env REPOS= / DATASETS= work too; positional args win.
+set -euo pipefail
+OUT=${1:?usage: gen_corpus.sh <out-dir> [repos] [datasets]}
+REPOS=${2:-${REPOS:-15}}
+DATASETS=${3:-${DATASETS:-30}}
+
+mkdir -p "$OUT"
+n=0
+for r in $(seq 1 "$REPOS"); do
+  rd=$(printf 'repo-%02d' "$r")
+  mkdir -p "$OUT/$rd"
+  for d in $(seq 1 "$DATASETS"); do
+    # Global counter across the whole corpus: column names never repeat, so
+    # every dataset has a distinct schema (and run.sh's per-dataset docs with
+    # them). 3 data rows = "a few small docs".
+    i=$((n + 1))
+    printf 'k_%d_id,k_%d_name,k_%d_val\n1,alpha-%d,10\n2,beta-%d,20\n3,gamma-%d,30\n' \
+      "$i" "$i" "$i" "$i" "$i" "$i" > "$OUT/$rd/$(printf 'ds_%03d.csv' "$d")"
+    n=$i
+  done
+done
+echo "generated $n datasets under $OUT ($REPOS repos x $DATASETS datasets, 3 docs each)"

benchmarks/idle-budget/results/result-citest-check.json

@@ -0,0 +1,106 @@
+{
+ "label": "citest-check",
+ "date_utc": "2026-09-21T23:12:41+00:00",
+ "binary": {
+  "version": "xerj v1.0.0-rc.77",
+  "path": "/root/xerj-874/engine/target/ci-test/xerj"
+ },
+ "corpus": {
+  "shape": "15x30",
+  "indices": 450,
+  "docs_per_index": 3
+ },
+ "port": 9610,
+ "window_s": 120.0,
+ "settle_target_s": 120.0,
+ "boot_ms": {
+  "empty_node": 54664,
+  "baseline_arm": 2214,
+  "n_indices": 1963
+ },
+ "wal_replay_lines_after_clean_flush": 0,
+ "doc_probe_hits": 3,
+ "baseline": {
+  "pid": 1861561,
+  "window_s": 121.5,
+  "clk_tck": 100,
+  "cpu_ticks": 13,
+  "cpu_percent_of_one_core": 0.107,
+  "wakeups_per_s": 25.2,
+  "voluntary_per_s": 24.3,
+  "nonvoluntary_per_s": 0.9,
+  "rss_kb": 31012,
+  "hwm_kb": 117824,
+  "rss_anon_kb": 1460,
+  "rss_file_kb": 29552,
+  "threads": 340,
+  "fds": 15,
+  "per_thread_cpu": [
+   {
+    "comm": "xerj-mem-sample",
+    "cpu_ticks": 7,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.058
+   },
+   {
+    "comm": "xerj-memtable-s",
+    "cpu_ticks": 3,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.025
+   },
+   {
+    "comm": "xerj-rt",
+    "cpu_ticks": 1,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.008
+   }
+  ],
+  "loadavg_start": [
+   "96.55",
+   "50.05",
+   "40.61"
+  ]
+ },
+ "measurement": {
+  "pid": 1709095,
+  "window_s": 120.0,
+  "clk_tck": 100,
+  "cpu_ticks": 25,
+  "cpu_percent_of_one_core": 0.2083,
+  "wakeups_per_s": 26.6,
+  "voluntary_per_s": 24.2,
+  "nonvoluntary_per_s": 2.5,
+  "rss_kb": 166272,
+  "hwm_kb": 180108,
+  "rss_anon_kb": 136556,
+  "rss_file_kb": 29716,
+  "threads": 336,
+  "fds": 15,
+  "per_thread_cpu": [
+   {
+    "comm": "xerj-memtable-s",
+    "cpu_ticks": 16,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.133
+   },
+   {
+    "comm": "xerj-mem-sample",
+    "cpu_ticks": 6,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.05
+   },
+   {
+    "comm": "xerj-rt",
+    "cpu_ticks": 1,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.008
+   }
+  ],
+  "loadavg_start": [
+   "35.77",
+   "67.91",
+   "60.14"
+  ]
+ },
+ "per_index_rss_kb": 300.6
+}

benchmarks/idle-budget/results/result-citest-check2.json

@@ -0,0 +1,106 @@
+{
+ "label": "citest-check2",
+ "date_utc": "2026-09-21T23:23:17+00:00",
+ "binary": {
+  "version": "xerj v1.0.0-rc.77",
+  "path": "/root/xerj-874/engine/target/ci-test/xerj"
+ },
+ "corpus": {
+  "shape": "15x30",
+  "indices": 450,
+  "docs_per_index": 3
+ },
+ "port": 9610,
+ "window_s": 120.0,
+ "settle_target_s": 120.0,
+ "boot_ms": {
+  "empty_node": 64,
+  "baseline_arm": 824,
+  "n_indices": 38
+ },
+ "wal_replay_lines_after_clean_flush": 0,
+ "doc_probe_hits": 3,
+ "baseline": {
+  "pid": 2027123,
+  "window_s": 120.0,
+  "clk_tck": 100,
+  "cpu_ticks": 12,
+  "cpu_percent_of_one_core": 0.1,
+  "wakeups_per_s": 26.0,
+  "voluntary_per_s": 24.4,
+  "nonvoluntary_per_s": 1.6,
+  "rss_kb": 82084,
+  "hwm_kb": 122280,
+  "rss_anon_kb": 54592,
+  "rss_file_kb": 27492,
+  "threads": 340,
+  "fds": 15,
+  "per_thread_cpu": [
+   {
+    "comm": "xerj-mem-sample",
+    "cpu_ticks": 7,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.058
+   },
+   {
+    "comm": "xerj-memtable-s",
+    "cpu_ticks": 3,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.025
+   },
+   {
+    "comm": "xerj-rt",
+    "cpu_ticks": 1,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.008
+   }
+  ],
+  "loadavg_start": [
+   "42.05",
+   "48.07",
+   "53.00"
+  ]
+ },
+ "measurement": {
+  "pid": 3926987,
+  "window_s": 120.0,
+  "clk_tck": 100,
+  "cpu_ticks": 26,
+  "cpu_percent_of_one_core": 0.2166,
+  "wakeups_per_s": 23.7,
+  "voluntary_per_s": 20.3,
+  "nonvoluntary_per_s": 3.3,
+  "rss_kb": 163820,
+  "hwm_kb": 178144,
+  "rss_anon_kb": 136272,
+  "rss_file_kb": 27548,
+  "threads": 336,
+  "fds": 15,
+  "per_thread_cpu": [
+   {
+    "comm": "xerj-memtable-s",
+    "cpu_ticks": 16,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.133
+   },
+   {
+    "comm": "xerj-mem-sample",
+    "cpu_ticks": 8,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.067
+   },
+   {
+    "comm": "xerj-rt",
+    "cpu_ticks": 1,
+    "threads": 1,
+    "cpu_percent_of_one_core": 0.008
+   }
+  ],
+  "loadavg_start": [
+   "35.14",
+   "41.42",
+   "49.37"
+  ]
+ },
+ "per_index_rss_kb": 181.6
+}