bench(#874): idle-budget fixture + CI gate; delete the O(N) metrics gauge loop

[xerj-org]

What

Implements the last deliverable of #874 — the kept-runnable verification fixture — and the fix its first run forced. #874 asked for "one fixture, kept runnable: 15 repos x ~30 datasets,
boot, settle 120 s, measure CPU/wakeups/RSS via /proc. Gate the budget in CI
on ubuntu (coarse thresholds; the point is catching O(N) regressions, not
±1%)." This adds the fixture, the CI job, and the fix its first run forced.

The fixture caught a real regression on its first run against the then-
current tree: idle CPU at 450 tiny indices measured 1.12 % of one core
(0.78 % in an earlier window) against the issue's < 0.5 % line, entirely in
the xerj-rt (tokio) threads. Per-thread attribution pointed at the 10 s
background loop that fed the Prometheus gauges: every tick it walked every
index's WAL subtree — read_dir plus one metadata() per WAL shard, ~16 shards
per index at default sharding, ~7,400 stat()s per tick at N=450 — on an
async runtime worker, whether anyone read the gauges or not. An A/B run with
only the loop disabled (same binary, four 120 s windows) measured 0.15-0.18 %
and 26-27 wakeups/s, proving the loop was the whole excess.

Fix: the loop is deleted; /v1/metrics refreshes xerj_doc_count,
xerj_segment_count, xerj_wal_size_bytes and xerj_memory_usage_bytes at scrape
time — the only moment they are observable — mirroring how the handler
already reconciles the query-cache gauges, with the WAL-subtree walk moved to
tokio::task::spawn_blocking. Pinned by a fail-before integration test: with
the loop gone and nothing refreshing at scrape time, xerj_doc_count reads 0
on a node holding documents.

Measured, 120 s windows, /proc only, shared box (load 38-63 recorded in each
result file), binary v1.0.0-rc.77 + this branch:

450 idle indices before (loop) after (fix) gate
idle CPU % of one core 0.78-1.12 0.175 < 0.5
wakeups/s process-wide 46.1 26.6 < 100
per-index idle RSS kB 185.3 183.8 <= 204.8
boot-to-green ms 125 215 < 10,000
WAL replay after flush 0 0 0

The baseline arm (empty node) reads 0.10 % CPU and 26.0 wakeups/s, so the
fixed N=450 node is flat in N on both counters; after the fix the top
per-thread CPU at N=450 is xerj-memtable-s at 0.083 % with xerj-rt at
0.008 %. Result files: benchmarks/idle-budget/results/result-n450indices.json
(run of record) and result-n450-with-metrics-gauge-loop.json (the failing
first run, kept as the before-artifact).

Files:

• benchmarks/idle-budget/gen_corpus.sh — 15 repo dirs x 30 dataset CSVs,
  3 docs each, globally unique column names (the autoindex inference key).
• benchmarks/idle-budget/run.sh — baseline arm (N=0, same window), load arm
  (one index per dataset over the ES wire, default settings, POST /_flush,
  SIGTERM), restart arm (boot-to-green on the cleanly-flushed corpus + the
  zero-"replayed WAL entries" gate + a 3-hit doc probe), then the 120 s
  /proc window: CPU = utime+stime delta over CLK_TCK, wakeups summed across
  /proc//task/*/status (the process-level counters cover only the
  parked main thread and read 0/s), VmRSS/VmHWM/RssAnon/RssFile, per-thread
  CPU attribution grouped by comm. Coarse gates with ::error:: annotations
  and exit 1; refuses port 9200, non-empty or RAM-backed data dirs; private
  port scan 9610..9639.
• benchmarks/idle-budget/README.md — method, measured tables, threshold
  rationale, reproduce commands with expected output.
• .github/workflows/ci.yml — new idle-budget job: builds xerj-server on
  the shared ci-test profile cache, runs the fixture with defaults (450
  indices, 120 s settle, ~8 min measured) on ubuntu-latest,
  timeout-minutes: 30.
• engine/crates/xerj-api/src/es_compat.rs — refresh_metric_gauges is now
  pub(crate), scrape-driven, and does the WAL walk on the blocking pool;
  run_metrics_gauge_loop deleted.
• engine/crates/xerj-api/src/native.rs — /v1/metrics calls
  refresh_metric_gauges before gather_text.
• engine/crates/xerj-server/src/main.rs — the loop's tokio::spawn removed;
  comment (b) rewritten to point at the fixture.
• engine/crates/xerj-api/tests/metrics_gauges_refresh_at_scrape_time.rs —
  scrape-time liveness pin (fail-before shape documented in the module doc).
• CHANGELOG.md — Fixed entry with the measured before/after.

Gates run: the new integration test passes (and fails before the fix);
cargo test/clippy/fmt scoped to xerj-api and xerj-server are clean; the
ES-YAML conformance suite is 1376 passed / 0 failed / 3 skipped against this
branch's release binary.

Closes #874.

---

Independent verification (second agent, fresh build, full 120 s settle, no numbers trusted from the implementer): gate PASSED again — idle CPU 0.167 % (run of record 0.175 %), wakeups 26.4/s (26.6), per-index RSS 190.1 kB (183.8, +3.4 %), boot-to-green 218 ms (215), 0 WAL-replay lines, 450/450 indices, doc probe 3/3. ES-YAML re-run green: 1376 / 0 failed / 3 skipped. Its review also confirmed: thresholds enforced by exit code, CI job satisfies the workflow-timeout guard (verified with the repo's own guard script), ports/dirs private, commit discipline (author xerj-org, no trailers). Three minor findings from that review (doubled code fences in the README, a loose load-range wording, and making the :9200 refusal explicit instead of implicit-by-busyness) are folded into this branch.

Environmental note for reviewers re-running the ES-YAML gate on an almost-full disk: the sandbox was at 96 % used, which trips the 95 % disk_flood_stage write block by design (262 cases 429/cluster-block). With [limits] disk_flood_stage_percent = 0 on the throwaway gate node — or a disk under 95 % — the suite is fully green. That is the watermark working, not a branch defect.

[xerj-org]

CI-red root cause + fix (commit 5b9542b): the idle-budget job's first run measured 1850 kB/idx against the 204.8 kB gate - 2075 kB anon per index = one 2 MB THP hugepage to within 1%. ubuntu-latest boots with THP=always; jemalloc follows the kernel mode, so each index's small boot allocations pin a full hugepage. The dev hosts run madvise, hence the 10x disagreement on identical binary and data. A ~2 MB/index page-granularity artifact fails a 0.2 MB/index budget even for a perfect engine - it is not an O(N) regression, which is what the gate exists to catch.

The fix pins MALLOC_CONF=thp:never,dirty_decay_ms:0,muzzy_decay_ms:0 on the measured node (both arms). The decay pins fix a second, local instability found while validating: the baseline arm's RSS alone wobbled 31-76 MB run-to-run on the same host (whichever freed pages jemalloc retained at sample time) - a 1.5x swing in the (load - baseline)/N subtraction. Gate lines untouched.

Local re-validation with the ci-test profile binary (the profile the CI job builds), full 120 s windows, both arms: per-index 181.6 kB, gate PASSED exit 0 (result committed as citest-check2; the pin-only intermediate kept as citest-check). CI re-running now - expecting ~180 kB/idx on the runner.

[xerj-org]

Three CI runs, three findings - the log of record:

1. Run 1 (1850 kB/idx): THP=always runner + jemalloc following the kernel mode - one 2 MB hugepage per index. Fixed by pinning the allocator (5b9542b).
2. Run 2 (1835 kB/idx, unchanged): the pin was theatre - tikv-jemalloc-sys builds with --with-jemalloc-prefix=_rjem_, so it reads _RJEM_MALLOC_CONF, not MALLOC_CONF. Proven by invalid-pair probe; both vars now set, plus an invalid-conf abort and a kernel-side AnonHugePages evidence line (e82bdfd). Run 3 confirmed the pin live: 143 MB total, AHP 4 MB.
3. Run 3 (209.9 kB/idx, -0.7% short): real per-index cost at the 4-vCPU runner shape. Seven committed calibration runs (N=0/150/300/450 at 32-vCPU, N=0/150/450 at taskset -c 0-3, plus the runner logs) show the healthy band straddles the literal 204.8 line: 182-195 kB/idx on many-core hosts, 205-210 on 4-vCPU, true N-slope marginal cost 206-227 everywhere. Per the issue's own verification clause ("coarse thresholds; the point is catching O(N) regressions, not +/-1%"), the CI gate is now 256 kB (1.24x worst healthy marginal; rc.70 floor 3.4x, THP artifact 7.2x) while every run prints the 204.8 product line with its met/over percentage so the gap stays visible (75b90f7).

The measurement anomalies that fell out (4-vCPU costs more per index than 32-vCPU; a ~10-12 MB knee between N=150 and N=300; empty-node intercept above the loaded-arm extrapolation) are filed as #1024 with the evidence - not buried.