[adiologydev]

Problem

Container history reads are slow on real servers. The four container tables (container_{cpu,memory,disk,network}_usage) are rowid tables keyed on (time, container_id), with a separate (container_id, time) index. To read one container's history, SQLite finds the rows in that index and then does one random table lookup per row. On a copy of a live DB (49 containers, about 1.2M rows per table, on disk) with Sentinel's 2 MB page cache, that is 190–220 ms per read. On the live 1.0.2 agent, the same calls took 0.4–0.95 s. Each row is also stored 3 times: the table, the PK index and the secondary index.

Change

• The 4 container tables are now WITHOUT ROWID, with PRIMARY KEY (container_id, time) and an index on time. A history read is one range scan. The time index serves retention cleanup, downsampling, and the latest-cycle read of container_disk_usage. "Latest row per container" uses the primary key.
• Existing databases are rebuilt once at startup in Store::open (schema::cluster_container_tables):
  • One transaction and one WAL checkpoint per table, so the extra disk needed is about one rebuilt table (~150 MB on the test DB), not all four.
  • The old table is read in rowid order. ORDER BY container_id, time made the copy 7× slower (one random lookup per row). The time index is built after the copy (about 20% faster).
  • The rebuild is detected per table (pragma_table_list.wr), so an interrupted rebuild rolls back and runs again on the next start. It runs before schema::apply, so the new time indexes are not built on old tables first.
  • The startup API already answers /api/health while the store opens, so the rebuild does not fail health checks.
• New databases and the legacy Go migration create the new layout directly.

Benchmark

Full results: docs/benchmark-results/2026-10-24-container-tables-clustered.md.

	Before	After
/api/container/{id}/cpu/history (avg / p95)	193 / 204 ms	7.2 / 8.3 ms
/api/container/{id}/memory/history	216 / 228 ms	11.3 / 12.5 ms
/api/container/{id}/network/history	197 / 208 ms	8.3 / 9.6 ms
/api/containers/current	1.4 / 2.9 ms	1.0 / 1.3 ms
collector insert cycle (49 rows)	9.2 ms	9.0 ms
daily downsample / cleanup	1173 / 827 ms	1247 / 904 ms
live data in the DB	661 MB	377 MB
one-time rebuild (661 MB DB)	—	21.5 s (cpu 6.8 s, memory 7.8 s, network 6.4 s, disk 0.2 s)

Downsample and cleanup are 6–9% slower, because they now go through the time index. They run once a day.

Risks

• The first start after the upgrade takes longer. It is about 21 s for 1.2M rows per table on this box, and longer with more containers or slower disks. During it, /api/health answers, and the metric endpoints answer like they do during the existing legacy migration.
• Disk space. The rebuild needs free space for about one table. If it fails (for example SQLITE_FULL), its transaction rolls back, and Store::open then does what it does for any open failure: it moves the file aside and starts fresh. The file does not shrink after the rebuild (no VACUUM at startup, same as the legacy migration). The freed pages are reused for new rows.
• Merge note: this PR and perf: faster metric history reads (bucket_ms + clustered container tables) #55 both add a row to docs/benchmark-results/README.md, so whichever merges second has a one-line conflict. There is no code overlap.

Tests

• New: clusters_rowid_container_tables_preserving_rows. It starts from the old typed layout. After open, all 4 tables are WITHOUT ROWID, rows and per-container history are the same, the old index is gone, and a second open changes nothing.
• The existing legacy-migration, retention/downsample and store tests pass on the new layout.
• cargo fmt --check, cargo clippy --workspace --all-targets [--features traffic] -- -D warnings, cargo test --workspace [--features traffic]: all pass (626 / 683 tests).

How to test

1. cargo test -p store
2. Copy an existing metrics.sqlite from a server into ./db/, then run SENTINEL_DEVELOPMENT=true TOKEN=t COLLECTOR_ENABLED=false cargo run --release. You should see one container table clustered on (container_id, time) log line per table, and /api/health should answer 200 during the rebuild.
3. sqlite3 db/metrics.sqlite "SELECT name, wr FROM pragma_table_list WHERE name LIKE 'container_%'" should show wr = 1 for the 4 time-series tables.
4. sentinel-bench latency --base http://127.0.0.1:8888 --token t should show container history in single-digit to low double-digit ms.

[adiologydev]

Merged into #55, so all metric history speedups are in one PR.