fix(#1019): _delete_by_query pages the whole match set, ids-only

[xerj-org]

What

Fixes #1019. Two defects, both in the by-query plumbing:

1. Full _source materialization on a path that only needs ids — the delete search hydrated a full Hit { source: Value } per match (10,100 Value trees to read only hit.id); _source: false did not avoid it. This was the 6 GiB sort-stack residual the engine: RSS pinned above the memory watermark after bulk ingest into 1,526 indices — 14.8 GB anonymous memory for 1.2 GB on disk, not released at idle; node refuses every write until restart #950 fix recorded as follow-up headroom.
2. size:10_000 single-shot cap — one call deleted at most 10,000 docs; total reported the truncated page length; batches was hardcoded 1. The autoindex client loops up to 1,000 passes as a workaround (each pass re-paying the full materializing search).

The fix, three layers: an internal SearchRequest::ids_only (serde(skip), hard-false in parse_request — cannot arrive from the wire) honored only when sort is _id/_doc-confined with no fields/highlight/aggs/etc.; keyset pagination (search_after on _id, default scroll_size 1000, ES-style max_docs); and a single-pass arm (Index::matching_ids_sorted) that reads only the #950 id-position maps + version map and never runs a search at all.

Root cause, two defects, both present in the HTTP runner
(es_compat.rs run_delete_by_query) and the engine loop
(Index::delete_by_query):

1. ONE search, {"query":…,"size":10000,"from":0}, then delete whatever
   came back. Beyond 10 000 matching docs the call silently under-deleted
   while reporting success: total was the page length, batches was
   hardcoded 1. The autoindex client works around it by looping up to
   1 000 delete passes, each re-paying the search (esclient.rs:1658-1669).
2. That search built a full Hit { source: Value } per match — 10 100
   hydrated Value trees per call to read nothing but hit.id.
   _source: false does not avoid this: the engine keeps the raw source
   for fields/highlight resolution, and the unsorted scan Value-parses
   every stored document anyway. This was the residual RSS transient
   engine: RSS pinned above the memory watermark after bulk ingest into 1,526 indices — 14.8 GB anonymous memory for 1.2 GB on disk, not released at idle; node refuses every write until restart #950 recorded as follow-up headroom.

Fail-before (both red on main, verbatim):

engine/crates/xerj-api/tests/delete_by_query_purges_entire_index.rs
delete_by_query_purges_past_ten_thousand_in_one_call
total must be the exact match count, not the page length:
{"total":10000,"deleted":10000,"batches":1,…}
left: Number(10000) right: Number(25000)
delete_by_query_honors_max_docs
with max_docs set, total is the processed count
left: Number(20) right: Number(7)
delete_by_query_reports_real_batch_count
10 docs at scroll_size 3 = 4 batches
left: Number(1) right: Number(4)
delete_by_query_paginates_selective_queries
20 matches at scroll_size 4 = 5 batches
left: Number(1) right: Number(5)
test result: FAILED. 0 passed; 4 failed

engine/crates/xerj-engine/tests/integration.rs
test_delete_by_query_purges_past_ten_thousand
every matching doc deleted, not just the first 10k page
left: 10000 right: 12000
test result: FAILED. 1 passed; 1 failed

Design.

• Internal ids-only projection. SearchRequest::ids_only
  (xerj-query/src/ast.rs) is serde(skip) and hard-set false in
  parse_request, so it can never arrive from the wire (same contract as
  savings). search_inner honours it only when the request provably
  needs nothing beyond ids: sort non-empty and confined to _id/_doc,
  no fields/script_fields/highlight/explain/collapse/rescore/
  aggs/min_score. Every other shape runs exactly as before. At the
  three admission sites (stored-section scan + both hydration arms) the
  stored _id is extracted by a borrowed prefix scan
  (extract_stored_id_str; escape-bearing layouts fall back to the full
  parse, which still admits source-Null) and the hit carries
  source: Value::Null with the scorer skipped. The version-map liveness
  filter is kept verbatim. A precomputed cursor (id, ascending) declines
  docs at/below search_after with one memcmp BEFORE the liveness lookup
  and every allocation.

• Keyset paged arm for arbitrary queries, modeled on the _reindex loop:
  flush the member first (paging and the id collection below are only
  correct over on-disk segments), then pull scroll_size pages sorted
  _id: asc with a search_after cursor. ES semantics: scroll_size
  default 1 000 clamped 1..=10_000, max_docs truncates processing and
  total then reports the processed count, batches is the real page
  count, total otherwise the exact page-1 match count,
  noops/version_conflicts 0, script resource failures fail-closed
  per batch before any id of that batch is deleted.

• Single-pass arm for match_all/ids selectors. Measurement-driven:
  the paged arm alone re-scans every stored section once per page —
  O(N x pages). On the 252k-doc corpus the default-params purge measured
  43.5 s, 28.9 s after the cursor fast-decline; still quadratic, because
  every page must consider every doc. Index::matching_ids_sorted now
  collects the whole live match set in ONE pass over engine: RSS pinned above the memory watermark after bulk ingest into 1,526 indices — 14.8 GB anonymous memory for 1.2 GB on disk, not released at idle; node refuses every write until restart #950's cached
  id_pos_map_for id maps plus the version map (one String per live
  match, ~48 B), sorts it, and the runner deletes it in scroll_size
  batches. Observable semantics unchanged (batches = ceil(n/scroll_size));
  like ES's by-query, the match set is the start-of-run snapshot and a
  concurrently-vanished id is tolerated. Any other query shape (or a
  segment without a complete id index) takes the paged arm; None falls
  back, never errors.

Reference-code retrieval, honestly: xc.py was run this session and
FAILED for this task's domains — the quickwit and elasticsearch corpora
are not in the sandbox index, and building them was outside this task's
hard constraints (the shared corpus server/index must not be touched).
The design is XERJ's own. The one adapted approach — resolving document
ids from an id map instead of the document store — was recorded at
commit d1e8056 (#950): meilisearch external_documents_ids()
(crates/meilisearch/src/routes/indexes/documents.rs:2284, MIT, approach
only) and quickwit warm_up_terms (quickwit-search/src/leaf.rs:475,
Apache-2.0); those file:line references are from the prior session and
approximate. #1019 builds on that #950 machinery. No Elasticsearch (AGPL/
SSPL/Elastic) or sonic (GPL) code was read for or copied into this change;
ES was consulted only for wire semantics (scroll_size, max_docs,
total, batches).

Measurement — synthetic corpus, this session, private ports + throwaway
data dirs, wall via curl time_total, RSS via /proc//status sampled
at 250 ms: 252 000 docs / 74.5 MB NDJSON ingested into one index
(16 segments). before = main 62fa237 release binary, after = this
branch's release binary.

single _delete_by_query, default params (match_all):
before wall 0.54 s deleted 10 000 / 252 000 (defect 1)
peak RSS 1 234 216 kB VmHWM 1 256 268 kB
before, client workaround loop until _count==0
wall 3.78 s deleted 252 000
peak RSS 1 285 416 kB VmHWM 1 354 104 kB
after wall 1.10 s deleted 252 000 / 252 000
total 252 000 batches 252 failures []
peak RSS 466 100 kB VmHWM 471 764 kB

3.4x faster than the pre-fix workaround loop AND complete, with ~62%
lower peak RSS. Intermediates on the same corpus (all this session):
ids-only + keyset paging alone 43.5 s default / 9.1 s at scroll 10k;

• cursor fast-decline 28.9 s / 7.4 s; + single-pass arm 1.10 s. The
  ~900 MB RSS transient common to both before runs is NOT the delete
  search itself: a single plain wire-level sorted search returned having
  added ~220 MB and grew to HWM 1.21 GB three seconds AFTER the response
  — background post-search merge/cache hydration over the 16-segment
  corpus (the engine: RSS pinned above the memory watermark after bulk ingest into 1,526 indices — 14.8 GB anonymous memory for 1.2 GB on disk, not released at idle; node refuses every write until restart #950 follow-up area, out of delete_by_query materializes every hit _source and caps at size:10_000 — 6 GiB sort stacks, ~30-min search phase on 252k docs #1019's scope). The single-pass
  arm never runs that search, so its peak is the delete path's own.

Gates (this branch, final code): cargo fmt --check clean; clippy
-p xerj-engine -p xerj-api -p xerj-query --all-targets clean; full
xerj-engine suite green; full xerj-api suite green (68 test binaries);
ES-YAML conformance suite 1378 passed / 0 failed / 3 skipped against a
private port with a throwaway data dir (main: 1376/0/3; +2 new
delete_by_query cases). New coverage: 4 HTTP-level tests, 1 engine
integration test, 1 engine lib test pinning source: Null + gap-free
dupe-free keyset pages, 2 ES-YAML cases, all fail-before on main.

Fixes #1019.

---

Independent verification (second agent, nothing trusted from the implementer): fail-before reproduced by reverting the four fix files to main (API 0/4 passed — total 10000 vs 25000; engine 10000 vs 12000); all new tests green on the branch; ES-YAML 1378 passed / 0 failed / 3 skipped (main is 1376/0/3 — the +2 are new delete_by_query pinning cases), run against a release binary rebuilt from the committed tree and md5-verified identical to the measured one; fmt + clippy clean, zero warnings; wire-level check: 15,000-doc _bulk then one default _delete_by_query purged all 15,000 in 118 ms.

Adversarial diff review confirmed: no _source on the delete path (pinned by test: source: Value::Null + gap-free/dupe-free keyset pages), pagination runs to exhaustion with a 10M backstop, error paths fail closed (malformed query 400s before any deletion; failed flush aborts; script-limit refusal checked per batch before that batch's deletes), and the one deliberate O(N) point (match set as Vec<String>, ~12 MB at 252k) is documented and bounded.

Environmental note: re-running the ES-YAML gate on this sandbox needed flood_stage=99% — disk sits at 95%, tripping the flood-stage write block by design. Not a branch defect.