perf: stop the banded search running out of memory MAPPINGS, and make pooling a byte copy

CHANGELOG.md

@@ -59,6 +59,45 @@ than a number. Both are recorded in every run's `manifest.json`.
 
 ### Changed
 
+- **The engine holds far fewer heap blocks, because that, not memory, is what a banded
+  search runs out of.** A grouped search of a 203M-precursor library died at about 290 GB
+  resident with 1.7 TB free, reporting a failed 3 KB allocation. Sampled on the live
+  process at 180 GB it held 129,393 memory mappings of the kernel's 1,048,576 per-process
+  limit, 92,030 of them 64-256 KB and 36,368 of them 256 KB-1 MB: one heap block per item,
+  which mimalloc maps individually. Six subsystems now use flat buffers where they used a
+  block per row: the library's `label` column is read as one bit per row rather than one
+  `String` (203M blocks), compete groups PSMs through one sorted vector rather than a
+  `Vec` per competition group, features holds one buffer rather than a `Vec<f64>` per PSM
+  and dense ids rather than a `HashSet` per peptidoform, quant reads chromatograms into
+  flat buffers with an axis store rather than three blocks per row, and rescoring's
+  per-fold training matrix is one buffer rather than one per training row. Rescore also
+  stops materialising the engine's own feature matrix when a sidecar classifier will read
+  the handoff parquet instead, which at experiment scale is about 250 GB that need not
+  exist. Artifacts are unchanged except where noted below.
+- **Pooling a grouped run's bands is a byte copy.** The rows are already in the pooled
+  table's order and already encoded, so `pool` splices the bands' parquet column chunks
+  into the output and decodes only the row groups whose `candidate_id` statistics say they
+  hold a candidate the overlap dedup drops. Measured on one run of the production
+  experiment, 63 bands and 136 GB: 3 minutes 12 seconds at 2.6 GB resident, against
+  3.5 MB/s at 110% CPU before, which had reached 42.7 GB in 2 hours 50 minutes when it was
+  stopped, on a disk that reads 221 MB/s. The pooled tables hold the same rows in the same
+  order with the same values; their row groups are the bands' own, so their content hashes
+  differ from a re-encoded pool's.
+- **A grouped run no longer pools `psms_extracted`.** Its only reader is the candidate
+  audit, so it is pooled when `extract.emit_candidate_audit` is set and left per band
+  otherwise. The band tables are written either way.
+- **`groups.parallel` is clamped to one less than the thread count.** A band in flight
+  parks a rayon worker on its extraction's accumulation channel, so as many bands as
+  threads leaves no worker to feed them and the run deadlocks with no error and no output
+  (reproduced on the fixture at `parallel = 2, --threads 2`). A larger value now warns and
+  uses `threads - 1`.
+- **`psms_extracted.parquet` and `run_windows.parquet` change content hash.** `write_table`
+  now encodes in 65,536-row chunks instead of building one Arrow copy of the whole table,
+  and on a column that is entirely NULL -- `apex_im` and the three ion-mobility columns
+  always are -- the definition levels are run-encoded, which moves the writer's internal
+  mini-batch size and so the page framing. Measured at 196,615 rows: 16 bytes, with every
+  row, value and row-group boundary unchanged.
+
 - **Library writers emit fragment tables sorted by `candidate_id`.** `import_diann_lib.py`,
   `make_reverse_decoys.py` and `make_shift_decoys.py` finish with a streaming bucket sort
   (`_lib_io.sort_fragments_by_candidate`: partition by candidate-id range into temporary
@@ -71,6 +110,9 @@ than a number. Both are recorded in every run's `manifest.json`.
   filtered scan, with a warning.
 ### Added
 
+- **`mumdia pool` pools a grouped run's band artifacts from the command line.** `run` does
+  this itself at the end of a grouped search; standalone it is for the case where the
+  search finished and the run did not, which now costs a pool rather than a re-search.
 - **`groups.window_groups` searches a run one isolation-window group at a time.** The
   run's windows are cut, in ascending m/z, into contiguous groups whose library bands hold
   about the same number of precursors (planned from the precursor table's row-group

THIRD_PARTY_LICENSES.md

@@ -9,7 +9,7 @@ accompanies it in every release archive and in the container image.
 
 ## Obligations
 
-177 third-party crates. Every one declares an SPDX expression; none is
+184 third-party crates. Every one declares an SPDX expression; none is
 unspecified.
 
 The following copyleft identifiers appear, in each case as one arm of a
@@ -24,17 +24,17 @@ Licence identifiers by crate count:
 
 | SPDX identifier | crates |
 |---|---|
-| `MIT` | 151 |
-| `Apache-2.0` | 146 |
+| `MIT` | 158 |
+| `Apache-2.0` | 152 |
 | `Apache-2.0 WITH LLVM-exception` | 4 |
 | `Unlicense` | 4 |
 | `CC0-1.0` | 3 |
 | `BSD-2-Clause` | 2 |
 | `BSD-3-Clause` | 2 |
+| `LGPL-2.1-or-later` | 2 |
 | `Zlib` | 2 |
 | `0BSD` | 1 |
 | `BSL-1.0` | 1 |
-| `LGPL-2.1-or-later` | 1 |
 | `MIT-0` | 1 |
 | `Unicode-3.0` | 1 |
 
@@ -112,6 +112,7 @@ Licence identifiers by crate count:
 | `futures-util` | 0.3.32 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-lang/futures-rs](https://github.com/rust-lang/futures-rs) |
 | `getrandom` | 0.2.17 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-random/getrandom](https://github.com/rust-random/getrandom) |
 | `getrandom` | 0.3.4 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-random/getrandom](https://github.com/rust-random/getrandom) |
+| `getrandom` | 0.4.3 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-random/getrandom](https://github.com/rust-random/getrandom) |
 | `half` | 2.7.1 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/VoidStarKat/half-rs](https://github.com/VoidStarKat/half-rs) |
 | `hashbrown` | 0.17.1 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-lang/hashbrown](https://github.com/rust-lang/hashbrown) |
 | `heck` | 0.5.0 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/withoutboats/heck](https://github.com/withoutboats/heck) |
@@ -123,6 +124,7 @@ Licence identifiers by crate count:
 | `indexmap` | 2.14.0 | Apache-2.0 OR MIT | `Apache-2.0` | [https://github.com/indexmap-rs/indexmap](https://github.com/indexmap-rs/indexmap) |
 | `is_terminal_polyfill` | 1.70.2 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/polyfill-rs/is_terminal_polyfill](https://github.com/polyfill-rs/is_terminal_polyfill) |
 | `itoa` | 1.0.18 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/dtolnay/itoa](https://github.com/dtolnay/itoa) |
+| `jobserver` | 0.1.35 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-lang/jobserver-rs](https://github.com/rust-lang/jobserver-rs) |
 | `js-sys` | 0.3.103 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys](https://github.com/wasm-bindgen/wasm-bindgen/tree/master/crates/js-sys) |
 | `lazy_static` | 1.5.0 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-lang-nursery/lazy-static.rs](https://github.com/rust-lang-nursery/lazy-static.rs) |
 | `lexical-core` | 1.0.6 | MIT/Apache-2.0 | `Apache-2.0` | [https://github.com/Alexhuszagh/rust-lexical](https://github.com/Alexhuszagh/rust-lexical) |
@@ -155,10 +157,12 @@ Licence identifiers by crate count:
 | `outref` | 0.5.2 | MIT | - | [https://github.com/Nugine/outref](https://github.com/Nugine/outref) |
 | `parquet` | 59.3.0 | Apache-2.0 | - | [https://github.com/apache/arrow-rs](https://github.com/apache/arrow-rs) |
 | `pin-project-lite` | 0.2.17 | Apache-2.0 OR MIT | `Apache-2.0` | [https://github.com/taiki-e/pin-project-lite](https://github.com/taiki-e/pin-project-lite) |
+| `pkg-config` | 0.3.34 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-lang/pkg-config-rs](https://github.com/rust-lang/pkg-config-rs) |
 | `proc-macro2` | 1.0.106 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/dtolnay/proc-macro2](https://github.com/dtolnay/proc-macro2) |
 | `quick-xml` | 0.41.0 | MIT | - | [https://github.com/tafia/quick-xml](https://github.com/tafia/quick-xml) |
 | `quote` | 1.0.46 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/dtolnay/quote](https://github.com/dtolnay/quote) |
 | `r-efi` | 5.3.0 | MIT OR Apache-2.0 OR LGPL-2.1-or-later | `Apache-2.0` | [https://github.com/r-efi/r-efi](https://github.com/r-efi/r-efi) |
+| `r-efi` | 6.0.0 | MIT OR Apache-2.0 OR LGPL-2.1-or-later | `Apache-2.0` | [https://github.com/r-efi/r-efi](https://github.com/r-efi/r-efi) |
 | `rayon` | 1.12.0 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rayon-rs/rayon](https://github.com/rayon-rs/rayon) |
 | `rayon-core` | 1.13.0 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rayon-rs/rayon](https://github.com/rayon-rs/rayon) |
 | `regex` | 1.12.4 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/rust-lang/regex](https://github.com/rust-lang/regex) |
@@ -219,6 +223,9 @@ Licence identifiers by crate count:
 | `zerocopy` | 0.8.52 | BSD-2-Clause OR Apache-2.0 OR MIT | `Apache-2.0` | [https://github.com/google/zerocopy](https://github.com/google/zerocopy) |
 | `zerocopy-derive` | 0.8.52 | BSD-2-Clause OR Apache-2.0 OR MIT | `Apache-2.0` | [https://github.com/google/zerocopy](https://github.com/google/zerocopy) |
 | `zmij` | 1.0.21 | MIT | - | [https://github.com/dtolnay/zmij](https://github.com/dtolnay/zmij) |
+| `zstd` | 0.13.3 | MIT | - | [https://github.com/gyscos/zstd-rs](https://github.com/gyscos/zstd-rs) |
+| `zstd-safe` | 7.2.4 | MIT OR Apache-2.0 | `Apache-2.0` | [https://github.com/gyscos/zstd-rs](https://github.com/gyscos/zstd-rs) |
+| `zstd-sys` | 2.0.16+zstd.1.5.7 | MIT/Apache-2.0 | `Apache-2.0` | [https://github.com/gyscos/zstd-rs](https://github.com/gyscos/zstd-rs) |
 
 ## Licence texts
 
@@ -330,7 +337,7 @@ licence text alone does not discharge the obligation: MIT requires the copyright
 notice in all copies, BSD requires it retained, and Apache-2.0 section 4(d)
 requires the contents of a NOTICE file to be propagated.
 
-Notices recovered for 126 of 177 crates (the remainder ship no copyright line in a licence file; see their repositories, linked above).
+Notices recovered for 133 of 184 crates (the remainder ship no copyright line in a licence file; see their repositories, linked above).
 
 **adler2 2.0.1**
 
@@ -551,6 +558,11 @@ Notices recovered for 126 of 177 crates (the remainder ship no copyright line in
 - Copyright (c) 2018-2025 The rust-random Project Developers
 - Copyright (c) 2014 The Rust Project Developers
 
+**getrandom 0.4.3**
+
+- Copyright (c) 2018-2026 The rust-random Project Developers
+- Copyright (c) 2014 The Rust Project Developers
+
 **hashbrown 0.17.1**
 
 - Copyright (c) 2016 Amanieu d'Antras
@@ -591,6 +603,10 @@ Notices recovered for 126 of 177 crates (the remainder ship no copyright line in
 
 - Copyright (c) Individual contributors
 
+**jobserver 0.1.35**
+
+- Copyright (c) 2014 Alex Crichton
+
 **js-sys 0.3.103**
 
 - Copyright (c) 2014 Alex Crichton
@@ -701,6 +717,10 @@ Notices recovered for 126 of 177 crates (the remainder ship no copyright line in
 
 - Copyright (c) 2022 Nugine
 
+**pkg-config 0.3.34**
+
+- Copyright (c) 2014 Alex Crichton
+
 **quick-xml 0.41.0**
 
 - Copyright (c) 2016 Johann Tuffe
@@ -711,6 +731,12 @@ Notices recovered for 126 of 177 crates (the remainder ship no copyright line in
 - Copyright (C) 2019-2023 Microsoft Corporation
 - Copyright (C) 2022-2023 David Rheinsberg
 
+**r-efi 6.0.0**
+
+- Copyright (C) 2017-2023 Red Hat, Inc.
+- Copyright (C) 2019-2023 Microsoft Corporation
+- Copyright (C) 2022-2023 David Rheinsberg
+
 **rayon 1.12.0**
 
 - Copyright (c) 2010 The Rust Project Developers
@@ -880,6 +906,19 @@ Notices recovered for 126 of 177 crates (the remainder ship no copyright line in
 - Copyright 2023 The Fuchsia Authors
 - Copyright 2019 The Fuchsia Authors.
 
+**zstd 0.13.3**
+
+- Copyright (c) 2016 Alexandre Bury
+
+**zstd-safe 7.2.4**
+
+- Copyright (c) 2016 Alexandre Bury
+
+**zstd-sys 2.0.16+zstd.1.5.7**
+
+- Copyright (c) 2016-present, Facebook, Inc. All rights reserved.
+- Copyright (c) 2016 Alexandre Bury
+
 ### NOTICE files, verbatim
 
 Apache-2.0 section 4(d) requires these to travel with the distribution.

ci/gen_cli_reference.py

@@ -48,6 +48,7 @@
 import os
 import re
 import shutil
+import json
 import subprocess
 import sys
 from pathlib import Path
@@ -88,6 +89,31 @@
 # ---------------------------------------------------------------------------
 
 
+def cargo_target_binaries() -> list[Path]:
+    """`release/mumdia` under the target directory cargo reports, if cargo answers.
+
+    Run from inside the workspace, not with `--manifest-path`: cargo discovers
+    `.cargo/config.toml` by walking up from the CURRENT directory, so from the repository
+    root it reports the in-tree `target/` and misses the redirect this exists to follow.
+    """
+    ws = REPO_ROOT / "rust" / "mumdia"
+    if not shutil.which("cargo") or not ws.is_dir():
+        return []
+    try:
+        out = subprocess.run(
+            ["cargo", "metadata", "--format-version", "1", "--no-deps"],
+            cwd=ws,
+            capture_output=True,
+            text=True,
+            timeout=120,
+            check=True,
+        ).stdout
+        target = Path(json.loads(out)["target_directory"])
+    except (subprocess.SubprocessError, OSError, ValueError, KeyError):
+        return []
+    return [target / "release" / "mumdia", target / "release" / "mumdia.exe"]
+
+
 def find_binary(explicit: str | None) -> Path:
     """Resolve the binary to interrogate, or exit with the search order shown."""
     if explicit:
@@ -101,10 +127,16 @@ def find_binary(explicit: str | None) -> Path:
         if not p.is_file():
             sys.exit(f"error: $MUMDIA_BIN={from_env} is not a file")
         return p
-    for rel in BIN_CANDIDATES:
-        p = Path(rel)
+    # The target directory cargo actually reports comes before the in-tree path. A
+    # redirected `build.target-dir` -- which `.cargo/config.toml.example` documents, to keep
+    # build output out of a synced folder -- leaves `rust/mumdia/target/` holding whatever
+    # was built before the redirect, and this generator then documents a months-old CLI.
+    # That is not hypothetical: it shipped a reference missing a whole subcommand, and CI
+    # caught it only because CI has no such stale directory. `ci/smoke.sh` avoids the same
+    # trap the same way.
+    for p in (cargo_target_binaries() + [Path(rel) for rel in BIN_CANDIDATES if rel]):
         if not p.is_absolute():
-            p = REPO_ROOT / rel
+            p = REPO_ROOT / p
         if p.is_file():
             return p
     on_path = shutil.which(PROG)

configs/config-schema.json

@@ -33,7 +33,7 @@
       "default_text": "0",
       "help": "",
       "gates": [],
-      "source_line": 1921
+      "source_line": 1934
     },
     {
       "path": "convert.thermo_raw_parser",
@@ -2446,6 +2446,19 @@
         "global",
         "per_group"
       ]
+    },
+    {
+      "path": "groups.parallel",
+      "name": "parallel",
+      "section": "groups",
+      "rust_type": "usize",
+      "kind": "integer",
+      "optional": false,
+      "default": 1,
+      "default_text": "1",
+      "help": "Bands searched at the same time inside one run. `1` (the default) is one band at a time, which is what bounds the memory: each band in flight holds its own extraction working set, so the peak is this many bands' worth. Raise it to fill a large machine, after checking one band's peak RSS: on a 203M-precursor library at 63 bands the largest band took 39 GB and the median far less. Results do not depend on it; bands are independent and their artifacts are pooled in band order either way. It must stay below the thread count: a band in flight parks one worker on its accumulation channel, so as many bands as there are threads leaves nothing to do the probing and the run deadlocks. A larger value is clamped to `threads - 1` with a warning rather than hanging.",
+      "gates": [],
+      "source_line": 1917
     }
   ],
   "list_sections": [

docs/23_cli_reference.md

@@ -46,6 +46,7 @@ Commands:
   sub-library     Subset a library to a set of candidates (a first pass's survivors, or prescan's), renumbered to the contiguous 0..n the fragment index requires
   extract         Targeted 3D extraction (peak-major cascade) -> psms_extracted, chromatograms
   features        Compute the minimal feature set -> features.parquet + PIN
+  pool            Pool a grouped run's band artifacts into the run-level tables
   compete         Keep the best candidate per competition group -> psms_competed.parquet
   rescore         Rescore + native target-decoy q-values -> psms_scored.parquet
   quant           Quantify identified peptides + roll up to protein groups
@@ -139,6 +140,7 @@ first sentence of the description, with the full text in the section below.
 | [`sub-library`](#sub-library) | yes | Subset a library to a set of candidates (a first pass's survivors, or prescan's), renumbered to the contiguous 0..n the fragment index requires |
 | [`extract`](#extract) | yes | Targeted 3D extraction (peak-major cascade) -> psms_extracted, chromatograms |
 | [`features`](#features) | yes | Compute the minimal feature set -> features.parquet + PIN |
+| [`pool`](#pool) | no | Pool a grouped run's band artifacts into the run-level tables |
 | [`compete`](#compete) | yes | Keep the best candidate per competition group -> psms_competed.parquet |
 | [`rescore`](#rescore) | yes | Rescore + native target-decoy q-values -> psms_scored.parquet |
 | [`quant`](#quant) | yes | Quantify identified peptides + roll up to protein groups |
@@ -157,8 +159,8 @@ first sentence of the description, with the full text in the section below.
 20 of the 23 documented subcommands accept `--config`:
  `align`, `compete`, `convert`, `digest`, `doctor`, `extract`, `features`, `mbr`, `peak-census`, `peptidoforms`, `predict-frag`, `prescan`, `quant`, `report`, `rescore`, `rt-im-train`, `run`, `run-experiment`, `search-seed`, `sub-library`.
 
-3 do not, so every setting they use comes from their own flags:
- `audit`, `inspect`, `quant-lfq`.
+4 do not, so every setting they use comes from their own flags:
+ `audit`, `inspect`, `pool`, `quant-lfq`.
 
 ## convert
 
@@ -408,6 +410,28 @@ Options:
 
 Plus the 5 repeated flags removed above: see "Global flags".
 
+## pool
+
+```text
+Pool a grouped run's band artifacts into the run-level tables.
+
+`run` does this itself at the end of a grouped search (`groups.window_groups`). Standalone it is for the case where the search finished and the run did not: the band directories hold everything, and pooling them is a byte copy of their parquet row groups, so a killed run costs a pool rather than a re-search.
+
+Usage: mumdia pool [OPTIONS] --groups-dir <GROUPS_DIR>
+
+Options:
+      --groups-dir <GROUPS_DIR>
+          The run's `groups/` directory, holding the `gNN/` band directories
+
+      --out-dir <OUT_DIR>
+          Where the pooled tables go. Default: the parent of `--groups-dir`, which is where a run writes them
+
+      --psms
+          Also pool `psms_extracted`, which only the candidate audit reads
+```
+
+Plus the 5 repeated flags removed above: see "Global flags".
+
 ## compete
 
 ```text