diff --git a/.github/dependabot.yml b/.github/dependabot.yml index 642d6cfc..5ad5fd8a 100644 --- a/.github/dependabot.yml +++ b/.github/dependabot.yml @@ -30,6 +30,22 @@ updates: - minor - patch + # The desktop application is a separate Cargo workspace with its own lockfile + # (desktop/Cargo.lock), which the entry above does not read (docs/29, work + # package C). Tauri and its toolkit crates are updated within their major version. + - package-ecosystem: cargo + directory: /desktop + schedule: + interval: monthly + open-pull-requests-limit: 3 + groups: + desktop-minor-and-patch: + patterns: + - "*" + update-types: + - minor + - patch + # Base images, so the digest pins in the Dockerfile stay current instead of # freezing the image on a base that stops receiving security updates. Pinning # without an update path is how a reproducible image becomes an unpatched one. diff --git a/.github/workflows/ci.yml b/.github/workflows/ci.yml index e69752b8..19036b3c 100644 --- a/.github/workflows/ci.yml +++ b/.github/workflows/ci.yml @@ -95,6 +95,17 @@ jobs: - name: Audit run: cargo audit --deny unsound --deny yanked + # The desktop application has its own lockfile, which the step above does not + # read (docs/29, work package C). One advisory is ignored deliberately: + # RUSTSEC-2024-0429, an unsound iterator in glib 0.18's `VariantStrIter`. Tauri + # 2's Linux toolkit is gtk 0.18, which pins glib 0.18; the fix is glib 0.20; and + # the application never touches a GVariant. Drop the ignore when Tauri moves to + # gtk4 / glib 0.20. + - name: Audit the desktop lockfile + run: >- + cargo audit --file ../../desktop/Cargo.lock --deny unsound --deny yanked + --ignore RUSTSEC-2024-0429 + build-test: name: build + test (${{ matrix.os }}) runs-on: ${{ matrix.os }} diff --git a/CHANGELOG.md b/CHANGELOG.md index a1c7a714..9ae66cfb 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -32,8 +32,35 @@ than a number. Both are recorded in every run's `manifest.json`. in the top 5,000; fragment charges 79% / 21%), `HCD2021` with charge-2 fragments only from precursor charge 3 14,412, the DIA-NN library 21,856. +- `peptides.tsv` and `proteins.tsv` (single-run and experiment-wide) carry + `is_transferred` and `transfer_q`, the acceptance basis of a match-between-runs row: + a transferred row keeps its grouped q next to the transfer q it was accepted at, a + tighter report threshold does not revoke a transfer that passed `mbr.q_transfer`, and a + protein group admitted through a transferred row carries the flag. The MBR worker's + augmented scored table gains `transfer_q` for it (docs/29 #19). +- `experiment_manifest.json` records the resolved `config_json` next to its hash, the + `model_identities` that produced the artifacts (RT source, fragment predictor, the + classifier that actually ran, feature schema, MBR strategy), the configured and + effective `quant.q_filter`, and input hashes taken at the start of the run rather than + at its end (docs/29 #15). +- Dependabot covers the desktop application's Cargo dependencies (`/desktop`), and CI + audits `desktop/Cargo.lock` with `cargo audit` next to the engine's lockfile, with one + documented ignore (RUSTSEC-2024-0429: glib 0.18 through Tauri 2's gtk 0.18). + ### Changed +- Model identities carry the installed predictor versions: `deeplc-4.1.1-base`, + `deeplc-4.1.1-finetuned`, `ms2pip-4.2.0-HCDch2` in the library report and the manifests, + in place of the family labels `deeplc-4.0-mt` and `ms2pip-` (docs/30). +- With nothing to transfer, the MBR worker writes a transfer table with its ten columns and + zero rows, and the requested augmented scored table with every row unflagged, instead of + a one-column placeholder and no scored table (docs/30). +- The candidate-audit rejection code `NO_PEAK_GROUP` is `DID_NOT_SURVIVE_EXTRACTION` + (`RejectionReason::DidNotSurviveExtraction`). The audit assigns it to every candidate + with no extracted row, and `extract` does not write the per-candidate table that would + separate presence, matched-fraction and gate failures, so the old name claimed a cause + the audit cannot see (docs/29 #16). The audit table has no versioned schema; the + metrics JSON gains `q_unit`. - MS2PIP 4.2.0 in every shipped environment (`env/docker-rescore.yml`, `env/console-ms2pip-requirements.txt`), and `env/mumdia-deeplc.yml` now carries `ms2pip==4.2.0` too, so one host environment serves DeepLC, MS2PIP and the `nn_torch` @@ -65,6 +92,207 @@ than a number. Both are recorded in every run's `manifest.json`. ### Fixed +- Code review A, data integrity (`docs/29_code_review_2026-09-07.md`, findings 1, 2, 4, + 9, 11, 17, 18, 21): + - quant's refusal of a pooled scored table read `source` as i32 while rescore writes + it as u32, treated the type error as "no such column", and so never ran on the + engine's own output; a pooled table quantified against one run's chromatograms + produced one identical row per run. The column is now read in its declared type and + a present column of another type is an error (#1). + - The streamed library loader checked fragment `mz` and `predicted_intensity` for + finiteness on the physical Arrow buffers, which ignore the validity bitmap, and then + turned NULL cells into NaN, NULL names into `""` and a NULL `candidate_id` into + candidate 0. Every required fragment column now rejects NULLs before its values are + read, through the same contract the typed getters enforce, with a fixture per + column (#2). + - `AtomicPath` removed the destination before renaming, so a failed publication had + already destroyed the previous artifact and readers saw a window with no file; two + writers for one destination in one process shared a temporary name. The rename + replaces in place on every platform, the temporary name carries a counter, and the + failure case is tested (#4). + - N-terminal methionine excision was skipped whenever the Met-retained peptide fell + outside the length window, so an N-terminal peptide of `max_len + 1` residues yielded + nothing although its excised form was in range. Both forms are judged on their own + length (#9). + - `rescore.max_feature_matrix_gib` was checked after the matrix had been filled, + against an estimate of the old `Vec>` layout, so it could neither prevent the + allocation nor describe it; it is now checked from the parquet footers and the + selected feature count before allocation, on the flat f32 layout, with checked + arithmetic (#11). + - A candidate DeepLC or MS2PIP returned nothing for received a substitute (iRT 0.0, or + the native intensities under an MS2PIP model identity). It is now dropped together + with its paired decoy or target, the counts are in the library report and a warning, + and a worker id that was not requested or appears twice is an error (#17). + - Numeric configuration domains are validated at load: thresholds and fractions within + their unit interval, positive multipliers and widths, ordered `min_len <= max_len` and + `charge_min <= charge_max`, counts at least one, with documented zero meanings kept + (#18). `quant.q_threshold = -0.1`, `rt_im_train.rt_window_multiplier = -1.0` and + `rescore.train_margin_frac = 2.0` were accepted before. + - `ci/gen_config_reference.py` and `ci/check_workflows.py` scan the files git tracks + rather than everything on disk, so scratch copies beside the sources no longer enter + the generated reference or the workflow check (#21). +- Code review B, workers (`docs/29`, findings 3, 6, 7, 8, 12, 20): + - The entrapment worker skipped a training fold whose training side held one class + and then scored that fold's held-out rows with the final model, trained on those + very rows, so in-sample scores entered the entrapment FDR. A single-class training + fold is now an error that names the condition; the final model scores the decoys + only (#3). + - The MBR worker printed an "empirical decoy fraction" over accepted transfers, a + population that cannot contain a decoy, and computed the transfer q as + `null / targets`, which is exactly 0 for any pool no permuted residual undercuts, so + a three-candidate pool was accepted whole at 1%. The q uses the engine's `+1` + pseudocount and the summary names the permuted-null draws inside the accepted window + instead (#6, #7). + - With `extract.retain_top_peaks` above 1 the MBR worker measured the transfer on the + last competed peak of a candidate, not the one rescore selected and quant integrates; + it now joins `selected_peak_rank` and falls back to the highest `prelim_score` peak + (#8). + - `augment_library.py` gave every added precursor a fresh `base_peptide_id`, so an + added charge state or modform of an existing peptide left its peptide's competition + group and fold; added forms of existing sequences keep the imported id (#12). + - `bench/feature_selection/fs_lib.py` hashed the peptide with its `DECOY_` prefix for + fold assignment, splitting pairs; it hashes the base sequence, and every benchmark + row records the code revision, fold rule, feature count, seed and training recipe + (#20). +- Code review C, desktop and output ownership (`docs/29`, findings 5, 13, 14): + - A repeated Start in the desktop application could launch a second engine into the + same results folder: the start flow had several awaits and no in-progress guard, and + the backend launched every request. A Start is now refused while one is in progress + or while the run the interface follows is still running, and the backend reserves a + run's results folder (by canonical path) before spawning the engine and releases it + when the run's end is published, so a request for an active folder is refused with + the owning run named (#5). + - `run-experiment --run-names` compared names case-sensitively, so `RunA` and `runa` + passed and addressed one directory on Windows, macOS and most network shares. Names + that differ only in case are rejected on every platform (#5). + - Desktop preflight asked the engine about converters without the request's + configuration, so a converter named in `convert.thermo_raw_parser` or + `convert.msconvert` was reported missing and the search refused, and it required + ThermoRawFileParser for Thermo `.raw` even when msconvert, the engine's own fallback + for a parser left at `auto`, was present. The probe now carries the configuration and + the verdict follows the engine's rule: only msconvert present runs, with a note; an + explicitly configured parser that is missing blocks, as it errors in the engine (#13). + - A cancelled desktop run could be published as failed. `cancel` and the process + waiter both wrote the terminal status and whichever ran second won, while the + cancellation flag was written and never read. The waiter is now the only writer: it + reads the intent after reaping the engine and publishes `cancelled`, `done` when the + engine had already finished, or `failed`; until then the run shows "Stopping" (#14). +- Code review F, whole-repository review (`docs/31_code_review_2026-09-08_full.md`, + F1 to F10): + - `prescan` read the infinite-bounds sentinel that `rt-im-train` writes for "calibration + unavailable, search the whole gradient" as "cannot be screened" and dropped the + candidate, so a run with no confident seeds discarded the entire library and exited 0 + with a zero-row survivors table. An unbounded window now screens over the whole + gradient, a candidate with no window row is treated the same, both are counted, and + screening every candidate away is an error (F1). + - A present-but-wrong-typed `is_transferred` was swallowed as "no transfers", silently + removing every match-between-runs identification from `peptides.tsv` and + `proteins.tsv` while the parquet still carried them. Present columns are read in their + declared type and a mismatch is an error; only an absent column falls back (F2). + - `sidecar::resolve_script` tried the working directory before the directory beside the + binary, the ordering `python::resolve_script_dir` was hardened against, so a `scripts/` + directory inside an untrusted dataset could have its worker executed. An absolute + directory is taken as given, then the executable's directory, then `/scripts`, and + the working directory last (F3). + - `Loess::predict` indexed before the start of its grid for a non-finite query, so one + library row with a null `predicted_irt` could abort or misread memory at rt-im-train. + It returns NaN, and rt-im-train treats a non-finite library iRT as "no calibrated RT" + and counts those rows (F4). + - The rescorer's in-memory TSV backend standardised with median/IQR while the parquet and + streaming backends used mean/std, so the same pool scored differently depending on + `rescore.handoff` and on the 4 GB streaming threshold. All three use mean/std, which + leaves the shipped parquet default and every published benchmark unchanged. The + `MUMDIA_NN_FOLD_KEYS` companion is length-checked instead of being sliced short, which + used to leave the tail rows unscored at a fabricated mid-rank score, and the estimate + that picks the backend counts feature columns by name (F5). + - `refuse_output_over_input` was wired into two of eighteen stages, so + `compete --features f.parquet --out f.parquet` replaced the widest artifact of the run + with the competed subset at exit 0. It now guards every output of `search-seed`, + `rt-im-train`, `extract`, `features`, `compete`, `rescore`, `quant` and `audit` (F6). + - The LOESS boundary extrapolation slope introduced in the previous package was the + pointwise local slope at the sparsest, most one-sided point of the fit: unbounded, free + to be negative, and multiplying an unbounded distance. It is the secant of the fitted + curve over its end decile, clamped non-negative and to at most four times the global + slope, and the test uses noisy anchors rather than a noiseless quadratic (F7). + Measured on HYE B01 against the previous behaviour, same library and settings: 48,533 + stripped peptides at 1%, 53,127 PSM-q 1% targets, 6,519 protein groups and 1,961,800 + extracted rows in both arms, identical to the row. The two extrapolations agree + wherever the anchors are dense and differ only outside the anchor range. + - A desktop stop arriving between the reap and the end of `publish_exit` could pass a + recycled process id to the tree kill. The waiter retires the id the instant `wait` + returns, before it reads the output directory (F8). + - The conversion lock added in the previous package spun without pause on an undeletable + stale lock, mistook clock skew and a peer's partial file for evidence about its own + holder, could be held by two processes at once, and left every interrupted conversion's + partial mzML behind for ever. Take-overs are bounded and paced, the holder is + identified by a token it reads back, a future modification time counts as fresh, the + partial-file probe matches this destination only, and abandoned partials are swept + under the lock (F9). + - Dropping an unpredicted candidate with everything sharing its pair key also removed + positional isomers that predicted correctly, bounded only by the library being emptied. + The direct misses and the collateral are counted separately and exceeding 2% of the + library is an error naming the sidecar. The key stays position-free deliberately: a + positional key would stop matching a reverse decoy to its target, trading a sensitivity + defect for an FDR one (F10). +- Code review E, follow-up (`docs/30_code_review_2026-09-08.md`, R1 to R9): + - Enabling DeepLC fine-tuning with its own defaults was rejected at load, because the + documented automatic batch size is `finetune_batch = 0` and the new validation demanded + a positive batch. Only the epoch count has a lower bound now (R1; a regression from + review A). + - `run-experiment --run-names` accepted `a` and `a.`, one directory on Windows, and the + second run overwrote the first with exit 0. Names ending in a dot or a space, containing + `<>:"|?*` or a control character, or naming a Windows reserved device are rejected on + every platform before anything is written (R2). + - A desktop stop could sweep temporary files that belonged to the next run in the same + folder: cancellation swept after the reservation had been released, and a stop on a + finished run swept as well. Cancellation is now intent and kill only and inert once the + run is terminal; the sweep happens in the waiter, after the reap and before the release, + and a stop still killing finishes before the folder changes hands (R3). + - Two searches converting the same vendor file concurrently shared one temporary output + and one could publish the other's bytes. Each conversion writes a unique partial file + under a lock beside the destination; a concurrent converter waits and reuses the + result (R4). + - Domain checks for the numeric settings review A left unchecked: `mbr.q_anchor`, + `min_anchor_runs`, `extract.min_matched_fraction`, `features.bound_peak_fraction`, + `quant.reliable_q` and the remaining fractions, correlations, tolerances and counts (R5). + - The DeepLC fine-tune and re-prediction worker zipped predictions with peptidoforms + without checking the count and silently kept the imported iRT for anything missing. A + count mismatch is an error; rows that keep their imported value are counted in + `.summary.json` and the engine warns when there are any (R6). + - The audit's `reported` flag repeated the precursor gate, so it could read `true` next to + `FAILED_PEPTIDE_FDR`, and a decoy could be `REPORTED`; the flag now follows the reason, + a decoy past both gates is `REMOVED_DURING_REPORTING`, and a present `precursor_q` of the + wrong type is an error rather than a fallback (R7). + - The desktop results-folder reservation compared exact folders only, so a search into a + child of an active experiment's folder was allowed; ancestors and descendants are + refused, siblings are not (R8). + - The Windows debug binary overflowed its 1 MiB main-thread stack on `--version`; the CLI + runs on a thread with a 256 MiB reservation and an integration test runs the built + binary (R9). +- Code review D, calibration, provenance, reporting (`docs/29`, findings 10, 15, 16, 19): + - LOESS retention-time calibration switched to the global least-squares line the + moment a query left the anchor range, while the grid just inside used the local fit, + and the two need not agree: on `y = 200 + 10x^2` (span 0.3) the prediction jumped + from 193.4 at `x = 1e-6` to 38.3 at `x = 0`, and from 1173.5 to 1018.4 at the top, + about 155 s discontinuities that misplaced gradient-edge peptides relative to their + extraction window. The map now continues the boundary local fit (its value and + slope) outside the range, and is continuous at both ends; the global line remains + only the degenerate fallback (#10). Measured on HYE B01 with the imported DIA-NN iRT + as the RT source and `native_tda`: 45,946 stripped peptides at `peptide_q_value` 1% + before, 45,957 after, at an unchanged 1.0% PSM-level decoy fraction and 6,410 protein + groups in both arms. 208,130 of 10.88 M candidates (1.9%) received a different window, + 194,698 of them with iRT above the anchor range, which the global line had placed + past the end of the 9,000 s run; the local fit places them at 8,578 to 9,100 s, and + extract accepted 454 more rows from them. A second pair on the DeepLC 4.1.1 + re-predicted precursor table (`w_rt` 414 s against 691 s): 48,533 stripped peptides in + both arms, PSM-q 1% targets 53,124 against 53,127 at the same decoy fraction, 6,519 + protein groups in both, 0.2% of candidates with a different window. Neutral on both RT + sources, which is what a boundary correction should be. + - The candidate audit's `passed_precursor_fdr` gate and `FAILED_PRECURSOR_FDR` reason + read the PSM `q_value`; they read `precursor_q`, the unit the label names, with the + PSM q as a recorded fallback on tables without it. A pooled scored table (several + `source` values) is refused, because the audit keys on `candidate_id` and would + attribute the last run's fate to every run (#16). - Desktop: the digest fields on the Search screen (missed cleavages, peptide length, charge range, carbamidomethyl, oxidation) now reach the engine on the built-in library path. They were read only by the DIA-NN library build, so with the built-in diff --git a/CLAUDE.md b/CLAUDE.md index 269396c2..d01520a4 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -137,8 +137,12 @@ Key semantics: eliminate a target against its decoy. Peptide-level q estimation subsequently performs picked target-decoy competition through the shared `base_peptide_id`; keep that pairing intact. -- `retain_top_peaks > 1` currently writes diagnostic peak alternatives only. - Those alternatives do not yet become feature/rescore rows. +- `extract.retain_top_peaks > 1` (default 1) writes the alternative peaks as + additional `psms_extracted` rows with `peak_rank >= 1` (plus a diagnostic + `.peaks.parquet`), `features` carries `peak_rank`, `compete` keys on it, and + `rescore` keeps one row per candidate and records `selected_peak_rank`. The + plumbing exists; what the default still lacks is entrapment validation on two + acquisitions. ## Validated sensitivity workflow @@ -203,9 +207,9 @@ The mechanism is peak-group formation rather than scoring: with most peaks gone, group. That reading comes from the cap dose-response, not from the audit ladder's own -label. `NO_PEAK_GROUP` cannot be used as evidence for it: `audit.rs` reads a +label. `DID_NOT_SURVIVE_EXTRACTION` cannot be used as evidence for it: `audit.rs` reads a per-candidate audit table that `extract` does not write (`emit_candidate_audit` -is unwired), so the reason map is always empty and the `_ => NoPeakGroup` +is unwired), so the reason map is always empty and the `_ => DidNotSurviveExtraction` catch-all absorbs presence failures, matched-fraction failures AND every extraction-gate rejection alike. Treat the label as "did not survive extract", and do not decompose it further until the audit table is actually produced. @@ -361,14 +365,15 @@ fine-tuning also is not guaranteed deterministic. single pooled run do not produce identical `q_value` columns. Batch to fit RAM, and compare per-run counts on `run_psm_q`. - Pooled rescore scales linearly, measured 0.834 ms/PSM on the streaming - backend. Two feature matrices, two widths: the Python worker's is - `n_psms x n_features x 4` bytes (f32), while the Rust `feats` that `rescore` - builds is `Vec>`, so `n_psms x n_features x 8` plus a heap allocation - and 24 bytes of spine per PSM. `native_tda` additionally runs all folds in + backend. Two feature matrices, one width: the Python worker's is + `n_psms x n_features x 4` bytes (f32), and the Rust `FeatureMatrix` that + `rescore` builds (`rescoring.rs`) is flat f32 as well, so the same + `n_psms x n_features x 4`. `native_tda` additionally runs all folds in parallel, each holding an owned standardised copy of its training slice, so its - peak is roughly `(1 + folds)x` the matrix. The stage logs the figure before it - allocates, and `rescore.max_feature_matrix_gib` turns exceeding a ceiling into - an error at startup rather than an OS kill hours in. + peak is roughly `(1 + folds)x` the matrix. `rescore.max_feature_matrix_gib` is + checked against that layout, from the parquet footers and the selected feature + count, before the allocation (docs/29 #11), so exceeding the ceiling is an error + at startup rather than an OS kill hours in. ### Rescore cost: handoff, feature selection, training-set reduction @@ -554,7 +559,8 @@ sensitivity result for it. Do not enable these by default from a single AIF count: -- model-visible top-K peaks (currently diagnostic sidecar only); +- model-visible top-K peaks (`extract.retain_top_peaks > 1`; implemented through + features, compete and rescore, default 1); - adaptive RT windows; - held-out RT window sizing (`rt_im_train.window_holdout_frac`). Implemented and measured on the AIF benchmark: +1.1% peptides with DeepLC 4.1.0 at unchanged @@ -581,8 +587,9 @@ and FDR population: the entrapment pool, HYE and AIF of docs/28 all ran under it The selected apex was historically correct/strongest only about 48-52% of the time while the correct peak appeared in the top five about 86-88%. Promoting top-K alternatives through features/rescore is therefore the best plausible -sensitivity project, but it needs a `candidate_id + peak_rank` contract and -entrapment validation before default activation. +sensitivity project. The `candidate_id + peak_rank` contract exists (`peak_rank` +on every extracted row, `selected_peak_rank` on the scored row, and the MBR worker +joins it); what default activation still needs is the entrapment validation. ## Coding conventions diff --git a/bench/feature_selection/fs_lib.py b/bench/feature_selection/fs_lib.py index cf656527..6a512f12 100644 --- a/bench/feature_selection/fs_lib.py +++ b/bench/feature_selection/fs_lib.py @@ -13,6 +13,7 @@ import hashlib import os import re +import subprocess import time os.environ.setdefault("KMP_DUPLICATE_LIB_OK", "TRUE") @@ -27,6 +28,38 @@ # train_fdr 0.01) are passed to the worker as MUMDIA_NN_FOLDS / ITERS / TRAIN_FDR # (rescore.rs), overriding the worker's own docstring default of 5 iterations. The measured # HYE run trained 10 iterations per fold and never hit the churn stop. +def recipe_metadata(cfg, n_features_used, seed): + """What a benchmark row was produced with, recorded beside it (docs/29 #20). + + A study evaluates a re-implementation of the worker, so the reader needs the code + revision, the fold rule, the feature count, the seed and the training settings to + compare it with what production later ran. Remaining differences from the worker + (its explicit base-peptide fold keys, its standardisation backend) are stated, not + hidden behind "faithful". + """ + here = os.path.dirname(os.path.abspath(__file__)) + try: + sha = subprocess.run( + ["git", "rev-parse", "--short", "HEAD"], cwd=here, capture_output=True, text=True, check=True + ).stdout.strip() + except (OSError, subprocess.CalledProcessError): + sha = "unknown" + c = {**WORKER_DEFAULTS, **(cfg or {})} + training = {k: (list(c[k]) if isinstance(c[k], tuple) else c[k]) for k in ( + "iters", "epochs", "hidden", "dropout", "lr", "wd", "batch", "train_fdr", + "neg_ratio", "neg_select", "margin_frac", "train_sub")} + return { + "code_sha": sha, + "folds": c["folds"], + "fold_key": "md5(base sequence, DECOY_ prefix stripped) % folds", + "fold_key_differs_from_worker": "the worker folds on explicit base_peptide_id pairs when the PIN carries them", + "n_features_used": int(n_features_used), + "seed": int(seed), + "preprocessing": "in-memory backend: median/IQR standardisation; worker's streaming backend uses mean/std", + "training": training, + } + + WORKER_DEFAULTS = dict( folds=3, iters=10, @@ -92,8 +125,11 @@ def load_pin( y = (tb.column("Label").to_numpy() == 1).astype(np.float32) cids = np.array([int(x.rsplit("_", 1)[-1]) for x in tb.column("SpecId").to_pylist()], np.int64) peps = [strip_pep(p) for p in tb.column("Peptide").to_pylist()] - pep_hash = np.array([int(hashlib.md5(p.encode()).hexdigest(), 16) for p in peps], dtype=object) base_seq = np.array([p[6:] if p.startswith("DECOY_") else p for p in peps], dtype=object) + # The fold key hashes the base sequence, so a target and its paired decoy share a + # fold as they do in the worker's explicit pairing. Until 2026-09-07 the hash was + # taken before the `DECOY_` prefix came off, and the pair could split (docs/29 #20). + pep_hash = np.array([int(hashlib.md5(p.encode()).hexdigest(), 16) for p in base_seq], dtype=object) tb_prot = tb.column("Proteins").to_pylist() if entrapment else None del tb n, nf = len(y), len(feat_cols) diff --git a/bench/feature_selection/fs_objective.py b/bench/feature_selection/fs_objective.py index 12121456..098b69d2 100644 --- a/bench/feature_selection/fs_objective.py +++ b/bench/feature_selection/fs_objective.py @@ -78,7 +78,8 @@ def main(): for sd in seeds: t0 = time.time() row, r = fs_lib.bench_subset(name, d, cols, cfg={**cfg, "seed_base": sd}) - row.update(dataset=args.tag, seed=sd, rows=int(len(d["y"])), when=time.strftime("%Y-%m-%d %H:%M:%S")) + row.update(dataset=args.tag, seed=sd, rows=int(len(d["y"])), when=time.strftime("%Y-%m-%d %H:%M:%S"), + recipe=json.dumps(fs_lib.recipe_metadata({**cfg, "seed_base": sd}, len(cols), sd), sort_keys=True)) pd.DataFrame([row]).to_csv(out_csv, mode="a", header=not os.path.exists(out_csv), index=False) if args.save_oof and name == args.save_oof: np.save(os.path.join(args.out_dir, f"{args.tag}_oof_{name}_seed{sd}.npy"), r["score"].astype(np.float32)) diff --git a/bench/feature_selection/fs_train_sweep.py b/bench/feature_selection/fs_train_sweep.py index 4ee66b17..746c3e6c 100644 --- a/bench/feature_selection/fs_train_sweep.py +++ b/bench/feature_selection/fs_train_sweep.py @@ -157,6 +157,9 @@ def main(): rows=int(len(d["y"])), n_features_used=len(cols) if cols else len(d["feat_cols"]), cfg=json.dumps(over, sort_keys=True), + recipe=json.dumps(fs_lib.recipe_metadata({**over, "seed_base": sd}, + len(cols) if cols else len(d["feat_cols"]), sd), + sort_keys=True), when=time.strftime("%Y-%m-%d %H:%M:%S"), ) pd.DataFrame([row]).to_csv(out_csv, mode="a", header=not os.path.exists(out_csv), index=False) diff --git a/ci/check_smoke.py b/ci/check_smoke.py index 340ff4ea..eb73301a 100644 --- a/ci/check_smoke.py +++ b/ci/check_smoke.py @@ -425,10 +425,13 @@ def hashes(path): "run-experiment writes the experiment-wide peptides.tsv and proteins.tsv") if rep_pep.is_file(): hdr = rep_pep.read_text(encoding="utf-8").splitlines()[0].split("\t") + # The two trailing columns are the acceptance basis of a match-between-runs + # row (docs/29 #19); without MBR they read `false` and empty. c.ok(hdr == ["precursor", "stripped_sequence", "charge", "protein", "q_value", - "score", "n_runs", "quantity_a", "quantity_b"], - "experiment peptides.tsv has the experiment-wide columns and one quantity " - "column per run", "\t".join(hdr)) + "score", "n_runs", "quantity_a", "quantity_b", "is_transferred", + "transfer_q"], + "experiment peptides.tsv has the experiment-wide columns, one quantity " + "column per run and the transfer basis", "\t".join(hdr)) # Rows are asserted on the standalone rewrite at q 0.05 (smoke.sh): the pooled # peptide-level q of this fixture cannot reach 1 percent, so the root pair is # legitimately header-only at the default threshold. @@ -444,8 +447,10 @@ def hashes(path): "identical inputs: every precursor is quantified in both runs") if rep_prot.is_file(): hdr = rep_prot.read_text(encoding="utf-8").splitlines()[0].split("\t") - c.ok(hdr == ["protein_group", "q_value", "n_runs", "lfq_a", "lfq_b"], - "experiment proteins.tsv has one LFQ column per run", "\t".join(hdr)) + c.ok(hdr == ["protein_group", "q_value", "n_runs", "lfq_a", "lfq_b", + "is_transferred", "transfer_q"], + "experiment proteins.tsv has one LFQ column per run and the transfer " + "basis", "\t".join(hdr)) for r in ("a", "b"): c.ok(not (exp / r / "peptides.tsv").exists(), f"no per-run peptides.tsv under {r}: the grouped q is experiment-wide") diff --git a/ci/check_workflows.py b/ci/check_workflows.py index c5374600..16360bca 100644 --- a/ci/check_workflows.py +++ b/ci/check_workflows.py @@ -27,6 +27,7 @@ class of error that reached the remote. from __future__ import annotations +import subprocess import sys from pathlib import Path @@ -61,8 +62,33 @@ def _no_duplicates(loader: StrictLoader, node, deep: bool = False) -> dict: ) +def workflow_files() -> list[Path]: + """The tracked workflow files; the directory glob only where git cannot answer. + + An untracked scratch copy beside the real workflows was checked as if it shipped + (docs/29 #21); the tracked set is what CI runs. + """ + try: + out = subprocess.run( + ["git", "ls-files", "-z", "--", str(WORKFLOWS)], + cwd=WORKFLOWS.parents[1], + capture_output=True, + check=True, + ).stdout + files = sorted( + WORKFLOWS.parents[1] / rel + for rel in out.decode("utf-8").split("\0") + if rel.endswith((".yml", ".yaml")) + ) + if files: + return files + except (OSError, subprocess.CalledProcessError): + pass + return sorted(WORKFLOWS.glob("*.yml")) + sorted(WORKFLOWS.glob("*.yaml")) + + def main() -> int: - files = sorted(WORKFLOWS.glob("*.yml")) + sorted(WORKFLOWS.glob("*.yaml")) + files = workflow_files() if not files: print(f"no workflows found under {WORKFLOWS}", file=sys.stderr) return 1 diff --git a/ci/gen_config_reference.py b/ci/gen_config_reference.py index fda28a13..caa1e295 100644 --- a/ci/gen_config_reference.py +++ b/ci/gen_config_reference.py @@ -40,6 +40,7 @@ import difflib import json import re +import subprocess import sys from pathlib import Path @@ -723,6 +724,35 @@ def reachable_enums( # --------------------------------------------------------------------------- +def tracked_files(root: Path, suffix: str, fallback, recursive: bool = True) -> list[Path]: + """Files under `root` with `suffix` that git tracks; `fallback()` when git cannot say. + + `git ls-files` is asked for `root` and the answer is filtered, so the input set of + this generator is the tracked source and nothing a developer left beside it. + """ + try: + out = subprocess.run( + ["git", "ls-files", "-z", "--", str(root)], + cwd=REPO_ROOT, + capture_output=True, + check=True, + ).stdout + except (OSError, subprocess.CalledProcessError): + return fallback() + files = [] + for rel in out.decode("utf-8").split("\0"): + if not rel or not rel.endswith(suffix): + continue + path = REPO_ROOT / rel + if not recursive and path.parent != root: + continue + if path.is_file(): + files.append(path) + if not files: + return fallback() + return sorted(files) + + def parse_profiles(text: str) -> dict[str, list[tuple[str, str]]]: """Extract `--profile NAME` overrides from `Config::apply_profile`.""" m = re.search(r"pub fn apply_profile\(.*?\n \}\n", text, re.S) @@ -1120,10 +1150,23 @@ def build_document() -> tuple[str, dict[str, object]]: if not (REPO_ROOT / target).is_file(): sys.exit(f"error: ITEM_STRUCT_DOC['{name}'] points at missing {target}") - rust_files = sorted( - p for p in CRATES_DIR.rglob("*.rs") if "target" not in p.parts + # Tracked sources only. A filesystem glob also picked up untracked scratch copies + # (`*-covr2.rs`, `*-covr2.py`) that sit beside the real files on a developer's + # machine, and their environment-variable reads then entered this document and + # failed `--check` in a workspace that was fine as far as git was concerned + # (docs/29 #21). Outside a git checkout, the release archive for instance, the + # glob is the only option and the archive holds tracked files only. + rust_files = tracked_files( + CRATES_DIR, + ".rs", + lambda: sorted(p for p in CRATES_DIR.rglob("*.rs") if "target" not in p.parts), + ) + py_files = tracked_files( + SCRIPTS_DIR, + ".py", + lambda: sorted(SCRIPTS_DIR.glob("*.py")), + recursive=False, ) - py_files = sorted(SCRIPTS_DIR.glob("*.py")) rust_reads, rust_sets, env_unresolved = scan_rust_env(rust_files) py_reads, py_sets = scan_python_env(py_files) diff --git a/desktop/README.md b/desktop/README.md index f5eb4eee..ddb66c20 100644 --- a/desktop/README.md +++ b/desktop/README.md @@ -63,6 +63,22 @@ next run would start in a dirty folder. Closing the window cancels every running search, for the same reason. +The terminal state is published in one place. `cancel` records the intent and kills +the tree; the thread that reaps the engine reads that intent and publishes `cancelled`, +`done` (the engine finished before the kill landed, so its outputs are complete) or +`failed`. Until then the status stays `running` with `cancel_requested` set and the +interface shows "Stopping". Two writers used to race here, and a stopped run could be +shown as failed with the last log line as its error (docs/29 #14). + +## Output ownership + +Two engines writing one results folder interleave their artifacts with no error from +either. A run reserves its results folder before the engine is spawned, by canonical +path so that spellings and, on Windows, case name one folder, and releases it when its +end is published; a second Start into an active folder is refused with the owning run +named. The frontend also refuses to start while a Start is in progress or while the run +it follows is still running, because it can show and stop only one (docs/29 #5). + ## How progress works No log parsing. Every engine stage writes `.report.json` beside its output, @@ -321,7 +337,12 @@ Two things worth knowing: they get it during the run rather than before it. - **Preflight blocks a vendor format whose converter is missing**, naming which converter, rather than letting the engine fail after the interface has switched to - the progress screen. + the progress screen. The converters are asked of the engine with the request's own + configuration (`doctor --json --config`), so one named in `convert.thermo_raw_parser` + or `convert.msconvert` counts, and the rule is the engine's: a Thermo `.raw` with the + parser at `auto` and only msconvert present runs, with a note; a parser the + configuration names and that is missing blocks, as it errors in the engine + (docs/29 #13). - **Bruker gets an ion-mobility warning** on the Setup screen and under the picker. MuMDIA's pipeline is 3D, so diaPASEF loses the separation that makes it selective. Saying so is the difference between a user reading a low count as a MuMDIA result diff --git a/desktop/src-tauri/src/main.rs b/desktop/src-tauri/src/main.rs index bc42357b..ebfa59a5 100644 --- a/desktop/src-tauri/src/main.rs +++ b/desktop/src-tauri/src/main.rs @@ -236,45 +236,37 @@ fn preflight( )); } - // A Thermo .raw with no converter fails inside the engine, after the run has - // been launched and the interface has switched to the progress screen. Caught - // here it is a sentence on the screen the user is already looking at. - // Every selected file, not just the first. A mixed selection where only one file - // needs a converter would otherwise pass preflight and fail mid-experiment, after - // the other runs had already been searched. - // Both answers come from the engine's own search (`doctor --json`), not from this - // application's install directory. Asking the narrow question blocked a user whose - // ThermoRawFileParser was on PATH, or named in the configuration, for a file the - // engine converts without complaint -- and the GUI was the only path that refused. - let thermo_missing = thermo::engine_thermo_parser().is_none(); - let msconvert_missing = thermo::msconvert_available().is_none(); - let mut needs_thermo: Vec<&str> = Vec::new(); - let mut needs_msconvert: Vec<&str> = Vec::new(); - for m in &req.mzml { - match thermo::needs(m) { - thermo::Needs::ThermoParser if thermo_missing => needs_thermo.push(m), - thermo::Needs::Msconvert if msconvert_missing => needs_msconvert.push(m), - _ => {} + // A vendor file with no converter fails inside the engine, after the run has been + // launched and the interface has switched to the progress screen. Caught here it is + // a sentence on the screen the user is already looking at, for every selected file, + // so a mixed selection cannot fail mid-experiment after the other runs were + // searched. The answer comes from the engine's own search (`doctor --json`) FOR + // THIS REQUEST'S CONFIGURATION, and the rule that turns it into a blocker or a note + // is the engine's (`thermo::converter_verdict`). + let mut warnings: Vec = Vec::new(); + match thermo::converters(req.config.as_deref()) { + Some(conv) => { + let (b, notes) = thermo::converter_verdict(&req.mzml, &conv); + blockers.extend(b); + warnings.extend(notes); + } + // The engine resolved a moment ago, so this is not "no engine". Refusing on it + // would block a search over a probe failure; the engine reports its own + // conversion errors, so let it be the judge and say the check did not run. + None => { + if req + .mzml + .iter() + .any(|m| thermo::needs(m) != thermo::Needs::Nothing) + { + warnings.push( + "Could not ask the engine which vendor converters it would use; the \ + selected vendor files will be converted when the search starts, or \ + fail there." + .into(), + ); + } } - } - if !needs_thermo.is_empty() { - blockers.push(format!( - "{} selected file(s) are Thermo .raw and the converter is not installed. -Install it on the Setup screen, or convert them to mzML yourself. -First: {}", - needs_thermo.len(), - needs_thermo[0] - )); - } - if !needs_msconvert.is_empty() { - blockers.push(format!( - "{} selected file(s) need ProteoWizard msconvert, which was not found. -MuMDIA does not install it; see the Setup screen. -First: {} ({})", - needs_msconvert.len(), - needs_msconvert[0], - thermo::label(needs_msconvert[0]) - )); } // Room on disk. The engine cannot resume, so filling the volume at hour three @@ -299,7 +291,6 @@ First: {} ({})", // roomy one passed preflight and then filled the acquisition drive mid-conversion, // and the engine cannot resume. let conversion = pf::conversion_space(&req.mzml); - let mut warnings: Vec = Vec::new(); for w in conversion { warnings.push(w); } diff --git a/desktop/src-tauri/src/run.rs b/desktop/src-tauri/src/run.rs index b06088a8..991d582a 100644 --- a/desktop/src-tauri/src/run.rs +++ b/desktop/src-tauri/src/run.rs @@ -38,6 +38,85 @@ const POLL: Duration = Duration::from_millis(700); /// Log lines kept in memory. The pane shows the tail; the full log is on disk. const LOG_TAIL: usize = 4000; +/// The results folders of the runs in flight, by canonical path, each with the id of +/// the run that owns it. +/// +/// Two engines writing one artifact set interleave their output with no error from +/// either (docs/29 #5): a repeated Start launched a second engine into the same folder +/// and the interface kept only the latest run id. A folder is reserved here before the +/// engine is spawned and released when the run's end is published, so while a run is +/// active nothing else can be started into its folder. +static ACTIVE_OUT_DIRS: Mutex> = Mutex::new(BTreeMap::new()); + +/// The identity of a results folder for ownership purposes. +/// +/// The canonical path folds the ways one directory can be spelled: relative against +/// absolute, `.` and `..` segments, symbolic links and, on Windows, case (`Out` and +/// `out` canonicalise to the on-disk spelling). The directory has to exist for that, +/// which is why `start` creates it first. When it cannot be canonicalised the path is +/// used as given, made absolute, so a reservation is still taken. +fn ownership_key(dir: &Path) -> String { + let p = std::fs::canonicalize(dir).unwrap_or_else(|_| { + if dir.is_absolute() { + dir.to_path_buf() + } else { + std::env::current_dir() + .map(|c| c.join(dir)) + .unwrap_or_else(|_| dir.to_path_buf()) + } + }); + let s = p.to_string_lossy().into_owned(); + if cfg!(windows) { + s.to_lowercase() + } else { + s + } +} + +/// Reserve `dir` for run `id`, or say which run already owns it. +/// +/// Returns the key to release with. Held by the `Run`, released by `publish_exit`. +pub fn reserve_out_dir(dir: &Path, id: &str) -> Result { + let key = ownership_key(dir); + let mut active = ACTIVE_OUT_DIRS.lock().unwrap_or_else(|e| e.into_inner()); + // Equal keys, and also one folder inside the other (docs/30 R8): an experiment writes + // into its per-run subfolders and cleanup walks its whole folder, so a search into a + // child of an active experiment, or an experiment over the parent of an active search, + // is an overlapping writer. Component-wise, so `out` and `out2` stay independent. + if let Some((held, owner)) = active + .iter() + .find(|(held, _)| **held == key || paths_nest(held, &key)) + { + let relation = if *held == key { + "is in use".to_string() + } else if Path::new(&key).starts_with(Path::new(held)) { + format!("is inside the results folder {held}, which is in use") + } else { + format!("contains the results folder {held}, which is in use") + }; + return Err(format!( + "the results folder {} {relation} by a search that is still running ({owner}). \ + Wait for it to finish or stop it, or choose another folder: two searches \ + writing one folder tree overwrite each other's results.", + dir.display() + )); + } + active.insert(key.clone(), id.to_string()); + Ok(key) +} + +/// True when one path is an ancestor of the other, by path components. +fn paths_nest(a: &str, b: &str) -> bool { + let (pa, pb) = (Path::new(a), Path::new(b)); + pa.starts_with(pb) || pb.starts_with(pa) +} + +/// Give a reserved results folder back. +pub fn release_out_dir(key: &str) { + let mut active = ACTIVE_OUT_DIRS.lock().unwrap_or_else(|e| e.into_inner()); + active.remove(key); +} + /// What the interface asks for when it starts a search. #[derive(Deserialize, Debug, Clone)] pub struct Request { @@ -126,8 +205,12 @@ pub struct Snapshot { /// /// The interface needs this to label the result counts: an experiment-wide /// rescore groups the q columns experiment-wide, so those counts are NOT per - /// file, and `run-experiment` writes no report at all. + /// file, and the `peptides.tsv` / `proteins.tsv` at the experiment root are the + /// experiment-wide report, not a per-run one. pub experiment: bool, + /// Stop was requested. The status stays `running` until the engine has actually + /// been reaped, and the interface shows "Stopping" meanwhile. + pub cancel_requested: bool, } pub struct Run { @@ -136,6 +219,8 @@ pub struct Run { /// it is spawned into a new group. pid: Mutex>, cancelled: AtomicBool, + /// The results-folder reservation, until `publish_exit` releases it. + reservation: Mutex>, } impl Run { @@ -152,26 +237,160 @@ impl Run { .unwrap_or_else(|e| e.into_inner().clone()) } - /// Stop the run: kill the process tree, then remove the rubble. + /// Stop the run: record the intent, kill the process tree, then remove the rubble. /// - /// Both halves matter. The engine spawns Python workers, so killing only the - /// engine would orphan a process that may hold tens of gigabytes. And a hard - /// kill skips destructors, so the atomic-write layer never removes its + /// Both halves of the kill matter. The engine spawns Python workers, so killing + /// only the engine would orphan a process that may hold tens of gigabytes. And a + /// hard kill skips destructors, so the atomic-write layer never removes its /// `.tmp-` files; without a sweep the next run starts in a dirty directory. + /// + /// What this does NOT do is write the terminal status. That is `publish_exit`'s, + /// once the process has been reaped, and it reads the intent recorded here. The + /// flag used to be written and never read while the status was written from here + /// as well, racing the waiter: it could wake from the dying process first and + /// publish `failed`, with the last log line as the "error", and this method then + /// declined to replace a terminal status (docs/29 #14). pub fn cancel(&self) { + // Inert once terminal: there is no process to kill, and the folder may already + // belong to a later run (docs/30 R3). Cleanup is not done here at all any more: + // it belongs to `publish_exit`, which runs after the reap and before the + // reservation is released, so it can only ever touch this run's own files. + if !self.is_active() { + return; + } self.cancelled.store(true, Ordering::SeqCst); - let pid = self.pid.lock().ok().and_then(|p| *p); - if let Some(pid) = pid { + self.set(|s| s.cancel_requested = true); + // The pid lock is held across the kill, and `publish_exit` retires the pid under + // the same lock before it sweeps and releases. A stop still in flight when the + // engine is reaped therefore finishes before the folder changes hands, and a stop + // that arrives after the reap finds no pid. + let guard = self.pid.lock().unwrap_or_else(|e| e.into_inner()); + if let Some(pid) = *guard { kill_tree(pid); } - let out_dir = self.snapshot().out_dir; - sweep_temp_files(Path::new(&out_dir)); - self.set(|s| { - if s.status == "running" || s.status == "starting" { - s.status = "cancelled".into(); + drop(guard); + } + + fn is_active(&self) -> bool { + matches!(self.snapshot().status.as_str(), "running" | "starting") + } + + /// Forget the process id, under the lock a stop takes before it kills. + /// + /// Called the instant `wait` returns, before anything else. The pid is free for the + /// operating system to reuse from that moment, and a stop landing later would + /// otherwise pass it to `kill_tree`, which on Windows terminates whatever now owns it + /// and its whole tree (docs/31 F8). Retiring it inside `publish_exit` was too late: + /// that function scans the output directory and reads the result reports first, so + /// the window was as long as that disk work. It is now a few instructions, and a stop + /// that reaches the lock inside it still finds the pid this run really owns, because + /// the reap has only just returned. + fn retire_pid(&self) { + if let Ok(mut p) = self.pid.lock() { + *p = None; + } + } + + /// Apply `f` only while the run is still active; returns whether it was. + fn set_if_active(&self, f: F) -> bool { + if let Ok(mut s) = self.snapshot.lock() { + if matches!(s.status.as_str(), "running" | "starting") { + f(&mut s); + return true; } + } + false + } + + /// Publish the terminal state of the run from how its process ended. + /// + /// The one place a run becomes terminal, so the outcome does not depend on which + /// thread ran first. Everything a finished run displays, its stages and results, + /// is read from disk BEFORE the status stops being `running`; the other way round + /// leaves a window in which the run says it is finished but has no stages, which an + /// interface polling for completion reliably catches. + /// + /// A process that exited successfully is `done` even under a cancel request: the + /// engine finished before the kill landed and its outputs are complete, and + /// calling them cancelled would hide a finished result. + fn publish_exit(&self, outcome: std::io::Result, out_dir: &Path) { + // Idempotent: the waiter retires the pid the moment `wait` returns (docs/31 F8), + // and this call is what makes `publish_exit` safe to reach from a test or any + // other path. Taking the lock here also waits for a stop that is still killing, + // so nothing below overlaps a kill. + self.retire_pid(); + let cancelled = self.cancelled.load(Ordering::SeqCst); + if cancelled { + // The only sweep: after the reap, before the release, inside this run's + // ownership of the folder. + sweep_temp_files(out_dir); + } + let stages = scan_stages(out_dir); + let results = read_results(out_dir); + let status = terminal_status(cancelled, &outcome); + // Released before the status is published, so a caller that sees the run end + // can start another into the same folder without being refused. + self.release_reservation(); + self.set(|s| { + s.stages = stages; + s.results = results; + s.exit_code = outcome.as_ref().ok().and_then(|st| st.code()); + s.error = match (&outcome, status) { + (_, "done" | "cancelled") => None, + // The last stderr line is almost always the anyhow error chain, which + // is the sentence worth showing. + (Ok(_), _) => s.log.iter().rev().find(|l| !l.trim().is_empty()).cloned(), + (Err(e), _) => Some(format!("could not wait for the engine: {e}")), + }; + s.status = status.into(); }); } + + fn release_reservation(&self) { + let key = self.reservation.lock().ok().and_then(|mut r| r.take()); + if let Some(key) = key { + release_out_dir(&key); + } + } +} + +/// The status a finished process publishes. Pure, so the interleavings of a Stop with +/// the engine's own exit can be pinned down in tests. +fn terminal_status( + cancelled: bool, + outcome: &std::io::Result, +) -> &'static str { + match outcome { + Ok(s) if s.success() => "done", + _ if cancelled => "cancelled", + _ => "failed", + } +} + +/// A run handle in its initial state. Shared by `start` and by the tests, which drive +/// `publish_exit` and `cancel` directly to pin down their interleavings. +fn new_run(id: &str, req: &Request, command: String, reservation: Option) -> Arc { + Arc::new(Run { + snapshot: Mutex::new(Snapshot { + id: id.to_string(), + status: "starting".into(), + exit_code: None, + error: None, + stages: Vec::new(), + log: Vec::new(), + out_dir: req.out_dir.clone(), + command, + started_unix_ms: now_ms(), + elapsed_ms: 0, + results: None, + library_mode: req.lib_precursors.is_some(), + experiment: req.experiment, + cancel_requested: false, + }), + pid: Mutex::new(None), + cancelled: AtomicBool::new(false), + reservation: Mutex::new(reservation), + }) } /// Kill a process and everything it spawned. @@ -433,6 +652,9 @@ pub fn start(id: String, req: Request) -> Result, String> { std::fs::create_dir_all(&req.out_dir) .map_err(|e| format!("cannot create the results folder {}: {e}", req.out_dir))?; + // Taken before the engine exists, so a second request for this folder is refused + // while this one is still being spawned, not only once it runs. + let reservation = reserve_out_dir(Path::new(&req.out_dir), &id)?; let display = format!( "{} {}", @@ -440,27 +662,7 @@ pub fn start(id: String, req: Request) -> Result, String> { args.iter().map(|a| quote(a)).collect::>().join(" ") ); - let library_mode = req.lib_precursors.is_some(); - let experiment = req.experiment; - let run = Arc::new(Run { - snapshot: Mutex::new(Snapshot { - id: id.clone(), - status: "starting".into(), - exit_code: None, - error: None, - stages: Vec::new(), - log: Vec::new(), - out_dir: req.out_dir.clone(), - command: display, - started_unix_ms: now_ms(), - elapsed_ms: 0, - results: None, - library_mode, - experiment, - }), - pid: Mutex::new(None), - cancelled: AtomicBool::new(false), - }); + let run = new_run(&id, &req, display, Some(reservation)); let mut cmd = engine::command(&exe); // Without this the managed Python environment and the managed .raw converter @@ -479,9 +681,13 @@ pub fn start(id: String, req: Request) -> Result, String> { cmd.process_group(0); } - let mut child = cmd - .spawn() - .map_err(|e| format!("could not start {}: {e}", exe.display()))?; + let mut child = match cmd.spawn() { + Ok(c) => c, + Err(e) => { + run.release_reservation(); + return Err(format!("could not start {}: {e}", exe.display())); + } + }; let pid = child.id(); if let Ok(mut p) = run.pid.lock() { @@ -531,15 +737,15 @@ pub fn start(id: String, req: Request) -> Result, String> { let started = Instant::now(); loop { let stages = scan_stages(&out_dir); - let running = { - let s = run.snapshot(); - s.status == "running" || s.status == "starting" - }; - run.set(|s| { + // Written only while the run is still active, under the snapshot lock: + // once `publish_exit` has published, a scan that was in flight must not + // replace the finished snapshot's stages with whatever the folder holds + // now, which may already be a later run's contents (docs/30). + let still_active = run.set_if_active(|s| { s.stages = stages; s.elapsed_ms = started.elapsed().as_millis() as u64; }); - if !running { + if !still_active { // The final scan belongs to the waiter, not here: it has to happen // BEFORE the status becomes terminal, or a caller that polls until // the run is finished can read a snapshot whose stages and results @@ -551,43 +757,15 @@ pub fn start(id: String, req: Request) -> Result, String> { }); } - // Reap the child, then publish the terminal state in one step. - // - // The order matters. Everything a finished run displays -- its stages and its - // results -- is read from disk here, BEFORE the status stops being `running`. - // Doing it the other way round leaves a window in which the run says it is - // finished but has no stages, which an interface polling for completion will - // reliably catch: the results screen renders empty and then fills in. + // Reap the child, then publish the terminal state in one step (`publish_exit`). { let run = Arc::clone(&run); let out_dir = PathBuf::from(&req.out_dir); std::thread::spawn(move || { let outcome = child.wait(); - let stages = scan_stages(&out_dir); - let results = read_results(&out_dir); - match outcome { - Ok(status) => run.set(|s| { - s.stages = stages; - s.results = results; - s.exit_code = status.code(); - if s.status == "cancelled" { - return; - } - if status.success() { - s.status = "done".into(); - } else { - s.status = "failed".into(); - // The last stderr line is almost always the anyhow error - // chain, which is the sentence worth showing. - s.error = s.log.iter().rev().find(|l| !l.trim().is_empty()).cloned(); - } - }), - Err(e) => run.set(|s| { - s.stages = stages; - s.status = "failed".into(); - s.error = Some(format!("could not wait for the engine: {e}")); - }), - } + // Before anything else: the pid is reusable from here (docs/31 F8). + run.retire_pid(); + run.publish_exit(outcome, &out_dir); }); } @@ -738,6 +916,251 @@ mod tests { } } + fn exit_status(code: i32) -> std::process::ExitStatus { + #[cfg(unix)] + { + use std::os::unix::process::ExitStatusExt; + std::process::ExitStatus::from_raw(code << 8) + } + #[cfg(windows)] + { + use std::os::windows::process::ExitStatusExt; + std::process::ExitStatus::from_raw(code as u32) + } + } + + fn scratch(name: &str) -> PathBuf { + let dir = std::env::temp_dir().join(format!("mumdia_run_{}_{name}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + dir + } + + /// A run in the `running` state with no process behind it, so `publish_exit` and + /// `cancel` can be interleaved by hand. + fn running(name: &str) -> (Arc, PathBuf) { + let dir = scratch(name); + let mut r = req(); + r.out_dir = dir.display().to_string(); + let run = new_run(&format!("run-{name}"), &r, "mumdia run ...".into(), None); + run.set(|s| s.status = "running".into()); + (run, dir) + } + + #[test] + fn a_stop_that_lands_before_the_waiter_wakes_is_published_as_cancelled() { + // The interleaving of docs/29 #14: Stop was pressed and the kill issued, and + // the waiter wakes from the dying process before `cancel` could have written + // anything. The waiter used to publish `failed` here, with the last log line + // as the error, and `cancel` then left that terminal status alone. + let (run, dir) = running("cancel_first"); + run.set(|s| s.log.push("thread 'main' panicked".into())); + run.cancelled.store(true, Ordering::SeqCst); + run.publish_exit(Ok(exit_status(1)), &dir); + let s = run.snapshot(); + assert_eq!(s.status, "cancelled"); + assert_eq!(s.error, None, "a stopped run has no error to show"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_run_that_failed_on_its_own_stays_failed_when_stop_arrives_late() { + let (run, dir) = running("failed_first"); + run.set(|s| s.log.push("Error: no such file".into())); + run.publish_exit(Ok(exit_status(1)), &dir); + run.cancel(); + let s = run.snapshot(); + assert_eq!(s.status, "failed"); + assert_eq!(s.error.as_deref(), Some("Error: no such file")); + // A stop that arrives after the end is inert and records nothing (docs/30 R3). + assert!(!s.cancel_requested); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_run_that_finished_before_the_kill_landed_is_done() { + let (run, dir) = running("done_under_cancel"); + run.cancelled.store(true, Ordering::SeqCst); + run.publish_exit(Ok(exit_status(0)), &dir); + assert_eq!(run.snapshot().status, "done"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_wait_error_is_failed_unless_a_stop_was_requested() { + let (run, dir) = running("wait_error"); + run.publish_exit(Err(std::io::Error::other("gone")), &dir); + let s = run.snapshot(); + assert_eq!(s.status, "failed"); + assert!( + s.error.as_deref().unwrap_or("").contains("could not wait"), + "{:?}", + s.error + ); + let _ = std::fs::remove_dir_all(&dir); + + let (run, dir) = running("wait_error_cancelled"); + run.cancelled.store(true, Ordering::SeqCst); + run.publish_exit(Err(std::io::Error::other("gone")), &dir); + assert_eq!(run.snapshot().status, "cancelled"); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_stop_after_the_reap_has_no_pid_to_kill() { + // docs/31 F8: the pid is reusable the moment `wait` returns, so the waiter retires + // it there rather than after reading the output directory. A stop arriving in + // between must find nothing, not a recycled pid. + let (run, dir) = running("reaped"); + *run.pid.lock().unwrap() = Some(4242); + run.retire_pid(); + assert_eq!(*run.pid.lock().unwrap(), None); + // The run is still active, so cancel proceeds and simply has nothing to signal. + run.cancel(); + assert!(run.snapshot().cancel_requested); + assert_eq!(*run.pid.lock().unwrap(), None); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn cancel_records_the_intent_without_publishing_a_terminal_status() { + // Until the engine is reaped the run is still running, whatever the button + // says; `cancel_requested` is what the interface shows "Stopping" from. + let (run, dir) = running("intent_only"); + run.cancel(); + let s = run.snapshot(); + assert_eq!(s.status, "running"); + assert!(s.cancel_requested); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_late_stop_after_the_run_ended_leaves_the_folder_alone() { + // docs/30 R3: A finished and released its folder; B took it and is writing. A + // stop delivered to A must neither sweep B's temporary file nor change A's state. + let (run, dir) = running("late_stop"); + let key = reserve_out_dir(&dir, "run-A").unwrap(); + *run.reservation.lock().unwrap() = Some(key); + run.publish_exit(Ok(exit_status(1)), &dir); + assert_eq!(run.snapshot().status, "failed"); + let key_b = reserve_out_dir(&dir, "run-B").expect("A released its folder"); + let b_file = dir.join("new.parquet.tmp-999-1"); + std::fs::write(&b_file, b"B's partial write").unwrap(); + run.cancel(); + assert!( + b_file.is_file(), + "a late stop must not sweep another run's files" + ); + let s = run.snapshot(); + assert_eq!(s.status, "failed"); + assert!(!s.cancel_requested, "a terminal run records no stop"); + release_out_dir(&key_b); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_stop_still_in_flight_finishes_before_the_folder_is_released() { + // docs/30 R3, the concurrent route: the engine is reaped while the stop thread is + // still inside the kill. Publication must wait for the kill to finish, so the + // reservation cannot be released, and taken by a new run, while the stop is + // still active in that folder. The kill is simulated by holding the pid lock. + let (run, dir) = running("inflight_stop"); + let key = reserve_out_dir(&dir, "run-A").unwrap(); + *run.reservation.lock().unwrap() = Some(key); + run.cancelled.store(true, Ordering::SeqCst); + let killing = run.pid.lock().unwrap(); + let (r2, d2) = (Arc::clone(&run), dir.clone()); + let waiter = std::thread::spawn(move || r2.publish_exit(Ok(exit_status(1)), &d2)); + std::thread::sleep(Duration::from_millis(300)); + assert!( + reserve_out_dir(&dir, "run-B").is_err(), + "the folder must stay reserved while the stop is in flight" + ); + assert_eq!( + run.snapshot().status, + "running", + "nothing is published mid-kill" + ); + drop(killing); + waiter.join().unwrap(); + assert_eq!(run.snapshot().status, "cancelled"); + let k = reserve_out_dir(&dir, "run-B").expect("released once the stop completed"); + release_out_dir(&k); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn overlapping_result_folders_are_refused_in_both_orders_but_siblings_are_not() { + // docs/30 R8: an experiment owns its per-run subfolders and its cleanup walks the + // whole tree, so a parent and a child are one writer. + let parent = scratch("nest"); + let child = parent.join("run1"); + let sibling = std::env::temp_dir().join(format!("mumdia_run_{}_nest2", std::process::id())); + std::fs::create_dir_all(&child).unwrap(); + std::fs::create_dir_all(&sibling).unwrap(); + let k = reserve_out_dir(&parent, "run-1").unwrap(); + let e = reserve_out_dir(&child, "run-2").unwrap_err(); + assert!(e.contains("is inside") && e.contains("run-1"), "{e}"); + let ks = reserve_out_dir(&sibling, "run-3").expect("a sibling is independent"); + release_out_dir(&k); + release_out_dir(&ks); + let kc = reserve_out_dir(&child, "run-2").unwrap(); + let e = reserve_out_dir(&parent, "run-1").unwrap_err(); + assert!(e.contains("contains") && e.contains("run-2"), "{e}"); + release_out_dir(&kc); + let _ = std::fs::remove_dir_all(&parent); + let _ = std::fs::remove_dir_all(&sibling); + } + + #[test] + fn a_results_folder_owned_by_an_active_run_is_refused_to_a_second() { + let dir = scratch("owned"); + let key = reserve_out_dir(&dir, "run-1").unwrap(); + let e = reserve_out_dir(&dir, "run-2").unwrap_err(); + assert!(e.contains("run-1") && e.contains("still running"), "{e}"); + // Another spelling of the same folder is the same folder. + let e2 = reserve_out_dir(&dir.join("."), "run-3").unwrap_err(); + assert!(e2.contains("run-1"), "{e2}"); + release_out_dir(&key); + let key2 = reserve_out_dir(&dir, "run-2").expect("free again once released"); + release_out_dir(&key2); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn a_case_alias_of_an_active_folder_is_refused_where_the_filesystem_folds_case() { + // `..._Case` and `..._case` are one directory on NTFS and APFS and two on ext4. + // The rule follows the filesystem, which is what the canonical path reports: + // the same directory is refused, a different one is free. + let dir = scratch("Case"); + let alias = std::env::temp_dir().join(format!("mumdia_run_{}_case", std::process::id())); + let key = reserve_out_dir(&dir, "run-1").unwrap(); + let same = std::fs::canonicalize(&alias).ok() == std::fs::canonicalize(&dir).ok(); + let second = reserve_out_dir(&alias, "run-2"); + if same { + let e = second.unwrap_err(); + assert!(e.contains("run-1"), "{e}"); + } else { + release_out_dir(&second.expect("a different directory is free")); + } + release_out_dir(&key); + let _ = std::fs::remove_dir_all(&dir); + } + + #[test] + fn publishing_the_end_of_a_run_releases_its_folder() { + let dir = scratch("release"); + let key = reserve_out_dir(&dir, "run-9").unwrap(); + let mut r = req(); + r.out_dir = dir.display().to_string(); + let run = new_run("run-9", &r, String::new(), Some(key)); + run.set(|s| s.status = "running".into()); + assert!(reserve_out_dir(&dir, "run-10").is_err()); + run.publish_exit(Ok(exit_status(0)), &dir); + let k = reserve_out_dir(&dir, "run-10").expect("released when the run ended"); + release_out_dir(&k); + let _ = std::fs::remove_dir_all(&dir); + } + #[test] fn fasta_and_library_together_is_rejected() { let mut r = req(); @@ -1102,10 +1525,10 @@ mod multifile_tests { #[test] fn a_pooled_experiment_reports_its_combined_table_and_says_so() { - // `run-experiment` writes `scored_combined.parquet` and never calls the report - // stage, so reading only `psms_scored.parquet.report.json` left the results - // screen blank after every experiment. And the counts it does yield are - // experiment-wide: the grouped q columns are grouped across the whole + // `run-experiment` writes `scored_combined.parquet` (and an experiment-wide + // TSV pair at the root), so reading only `psms_scored.parquet.report.json` left + // the results screen blank after every experiment. And the counts it does + // yield are experiment-wide: the grouped q columns are grouped across the whole // experiment, so a per-file reading of them is diluted by ~1/n_runs. let dir = std::env::temp_dir().join("mumdia-results-experiment"); let _ = std::fs::remove_dir_all(&dir); @@ -1124,7 +1547,7 @@ mod multifile_tests { assert!(r.experiment_wide, "a combined table is experiment-wide"); assert_eq!(r.peptides_1pct, 7); assert_eq!(r.precursors_1pct, 8); - // And it writes no report, so neither TSV exists. + // This fixture wrote no TSV, so neither is reported present. assert!(!r.has_peptides_tsv && !r.has_proteins_tsv); // A single run's own report wins, and is not labelled experiment-wide. diff --git a/desktop/src-tauri/src/thermo.rs b/desktop/src-tauri/src/thermo.rs index daa50c51..d4156ba3 100644 --- a/desktop/src-tauri/src/thermo.rs +++ b/desktop/src-tauri/src/thermo.rs @@ -131,31 +131,153 @@ pub fn is_raw(path: &str) -> bool { /// directories under Program Files, then `PATH`, and a second implementation here /// would drift from that. pub fn msconvert_available() -> Option { - converter_path("msconvert") + converters(None).and_then(|c| c.msconvert.path) } -/// The Thermo converter the ENGINE would use, which is not the same question as -/// whether this application installed one. -/// -/// `Installer::refresh` only ever looks in `data_dir()/ThermoRawFileParser/`, while -/// the engine's `raw::locate_parser` also accepts an explicit -/// `convert.thermo_raw_parser`, `MUMDIA_THERMO_PARSER`, a binary beside the engine or -/// one on `PATH`. Preflight asked the narrow question and hard-blocked users whose -/// converter the engine would have found perfectly well -- and the GUI was the only -/// path that refused. `doctor --json` already reports both converters from the -/// engine's own search; nothing was reading the `thermo` half of it. -pub fn engine_thermo_parser() -> Option { - converter_path("thermo") +/// One converter as the engine reports it for a configuration: what was configured, +/// what was found, and when nothing was, why. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct Converter { + /// `convert.thermo_raw_parser` or `convert.msconvert` as the engine read it: + /// `auto`, or a path. + pub configured: String, + pub path: Option, + pub detail: Option, +} + +impl Converter { + /// True when the configuration names a converter rather than leaving the search + /// to the engine. The engine treats a wrong explicit path as an error and never as + /// a reason to use a different converter (`raw::ensure_mzml`), because vendor + /// conversion is not reproducible across converters; preflight says the same. + pub fn explicit(&self) -> bool { + !self.configured.is_empty() && self.configured != "auto" + } +} + +/// Both converters, as the engine resolves them for one configuration. +#[derive(Clone, Debug, Default, PartialEq, Eq)] +pub struct Converters { + pub thermo: Converter, + pub msconvert: Converter, } -/// One `doctor --json` probe, shared by both converter questions. -fn converter_path(key: &str) -> Option { +/// Ask the engine which converters a request with this configuration would use. +/// +/// `doctor --json --config `: the engine resolves `convert.thermo_raw_parser` +/// and `convert.msconvert` from that file, its environment (`MUMDIA_THERMO_PARSER`, +/// `MUMDIA_MSCONVERT`), its own directory and `PATH`, so a converter is found here +/// exactly when the run would find it. The probe used to run without the configuration +/// (docs/29 #13): it answered for the defaults, so a converter the configuration named +/// at an off-`PATH` location was reported missing and the search refused, while the +/// engine would have converted without complaint. `None` when the engine could not be +/// asked at all, which is a different situation from "asked, and nothing found". +pub fn converters(config: Option<&str>) -> Option { let (exe, _) = crate::engine::resolve().ok()?; let mut cmd = crate::engine::command(&exe); crate::components::stamp_env(&mut cmd); - let out = cmd.arg("doctor").arg("--json").output().ok()?; + // `doctor` exits non-zero when the configuration's interpreters do not resolve, + // but it prints the report first, and the converter half is what is wanted here. + let out = cmd.args(doctor_args(config)).output().ok()?; let v: serde_json::Value = serde_json::from_slice(&out.stdout).ok()?; - v.get(key)?.get("path")?.as_str().map(|s| s.to_string()) + Some(Converters { + thermo: converter_of(v.get("thermo")), + msconvert: converter_of(v.get("msconvert")), + }) +} + +/// The `doctor` invocation for a configuration, or for the defaults without one. +fn doctor_args(config: Option<&str>) -> Vec { + let mut args = vec!["doctor".to_string(), "--json".to_string()]; + if let Some(c) = config { + args.push("--config".to_string()); + args.push(c.to_string()); + } + args +} + +/// One converter entry of the `doctor --json` report; an absent entry is "nothing". +fn converter_of(v: Option<&serde_json::Value>) -> Converter { + let field = |k: &str| { + v.and_then(|c| c.get(k)) + .and_then(|s| s.as_str()) + .map(|s| s.to_string()) + }; + Converter { + configured: field("configured").unwrap_or_default(), + path: field("path"), + detail: field("detail"), + } +} + +/// What preflight says about the converters the selected files need: hard blockers, +/// and notes worth reading before an hour is spent. +/// +/// The rule is the engine's own (`raw::ensure_mzml`), restated rather than imported +/// because this application spawns the engine and does not link it: +/// +/// - a Thermo `.raw` goes to ThermoRawFileParser when one is found; +/// - when `convert.thermo_raw_parser` is `auto` and none is found, msconvert converts +/// it instead, with a note, because that is what the engine will do; +/// - a parser the configuration names explicitly and that is not found is an error +/// and never a fallback, matching the engine; +/// - every other vendor format needs msconvert. +/// +/// Preflight used to require the Thermo parser specifically, so a machine with only +/// msconvert was refused a search the engine would have run (docs/29 #13). +pub fn converter_verdict(files: &[String], conv: &Converters) -> (Vec, Vec) { + let mut blockers = Vec::new(); + let mut notes = Vec::new(); + let thermo: Vec<&str> = files + .iter() + .filter(|m| needs(m) == Needs::ThermoParser) + .map(|s| s.as_str()) + .collect(); + let other: Vec<&str> = files + .iter() + .filter(|m| needs(m) == Needs::Msconvert) + .map(|s| s.as_str()) + .collect(); + + if let (Some(first), None) = (thermo.first(), &conv.thermo.path) { + let n = thermo.len(); + if conv.thermo.explicit() { + let detail = conv + .thermo + .detail + .as_deref() + .map(|d| format!(": {d}")) + .unwrap_or_default(); + blockers.push(format!( + "{n} selected file(s) are Thermo .raw and the converter named in the \ + configuration was not found (convert.thermo_raw_parser = {}){detail}.\n\ + Fix that path, or set it to \"auto\" to let MuMDIA search for a converter.\n\ + First: {first}", + conv.thermo.configured + )); + } else if let Some(ms) = conv.msconvert.path.as_deref() { + notes.push(format!( + "{n} Thermo .raw file(s) will be converted with ProteoWizard msconvert ({ms}) \ + because ThermoRawFileParser was not found.\nInstall it on the Setup screen to \ + use the licence-free converter instead.\nFirst: {first}" + )); + } else { + blockers.push(format!( + "{n} selected file(s) are Thermo .raw and no converter is installed.\n\ + Install ThermoRawFileParser on the Setup screen, install ProteoWizard \ + msconvert, or convert them to mzML yourself.\nFirst: {first}" + )); + } + } + if let (Some(first), None) = (other.first(), &conv.msconvert.path) { + blockers.push(format!( + "{} selected file(s) need ProteoWizard msconvert, which was not found.\n\ + MuMDIA does not install it; see the Setup screen.\nFirst: {first} ({})", + other.len(), + label(first) + )); + } + (blockers, notes) } /// State of the converter: installed or not, and the last install's progress. @@ -497,6 +619,131 @@ fn unzip(archive: &std::path::Path, target: &PathBuf) -> Result<(), String> { mod tests { use super::*; + fn conv(thermo: (&str, Option<&str>), msconvert: Option<&str>) -> Converters { + Converters { + thermo: Converter { + configured: thermo.0.into(), + path: thermo.1.map(String::from), + detail: thermo + .1 + .is_none() + .then(|| "no ThermoRawFileParser found".to_string()), + }, + msconvert: Converter { + configured: "auto".into(), + path: msconvert.map(String::from), + detail: None, + }, + } + } + + fn files(xs: &[&str]) -> Vec { + xs.iter().map(|s| s.to_string()).collect() + } + + #[test] + fn a_thermo_file_with_only_msconvert_is_allowed_with_a_note() { + // The engine's fallback for a parser left at `auto` (docs/29 #13): preflight + // used to demand the parser and block this. + let (blockers, notes) = converter_verdict( + &files(&["a.raw"]), + &conv(("auto", None), Some("C:/pwiz/msconvert.exe")), + ); + assert!(blockers.is_empty(), "{blockers:?}"); + assert_eq!(notes.len(), 1); + assert!(notes[0].contains("msconvert"), "{}", notes[0]); + } + + #[test] + fn an_explicit_parser_that_is_missing_blocks_even_with_msconvert_present() { + // The engine's rule: a configured path that is wrong is an error, not a reason + // to convert with a program the configuration did not name. + let (blockers, notes) = converter_verdict( + &files(&["a.raw"]), + &conv( + ("D:/tools/ThermoRawFileParser.exe", None), + Some("msconvert"), + ), + ); + assert_eq!(blockers.len(), 1, "{blockers:?}"); + assert!( + blockers[0].contains("D:/tools/ThermoRawFileParser.exe"), + "{}", + blockers[0] + ); + assert!( + blockers[0].contains("no ThermoRawFileParser found"), + "{}", + blockers[0] + ); + assert!(notes.is_empty(), "{notes:?}"); + } + + #[test] + fn converters_the_configuration_names_off_path_are_enough() { + // The probe carries the request's configuration, so a converter found only + // through it arrives with a path, and nothing is blocked. + let mut c = conv( + ( + "D:/tools/ThermoRawFileParser.exe", + Some("D:/tools/ThermoRawFileParser.exe"), + ), + Some("E:/pwiz/msconvert.exe"), + ); + c.msconvert.configured = "E:/pwiz/msconvert.exe".into(); + assert!(c.thermo.explicit() && c.msconvert.explicit()); + let (blockers, notes) = converter_verdict(&files(&["a.raw", "b.d"]), &c); + assert!( + blockers.is_empty() && notes.is_empty(), + "{blockers:?} {notes:?}" + ); + } + + #[test] + fn nothing_installed_blocks_thermo_and_bruker_and_leaves_mzml_alone() { + let (blockers, _) = converter_verdict( + &files(&["a.raw", "b.d", "c.raw"]), + &conv(("auto", None), None), + ); + assert_eq!(blockers.len(), 2, "{blockers:?}"); + assert!( + blockers[0].starts_with("2 selected file(s) are Thermo"), + "{}", + blockers[0] + ); + assert!(blockers[0].contains("msconvert"), "{}", blockers[0]); + assert!(blockers[1].contains("msconvert"), "{}", blockers[1]); + let (b, n) = converter_verdict(&files(&["x.mzML"]), &conv(("auto", None), None)); + assert!(b.is_empty() && n.is_empty()); + } + + #[test] + fn the_probe_carries_the_requests_configuration() { + assert_eq!(doctor_args(None), vec!["doctor", "--json"]); + assert_eq!( + doctor_args(Some("C:/cfg/run.json")), + vec!["doctor", "--json", "--config", "C:/cfg/run.json"] + ); + } + + #[test] + fn the_report_is_read_as_the_engine_writes_it() { + let v = serde_json::json!({ + "thermo": {"status": "none", "configured": "auto", "path": null, + "detail": "no ThermoRawFileParser found"}, + "msconvert": {"status": "ok", "configured": "auto", + "path": "/opt/pwiz/msconvert", "detail": null} + }); + let t = converter_of(v.get("thermo")); + assert_eq!(t.configured, "auto"); + assert!(!t.explicit()); + assert_eq!(t.path, None); + assert_eq!(t.detail.as_deref(), Some("no ThermoRawFileParser found")); + let m = converter_of(v.get("msconvert")); + assert_eq!(m.path.as_deref(), Some("/opt/pwiz/msconvert")); + assert_eq!(converter_of(None), Converter::default()); + } + #[test] fn the_download_is_pinned_to_the_publishers_own_release() { let Some(a) = asset() else { return }; diff --git a/desktop/ui/app.js b/desktop/ui/app.js index ad2b35b3..d777bb6a 100644 --- a/desktop/ui/app.js +++ b/desktop/ui/app.js @@ -58,6 +58,10 @@ const state = { lastStatus: null, outDir: "", componentsReady: false, + // The engine was found at startup. Start is only ever enabled when it was. + engineOk: false, + // A Start is in progress: library build, derived configuration, preflight, launch. + starting: false, setupTimer: null, schema: null, overrides: {}, @@ -128,6 +132,7 @@ async function init() { const info = await invoke("engine_info"); $("engine-line").textContent = `${info.version}\n${info.path}`; $("engine-line").title = `${info.version} — found via ${info.source}\n${info.path}`; + state.engineOk = true; } catch (e) { $("engine-line").textContent = "engine not found"; banner($("engine-error"), String(e)); @@ -1319,7 +1324,35 @@ function renderBatchSummary() { } } +// Start is asynchronous with several awaits before the run exists (library build, +// derived configuration, preflight), and a second click during any of them launched a +// second engine into the same results folder while the interface kept only the latest +// run id (docs/29 #5). This guard sits before the first await and is the first line of +// defence; the backend's reservation of active results folders is the second. async function start() { + if (state.starting) return; + state.starting = true; + $("start").disabled = true; + try { + await startSearch(); + } finally { + state.starting = false; + $("start").disabled = !state.engineOk; + } +} + +// The run the interface follows, when it has not reached a terminal state. +async function activeRun() { + if (!state.runId) return null; + try { + const s = await invoke("run_state", { id: state.runId }); + return s && (s.status === "running" || s.status === "starting") ? s : null; + } catch { + return null; + } +} + +async function startSearch() { banner($("start-error"), ""); const p = state.picks; const threads = parseInt($("threads").value, 10); @@ -1329,6 +1362,17 @@ async function start() { return; } + // The interface follows one run. Starting another while it is in progress would + // leave the first running with nothing showing it and nothing able to stop it. + const active = await activeRun(); + if (active) { + banner( + $("start-error"), + `A search is still running in ${active.out_dir}. Stop it, or wait for it to finish, before starting another.` + ); + return; + } + // FASTA mode with DIA-NN: the search is a library-mode search whose library is // produced first. Everything after this point is the ordinary library path, which // is also the tested one. Built ONCE for the whole selection, not per file: the @@ -1577,6 +1621,11 @@ function render(s) { cancelled: "Search stopped", }; $("prog-title").textContent = titles[s.status] || s.status; + // Stop was pressed but the engine has not been reaped yet: the status stays + // `running` until the waiter publishes the outcome, and the title says so. + if (s.cancel_requested && (s.status === "running" || s.status === "starting")) { + $("prog-title").textContent = "Stopping"; + } const done = expected.filter(([k]) => seen.has(k)).length; const parts = []; diff --git a/docs/02_config_and_data_model.md b/docs/02_config_and_data_model.md index 966423dd..a2e14065 100644 --- a/docs/02_config_and_data_model.md +++ b/docs/02_config_and_data_model.md @@ -385,7 +385,7 @@ the smaller `stage_order` and `Reported` never overrides a real rejection; | 6 | `RtPruned` / `RT_PRUNED` | candidate generation (B) | | 7 | `CandidateCapReached` / `CANDIDATE_CAP_REACHED` | candidate generation (B) | | 8 | `NoFragmentTraces` / `NO_FRAGMENT_TRACES` | extraction (C, D) | -| 9 | `NoPeakGroup` / `NO_PEAK_GROUP` | extraction (C, D) | +| 9 | `DidNotSurviveExtraction` / `DID_NOT_SURVIVE_EXTRACTION` (was `NoPeakGroup` / `NO_PEAK_GROUP` before docs/29 #16) | extraction (C, D) | | 10 | `PeakNotSelected` / `PEAK_NOT_SELECTED` | peak/peptide ranking (E) | | 11 | `OutcompetedByTarget` / `OUTCOMPETED_BY_TARGET` | competition (G) | | 12 | `OutcompetedByDecoy` / `OUTCOMPETED_BY_DECOY` | competition (G) | diff --git a/docs/04_convert.md b/docs/04_convert.md index e68009fa..1bb25045 100644 --- a/docs/04_convert.md +++ b/docs/04_convert.md @@ -321,10 +321,10 @@ and 80.3% of what a library-free DIA-NN 2.2.0 search reports on the same file. The mechanism is downstream: with most peaks gone, candidates cannot assemble enough distinct matched fragments to satisfy `extract.presence_min_fragments` (`config.rs:523`, default 3 at `config.rs:690`), so they fail peak-group -formation and are recorded with rejection code `NO_PEAK_GROUP` +formation and are recorded with rejection code `DID_NOT_SURVIVE_EXTRACTION` (`rejection.rs:62`). This was confirmed with `mumdia audit` on the capped arm, restricted to the peptides that same DIA-NN search reports as present (78,782 -distinct `Stripped.Sequence` at DIA-NN `Q.Value` <= 0.01): 49,105 of 78,782 (62.3%) stopped at `candidate_generated` with `NO_PEAK_GROUP`, against only +distinct `Stripped.Sequence` at DIA-NN `Q.Value` <= 0.01): 49,105 of 78,782 (62.3%) stopped at `candidate_generated` with `DID_NOT_SURVIVE_EXTRACTION`, against only 5,380 lost to FDR and 355 to competition, and a counterfactual replay on the uncapped artifact recovered 41,948 (85.4%) of them. The loss is therefore extraction-side, not a scoring or competition effect. See docs/09_extract.md for @@ -540,7 +540,13 @@ than its input is either from a different acquisition of the same name or from before the input was re-acquired, and searching it would search the wrong data. An unreadable timestamp counts as not reusable. -The converter writes to `.partial.mzML` and the engine renames it to +The converter writes to `.partial--.mzML`, a name unique to this +conversion, under a `.mzML.converting` lock beside the destination; a second +process converting the same input waits for the lock and reuses the finished mzML +instead of converting into the same destination (docs/30 R4: two concurrent +conversions used to share one partial file, and one of them published the other's +bytes). A lock whose holder stopped writing for fifteen minutes is broken. The engine +then renames the partial file to `.mzML` only after a zero exit and a file at that path, so a killed run or a converter crash leaves nothing the reuse rule can mistake for a finished conversion; a stale `.partial.mzML` is removed at the next attempt. The marker sits in the stem diff --git a/docs/06_predict_frag_index_matchers.md b/docs/06_predict_frag_index_matchers.md index 5a2b7510..17c721d7 100644 --- a/docs/06_predict_frag_index_matchers.md +++ b/docs/06_predict_frag_index_matchers.md @@ -149,14 +149,17 @@ struct so intensity/iRT assignment reuses it instead of re-parsing `NativeRt::predict_irt` per candidate. DeepLC path deduplicates by peptidoform string (RT is charge-independent, `predict_frag.rs:324-334`), runs the sidecar once over the unique set, then maps results back. Peptidoforms DeepLC returns no -prediction for are anchored at `irt = 0.0` and counted; if any are missing a -`tracing::warn!` fires (`predict_frag.rs:347-352`). This is the DeepLC-miss iRT -warning: it makes the silent "unmatched peptidoform gets iRT 0.0" failure visible, -because an iRT-0 anchor collapses the RT window onto the gradient origin and -misplaces the candidate at extraction. The DeepLC branch requires `deeplc_python` -and errors otherwise (`predict_frag.rs:318-321`); its returned model id is the -hardcoded string `"deeplc-4.0-mt"` (`predict_frag.rs:353`), not a trait -`identity()` (the sidecar path has no `RtPredictor` impl to query). +prediction for are reported to `run`, which drops them together with every candidate +sharing their pair key (base peptide, charge, modification set), so a target and its +paired decoy leave together; the counts land in the library report as +`candidates_dropped_unpredicted` and `pairs_dropped_unpredicted` and in a warning. They +used to be anchored at `irt = 0.0` with a warning, which collapsed the RT window onto +the gradient origin for those candidates (docs/29 #17). `run_deeplc` also rejects a +returned id that was not requested or that appears twice. The DeepLC branch requires `deeplc_python` +and errors otherwise (`predict_frag.rs:318-321`); its returned model id is +`deeplc--base`, the version read from the interpreter, not a trait +`identity()` (the sidecar path has no `RtPredictor` impl to query). It was the literal +`"deeplc-4.0-mt"` until docs/30, which named a family, not the release that predicted. **intensity assignment** (`assign_intensities`, `predict_frag.rs:359`). Native path calls `NativeFrag::predict_intensities`. MS2PIP path runs the sidecar over @@ -177,8 +180,9 @@ precursor charge 2) 78.6% of the kept fragments were charge-2 heuristics tied at among the top 1,000 seed scores. `HCDch2` with 12 fragments gave 19,308 confident seeds on the same run (DIA-NN library: 21,856; `HCD2021`, 12 fragments, charge-2 only from charge 3: 14,412). A candidate MS2PIP returns nothing for (absent from -the map, or an empty per-candidate map) falls back wholesale to native in either -regime; a fragment at `0.0` can then be dropped by top-N. MS2PIP requires `ms2pip_python` and errors +the map, or an empty per-candidate map) is dropped with its pair, exactly like a +DeepLC miss, rather than receiving the native heuristic under an MS2PIP model identity +(docs/29 #17); a fragment at `0.0` can still be dropped by top-N. MS2PIP requires `ms2pip_python` and errors otherwise (`predict_frag.rs:371-374`); its model id is `format!("ms2pip-{model}")` (`predict_frag.rs:446`). @@ -419,13 +423,13 @@ m/z (`Library::local_frag_index`, `index.rs:325`). | `PredictFragParams` | `predict_frag.rs:24` | Stage C entry args: in/out paths, `cfg`, `work_dir`, `config_hash` | | `predict_frag::run` | `predict_frag.rs:50` | Stage C entry: parse, fragment, assign intensity/iRT, top-N, sort, write; returns `(n_prec, n_frag)` | | `Raw` | `predict_frag.rs:34` | one candidate pre-assignment; caches the `ParsedPeptidoform` so RT/intensity reuse the parse | -| `assign_rt` | `predict_frag.rs:308` | native or DeepLC iRT; emits the DeepLC-miss warning; DeepLC id `"deeplc-4.0-mt"` | -| `assign_intensities` | `predict_frag.rs:359` | native or MS2PIP intensity with per-charge-group normalization + native charge-2 fallback; MS2PIP id `"ms2pip-{model}"` | +| `assign_rt` | `predict_frag.rs:308` | native or DeepLC iRT; emits the DeepLC-miss warning; DeepLC id `deeplc--base` | +| `assign_intensities` | `predict_frag.rs:359` | native or MS2PIP intensity with per-charge-group normalization + native charge-2 fallback; MS2PIP id `ms2pip--{model}` | | `fragment_cardinality` | `predict_frag.rs:457` | distinct precursors per 0.01 Da fragment-m/z bin, per fragment row; diagnostic column, no consumer yet | | `RtPredictor` / `FragmentPredictor` | `predict.rs:13` / `predict.rs:19` | predictor traits (predict + `identity`); implemented only by the native structs | | `NativeRt` | `predict.rs:25` | additive retention-coefficient model + `sqrt(len)` + `0.01*mod` term, `identity` `native-rt-v1` | | `NativeFrag` | `predict.rs:73` | heuristic b/y intensity model (y=1.0, b=0.75, mid-seq positional, charge-2 x0.5), max-normalized, `identity` `native-frag-v1` | -| `resolve_script` | `sidecar.rs:20` | locate a worker script (CWD, exe dir/dir, exe dir/scripts, else CWD-relative) | +| `resolve_script` | `sidecar.rs:20` | locate a worker script (an absolute directory as given, else exe dir/dir, exe dir/scripts, and the working directory LAST; docs/31 F3) | | `run_ms2pip` | `sidecar.rs:42` | MS2PIP client; in `id`/`peptidoform`/`charge`, out `id`/`ion_type`/`ordinal`/`intensity`; returns `cid -> (ion_byte, ordinal) -> intensity` | | `run_deeplc` | `sidecar.rs:81` | DeepLC client; in `id`/`peptidoform`, out `id`/`predicted_rt`; returns `id -> predicted_rt` | | `run_deeplc_finetune` | `sidecar.rs:111` | DeepLC multitask fine-tune; `deeplc_finetune.py ` + epoch/patience/q-train/batch flags (called by `run`, not predict-frag) | @@ -520,10 +524,11 @@ Matcher selection (both stages default to `Fragindex`): sums `obs_sum` in the caller's fixed peak order and `touched()` is in first-touch order, so callers sort before any float reduction (`fragindex.rs:346-350`). -- **DeepLC nondeterminism and the iRT-0 anchor.** The DeepLC sidecar and fine-tune - are not seeded, so iRT values vary run to run. Any peptidoform DeepLC does not - return lands at iRT 0.0; the warning at `predict_frag.rs:347-352` reports the - count so a large miss is visible rather than silent. +- **DeepLC nondeterminism and unpredicted peptidoforms.** The DeepLC sidecar and + fine-tune are not seeded, so iRT values vary run to run. Any peptidoform DeepLC + does not return is dropped from the library together with its pair (it used to be + anchored at iRT 0.0); the library report's `candidates_dropped_unpredicted` counts + them so a large miss is visible rather than silent. - **An imported library can carry one iRT per stripped peptide.** The importer copies `predicted_irt` verbatim from the source library's RT column (`import_diann_lib.py:148`) and library load accepts it unchecked diff --git a/docs/08_rt_im_train.md b/docs/08_rt_im_train.md index 28141cf3..9f02f7c8 100644 --- a/docs/08_rt_im_train.md +++ b/docs/08_rt_im_train.md @@ -179,9 +179,10 @@ rt_im_train.rs:136-140). `predict` is a closure (rt_im_train.rs:142-150): it returns `NaN` when calibration is unavailable, otherwise `loess.predict(irt)` when a LOESS model exists, else -`slope * irt + intercept`. The linear coefficients are therefore both the primary -map (Linear method) and the extrapolation fallback (LOESS outside its training -range; see below). +`slope * irt + intercept`. The linear coefficients are the primary map under the +Linear method and, under LOESS, only the degenerate fallback (fewer than four anchors, +or a local window without spread). Outside its training range LOESS continues the +local fit at the nearest boundary, not the global line (see below). The math: @@ -191,7 +192,7 @@ The math: returns `(0, mean(ys))` (a constant map, calibrate.rs:8-16); a near-zero denominator (all `x` equal) returns `(0, Sy/n)` (calibrate.rs:22-24). - **`Loess::fit`** (calibrate.rs:42-83) first computes the global linear fit as - the extrapolation fallback (calibrate.rs:43), sorts the points by `x` + the degenerate fallback (calibrate.rs:43), sorts the points by `x` (calibrate.rs:45-48), and if fewer than 4 points are present just fills the grid from the linear line (calibrate.rs:50-66). Otherwise the local window size is `k = clamp(ceil(span * n), 3, n)` (calibrate.rs:67) and it evaluates a @@ -207,8 +208,10 @@ The math: contribute. If the weighted system is degenerate, meaning `sw < 1e-12` (all weights vanished) or `sw*swxx - swx^2` near zero (calibrate.rs:146-149), it falls back to the global line. -- **`Loess::predict`** (calibrate.rs:87-102) uses the linear fallback for `x` at or - outside the grid ends, and also when the grid has fewer than 2 nodes +- **`Loess::predict`** (calibrate.rs:87-102) returns NaN for a non-finite `x` (a null + library iRT reads as NaN, and interpolating it used to index before the start of the + grid; docs/31 F4), extrapolates from the nearer grid end for `x` outside the range, + uses the linear fallback when the grid has fewer than 2 nodes (calibrate.rs:89-94), and otherwise linearly interpolates between the two bracketing grid nodes found by `partition_point` (calibrate.rs:95-101). When the two bracketing nodes are within `1e-12` in `x` it returns the lower node's `y` @@ -513,9 +516,14 @@ cheaper option with equal RT residuals on the one run measured here. | `candidate_window` | rt_im_train.rs:65-70 | Builds `(cal, lo, hi)`; returns `(NaN, -inf, +inf)` when calibrated RT or width is absent. | | `rt_im_train::run` | rt_im_train.rs:72-354 | The stage: join iRT, select anchors, fit, window, apply, write. | | `linear_fit` | calibrate.rs:6-28 | OLS `y = slope*x + intercept` with degenerate-case guards. | -| `Loess` | calibrate.rs:31-37 | Grid-based local-linear smoother; carries a linear fallback for extrapolation. | +| `Loess` | calibrate.rs:31-37 | Grid-based local-linear smoother; carries the two boundary extrapolation slopes and the global line as the degenerate fallback. | | `Loess::fit` | calibrate.rs:42-83 | Sorts anchors, builds a `grid_n`-point local-linear grid, `k = clamp(ceil(span*n),3,n)`. | -| `Loess::predict` | calibrate.rs:87-102 | Grid interpolation inside range, linear extrapolation outside. | +| `Loess::predict` | calibrate.rs:87-102 | NaN for a non-finite query (docs/31 F4). Grid interpolation inside range; outside it, the boundary grid value continued with the boundary extrapolation slope, so the map is continuous at both ends. Until docs/29 #10 it switched to the global line there, which on `y = 200 + 10x^2` (span 0.3) jumped from 193.4 to 38.3 at `x = 0` and from 1173.5 to 1018.4 at the top: about 155 s discontinuities for gradient-edge peptides. The extrapolation slope is the +secant of the fitted curve over its end decile, clamped non-negative and to at most four +times the global slope: the pointwise local slope it first used comes from the sparsest, +most one-sided window in the fit, and on noisy anchors it was free to be negative (which +inverts the iRT-to-RT map) or several times the global slope, multiplying a distance that +is unbounded by construction (docs/31 F7). Measured on HYE B01 with the imported iRT and `native_tda`: 45,946 stripped peptides at 1% before, 45,957 after, decoy fraction unchanged; 1.9% of candidates got a different window, almost all with iRT above the anchor range, which the global line had placed past the end of the run. With the DeepLC 4.1.1 re-predicted precursors (`w_rt` 414 s): 48,533 in both arms, 0.2% of windows moved. | | `local_linear` | calibrate.rs:107-153 | Tricubic-weighted local least squares at one point. | | `percentile` | calibrate.rs:156-164 | Nearest-rank percentile: sorts a copy, `rank = round(p.clamp(0,1)*(len-1))`. Not interpolated. Empty input returns 0.0. | | `CalibrationMethod` | config.rs:56-61 | Enum `{ Loess, Linear, None }`; default `Loess`. `None` is rejected at load. | @@ -618,8 +626,10 @@ though the enum variant still exists. and therefore the whole run, non-reproducible. Treat it as an accuracy lever, not a deterministic default. - **`slope`/`intercept` are emitted in `cal.json` whenever calibration is available, - including under LOESS** (rt_im_train.rs:128, 286-287, 313-314). They are the LOESS - extrapolation fallback, not dead values; do not assume they were unused when + including under LOESS** (rt_im_train.rs:128, 286-287, 313-314). Under LOESS they + are the degenerate fallback (fewer than four anchors, or a local window without + spread), not the extrapolation model: since docs/29 #10 the map continues the fitted + curve outside the anchor range. Do not read them as the calibration when `method == "loess"`. They are serialized as `null` only when calibration is unavailable (`n_train < 2`), since the fit is not computed in that case. - **`CalibrationMethod::None` still exists but is rejected** at config load diff --git a/docs/09_extract.md b/docs/09_extract.md index 5efb1e26..1cc7e5d6 100644 --- a/docs/09_extract.md +++ b/docs/09_extract.md @@ -327,21 +327,22 @@ rejected before any gate or score is reached. Measured on a 50-window Orbitrap DIA run, `--top-peaks-ms2 300` discarded 78.6% of MS2 peaks; a `mumdia audit` ladder restricted to peptides that an external library-free search confirms are present in the run showed 49,105 of 78,782 (62.3%) stopping at -`candidate_generated` with rejection code `NO_PEAK_GROUP` (`rejection.rs:62`). +`candidate_generated` with rejection code `DID_NOT_SURVIVE_EXTRACTION` (`rejection.rs:62`). Only 5,380 were lost to FDR and 355 to competition. Replaying those candidates against the uncapped artifact recovered 41,948 of the 49,105 (85.4%), which is what identifies the cap as the cause. -`NO_PEAK_GROUP` does not mean "never assembled `presence_min_fragments` distinct -fragments", although this document previously said so. `audit.rs` reads a +`DID_NOT_SURVIVE_EXTRACTION` (named `NO_PEAK_GROUP` until docs/29 #16) does not mean +"never assembled `presence_min_fragments` distinct fragments", although this document +previously said so. `audit.rs` reads a per-candidate audit table that `extract` does not write -- `emit_candidate_audit` -is unwired -- so the reason map is always empty and the `_ => NoPeakGroup` +is unwired -- so the reason map is always empty and the `_ => DidNotSurviveExtraction` catch-all absorbs presence failures, matched-fraction failures and every extraction-gate rejection alike. Read it as "did not survive extract". The attribution to the peak cap above stands on the replay experiment, not on the label. -When `NO_PEAK_GROUP` dominates an audit ladder, check the conversion cap before +When `DID_NOT_SURVIVE_EXTRACTION` dominates an audit ladder, check the conversion cap before tuning the presence thresholds or gates here; see docs/04_convert.md for the peak census and the cap dose-response. diff --git a/docs/12_quant_lfq_align_mbr_report_audit.md b/docs/12_quant_lfq_align_mbr_report_audit.md index 0243b4e4..796452d4 100644 --- a/docs/12_quant_lfq_align_mbr_report_audit.md +++ b/docs/12_quant_lfq_align_mbr_report_audit.md @@ -110,12 +110,15 @@ and one `psms.parquet` per run in `source` order. Produces `.parquet`, one per accepted transfer (`mbr_worker.py:254`): `candidate_id`, `source`, `peptidoform`, `charge`, `protein_group`, `label`, `expected_rt`, `observed_rt`, `rt_delta`, `transfer_q` (10 columns). When there are no transfer candidates at all the worker -short-circuits and writes a placeholder table with a single empty `candidate_id` -column (`pa_write_empty`, `mbr_worker.py:289`), so `.parquet` always exists. -Optionally writes an augmented scored table (`--out-psms-scored`) that lowers each accepted -transfer's `q_value` to `min(q_value, transfer_q)` on the matching `(candidate_id, -source)` row and adds an `is_transferred` bool (`mbr_worker.py:272`); this requires the -scored table to carry a `source` column. All MBR outputs are written by Python and +short-circuits and writes the same ten columns with zero rows (`write_empty_transfers`), +and, when `--out-scored` was asked for, the scored table unchanged with `is_transferred` +false and `transfer_q` NaN on every row (`write_unflagged_scored`), so a downstream stage +never meets a missing file or a one-column placeholder (docs/30). +Optionally writes an augmented scored table (`--out-scored`) that lowers each accepted +transfer's PSM q columns to `min(q, transfer_q)` on the matching `(candidate_id, +source)` row and adds an `is_transferred` bool plus a `transfer_q` column (the accepted +q on transferred rows, NaN elsewhere; `mbr_worker.py`); this requires the scored table +to carry a `source` column. All MBR outputs are written by Python and have no `report.json`. The unreachable re-extraction tier (`--emit-transfer-targets `) instead writes @@ -130,8 +133,14 @@ Consumes `psms_scored.parquet` and optionally the two quant tables. The CLI take (`main.rs:796-797`); `run` passes the run out-dir and `q_threshold = quant.q_threshold` (`run.rs:484-491`). `peptides.tsv` header (`report.rs:95`): `precursor`, `stripped_sequence`, `charge`, `protein`, -`q_value`, `score`, `quantity`. `proteins.tsv` header (`report.rs:134`): -`protein_group`, `q_value`, `quantity`. No Parquet or `report.json` is written; +`q_value`, `score`, `quantity`, `is_transferred`, `transfer_q`. `proteins.tsv` header +(`report.rs:134`): `protein_group`, `q_value`, `quantity`, `is_transferred`, +`transfer_q`. The last two export the acceptance basis: a row is written when it is a +target and either its grouped q passes the threshold or it is an accepted +match-between-runs transfer, and a transferred row keeps its grouped q (usually 1.0) +next to the transfer q it was accepted at. A tighter threshold does not revoke a +transfer; a transfer is not protein-group confidence (docs/29 #19). No Parquet or +`report.json` is written; `report::run` returns `(n_precursors, n_protein_groups)`, and the `mumdia report` handler prints a one-line summary (`main.rs:806`). @@ -423,9 +432,17 @@ The worker (`scripts/mbr_worker.py`) implements two tiers: The M5 augmented scored output (`--out-scored`, `mbr_worker.py:272`) requires the scored table to have a `source` column and matches transfers on `(candidate_id, -source)`, taking `min(q_value, transfer_q)` and setting `is_transferred`. The worker -prints a validation summary (accepted counts per run, empirical decoy fraction, and the -RT window `delta_star` at `q_transfer`, `mbr_worker.py:245`). +source)`, taking `min(q, transfer_q)` on the PSM q columns and setting `is_transferred` +and `transfer_q`. The worker +prints a validation summary: accepted counts per run, how many permuted-RT null draws +fall inside the accepted RT window, and the window `delta_star` at `q_transfer`. It +no longer prints a "decoy fraction" among accepted transfers: every transfer candidate +is a confident target of another run, so that fraction was structurally zero and said +nothing about calibration (docs/29 #6). The transfer q uses the engine's `+1` +pseudocount, `(null <= delta + 1) / (targets <= delta)`, so a pool no permuted residual +undercuts no longer gets q = 0 (docs/29 #7), and the RT residual is measured on the +peak rescore selected (`selected_peak_rank`) when a competed table carries several +peaks per candidate (docs/29 #8). ### report (`report.rs:49`, `run`) @@ -456,16 +473,20 @@ presentation value rounded to one decimal. The search space is every library precursor (`candidate_id`, `peptidoform`, `charge`, `label`, `protein`). Survivor sets are built as `HashSet` of `candidate_id` from -`psms` (extracted), `competed`, and `scored`; scored also yields `q_by_cid` and -optional `pepq_by_cid` (peptide-level q, only present in some scored schemas, -`audit.rs:90`). `load_extract_reasons` (`audit.rs:51`) tries to read a +`psms` (extracted), `competed`, and `scored`; scored also yields `q_by_cid`, read from +`precursor_q` (the unit the `passed_precursor_fdr` label names; the PSM `q_value` only +on an older table without that column, recorded as `q_unit` in the metrics JSON; +docs/29 #16) and optional `pepq_by_cid` (peptide-level q, only present in some scored +schemas). A scored table with more than one `source` is refused: the audit keys on +`candidate_id`, so a pooled table would attribute one run's fate to every run; audit +the per-run split tables. `load_extract_reasons` (`audit.rs:51`) tries to read a `.audit.parquet` sidecar to refine the extract-stage bucket, but nothing in the current chain writes that file (see the gotchas), so the map is empty and every -extract-stage loss buckets to `NO_PEAK_GROUP`. For each candidate the earliest +extract-stage loss buckets to `DID_NOT_SURVIVE_EXTRACTION`. For each candidate the earliest rejection reason is assigned along the ladder (`audit.rs:136`): - not in `extracted` -> refined from the sidecar (`NO_FRAGMENT_TRACES`/`NO_VALID_FRAGMENTS`/`PEAK_NOT_SELECTED`/`RT_PRUNED`/ - `WRONG_ISOLATION_WINDOW`, `audit.rs:139`) or the generic `NO_PEAK_GROUP` when no + `WRONG_ISOLATION_WINDOW`, `audit.rs:139`) or the generic `DID_NOT_SURVIVE_EXTRACTION` when no sidecar (the only outcome today); - extracted but not in `competed` -> `OUTCOMPETED_BY_DECOY` (decoy) or `OUTCOMPETED_BY_TARGET` (target); @@ -516,7 +537,7 @@ FDR/reporting). | `run_mbr` | sidecar.rs:162 | build argv and spawn `mbr_worker.py` | | `binned_map` | mbr_worker.py:31 | monotone binned-median RT calibration | | `expected_rt` | mbr_worker.py:116 | cross-run predicted RT for a candidate in a run | -| `pa_write_empty` | mbr_worker.py:289 | placeholder output when there are no transfer candidates | +| `write_empty_transfers` / `write_unflagged_scored` | mbr_worker.py | full-schema outputs when there are no transfer candidates | | `ReportParams` / `report::run` | report.rs:13 / 49 | TSV writer | | `strip` | report.rs:24 | stripped sequence from a peptidoform | | `qcell` | report.rs:39 | quantity cell formatting (1 decimal; empty on NaN) | @@ -629,19 +650,23 @@ but do not affect the wired `mumdia mbr` path. `traces_extracted` are all set from the same `traces` flag (`audit.rs:170`) because the artifacts only record presence in `psms`; the (not-yet-written) in-extract sidecar would be the only way to split "no traces" from "traces but no accepted peak". -- **audit `reported` vs `REPORTED`.** The `reported` bool column is set from - `passed_prec` alone (`audit.rs:176`), while the `REPORTED` rejection reason additionally - requires the peptide gate. A candidate can therefore have `reported=true` yet - `rejection_reason=FAILED_PEPTIDE_FDR`. Treat `rejection_reason` as authoritative. +- **audit `reported` and `REPORTED` agree.** The `reported` bool is `rejection_reason == + REPORTED`: a target that passed the precursor and the peptide gate. A decoy that passes + both is `REMOVED_DURING_REPORTING`, because the report never writes a decoy. The two + gate columns (`passed_precursor_fdr`, `passed_peptide_fdr`) remain the diagnostics. + Until docs/30 R7 the flag repeated the precursor gate alone, so a row could read + `reported=true` next to `FAILED_PEPTIDE_FDR`, and a decoy could be `REPORTED`. A present + `precursor_q` column of the wrong type is an error; only an absent column falls back to + the PSM `q_value` (recorded as `q_unit`). - **audit reason coverage.** In the current chain `audit.rs` can only ever emit - `NO_PEAK_GROUP`, `OUTCOMPETED_BY_TARGET`/`OUTCOMPETED_BY_DECOY`, `FAILED_PRECURSOR_FDR`, + `DID_NOT_SURVIVE_EXTRACTION`, `OUTCOMPETED_BY_TARGET`/`OUTCOMPETED_BY_DECOY`, `FAILED_PRECURSOR_FDR`, `FAILED_PEPTIDE_FDR`, and `REPORTED`. The five refined extract codes (`NO_FRAGMENT_TRACES`, `NO_VALID_FRAGMENTS`, `PEAK_NOT_SELECTED`, `RT_PRUNED`, `WRONG_ISOLATION_WINDOW`) are matched by `load_extract_reasons` (`audit.rs:139`) but need the absent sidecar to fire. The six remaining enum variants (`PEPTIDE_NOT_GENERATED`, `MODIFICATION_NOT_ALLOWED`, `CHARGE_OUT_OF_RANGE`, `PRECURSOR_MZ_OUT_OF_RANGE`, `CANDIDATE_CAP_REACHED`, `REMOVED_DURING_REPORTING`) have - no producer at all: even a sidecar string of that name falls to the `_ => NoPeakGroup` + no producer at all: even a sidecar string of that name falls to the `_ => DidNotSurviveExtraction` arm (`audit.rs:145`). - **report.json coverage is partial.** `quant` writes reports for `peptide_quant`, `protein_quant`, and emitted `fragment_quant`; the peak-bounds diff --git a/docs/13_sidecars.md b/docs/13_sidecars.md index 0bfb730b..2a5365d6 100644 --- a/docs/13_sidecars.md +++ b/docs/13_sidecars.md @@ -155,8 +155,10 @@ code. - OUT `.parquet`: one row per accepted transfer with `candidate_id, source, peptidoform, charge, protein_group, label, expected_rt, observed_rt, rt_delta, transfer_q` (`mbr_worker.py:254-265`). Optional - `--out-scored` writes the scored table with accepted transfers' `q_value` - lowered to `transfer_q` and an `is_transferred` flag added. Optional + `--out-scored` writes the scored table with accepted transfers' PSM q columns + lowered to `transfer_q`, an `is_transferred` flag and a `transfer_q` column (NaN + on non-transferred rows) added; with no transfer candidates the transfer table has + its ten columns and zero rows and the augmented table is the input, unflagged. Optional `--emit-transfer-targets` writes per-run `run_windows`-format tables (`candidate_id, rt_pred_cal, rt_lo, rt_hi, im_*`) plus a permuted-RT decoy file for the re-extraction tier (`mbr_worker.py:142-151`). @@ -207,10 +209,12 @@ MS2PIP charge-1 (TIC-fraction, ~0.02-0.3) and the native charge-2 fallback top-N would bury MS2PIP, so each charge group is max-normalized to its own peak before they compete (`predict_frag.rs:365-384`). Two native-fallback edge cases: `run_ms2pip` returning an empty map is a hard error (`bail!("MS2PIP returned no -predictions")`, `predict_frag.rs:342-344`), while a single candidate that MS2PIP -returned nothing for (missing/empty per-id entry) falls back wholesale to the -native intensities for that candidate (`predict_frag.rs:387-389`). The -worker builds `psm_utils.PSMList` in chunks of `max(100k, 20k x processes)` rows, +predictions")`), while a single candidate that MS2PIP returned nothing for +(missing/empty per-id entry) is dropped from the library together with its pair +(`drop_unpredicted`; docs/29 #17) instead of falling back to the native intensities +under an MS2PIP model identity. `run_ms2pip` rejects an id it did not request and a +fragment reported twice. The worker builds `psm_utils.PSMList` in chunks of +`max(100k, 5k x processes)` rows, calls `ms2pip.predict_batch(model, processes=N)` with `N` the engine's thread count passed as the fourth argument (the old cap `min(8, cpu_count)` applies only when the argument is absent), and converts MS2PIP's log2 intensities to linear via @@ -224,9 +228,11 @@ start method safe for multiprocessing. **DeepLC predict** (`predict_frag.rs:274-312`, worker `deeplc_worker.py`). Selected by `predict_frag.rt_predictor = "deeplc"`. `assign_rt` deduplicates by peptidoform (RT is charge-independent, `predict_frag.rs:281-290`), calls -`run_deeplc`, and writes `r.irt`. Peptidoforms with no returned iRT are anchored -at `0.0` with a warning (`predict_frag.rs:293-308`; this is the "unmatched -peptidoforms silently get iRT 0.0" foot-gun noted in CLAUDE.md). The worker calls +`run_deeplc`, and writes `r.irt`. Peptidoforms with no returned iRT are dropped +from the library with their pairs and counted in the library report (they used to +be anchored at `0.0` with a warning, the "unmatched peptidoforms silently get iRT +0.0" foot-gun; docs/29 #17). `run_deeplc` rejects an id it did not request and a +duplicate id. The worker calls `deeplc.predict` in 200k chunks and, when the multitask model returns an ensemble matrix `(N, n_models)`, averages across models (`deeplc_worker.py:44-47`). Predictions are uncalibrated; rt-im-train's per-run LOESS/linear maps them onto @@ -327,9 +333,9 @@ rescorer or `native_tda` (`rescore.rs:217-276`). `classify_entrapment` marker, does not contain `entrapment_exclude`, and matches none of `entrapment_contaminant_markers`. The worker trains real-target (positive) vs spike-in (negative), decoys excluded from training (`entrapment_worker.py:80-83`), -out-of-fold with `GroupKFold` grouped by `base_peptide_id` -(`entrapment_worker.py:93-101`); a final model fit on all non-decoy PSMs scores -decoys and any single-class-fold gaps (`:103-108`). Model is `gbm` +out-of-fold with `GroupKFold` grouped by `base_peptide_id`; a training fold with a +single class is an error, never a gap filled with in-sample scores (docs/29 #3), and +a final model fit on all non-decoy PSMs scores the decoys only. Model is `gbm` (`HistGradientBoostingClassifier`, `early_stopping=False` so `random_state=0` is reproducible) or `nn` (StandardScaler + MLP pipeline) via `MUMDIA_ENTRAPMENT_MODEL` (`entrapment_worker.py:28-60`). The rationale: spike-in diff --git a/docs/14_build_test_deploy_gotchas.md b/docs/14_build_test_deploy_gotchas.md index beccf2d7..0f0422cc 100644 --- a/docs/14_build_test_deploy_gotchas.md +++ b/docs/14_build_test_deploy_gotchas.md @@ -31,7 +31,7 @@ configurations in `configs/`, the container definition in `Dockerfile` + | `rust/mumdia/crates/mumdia-io/Cargo.toml` | I/O crate; adds `arrow`/`parquet`/`blake3` over `mumdia-core` | | `rust/mumdia/crates/mumdia/tests/pipeline.rs` | The only integration test file: extract -> features -> compete -> rescore on crafted Parquet | | `rust/mumdia/crates/mumdia-core/build.rs` | Stamps the git commit and build date into the crate so `manifest.json` can record them | -| `.github/workflows/ci.yml` | Eight jobs: `lint` (fmt + clippy `-D warnings` + rustdoc), `audit` (`cargo audit`/`cargo deny`), `build-test` matrix on ubuntu/macos/windows, `smoke` (end-to-end `run` and `run-experiment` on a generated fixture, ubuntu + windows), `sidecar-imports` (a real conda env per sidecar, matrixed, plus `pip-audit`), `smoke-cross-platform` (asserts the two platforms produced byte-identical `peptides.tsv` and `proteins.tsv`), `desktop` (the console's own workspace: fmt, clippy, unit tests, and a frontend/backend consistency check), `sidecars` (compileall + JSON/YAML parse + doc-reference check + generated-document freshness); on push-to-`main`, every PR, weekly, and on demand | +| `.github/workflows/ci.yml` | Eight jobs: `lint` (fmt + clippy `-D warnings` + rustdoc), `audit` (`cargo audit` on both lockfiles, engine and desktop), `build-test` matrix on ubuntu/macos/windows, `smoke` (end-to-end `run` and `run-experiment` on a generated fixture, ubuntu + windows), `sidecar-imports` (a real conda env per sidecar, matrixed, plus `pip-audit`), `smoke-cross-platform` (asserts the two platforms produced byte-identical `peptides.tsv` and `proteins.tsv`), `desktop` (the console's own workspace: fmt, clippy, unit tests, and a frontend/backend consistency check), `sidecars` (compileall + JSON/YAML parse + doc-reference check + generated-document freshness); on push-to-`main`, every PR, weekly, and on demand | | `.github/workflows/release.yml` | Dormant until a `v*` tag; `validate-tag` gates on tag-equals-workspace-version, ancestry from `main` and a green `ci.yml` for that exact SHA, then builds three target binaries, smoke-tests each, unpacks each archive into a clean directory and runs that archive's own `ci/smoke.sh`, and attaches archives + `.sha256` to the Release. `workflow_dispatch` rehearses everything except the upload | | `.github/workflows/docker.yml` | Builds the image into the local daemon, smoke-tests it, then pushes to GHCR only on a `v*` tag; build-and-smoke-only on `workflow_dispatch` | | `.github/dependabot.yml` | Monthly grouped Cargo + GitHub Actions + Docker base-image updates; `arrow*`/`parquet*` grouped apart because they carry the on-disk contract. No `pip` entry: the Python pins live in the pip sections of the `env/` conda specifications, which Dependabot cannot parse | diff --git a/docs/15_data_dictionary.md b/docs/15_data_dictionary.md index 14279fb6..a260ef51 100644 --- a/docs/15_data_dictionary.md +++ b/docs/15_data_dictionary.md @@ -817,11 +817,19 @@ Each `ArtifactRecord`: Not Parquet, but these are the files most users actually read, so they are documented here rather than left to the source. Written by `report` -(`stages/report.rs`), which has two call sites: inside `run` (`run.rs:484`) and -the `mumdia report` handler (`main.rs:798`). `run-experiment` never calls it, so -an experiment output tree contains neither TSV at any level; take per-run counts -from the per-run split `scored.parquet` tables on `run_psm_q`, or invoke -`mumdia report` yourself. +(`stages/report.rs`): `report::run` inside `run` and the `mumdia report` handler, +and `report::run_experiment` inside `run-experiment` (also `mumdia report +--experiment-dir`), which writes one experiment-wide pair at the experiment root +with an `n_runs` column and one quantity column per run and no per-run TSVs; take +per-run counts from the per-run split `scored.parquet` tables on `run_psm_q`. + +**Acceptance rule, both writers.** A row is written when it is a target and either +its grouped q is at or under the threshold or it is flagged `is_transferred` by +`mumdia mbr`. The two columns `is_transferred` and `transfer_q` export that basis +(docs/29 #19): a transferred row keeps its grouped q, which is usually 1.0, and a +tighter `--q` does not revoke a transfer that already passed `mbr.q_transfer`. A +transfer is a peptide-level acceptance, not protein-group confidence; a protein group +admitted through a transferred row carries the flag so the reader can tell. Values are formatted for reading, not for analysis: `q_value` is printed to six decimals, `score` to four, and `quantity` to one @@ -843,6 +851,8 @@ the best q (`report.rs:90,105-108`). Targets only, filtered on | `q_value` | `peptide_q_value` | **base-peptide** q, six decimals | | `score` | `score` | rescorer score, four decimals | | `quantity` | `peptide_quant.parquet` joined on `(peptidoform, charge)` | empty when the precursor was not quantifiable or no `--peptide-quant` was passed (`report.rs:39-45,61-72`) | +| `is_transferred` | `is_transferred` (`mumdia mbr`), `false` without it | `true` when the row was admitted as a match-between-runs transfer rather than on its q | +| `transfer_q` | `transfer_q` (`mumdia mbr`) | the transfer q the row was accepted at, six decimals; empty on every other row | The row unit and the filter column deliberately disagree, and this is the single most common misreading of MuMDIA output: rows are precursors @@ -861,6 +871,8 @@ with a non-empty group, filtered on `pg_q_value <= --q` (`report.rs:137`). | `protein_group` | `protein_group` | accession set as grouped by rescore | | `q_value` | `pg_q_value` | protein-group q, six decimals | | `quantity` | `protein_group_quant.parquet` joined on `protein_group` | empty when not quantifiable or no `--protein-quant` was passed | +| `is_transferred` | `is_transferred` of the admitting row | `true` when the group entered through a transferred peptide row; a transfer is not protein-group confidence | +| `transfer_q` | `transfer_q` of the admitting row | empty unless `is_transferred` | Both grouped q columns (`peptide_q_value`, `pg_q_value`) are written only to each group's single winning row, so under an experiment-wide rescore the grouping is diff --git a/docs/17_troubleshooting.md b/docs/17_troubleshooting.md index f399d228..f12a41bd 100644 --- a/docs/17_troubleshooting.md +++ b/docs/17_troubleshooting.md @@ -27,7 +27,7 @@ is the source of truth if it has moved. | Two runs of the same command give slightly different identification counts | A nondeterministic opt-in path is on: DeepLC fine-tune (no seed) or the PyTorch NN rescorer (approximately reproducible) | Accept the trade for the ID gain, or leave those paths off; set `MUMDIA_NN_SEEDS > 1` to average out NN variance | | Peptidoforms silently get iRT 0.0 | DeepLC returned no iRT for some peptidoforms; they are anchored at 0.0 with a warning | Check the DeepLC env and the `n_irt_missing` warning; unmatched peptidoforms are a known foot-gun | | Peptide count is a fraction of expectation while the empirical decoy fraction still sits at the target | `--top-peaks-ms2` truncated the MS2 peaks at convert time and baked the truncation into the spectra artifact (`convert.rs:76-79`) | Reconvert uncapped or with a much larger cap; the right cap is acquisition-specific, not a universal preset | -| `mumdia audit` attributes most missed peptides to `NO_PEAK_GROUP` at `candidate_generated` | The same peak truncation leaves too few surviving fragments to satisfy `extract.presence_min_fragments` (`extract.rs:1926-1931`) | Raise or remove `--top-peaks-ms2` and reconvert; do not lower `presence_min_fragments` to compensate | +| `mumdia audit` attributes most missed peptides to `DID_NOT_SURVIVE_EXTRACTION` at `candidate_generated` | The same peak truncation leaves too few surviving fragments to satisfy `extract.presence_min_fragments` (`extract.rs:1926-1931`) | Raise or remove `--top-peaks-ms2` and reconvert; do not lower `presence_min_fragments` to compensate | | Every modified form of a peptide is missing from the output; `precursor_q` reports exactly 1.000 precursors per peptide | `compete.group_by = base_peptide` keys the stripped sequence, not the precursor (`compete.rs:88`), and deletes all but the top-scoring sibling before rescore (`compete.rs:319-340`) | Set `compete.group_by = peptidoform_charge` (`compete.rs:93-98`); required for any PTM search | | `cal.json` reports a small `rt_residual_abs_median_s` but RT windows still miss peptides | The residual is measured on the same anchors the calibration was fitted to (`rt_im_train.rs:137`, `rt_im_train.rs:177-185`), so it is in-sample | Treat it as a fit diagnostic; size external RT tolerances from an out-of-sample comparison | | RT windows in a PTM search behave as if the modification were absent | The imported library gave every modform of a stripped peptide the same `predicted_irt` | Check per-stripped-peptide variance of `predicted_irt` in the library; re-predict iRT per peptidoform | @@ -222,7 +222,7 @@ The mechanism is peak-group formation, not scoring. With most peaks gone, `extract.presence_min_fragments` (default 3) cannot be met and the candidate returns no peak group (`extract.rs:1926-1931`). `mumdia audit` on the capped arm, restricted to peptides the reference search confirms are present, put -49,105 of 78,782 (62.3%) at `candidate_generated` with `NO_PEAK_GROUP`; only +49,105 of 78,782 (62.3%) at `candidate_generated` with `DID_NOT_SURVIVE_EXTRACTION`; only 5,380 were lost to FDR and 355 to competition. A counterfactual replay against the uncapped artifact recovered 41,948 of those 49,105 (85.4%). Do not lower `presence_min_fragments` to compensate: that trades a real fragment requirement diff --git a/docs/18_findings_and_decisions.md b/docs/18_findings_and_decisions.md index 741f1b15..c5e149ce 100644 --- a/docs/18_findings_and_decisions.md +++ b/docs/18_findings_and_decisions.md @@ -150,7 +150,7 @@ The mechanism is peak-group formation, not scoring. With most peaks removed, (`rust/mumdia/crates/mumdia-core/src/config.rs:523`, default at `:690`) cannot be satisfied, so real peptides never form a peak group. `mumdia audit` on the capped arm, restricted to peptides DIA-NN confirms are present, shows 49,105 of 78,782 -(62.3 percent) stopped at `candidate_generated` with `NO_PEAK_GROUP`, against +(62.3 percent) stopped at `candidate_generated` with `DID_NOT_SURVIVE_EXTRACTION`, against only 5,380 lost to FDR and 355 lost to competition. A counterfactual replay on the uncapped artifact recovered 41,948 of those 49,105 (85.4 percent). diff --git a/docs/20_sensitivity_and_quantification_playbook.md b/docs/20_sensitivity_and_quantification_playbook.md index 8af4b227..ceba92b9 100644 --- a/docs/20_sensitivity_and_quantification_playbook.md +++ b/docs/20_sensitivity_and_quantification_playbook.md @@ -131,7 +131,7 @@ real peptides never form a peak group. On a 50-window Orbitrap DIA run, a cap of 300 discarded 78.6% of all MS2 peaks and truncated 85.5% of spectra, and `mumdia audit` restricted to peptides DIA-NN 2.2.0 confirms are present showed 49,105 of 78,782 (62.3%) stopped at `candidate_generated` with -`NO_PEAK_GROUP`, versus 5,380 lost to FDR and 355 lost to competition. Peptides.tsv rows at `peptide_q_value` <= 0.01 +`DID_NOT_SURVIVE_EXTRACTION`, versus 5,380 lost to FDR and 355 lost to competition. Peptides.tsv rows at `peptide_q_value` <= 0.01 fell from 63,237 uncapped to 25,425 at cap 300 with the decoy fraction at 0.99% in both arms, so this is lost sensitivity and not a changed threshold. The canonical treatment of the flag, including the full peak census and the cap @@ -295,7 +295,7 @@ remaining lever, then rescore, then seed, all on faint signal. Before reading a presence/apex loss on any other run as an extraction problem, check the conversion cap. A cap that truncates most spectra produces exactly the -same signature (`NO_PEAK_GROUP` at `candidate_generated`) while the cause is the +same signature (`DID_NOT_SURVIVE_EXTRACTION` at `candidate_generated`) while the cause is the converted artifact, not the extraction thresholds. On the 50-window run above, 62.3% of the confirmed-present peptides were lost this way at a cap of 300. diff --git a/docs/24_config_reference.md b/docs/24_config_reference.md index af4a3e39..4bcbfb18 100644 --- a/docs/24_config_reference.md +++ b/docs/24_config_reference.md @@ -707,55 +707,55 @@ listed with the file it is in. | `CONDA_PREFIX` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/python.rs:207` | | `DEEPLC_FT_THREADS` | sidecar | `"8"` | `scripts/deeplc_finetune.py:27` | | `MUMDIA_BREW_ITERS` | sidecar | `"20"` | `scripts/mokapot_worker.py:38` | -| `MUMDIA_ENTRAPMENT_MODEL` | sidecar | `"gbm"` | `scripts/entrapment_worker.py:34` | +| `MUMDIA_ENTRAPMENT_MODEL` | sidecar | `"gbm"` | `scripts/entrapment_worker.py:39` | | `MUMDIA_LR_C` | sidecar | `"1.0"` | `scripts/mokapot_worker.py:48` | | `MUMDIA_LR_MAX_ITER` | sidecar | `"1000"` | `scripts/mokapot_worker.py:49` | | `MUMDIA_MOKAPOT_WORKERS` | sidecar | `"3"` | `scripts/mokapot_worker.py:120` | | `MUMDIA_MSCONVERT` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/raw.rs:193` | | `MUMDIA_NN_ALPHA` | sidecar | `"1e-4"` | `scripts/mokapot_worker.py:90` | -| `MUMDIA_NN_BATCH` | sidecar | `4096` | `scripts/nn_rescore_worker.py:280` | -| `MUMDIA_NN_CHUNK` | sidecar | `250000` | `scripts/nn_rescore_worker.py:284` | -| `MUMDIA_NN_DEVICE` | sidecar | `"auto"` | `scripts/nn_rescore_worker.py:292` | -| `MUMDIA_NN_DROPOUT` | sidecar | `0.3` | `scripts/nn_rescore_worker.py:277` | -| `MUMDIA_NN_EARLY_STOP` | sidecar | `1` | `scripts/nn_rescore_worker.py:285` | -| `MUMDIA_NN_EARLY_STOP_TOL` | sidecar | `0.01` | `scripts/nn_rescore_worker.py:286` | -| `MUMDIA_NN_EPOCHS` | sidecar | `25` | `scripts/nn_rescore_worker.py:275` | -| `MUMDIA_NN_FEATURES` | sidecar | `""` | `scripts/nn_rescore_worker.py:368` | -| `MUMDIA_NN_FOLDS` | sidecar | `3` | `scripts/nn_rescore_worker.py:263` | -| `MUMDIA_NN_FOLD_KEYS` | sidecar | `""` | `scripts/nn_rescore_worker.py:407` | -| `MUMDIA_NN_HIDDEN` | sidecar | `"128,64"` in nn_rescore_worker.py; `"128,64,64,32"` in mokapot_worker.py | `scripts/mokapot_worker.py:83`, `scripts/nn_rescore_worker.py:276` | -| `MUMDIA_NN_INIT_SAMPLE` | sidecar | `300000` | `scripts/nn_rescore_worker.py:537` | -| `MUMDIA_NN_INIT_TOPK` | sidecar | `0` | `scripts/nn_rescore_worker.py:659` | -| `MUMDIA_NN_ITERS` | sidecar | `5` | `scripts/nn_rescore_worker.py:274` | -| `MUMDIA_NN_LR` | sidecar | `1e-3` | `scripts/nn_rescore_worker.py:278` | -| `MUMDIA_NN_MARGIN_FRAC` | sidecar | `0.5` | `scripts/nn_rescore_worker.py:273` | +| `MUMDIA_NN_BATCH` | sidecar | `4096` | `scripts/nn_rescore_worker.py:296` | +| `MUMDIA_NN_CHUNK` | sidecar | `250000` | `scripts/nn_rescore_worker.py:300` | +| `MUMDIA_NN_DEVICE` | sidecar | `"auto"` | `scripts/nn_rescore_worker.py:308` | +| `MUMDIA_NN_DROPOUT` | sidecar | `0.3` | `scripts/nn_rescore_worker.py:293` | +| `MUMDIA_NN_EARLY_STOP` | sidecar | `1` | `scripts/nn_rescore_worker.py:301` | +| `MUMDIA_NN_EARLY_STOP_TOL` | sidecar | `0.01` | `scripts/nn_rescore_worker.py:302` | +| `MUMDIA_NN_EPOCHS` | sidecar | `25` | `scripts/nn_rescore_worker.py:291` | +| `MUMDIA_NN_FEATURES` | sidecar | `""` | `scripts/nn_rescore_worker.py:387` | +| `MUMDIA_NN_FOLDS` | sidecar | `3` | `scripts/nn_rescore_worker.py:279` | +| `MUMDIA_NN_FOLD_KEYS` | sidecar | `""` | `scripts/nn_rescore_worker.py:426` | +| `MUMDIA_NN_HIDDEN` | sidecar | `"128,64"` in nn_rescore_worker.py; `"128,64,64,32"` in mokapot_worker.py | `scripts/mokapot_worker.py:83`, `scripts/nn_rescore_worker.py:292` | +| `MUMDIA_NN_INIT_SAMPLE` | sidecar | `300000` | `scripts/nn_rescore_worker.py:564` | +| `MUMDIA_NN_INIT_TOPK` | sidecar | `0` | `scripts/nn_rescore_worker.py:686` | +| `MUMDIA_NN_ITERS` | sidecar | `5` | `scripts/nn_rescore_worker.py:290` | +| `MUMDIA_NN_LR` | sidecar | `1e-3` | `scripts/nn_rescore_worker.py:294` | +| `MUMDIA_NN_MARGIN_FRAC` | sidecar | `0.5` | `scripts/nn_rescore_worker.py:289` | | `MUMDIA_NN_MAX_ITER` | sidecar | `"200"` | `scripts/mokapot_worker.py:91` | -| `MUMDIA_NN_NEG_RATIO` | sidecar | `0.0` | `scripts/nn_rescore_worker.py:267` | -| `MUMDIA_NN_NEG_SELECT` | sidecar | `"random"` | `scripts/nn_rescore_worker.py:268` | -| `MUMDIA_NN_PREGATHER_GB` | sidecar | `8` | `scripts/nn_rescore_worker.py:287` | -| `MUMDIA_NN_SEED` | sidecar | `0` | `scripts/nn_rescore_worker.py:283` | -| `MUMDIA_NN_SEEDS` | sidecar | `1` | `scripts/nn_rescore_worker.py:282` | +| `MUMDIA_NN_NEG_RATIO` | sidecar | `0.0` | `scripts/nn_rescore_worker.py:283` | +| `MUMDIA_NN_NEG_SELECT` | sidecar | `"random"` | `scripts/nn_rescore_worker.py:284` | +| `MUMDIA_NN_PREGATHER_GB` | sidecar | `8` | `scripts/nn_rescore_worker.py:303` | +| `MUMDIA_NN_SEED` | sidecar | `0` | `scripts/nn_rescore_worker.py:299` | +| `MUMDIA_NN_SEEDS` | sidecar | `1` | `scripts/nn_rescore_worker.py:298` | | `MUMDIA_NN_SOLVER` | sidecar | `"adam"` | `scripts/mokapot_worker.py:89` | -| `MUMDIA_NN_STREAM` | sidecar | `"auto"` | `scripts/nn_rescore_worker.py:329` | -| `MUMDIA_NN_STREAM_GB` | sidecar | `4` | `scripts/nn_rescore_worker.py:338` | -| `MUMDIA_NN_THREADS` | both | `16` | `rust/mumdia/crates/mumdia/src/main.rs:93`, `scripts/nn_rescore_worker.py:313`, `scripts/nn_rescore_worker.py:314` | -| `MUMDIA_NN_TRAIN_FDR` | sidecar | `0.01` | `scripts/nn_rescore_worker.py:281` | -| `MUMDIA_NN_TRAIN_SUB` | sidecar | `0.0` | `scripts/nn_rescore_worker.py:264` | -| `MUMDIA_NN_WARM_EPOCHS` | sidecar | `0` | `scripts/nn_rescore_worker.py:266` | -| `MUMDIA_NN_WARM_START` | sidecar | `0` | `scripts/nn_rescore_worker.py:265` | -| `MUMDIA_NN_WD` | sidecar | `1e-4` | `scripts/nn_rescore_worker.py:279` | +| `MUMDIA_NN_STREAM` | sidecar | `"auto"` | `scripts/nn_rescore_worker.py:345` | +| `MUMDIA_NN_STREAM_GB` | sidecar | `4` | `scripts/nn_rescore_worker.py:357` | +| `MUMDIA_NN_THREADS` | both | `16` | `rust/mumdia/crates/mumdia/src/main.rs:93`, `scripts/nn_rescore_worker.py:329`, `scripts/nn_rescore_worker.py:330` | +| `MUMDIA_NN_TRAIN_FDR` | sidecar | `0.01` | `scripts/nn_rescore_worker.py:297` | +| `MUMDIA_NN_TRAIN_SUB` | sidecar | `0.0` | `scripts/nn_rescore_worker.py:280` | +| `MUMDIA_NN_WARM_EPOCHS` | sidecar | `0` | `scripts/nn_rescore_worker.py:282` | +| `MUMDIA_NN_WARM_START` | sidecar | `0` | `scripts/nn_rescore_worker.py:281` | +| `MUMDIA_NN_WD` | sidecar | `1e-4` | `scripts/nn_rescore_worker.py:295` | | `MUMDIA_PYTHON` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/python.rs:198` | | `MUMDIA_PYTHON_DEEPLC` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/python.rs:197` | | `MUMDIA_PYTHON_MBR` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/python.rs:197` | | `MUMDIA_PYTHON_MS2PIP` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/python.rs:197` | | `MUMDIA_PYTHON_RESCORE` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/python.rs:197` | -| `MUMDIA_RESCORE_MODEL` | both | `"nn"` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:320`, `scripts/mokapot_worker.py:181`, `scripts/mokapot_worker.py:37` | +| `MUMDIA_RESCORE_MODEL` | both | `"nn"` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:353`, `scripts/mokapot_worker.py:181`, `scripts/mokapot_worker.py:37` | | `MUMDIA_THERMO_PARSER` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/raw.rs:171` | | `MUMDIA_XGB_DEPTH` | sidecar | `"6"` | `scripts/mokapot_worker.py:63` | | `MUMDIA_XGB_JOBS` | sidecar | `"0"` | `scripts/mokapot_worker.py:68` | | `MUMDIA_XGB_LR` | sidecar | `"0.1"` | `scripts/mokapot_worker.py:64` | | `MUMDIA_XGB_TREES` | sidecar | `"200"` | `scripts/mokapot_worker.py:62` | -| `OMP_NUM_THREADS` | both | `16` | `rust/mumdia/crates/mumdia/src/main.rs:93`, `scripts/nn_rescore_worker.py:316`, `scripts/nn_rescore_worker.py:317` | +| `OMP_NUM_THREADS` | both | `16` | `rust/mumdia/crates/mumdia/src/main.rs:93`, `scripts/nn_rescore_worker.py:332`, `scripts/nn_rescore_worker.py:333` | | `PATH` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/raw.rs:293` | | `ProgramFiles` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/raw.rs:201` | | `ProgramFiles(x86)` | engine | none (unset means off) | `rust/mumdia/crates/mumdia/src/raw.rs:202` | @@ -774,23 +774,23 @@ one exception noted in its own help text: it sets `MUMDIA_NN_THREADS` and |---|---|---|---| | `KMP_DUPLICATE_LIB_OK` | sidecar | `"TRUE"` | `scripts/deeplc_finetune.py:28` | | `MKL_NUM_THREADS` | sidecar | `"1"` | `scripts/deeplc_finetune.py:31` | -| `MUMDIA_NN_FOLDS` | engine | `p.cfg.folds.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1322` | -| `MUMDIA_NN_FOLD_KEYS` | engine | `&foldkeys` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1349` | -| `MUMDIA_NN_ITERS` | engine | `p.cfg.num_iter.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1323` | -| `MUMDIA_NN_MARGIN_FRAC` | engine | `p.cfg.train_margin_frac.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1347` | -| `MUMDIA_NN_NEG_RATIO` | engine | `p.cfg.train_neg_ratio.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1328` | -| `MUMDIA_NN_NEG_SELECT` | engine | `match p.cfg.train_neg_select { mumdia_core::config::NegSelect::Random => "random", mumdia_core::config::NegSelect::Margin => "margin", mumdia_core::config::NegSelect::Hybrid => "hybrid", }` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1329` | -| `MUMDIA_NN_SEEDS` | engine | `p.cfg.seeds.max(1).to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1348` | +| `MUMDIA_NN_FOLDS` | engine | `p.cfg.folds.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1355` | +| `MUMDIA_NN_FOLD_KEYS` | engine | `&foldkeys` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1382` | +| `MUMDIA_NN_ITERS` | engine | `p.cfg.num_iter.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1356` | +| `MUMDIA_NN_MARGIN_FRAC` | engine | `p.cfg.train_margin_frac.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1380` | +| `MUMDIA_NN_NEG_RATIO` | engine | `p.cfg.train_neg_ratio.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1361` | +| `MUMDIA_NN_NEG_SELECT` | engine | `match p.cfg.train_neg_select { mumdia_core::config::NegSelect::Random => "random", mumdia_core::config::NegSelect::Margin => "margin", mumdia_core::config::NegSelect::Hybrid => "hybrid", }` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1362` | +| `MUMDIA_NN_SEEDS` | engine | `p.cfg.seeds.max(1).to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1381` | | `MUMDIA_NN_THREADS` | engine | `n.to_string()` | `rust/mumdia/crates/mumdia/src/main.rs:94` | -| `MUMDIA_NN_TRAIN_FDR` | engine | `p.cfg.train_fdr.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1324` | -| `MUMDIA_NN_TRAIN_SUB` | engine | `p.cfg.train_subsample.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1337` | -| `MUMDIA_NN_WARM_EPOCHS` | engine | `p.cfg.train_warm_epochs.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1346` | -| `MUMDIA_NN_WARM_START` | engine | `if p.cfg.train_warm_epochs > 0 { "1" } else { "0" }` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1338` | +| `MUMDIA_NN_TRAIN_FDR` | engine | `p.cfg.train_fdr.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1357` | +| `MUMDIA_NN_TRAIN_SUB` | engine | `p.cfg.train_subsample.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1370` | +| `MUMDIA_NN_WARM_EPOCHS` | engine | `p.cfg.train_warm_epochs.to_string()` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1379` | +| `MUMDIA_NN_WARM_START` | engine | `if p.cfg.train_warm_epochs > 0 { "1" } else { "0" }` | `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1371` | | `NUMEXPR_NUM_THREADS` | sidecar | `"1"` | `scripts/deeplc_finetune.py:32` | | `OMP_NUM_THREADS` | both | `"1"` in deeplc_finetune.py; `n.to_string()` in main.rs | `rust/mumdia/crates/mumdia/src/main.rs:94`, `scripts/deeplc_finetune.py:29` | | `OPENBLAS_NUM_THREADS` | sidecar | `"1"` | `scripts/deeplc_finetune.py:30` | -| `PYTHONIOENCODING` | engine | `"utf-8"` | `rust/mumdia/crates/mumdia/src/sidecar.rs:332` | -| `PYTHONUTF8` | engine | `"1"` | `rust/mumdia/crates/mumdia/src/sidecar.rs:332`, `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1078`, `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1316` | +| `PYTHONIOENCODING` | engine | `"utf-8"` | `rust/mumdia/crates/mumdia/src/sidecar.rs:425` | +| `PYTHONUTF8` | engine | `"1"` | `rust/mumdia/crates/mumdia/src/sidecar.rs:425`, `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1111`, `rust/mumdia/crates/mumdia/src/stages/rescore.rs:1349` | ## Unresolved by the generator @@ -806,8 +806,8 @@ Every field whose struct has an `impl Default` resolved from the source. 2 environment read(s) whose name is not a literal: -- `rust/mumdia/crates/mumdia/src/stages/extract.rs:2685: env read via closure of `&mut flushed`` -- `rust/mumdia/crates/mumdia/src/stages/extract.rs:2703: env read via closure of `&mut cand_hits`` +- `rust/mumdia/crates/mumdia/src/stages/extract.rs:2693: env read via closure of `&mut flushed`` +- `rust/mumdia/crates/mumdia/src/stages/extract.rs:2711: env read via closure of `&mut cand_hits`` ## Coverage diff --git a/docs/28_feature_selection_analysis.md b/docs/28_feature_selection_analysis.md index 9827aeba..a7d78d44 100644 --- a/docs/28_feature_selection_analysis.md +++ b/docs/28_feature_selection_analysis.md @@ -94,8 +94,13 @@ labels, masses and peptides in the same order. The objective is measured with `bench/feature_selection/fs_lib.py::run_rescoring`, a re-implementation of `scripts/nn_rescore_worker.py` under the configuration the engine -ships: md5(stripped peptide) folds (3), per-fold init-feature scan over both signs, +ships: md5(base sequence, `DECOY_` prefix stripped) folds (3), per-fold init-feature scan over both signs, Percolator-style self-training on targets at 1% vs all decoys for `num_iter = 10` +(fold rule since 2026-09-07; the studies in this document hashed the peptide with its +`DECOY_` prefix, so a target and its paired decoy could fall in different folds, unlike +the worker's explicit base-peptide pairing; docs/29 #20. Each benchmark row now carries a +`recipe` column with the code revision, fold rule, feature count, seed and training +settings it was produced with.) Then iterations (the `RescoreConfig` default, passed as `MUMDIA_NN_ITERS`; the worker's own docstring default of 5 is never what the engine runs), a fresh MLP 128-64 with dropout 0.3 per iteration, churn early stop at 1% (never triggered here, as in the real run), mean/std diff --git a/docs/29_code_review_2026-09-07.md b/docs/29_code_review_2026-09-07.md new file mode 100644 index 00000000..a7263a0b --- /dev/null +++ b/docs/29_code_review_2026-09-07.md @@ -0,0 +1,298 @@ +# MuMDIA code review — 2026-09-07 + +Review baseline: `ab7049e` (`desktop: digest fields reach the built-in library path; settings start from the preset`). Revised after maintainer spot-checking and scope decisions. The code references and reproduction results describe that review baseline; the changes below are planned, not reported as implemented. This revision changes only the review document. + +The tracked implementation contains 81 Rust files (43,102 lines), 41 Python files (11,910 lines), and the desktop JavaScript/HTML/CSS (2,407 lines). Review scope covered the engine stages, shared configuration/IO/numerical code, workers and library helpers, desktop supervision/UI, tests, benchmark tooling, and delivery configuration. Local data, results, and the third-party `alphadia/` comparison checkout were excluded. Existing `-covr2` scratch copies were inventoried because some affect local tooling. + +**Main conclusion:** prioritize data integrity and confidence estimation before further sensitivity tuning. Several checks described as already fixed in the historical review have either regressed during streaming changes or remain incomplete at adjacent entry points. Green contract tests currently miss important violations of those contracts. + +The maintainer independently confirmed findings **1, 2, 3, 4, 6, 7, 9, 10, 11, and 21** as written. The remaining findings were consistent with the code they spot-checked. The documentation contradiction is also confirmed: top-K promotion already exists behind `retain_top_peaks`; rescore selects the best peak per candidate and writes `selected_peak_rank`. It is not a missing implementation. + +The agreed scope is **four PRs plus local hygiene**. Estimates below are the maintainer's planning estimates, not measured completion times. Reproductions remain evidence for the defects; the accepted changes and explicit deferrals replace the original open-ended recommendations. + +| Work package | Scope | Estimate | +|---|---|---| +| A. Engine data integrity | #1, #2, #4, #9, #11, #17, #18: typed/null contracts, safe publication, digest boundary, memory preflight, predictor coverage, numeric validation | About 1 day | +| B. Workers | #3, #6, #7, #8, #12, #20: held-out scoring, transfer confidence/diagnostics, selected apex, base IDs, minimal benchmark correction | About 1 day | +| C. Desktop and output ownership | #5, #13, #14, plus desktop Dependabot/cargo-audit coverage; includes CLI case-insensitive run-name uniqueness | About 1 day | +| D. Provenance, reporting, calibration, docs | #10, #15, #16, #19 and stale guide statements; HYE B01 before/after count for LOESS before merge | About 1 day | +| Local hygiene | #21: move the 113 identified scratch copies to the existing backup folder without deletion; switch the two scanners to `git ls-files` | Separate from A–D; no scratch-move code review needed | + +**Evidence labels:** *Reproduced* means a small executable probe demonstrated the failure using current code. *Source-confirmed* means the trigger and consequence follow from the current implementation; no full application or biological-data reproduction was performed. *Improvement* denotes an explicit design or maintenance recommendation rather than an unintended behavior. + +**Priorities:** P1 = fix before relying on the affected workflow; P2 = next correctness/reliability work; P3 = maintenance or usability. These priorities apply to the stated trigger, not to every MuMDIA run. + +| Priority | Finding | Affected workflow | +|---|---|---| +| P1 | 1. Pooled quant guard reads the wrong integer type | Manual quantification of pooled scored tables | +| P1 | 2. Streaming fragment loading accepts NULLs as NaNs | Imported/external fragment libraries | +| P1 | 3. Entrapment worker substitutes in-sample scores | Entrapment with class-deficient training folds | +| P1 | 4. Artifact replacement can destroy the previous output | Failed replacement/concurrent writers | +| P1 | 5. Output-directory ownership is not enforced | Desktop duplicate starts; Windows run-name aliases | +| P2 | 6. MBR empirical decoy fraction cannot validate transfers | MBR validation | +| P2 | 7. MBR can assign zero q-values to a tiny transfer pool | MBR confidence estimation | +| P3 | 8. MBR loses the selected-peak identity | Opt-in top-K + MBR; not triggered at default `retain_top_peaks = 1` | +| P2 | 9. Met excision misses the maximum-length boundary | Native digest/library augmentation | +| P2 | 10. LOESS has discontinuous endpoint predictions | RT calibration and alignment | +| P2 | 11. Rescore memory limit runs after allocation and uses the wrong width | Large/pooled rescoring | +| P2 | 12. Library augmentation splits one peptide across base IDs | `--match-level peptidoform_charge` | +| P2 | 13. Desktop converter preflight disagrees with the engine | Explicit converters; Thermo via msconvert | +| P2 | 14. Cancellation can finish as failed | Desktop stop action | +| P2 | 15. Experiment provenance discards resolved configuration | Reproduction and long-running experiments | +| P2 | 16. Audit labels PSM q as precursor FDR | Candidate loss analysis | +| P2 | 17. Incomplete predictor output can become a mixed/incorrect library | Predictor sidecar failures | +| P2 | 18. Numeric configuration validation is incomplete | Invalid but accepted configuration | +| P2 | 19. Transferred reports conceal their acceptance basis | MBR reporting and threshold changes | +| P3 | 20. Feature-selection benchmark has drifted from production | Benchmark interpretation and future tuning | +| P3 | 21. Scratch files affect supposedly source-derived tooling | Local tests/docs generation | + +**1. P1 — pooled quantification's safety check is bypassed by native output. Reproduced.** + +`rust/mumdia/crates/mumdia/src/stages/quant.rs:522` uses `if let Ok(source) = ps.i32("source")`. Rescore writes `source` as `U32`, and the IO accessor only accepts `Int32Array` (`mumdia-io/src/table.rs:766`). The type error is treated as an absent optional column, so the pooled-table refusal never runs for normal engine output. + +A two-row table with candidate 0 in sources 0 and 1 was rejected when `source` was signed, but accepted when it was unsigned. Against a single chromatogram, quant returned two peptide rows with quantities `[15.0, 15.0]`. This is a real duplicate-quantity path. The ordinary `run-experiment` split protects its own calls; manual `rescore` followed by `quant` does not. + +Accepted change (A): test column presence separately, read its declared unsigned type, and propagate errors on malformed present columns. Test with a table emitted by rescore, not an independently constructed signed approximation. + +**2. P1 — streaming library loading bypasses NULL validation. Reproduced.** + +`rust/mumdia/crates/mumdia/src/index.rs:315` validates `a_mz.values()` and `a_int.values()`, which are the physical Arrow value buffers and do not apply validity bits. At lines 350–359 it subsequently converts NULL cells into NaNs. The candidate-ID counting/fill passes also read raw IDs without rejecting NULLs, and NULL names become empty strings. + +A valid two-candidate precursor table plus NULL fragment m/z and intensity loaded successfully with `Library::load_with(..., false)`. Its resulting arrays were `mz=[NaN, 250.0]` and `intensity=[1.0, NaN]`. Thus the check does not enforce the library invariant it describes. Corrupt fragment values can reach matching and similarity calculations; nullable IDs can attach fragments to the wrong candidate. + +Accepted change (A): reject NULLs in every required fragment column before accessing buffers. Share the validity check with the typed getters, retain numeric finiteness checks, and add a NULL fixture for each required column. + +**3. P1 — entrapment out-of-fold gaps are filled by training on the held-out rows. Reproduced.** + +`scripts/entrapment_worker.py:95` skips a fold whose training set has only one class. Lines 104–110 then fit a full model and use it to fill every missing score, including the skipped validation rows. Those scores are consumed as confidence-estimation input. + +The probe used two groups, one real-target group and one entrapment group. Both training folds were single-class. A model spy replacing only the learner recorded that all four output rows were scored by a model trained on those exact rows. The worker still completed successfully. + +Accepted change (B): a single-class training fold is an error. Never fill held-out target/entrapment rows with in-sample predictions. Keep full-model scoring of excluded decoys separate, and test group exclusion itself rather than only finite coverage and score separation. + +**4. P1 — the common artifact publisher is not atomic and is not failure-preserving. Reproduced.** + +`rust/mumdia/crates/mumdia-io/src/table.rs:377` removes an existing final file before attempting the rename. This happens on every platform. If the rename fails, the previous result is already gone. Readers also have a window in which no final file exists, contrary to the `AtomicPath`/JSON writer contract. + +A forced publication failure demonstrated `previous_output_survives=false`. Two live `AtomicPath` objects for the same destination also produced identical temporary paths: the suffix contains only the process ID. Concurrent invocations in one process can therefore overwrite or remove one another's temporary file. + +Accepted change (A): remove the pre-delete and use Rust's existing rename replacement behavior, including on Windows. Use unique temporary names containing a PID and counter. Test preservation of the previous artifact on failure and independent temporary names for writers in the same process. + +Correction to the original recommendation: this does not promise successful replacement while another process holds the destination open on Windows with incompatible sharing permissions. Such a reader can still make rename fail. The required guarantee is that failure leaves the previous result intact, not that every concurrent-reader scenario succeeds. + +**5. P1 — prevent two active runs from owning the same output directory. Source-confirmed; Windows alias reproduced.** + +`desktop/ui/app.js:1322` starts an asynchronous flow with several awaits but no start-in-progress guard. `desktop/src-tauri/src/main.rs:392` launches every request before inserting the handle, without checking active output-directory ownership. `run.rs:430` simply creates/reuses the directory. A repeated start can therefore launch two engines writing the same artifact set; the frontend retains only the latest run ID. + +There is a related CLI issue in `rust/mumdia/crates/mumdia/src/stages/run_experiment.rs:359`: run-name uniqueness is checked as case-sensitive strings. `RunA` and `runa` pass that check but address the same directory on this Windows filesystem. The filesystem alias was reproduced. Sequential processing overwrites earlier artifacts; parallel processing can interleave runs. + +Accepted change (C): add a start-in-progress guard before the frontend's first await and reserve active output directories in the backend before spawning. Reject conflicting requests and release reservations on terminal completion. For CLI run names, compare uniqueness case-insensitively on every platform. This deliberately conservative rule replaces the original recommendation to probe the destination filesystem. Test duplicate starts and case-only name differences. + +**6. P2 — MBR's printed empirical decoy fraction is structurally zero. Reproduced.** + +`scripts/mbr_worker.py:97` builds only confident target sets. At line 119, `allc` is their union, and every transfer candidate comes from it. Nevertheless, lines 251–254 print the fraction of accepted rows labelled `decoy` as an empirical validation statistic. With a valid library's stable candidate labels, decoys cannot enter that calculation. + +A fixture containing an otherwise equally eligible decoy produced three target candidates, excluded the decoy, and printed `empirical decoy-frac=0.00%`. This does not establish that transfer confidence is calibrated. The worker does construct a separate permuted-RT null; the specific bug is presenting the unreachable label count as independent validation. + +Accepted change (B): replace the structurally zero “empirical decoy fraction” with explicitly named permuted-null counts. A paired decoy-transfer population is research and is deferred; it is not a requirement for this bug fix. Neither the old percentage nor the replacement count alone establishes biological calibration. + +**7. P2 — transfer q estimation has no small-sample safeguard. Reproduced.** + +`scripts/mbr_worker.py:197` computes `dec_cum / tgt_cum`, whereas native target-decoy q estimation uses a +1 safeguard. If no permuted residual is as good as the first observed residuals, their estimated q is exactly zero, however small the pool. + +The three-target probe returned `transfer_q=[0.0, 0.0, 0.0]` and accepted all three at 1%. This demonstrates zero-tail estimation, not a measured biological false-discovery rate. The current tests use hundreds of well-separated planted transfers and do not exercise this small-pool behavior. + +Accepted change (B): use a +1 numerator for transfer q estimation and add a small-pool regression test. This removes the reproduced zero-tail behavior; it is not a claim that the entire transfer procedure has thereby been scientifically validated. Redesigning the null population is deferred with #6. + +**8. P3 — MBR discards the peak selected by rescoring. Source-confirmed; lower urgency.** + +`scripts/mbr_worker.py:104` reduces per-run competed rows to `dict(candidate_id -> apex_rt)`, retaining the last row. With promoted top-K peaks, a competed table can contain multiple peaks for one candidate. Rescore explicitly selects the best scoring peak per `(source, candidate_id)` and writes its apex and `selected_peak_rank` (`stages/rescore.rs:451`). The experiment passes the original competed tables to MBR (`stages/run_experiment.rs:609`), so the worker can evaluate transfer RT using a different peak. + +Its augmentation then changes the scored row by `(candidate_id, source)` without moving that row's apex. Consequently the peak whose RT justified a transfer can differ from the peak subsequently quantified. + +Accepted change (B): use the rescore-selected apex, joining the candidate/source and `selected_peak_rank` to the corresponding competed peak. Test a top-K + MBR fixture in which the last competed peak is not the rescore winner. This mismatch cannot trigger at the default `retain_top_peaks = 1`, so it has lower urgency than the worker confidence defects. + +**9. P2 — N-terminal Met excision is skipped when the original peptide is one residue too long. Reproduced.** + +`rust/mumdia/crates/mumdia/src/stages/digest.rs:86` rejects the original peptide by length before considering the excised form at line 102. An original length of `max_len + 1` is rejected even when removing its initial M produces an allowed peptide. + +With FASTA `MPEPTIDK`, `min_len=max_len=7`, and Met excision enabled, the digest returned no peptides instead of `PEPTIDK`. Decoys were disabled only to isolate the digest-boundary probe. + +Accepted change (A): independently validate the ordinary and Met-excised intervals. Test exactly `max_len + 1`, `min_len`, and nonterminal M cases. + +**10. P2 — LOESS jumps to an unrelated line at both training boundaries. Reproduced.** + +`rust/mumdia/crates/mumdia/src/calibrate.rs:86` uses a global linear fit for `x <= min(grid)` and `x >= max(grid)`, but local-fit interpolation just inside those endpoints. The two models need not agree at either endpoint. + +For `x=0,0.1,...,9.9` and `y=200+10*x²`, span 0.3, the actual implementation returned 38.30 seconds at x=0 and 193.38 seconds at x=0.000001; at the upper boundary it jumped from 1173.48 to 1018.40 seconds. These approximately 155-second discontinuities can misplace edge peptides relative to a narrow extraction window and distort calibration diagnostics. + +Accepted change (D): extrapolate continuously from the boundary local fit. Test endpoint continuity with nonlinear data. Because this changes RT windows at gradient edges, run a **HYE B01 before/after identification count before merge**, recording the row and q-value unit. That comparison is a merge check for this correction, not evidence for an unrelated sensitivity-default change. + +**11. P2 — the rescore memory ceiling checks the wrong allocation after it has happened. Source-confirmed.** + +`rust/mumdia/crates/mumdia/src/stages/rescore.rs:146` reserves the full matrix, fills it, and calls `finish()` before estimating its size at line 269 and checking `max_feature_matrix_gib`. The estimate still assumes `n * features * 8 + n * 24`, despite `rescoring.rs:22` storing one contiguous f32 buffer. + +The ceiling therefore cannot prevent the initial allocation/OOM, and it can reject a matrix using roughly half the estimated storage. Its diagnostic also describes an obsolete representation. + +Accepted change (A): check the matrix size before allocation using metadata, the selected feature count, checked arithmetic, and the actual flat f32 layout. Test a threshold between the actual and old estimated sizes. The setting remains a matrix limit; a comprehensive process/worker memory estimator is not required for this correction. + +**12. P2 — augmentation can give sibling forms different base-peptide IDs. Source-confirmed.** + +`scripts/augment_library.py:145` offsets every newly predicted `base_peptide_id`. This is appropriate for wholly new sequences, but `--match-level peptidoform_charge` also adds missing charge states/modforms of sequences already present. Their new base IDs no longer match the imported sibling's ID. The subsequent shift-decoy helper preserves that base-ID distinction. + +This splits one stripped peptide across peptide-level competition groups and potentially CV folds. The helper's comment that it keeps base linkage is only true within the newly added subset. + +Accepted change (B): construct a canonical stripped-sequence-to-base-ID map over imported and added targets, reuse existing IDs for existing sequences, allocate IDs only for new sequences, and preserve paired decoy IDs. Test adding a charge and a modification to an existing peptide. + +**13. P2 — desktop converter discovery ignores the request's configuration and a supported fallback. Source-confirmed.** + +`desktop/src-tauri/src/thermo.rs:152` invokes `doctor --json` without the selected config. `desktop/src-tauri/src/main.rs:258` uses that result to hard-block raw inputs. A converter available only through `convert.thermo_raw_parser` or `convert.msconvert` in the request's config is therefore treated as missing. + +The same preflight requires the Thermo parser specifically even when msconvert is present; the engine explicitly supports msconvert as the fallback for Thermo when the parser is left at `auto` (`rust/mumdia/crates/mumdia/src/raw.rs:584`). + +Accepted change (C): probe with the request's effective config and honor the msconvert fallback when the Thermo parser is `auto`. Test an explicit off-PATH converter and Thermo with only msconvert. An invalid explicitly selected parser should still be reported as an error, matching the engine. + +**14. P2 — a cancelled desktop run can be published as failed. Source-confirmed.** + +`desktop/src-tauri/src/run.rs:159` sets an atomic cancellation flag, kills the tree, sweeps temporary files, and only then changes snapshot status if it is still running/starting. The waiter at line 565 can wake during that interval and publish `failed`. The cancel method then refuses to replace that terminal status. The cancellation flag is written but never read. + +Accepted change (C): the waiter consults cancellation intent and uses one terminal-state transition. Add a controlled race test that lets the waiter complete while cancellation is in progress. + +**15. P2 — experiment provenance omits information needed to reproduce the run. Source-confirmed.** + +`rust/mumdia/crates/mumdia/src/stages/run_experiment.rs:746` creates a `Manifest`, but the final hand-built JSON at line 824 drops its `config_json` and model identities. It records a configuration hash without the resolved configuration itself. The shared library and per-run intermediate artifacts are also outside the experiment artifact inventory, as the source acknowledges. + +Input hashes are collected at the end, unlike the single-run orchestrator's beginning-of-run hashes. If an input changes during a long experiment, the recorded bytes may not be the bytes used by the search. Quantification also overrides `q_filter` while artifact provenance continues to use the original configuration hash. + +Accepted change (D): include the resolved config, effective `q_filter`, actual model identities, and input hashes captured at the start. Relocatable references are deferred. Completing the broader shared/per-run artifact inventory can follow with the executor/provenance refactor; that refactor is not required for this PR. A config hash alone is not a replayable configuration. + +**16. P2 — the audit's precursor-FDR label actually uses PSM q. Source-confirmed.** + +`rust/mumdia/crates/mumdia/src/stages/audit.rs:89` reads `q_value` into `scored_q`; the later `passed_prec` flag and `FailedPrecursorFdr` reason use that value. The scored schema also has a separate `precursor_q`. These columns are not interchangeable. + +The audit also reduces rows by candidate ID alone and does not select a source, so a pooled scored input overwrites earlier runs' entries. Finally, extraction rejection categories remain coarse because `emit_candidate_audit` does not produce the promised per-candidate table. + +Accepted change (D): read the q unit named by the label, require one source, and call the coarse extraction bucket “did not survive extraction.” Test a row whose PSM and precursor q values lie on opposite sides of the threshold. An additional source-selection interface and precise extraction-reason instrumentation are not required for this correction. + +**17. P2 — prediction workers do not have the rescorer's complete-output contract. Source-confirmed.** + +`rust/mumdia/crates/mumdia/src/sidecar.rs:158` collects returned DeepLC IDs into a map without duplicate/coverage validation. `stages/predict_frag.rs:339` substitutes iRT 0.0 for missing predictions, warning but continuing. The MS2PIP branch at line 415 substitutes native intensities for an entirely missing candidate, while reporting the MS2PIP model identity for the library. + +A partially successful worker can therefore yield a plausible, finite library containing an unintended mixture of prediction sources. The final finiteness check does not detect missing rows replaced by valid numbers. + +Accepted change (A): validate returned IDs and coverage, then **drop uncovered candidates with a counted warning** and record the dropped count in the library report. Do not substitute iRT 0.0 or native intensities for a missing whole prediction. A handful of unpredicted peptidoforms should not abort a whole-proteome build; the original fail-by-default recommendation is withdrawn. Keep unsupported-ion behavior distinct from missing candidates, and retain the library's target/decoy pairing invariants when filtering. Test missing and duplicate IDs as well as the reported drop count. + +**18. P2 — accepted numeric configurations can silently disable the intended analysis. Reproduced at config load.** + +`rust/mumdia/crates/mumdia-core/src/config.rs:1763` validates some numeric fields but leaves several important domains unchecked. Current `Config::from_json` accepts all of: + +```json +{"quant":{"q_threshold":-0.1}} +{"rt_im_train":{"rt_window_multiplier":-1.0}} +{"rescore":{"train_margin_frac":2.0}} +``` + +The negative q threshold excludes ordinary discoveries. The negative RT multiplier is later clamped into a one-second window (`stages/rt_im_train.rs:293`), potentially suppressing real candidates rather than rejecting the bad setting. A fraction above one is outside the documented training recipe's domain. + +Accepted change (A): validate numeric domains at load: thresholds and fractions in `[0, 1]`, positive multipliers, and ordered min/max pairs. Preserve any stricter existing field contract and distinguish documented count/disabled-value semantics from fractions. Invalid settings must fail before expensive stages, rather than be clamped into a different analysis. A wider configuration/schema refactor is not required for this PR. + +**19. P2 — MBR reporting does not expose the acceptance rule it applies. Source-confirmed.** + +`stages/report.rs:72` and its experiment counterpart accept any target with `is_transferred=true`, regardless of the selected report threshold or grouped q. The protein report uses the same route. The TSV then prints the unchanged peptide/protein grouped q, which can be 1.0, but does not include a transfer flag or transfer q. Quantification similarly treats the flag as an independent unconditional acceptance route. + +This may be an intentional separate transfer threshold, but it is not apparent from the exported table. Re-reporting at a stricter q does not apply that stricter threshold to these rows, and peptide transfer acceptance is not itself protein-group confidence. + +Accepted change (D): add `is_transferred` and `transfer_q` columns to the TSVs and explicitly state the existing acceptance rule, while preserving the original q columns. Document that a tighter report threshold does not revoke prior transfer acceptance, and that a transfer is not itself protein-group confidence. Separate identification/transfer thresholds and changes to the protein acceptance rule wait for MBR to become a default candidate. Tests should make the retained behavior and exported acceptance basis explicit. + +**20. P3 — benchmark rescoring is no longer a faithful copy of production. Source-confirmed.** + +`bench/feature_selection/fs_lib.py:95` hashes the peptide string, including `DECOY_`, before removing that prefix for another field. Its fold assignment at line 309 uses those hashes. Production `scripts/nn_rescore_worker.py:158` prefers explicit paired base-peptide fold keys. The benchmark also keeps old training defaults and has its own q-value implementation and preprocessing. + +This means a feature study can evaluate a different split/training protocol from the one subsequently deployed. It does not establish that every historical benchmark number is wrong; their exact inputs and recipes must be inspected before drawing that conclusion. + +Accepted change (B), minimal scope: strip `DECOY_` before hashing and record recipe metadata with benchmark results. Include the code SHA, folds, feature list, seed, preprocessing, and training settings. This corrects the identified prefix mismatch; it does not make sequence hashing identical to production's explicit base-ID folding for every decoy strategy. Sharing training utilities or routing studies through the production worker is deferred, and remaining protocol differences should be stated rather than called exact equivalence. + +**21. P3 — local scratch copies contaminate source discovery. Reproduced.** + +The ignored `rust/mumdia/crates/mumdia/tests/pipeline-covr2.rs` is still automatically discovered by Cargo; it ran alongside the tracked integration tests. Git ignore rules do not govern Cargo discovery. + +`python ci/gen_config_reference.py --check` failed in this workspace because `ci/gen_config_reference.py:1124` scans filesystem Rust/Python globs and includes `-covr2` copies. The generated delta added scratch-worker references and duplicated/default-conflicting metadata. The workflow checker also scanned all six YAML files, including three untracked variants. This is a dirty-workspace tooling problem, not evidence that a clean checkout's documentation check fails. + +Accepted local hygiene: move the **113 scratch copies identified by the maintainer** out of source, test, and workflow trees into the existing backup folder. They are untracked and nothing is to be deleted. The move needs no code review. Switch `gen_config_reference.py` and `check_workflows.py` to `git ls-files` so their input set is tracked source. This revision records those actions only; it does not perform the move or change either scanner. + +**Accepted supporting work and explicit deferrals** + +- **Developer guide — D.** Correct `CLAUDE.md`: top-K promotion exists behind `retain_top_peaks`, rescore writes `selected_peak_rank`, the feature matrix is flat f32, and experiment reports exist. These are documentation corrections to implemented behavior. +- **Desktop dependency maintenance — C.** Add `/desktop` to Dependabot and include its separate lockfile in cargo-audit coverage. No specific vulnerability is alleged. Keep tests for the start guard, converter configuration, and cancellation race with their fixes; a broader GUI/platform testing project is not a prerequisite for A–D. +- **Checked IO contracts — A.** Replace broad optional-field fallbacks where the affected fixes land: distinguish an absent optional column from a malformed present one, and share NULL checks with the typed getters. A wholesale rewrite of every streaming reader is deferred. +- **Separate peptide and precursor exports — follow-up PR.** The current TSV is precursor-shaped but selects on peptide q assigned only to a base peptide's winning row (`stages/report.rs:57`, `stages/rescore.rs` grouped-q mapping). Document the resulting omission of sibling charges/modforms now. A separate precursor export selected on `precursor_q` is follow-up work, not part of A–D. Changing competition to `peptidoform_charge` alone does not fix the export semantics. +- **Unified per-run executor — after A–D.** A shared executor could reduce duplication between `stages/run.rs` and `stages/run_experiment.rs::process_run`, but it is a multi-day refactor of tested orchestrators, not a prerequisite for correcting the identified defects. +- **Shared benchmark/worker utilities — deferred.** B contains the prefix-hashing correction and recipe metadata. The training-utility refactor remains a separate project. +- **MBR redesign — deferred.** A paired decoy-transfer population is research. Separate reporting thresholds, revised protein acceptance, and deprecation of inert strategy variants belong with the later redesign. The maintainer reports that the settings editor already labels inert options, so that UI labeling is not an outstanding request. +- **Relocatable experiment references — deferred.** D records resolved configuration, effective filtering, model identities, and input hashes at start. Relocation support is outside that PR. +- **Resume/recovery — out of scope.** The current manifest is provenance rather than a resume database. This review does not request a checkpoint/resume implementation. +- **Inline incident history — house style retained.** The recommendation to remove or relocate incident-history comments is withdrawn. Correct stale technical claims and add regression tests alongside A–D; no stylistic rewrite or module split is required in this pass. + +**Original review validation and limits** + +The results below were collected during the original review. They are not validation of the planned fixes. No code tests, benchmarks, scratch moves, or implementation work were performed for this document-only revision. + +| Check | Result | +|---|---| +| `cargo test --workspace --locked` in `rust/mumdia` | Passed; local ignored `pipeline-covr2` tests were also auto-discovered | +| `cargo fmt --check` | Passed | +| `cargo clippy --workspace --all-targets --locked -- -D warnings` | Passed | +| Base Python contract suite | 63 passed, 12 skipped; skipped optional ML dependencies limit coverage | +| `python ci/check_desktop_ui.py` | Passed: 106 element IDs, 30 commands; checks names, not runtime payload/behavior | +| `python ci/check_doc_refs.py` | Passed before adding this report | +| `python ci/check_workflows.py` | Passed for tracked and local scratch workflow files | +| `node --check desktop/ui/app.js` | Passed | +| `python ci/gen_config_reference.py --check` | Failed due to local scratch-source discovery described in finding 21 | +| Direct Rust probes | Reproduced findings 1, 2, 4, 9, 10, and config acceptance in 18 | +| Direct Python probes | Reproduced in-sample gap filling, unreachable MBR decoy diagnostic, and zero transfer q tail | +| Windows filesystem probe | Confirmed case-different run names address the same directory here | + +The probes were written under `C:/Users/robbi/AppData/Local/Temp/mumdia_review_20260907/`; the LOESS probe is under `C:/Users/robbi/AppData/Local/Temp/mumdia-core-review/`. The Rust probes linked the current built workspace libraries, except the small LOESS probe which compiled the current module directly. The entrapment probe replaced the learner with a membership spy while retaining the worker's actual split/fill control flow. + +No full real-data DIA experiment, model retraining benchmark, desktop GUI end-to-end run, fresh release/container build, or current external vulnerability scan was completed for this review. Optional-sidecar-enabled test completion was not established, so it is not counted as validation. Static desktop findings remain source-confirmed rather than reproduced in a running GUI. The findings support the concrete corrections above; they do not quantify their prevalence or sensitivity effect on biological datasets. + +**Agreed delivery:** A–D plus the separate local hygiene action, with P1 data-integrity defects first. Add the regression fixtures where each finding is fixed. D's LOESS correction requires the HYE B01 before/after count before merge. Deferred research, architecture, resume, and export projects are not prerequisites for these four PRs. Future promotion of sensitivity or MBR defaults still requires the repository's scientific validation; this maintenance plan does not claim that validation has already happened. + +--- + +## Implementation status (2026-09-07) + +All four work packages are implemented as stacked pull requests, each on the previous one and +the first on #61 (`perf/ms2pip-worker-throughput`); merge in order. Each PR carries its tests +and CHANGELOG entries; CI was green on every one at the time of writing. + +| Package | PR | Findings | Notes | +|---|---|---|---| +| A. Engine data integrity | #62 `review/a-data-integrity` | 1, 2, 4, 9, 11, 17, 18, 21 | Predictor misses drop the candidate and its pair; numeric domains validated at load; scanners on `git ls-files`. | +| B. Workers | #63 `review/b-workers` | 3, 6, 7, 8, 12, 20 | Single-class fold is an error; MBR q uses the `+1` pseudocount and prints the permuted-null draws; `selected_peak_rank` joined. | +| C. Desktop and output ownership | #64 `review/c-desktop` | 5, 13, 14 | Frontend start guard plus backend results-folder reservation; converter probe with the request's configuration and the engine's msconvert fallback rule; single terminal-state writer; Dependabot and `cargo audit` for `desktop/Cargo.lock` (one documented ignore). | +| D. Provenance, reporting, calibration, docs | #65 `review/d-provenance` | 10, 15, 16, 19 | Boundary-continuous LOESS; experiment manifest with `config_json`, model identities, effective `q_filter`, inputs hashed at start; audit on `precursor_q`, single source, `NO_PEAK_GROUP` renamed `DID_NOT_SURVIVE_EXTRACTION`; `is_transferred` / `transfer_q` in both TSVs. | +| Local hygiene | (no PR) | 21 | 173 scratch copies moved to `C:/Users/robbi/mumdia_bench/untracked_backup_2026-09-06/covr2/`, nothing deleted. | + +**Finding 10 merge check.** Merge check, HYE B01 (`LFQ_Orbitrap_AIF_Condition_B_Sample_Alpha_01`, DIA-NN library with +its imported iRT as the RT source, `native_tda`, 32 threads, one binary changed): stripped +peptides at `peptide_q_value` 1% 45,946 before and 45,957 after; PSM-q 1% targets 50,332 +against 50,335 at an unchanged 1.0% decoy fraction; protein groups 6,410 both. 208,130 of +10.88 M candidates (1.9%) received a different RT window, 194,698 of them with iRT above the +anchor range (170), which the old global line had placed at 10,625 to 11,946 s in a 9,000 s +run; the continued local fit places them at 8,578 to 9,100 s. Extract accepted 454 more rows +and the run took 37:30 against 34:14. +Second pair, same file and settings with the DeepLC 4.1.1 base-model re-predicted precursor +table as the library (`rt_im_train.library_irt = auto` output of 2026-09-06, passed as +`--lib-precursors`; `w_rt` 414 s against 691 s, in-sample residual median 78 s against 130 s): +stripped peptides at 1% 48,533 in both arms; PSM-q 1% targets 53,124 against 53,127 at an +unchanged 1.0% decoy fraction; protein groups 6,519 in both; 22,779 of 10.88 M candidates +(0.2%) received a different window (median move 424 s); extract accepted 27 more rows; wall +26:06 against 25:46. The correction is neutral on both RT sources, which is the expected +result for a boundary continuity fix: the anchors are inside the range and the candidates it +moves are the ones the old line had placed outside the acquisition. + +Row and q-value unit of every count above: `peptides.tsv` rows selected on the stripped-peptide +`peptide_q_value` at 1% (one row per stripped sequence here, since `native_tda` was run with +the default `compete.group_by = peptidoform_charge` and the report deduplicates on the winning +row), PSM counts on the pooled `q_value`. diff --git a/docs/30_code_review_2026-09-08.md b/docs/30_code_review_2026-09-08.md new file mode 100644 index 00000000..6ce3a1a9 --- /dev/null +++ b/docs/30_code_review_2026-09-08.md @@ -0,0 +1,265 @@ +# MuMDIA follow-up code review — 2026-09-08 + +Baseline: `d7874f2099e9bfb9ced3069e2f53ea8e942a2584`. Compared with the original review baseline `ab7049e`, including the four implemented work packages and their surrounding callers. This review changes no implementation. + +**The earlier corrections are substantially in place, but the tree is not ready for an “everything fixed” sign-off.** Nine remaining findings are listed below. The most urgent are a new fine-tuning configuration regression and three output-integrity problems. The ordinary native release pipeline passes its smoke tests. + +Scope: first-party engine/core/IO/numerical code, Python workers and library helpers, desktop supervision and UI, tests, benchmark utilities, and delivery configuration. The tracked implementation inventory is 81 Rust files (44,839 lines), 42 Python files (12,332 lines), and three JavaScript/HTML/CSS files (2,456 lines). Verification concentrated on the 21 previous findings, changed code, and contracts between components. External comparison checkouts, acquisition datasets, and historical result directories are outside the source review. This is a code and test review, not proof that every scientific workflow is correct. + +P1 means fix before relying on the affected workflow; P2 means a correctness or reliability follow-up. “Reproduced” below identifies an executed probe; “source-confirmed” identifies a code path whose consequence was established without a full biological-data run. + +| Priority | Finding | Trigger | +|---|---|---| +| P1 | R1. Validation rejects the documented automatic fine-tune batch | Enable DeepLC fine-tuning without overriding its default batch | +| P1 | R2. Windows run-name aliases still overwrite runs | Experiment run names such as `a` and `a.` | +| P1 | R3. Finished-run cancellation can delete the next run's files | Late or still-running cancellation after output-directory release | +| P1 | R4. Concurrent vendor conversions share a temporary output | Two searches convert the same input concurrently | +| P2 | R5. Several active numeric settings still bypass domain checks | Invalid extraction fractions, anchor thresholds, or reliability thresholds | +| P2 | R6. The library RT rewrite still accepts incomplete predictions | Short prediction result in the fine-tune/re-prediction worker | +| P2 | R7. Audit reporting flags, reasons, and counts disagree | Precursor/peptide q disagreement, decoys, or malformed precursor q | +| P2 | R8. Output reservations allow overlapping directory trees | Concurrent parent experiment and child-directory search through the backend | +| P2 | R9. The Windows debug CLI overflows its initial stack | Normal debug binary invoked even with `--version` | + +**R1. Preserve `finetune_batch = 0` as automatic sizing. Reproduced; regression in package A.** + +`rust/mumdia/crates/mumdia-core/src/config.rs:2110` rejects `finetune_batch == 0` whenever `finetune_deeplc` is enabled. However, the field documentation at line 592 and the default at line 677 explicitly define zero as automatic sizing. `scripts/deeplc_finetune.py` implements that automatic mode. + +This minimal configuration now fails `Config::from_json`: + +```json +{"rt_im_train":{"finetune_deeplc":true}} +``` + +The returned error is `rt_im_train.finetune_epochs and finetune_batch must be >= 1 when finetune_deeplc is on`. Thus simply enabling a supported feature makes its defaults invalid, before any model is launched. Changing the batch to an arbitrary positive number also changes the intended training recipe. + +Keep the positive-epoch check, retain zero's documented batch semantics, and test enabling fine-tuning with an omitted batch, an explicit zero, and a positive override. + +**R2. Reject trailing-dot/space run-name aliases on Windows. Reproduced end to end; remaining gap in original finding 5.** + +`rust/mumdia/crates/mumdia/src/stages/run_experiment.rs:331` now compares names case-insensitively, but the directory-name check at line 419 only rejects empty names, separators, `.` and `..`. It accepts `a` and `a.`, which address the same directory in this Windows execution path. + +I ran the release binary over the generated smoke fixtures with `--run-names a --run-names a.`. The command exited **0**, created only one physical `a` directory, and both logical split paths contained `source = 1`. Source 0 had been overwritten. This is a successful experiment with broken run identity, not merely a naming error that stops execution. + +Validate portable directory names before computation. In particular, reject trailing dots/spaces and Windows-invalid names, in addition to case-insensitive duplicates. Test the actual per-run paths and their source columns after a valid experiment, and reject the alias pair before writing artifacts. Filesystem probing is unnecessary for these syntactic aliases. + +**R3. Keep cancellation cleanup inside the run's ownership lifetime. Reproduced; remaining gap in original finding 14.** + +`desktop/src-tauri/src/run.rs:233` performs cancellation and a recursive temporary-file sweep even if the run is already terminal. `publish_exit` releases the directory reservation at line 267 without coordinating with an outstanding `cancel()` call. The stored PID is never cleared. The command at `desktop/src-tauri/src/main.rs:403` also accepts cancellation of retained, finished run handles. + +A probe using the production `Run` methods performed this sequence: + +1. Finish run A and release its reservation. +2. Reserve the same directory for run B and create `new.parquet.tmp-999-1`. +3. Deliver a late cancellation to run A. + +The new file was deleted while A remained `failed`. No process was killed in this reproduction: A deliberately had no PID. The deletion follows from `sweep_temp_files` at line 401 matching every `.tmp-` filename, regardless of owner. + +There is also an ordinary concurrent route: A's waiter can reap the engine and release the folder while A's cancellation thread is still finishing `kill_tree`, after which that thread sweeps the reused folder. Unix's TERM/KILL delay makes the separation especially visible. + +Make terminal cancellation inert, retire the PID when the child is reaped, and synchronize cancellation completion with final publication and reservation release. Cleanup should affect only artifacts owned by the ending run. Test late cancellation and an in-flight cancellation overlapping a subsequent start; the existing status-only race tests miss file ownership. + +**R4. Give concurrent vendor conversions independent output ownership. Reproduced with converter stubs.** + +`rust/mumdia/crates/mumdia/src/raw.rs:637` always uses `.partial.mzML`, removes any existing file at that path, and passes it to the converter. Publication at line 774 then renames that shared file. Different search output directories do not isolate this path: conversion normally writes beside the acquisition input. + +A deterministic probe ran the actual `raw::ensure_mzml` concurrently with two small converter executables. Converter A wrote its bytes; converter B replaced the shared temporary file; A then completed. **A returned success with converter B's bytes.** B subsequently failed because its temporary output had already been renamed away. This demonstrates the ownership failure without depending on proprietary acquisition data or converter timing. + +Use unique temporary paths and coordinate ownership of the converted destination. Unique temporary names alone do not fully solve the problem if two conversion recipes can replace the same final mzML while another search is about to read it. An input/recipe-specific conversion cache or an appropriately scoped conversion reservation can make the consumed artifact stable. Preserve the existing `.mzML` extension requirement. + +**R5. Finish numeric validation for the active configuration surface. Reproduced; original finding 18 remains partial.** + +The domain checks added at `rust/mumdia/crates/mumdia-core/src/config.rs:1988` validate several important settings, but leave other active fractions and thresholds unchecked. All of these still pass `Config::from_json`: + +```json +{"mbr":{"q_anchor":-0.1}} +{"mbr":{"q_anchor":2.0,"min_anchor_runs":0}} +{"extract":{"min_matched_fraction":2.0}} +{"features":{"bound_peak_fraction":-1.0}} +{"quant":{"reliable_q":2.0}} +``` + +These fields are used. For example, `extract.rs:2132` compares the matched fraction with `min_matched_fraction`, so a value above one rejects the candidates. `quant.rs:769` uses `reliable_q` to select the supposedly confident population for consensus bounds. The MBR worker directly receives `q_anchor` from `run_experiment.rs:678`. + +Audit the remaining active numeric fields by their documented domains, including anchor support and correlation thresholds. Apply checks at the relevant entry points; standalone command-line thresholds should not bypass the same contract. Preserve intentional zero sentinels, as R1 illustrates. Test invalid values alongside supported boundary values rather than adding only rejection cases. + +**R6. Apply prediction coverage checks to the library rewrite too. Reproduced with a predictor stub; adjacent gap to original finding 17.** + +The new `predict-frag` handling correctly drops uncovered candidates with their pairs. The separate full-library path in `scripts/deeplc_finetune.py:274` still zips `batch` with predictions without checking length, then uses `preds.get(base_pf(pf), orig[i])` at line 287. Both fine-tuning and `run_deeplc_repredict` use this worker. + +I executed the current rewrite control flow with three standard peptidoforms, original iRT values `[10, 20, 30]`, and a predictor returning just `[100]`. It exited normally and wrote `[100, 20, 30]`, while announcing a library re-predicted with the DeepLC base model. That mixes RT scales and hides missing predictions. The probe replaced the ML implementation, not the loop and fallback responsible for the result. + +Check result shape and coverage before rewriting the library. Preserve the agreed policy of dropping unsupported candidates with a counted warning rather than aborting a whole-proteome build for a few misses. This rewrite must coordinate precursor/fragment references and target-decoy pairing when dropping rows. Treat a structurally malformed predictor response distinctly from a documented unsupported peptidoform. If retained imported values remain an intentional mode, make their count and RT-source mixture explicit. + +**R7. Make audit's `reported` field agree with its reasons and actual report eligibility. Reproduced; adjacent gaps in original finding 16.** + +`rust/mumdia/crates/mumdia/src/stages/audit.rs:188` assigns `FAILED_PEPTIDE_FDR` when precursor q passes but peptide q fails. Line 208 nevertheless writes `reported = passed_prec`. The metrics count `REPORTED` reasons at line 237, producing a different reported count. Decoys can also receive `REPORTED`, although the report stage excludes them. + +A three-row probe produced: + +```text +precursor_id: [1, 2, 3] +reported: [true, true, true] +rejection_reason: [FAILED_PEPTIDE_FDR, REPORTED, REPORTED] +metrics.reported: 2 +``` + +Row 3 was a decoy. Thus neither the row flags nor the reason/count pair consistently represents the final target report. + +There is a related error-handling gap at line 109: any failure to read `precursor_q`, including a wrong declared type, triggers the legacy PSM-q fallback. An explicitly present Int32 precursor-q column was accepted, and metrics recorded `q_unit = q_value` instead. This repeats the absent-versus-malformed mistake fixed in quant. + +Use one explicit acceptance definition for the audit's reporting flag, rejection reason, and summary. Keep precursor-gate diagnostics separate from final-report membership, which must follow the report's existing target/q/transfer rules. Restrict legacy fallback to genuinely absent columns. Test opposite-side precursor/peptide q values, decoy rows, and a malformed present precursor-q column. This does not require the deferred export redesign. + +**R8. Reserve overlapping output trees, not just equal directory strings. Reproduced at the backend API.** + +`desktop/src-tauri/src/run.rs:79` only checks `active.get(&key)`. Reserving a directory for an experiment and then reserving its `run1` child for another search both succeed. An experiment writes inside its per-run subdirectories, and cancellation cleanup walks the entire reserved parent recursively, so these are overlapping writers even though the keys differ. + +The frontend's one-active-run check limits the ordinary single-window trigger; this finding concerns the backend reservation contract and callers of `run::start`. It is separate from duplicate-click protection, which is implemented. + +Check ancestor/descendant overlap using canonical path components while holding the reservation lock. Avoid plain string-prefix comparison, which would incorrectly equate siblings such as `out` and `out2`. Test both reservation orders and a non-overlapping sibling. + +**R9. Make the normal Windows debug CLI runnable. Reproduced.** + +After `cargo build --locked`, `C:/Users/robbi/mumdia_build/debug/mumdia.exe --version` aborts with `thread 'main' has overflowed its stack`. This happens directly from PowerShell as well as from the smoke harness, before a search begins. The CLI entry point is `rust/mumdia/crates/mumdia/src/main.rs:1036`. + +The PE header reserves 1 MiB of stack. Changing only that reservation to 16 MiB in a temporary copy makes the identical debug binary print its version successfully. This establishes a stack-size problem; the exact allocation responsible was not isolated. The release binary works and passed the full smoke test. + +Reduce the debug entry/dispatch stack requirement, or establish and document an appropriate Windows stack reservation. Add a subprocess test of the normal debug executable's help/version path. Library unit tests and release-only CLI smoke coverage currently miss this developer-build failure. + +**Status of the original 21 findings** + +| Original findings | Current assessment | +|---|---| +| 1, 2, 4, 9 | Fixed for the reported triggers; the previous executable probes now reject pooled U32 quant and NULL fragments, preserve an existing artifact after publication failure, use distinct temp paths, and retain the Met-excised boundary peptide. | +| 3 | Fixed: class-deficient training folds error and non-decoy held-out gaps are not filled in sample. | +| 5 | Duplicate-start and case-only fixes implemented; remaining output-ownership gaps are R2 and R8. | +| 6, 7, 8 | Implemented: named permuted-null diagnostics, `+1` numerator, and selected-peak join. MBR/top-K tests pass. | +| 10 | Endpoint-continuity fix implemented and tested. Both HYE B01 comparison pairs are documented in the previous review's implementation-status section; those biological-data runs were not repeated here. | +| 11 | Fixed: matrix sizing uses the flat f32 layout and checks before allocation. | +| 12 | Implemented: augmentation reuses the canonical stripped sequence's base ID. | +| 13 | Implemented: converter probe carries request configuration and mirrors the auto-parser/msconvert fallback. | +| 14 | Single terminal-state writer implemented, but cancellation lifetime remains unsafe: R3. | +| 15 | Resolved config, effective quant filter, start-time input hashes, and model-identity fields added. More precise model provenance is still useful; see below. | +| 16 | Correct precursor-q read, pooled-table rejection, and coarse extraction label added; R7 remains. | +| 17 | Original `predict-frag` gap corrected; the adjacent library RT rewrite remains exposed: R6. | +| 18 | Partial, with a new valid-config regression: R1 and R5. | +| 19 | Accepted minimal scope implemented: TSV transfer flag/q columns and documented acceptance rule. The separate-threshold/protein-confidence redesign remains deferred. | +| 20 | Accepted minimal scope implemented: strip `DECOY_` before benchmark hashing and record recipe metadata. Production/benchmark fold identity is still explicitly different where expected. | +| 21 | Tracked-file scanners implemented; no `-covr2` copies found in the checked source/test/workflow trees. No scratch files were moved or deleted during this review. | + +**Other focused cleanup opportunities** + +- Model identities remain mostly family or recipe labels. `predict_frag.rs:378` returns the literal `deeplc-4.0-mt`; experiment provenance reconstructs RT/fragment names from configuration at `run_experiment.rs:822`. Preserve installed worker versions and checkpoint identifiers/hashes where available, alongside the intended recipe. Do not interpret the current strings as a complete software/model identity. The rescorer now records the actual executed fallback correctly. +- `scripts/mbr_worker.py:232` exits successfully without writing a requested `--out-scored` when there are no candidates, and its empty transfer file has only `candidate_id`. The experiment orchestrator handles absence, but the standalone worker has an unstable output contract. Its current test explicitly pins this behavior. A full-schema empty transfer table and an unchanged augmented scored output would make downstream use simpler and safer. +- Several fixed-contract tests still construct lookalike fixtures instead of connecting the producer and consumer. Add the originally requested rescore-emitted-table-to-quant check, a worker-to-report transfer-q check, and lifecycle tests that assert artifact ownership. The fine-tune regression also calls for testing valid non-default feature combinations. +- The desktop progress thread can still write stage snapshots after the waiter publishes completion (`run.rs:677` onward). Stop updates once terminal and coordinate the final scan with publication, so a stale scan or a new run's directory contents cannot replace the old run's finished snapshot. This fits the R3 lifecycle work. + +The earlier deferrals remain deferrals: unified orchestration, resume/recovery, relocatable provenance, separate peptide/precursor exports, shared benchmark utilities, MBR redesign, and rewriting the repository's inline incident-history style. They are not prerequisites for this correctness pass. + +**Verification performed** + +| Check | Result | +|---|---| +| Engine `cargo test --workspace --locked --quiet` | 258 tests passed across workspace unit, integration, and documentation test groups. | +| Engine formatting, Clippy (`--workspace --all-targets`, warnings denied), Rustdoc (warnings denied) | Passed. | +| Engine debug and release builds | Built successfully; debug runtime failure is R9. | +| Release native smoke workflow | Passed all 144 assertions, including repeatability, malformed-RT handling, the single-run and experiment orchestrators, splitting, quantification, LFQ, manifests, and reports. | +| Desktop library tests | 78 passed using a separate build directory outside OneDrive. | +| Desktop integration tests with the generated fixture and current release engine | Search completion and cancellation tests passed. The installation test returned early because `MUMDIA_TEST_INSTALL=1` was not set; it was not exercised. | +| Base Python suite | 67 passed, 12 skipped for optional dependencies. | +| Mokapot/NN subset in existing `py312_mumdia` environment | 11 passed, including previously skipped rescorer tests. This overlaps some base-suite tests; counts are not additive. | +| Predictor suite in existing `deeplc_mt` environment | 11 passed, 3 skipped. That environment has an older DeepLC than the required version and an MS2PIP/Pydantic import incompatibility. Real current DeepLC/MS2PIP prediction remains unverified here. | +| Workflow structure, desktop UI references, documentation references, generated CLI/configuration references and configuration schema | Passed. | +| Current engine dependency audit under repository policy | Passed; one allowed unmaintained-crate warning. | +| Current desktop dependency audit under repository policy | Passed with the repository's explicit `RUSTSEC-2024-0429` ignore and 16 allowed unmaintained-crate warnings. This is not an assertion of zero advisories. | +| Targeted regression probes | Confirmed R1–R9 at the levels specified above, and rechecked original integrity failures. | + +Probe sources, fixtures, logs, and temporary executable copies are under `C:/Users/robbi/AppData/Local/Temp/mumdia_review_20260908/`. Production source and user results were not modified. The smoke script was copied there with its working/output paths redirected and its initial recursive deletion removed; the pipeline commands and assertions were retained. Dependency environments were used as found, without installing or upgrading packages. + +No fresh Linux/macOS execution, packaged GUI interaction, container/release packaging test, real vendor conversion, full biological-data experiment, or model retraining benchmark was completed in this pass. The converter and incomplete-prediction reproductions used controlled stubs as stated. These limits do not weaken the demonstrated ownership/configuration failures, but they do limit broader scientific and platform sign-off. + +--- + +## Implementation status (2026-09-08) + +All nine findings are addressed in one further stacked pull request, E (`review/e-followup`, +on D), with a test per finding. Two of the accepted changes were narrowed on purpose: + +- **R6.** The library rewrite does not drop rows. A row's `candidate_id` is the contiguous + index the fragment table references, so dropping rows there would mean renumbering both + tables for a rare event. Instead a prediction count that differs from the batch is a hard + error, a peptidoform without a finite prediction keeps its imported value, and the counts + (`repredicted`, `retained_imported`, split into non-standard residues and no finite + prediction) are written to `.summary.json` and warned about by the engine. +- **R4.** No recipe-keyed conversion cache. The temporary name is unique per conversion + (`.partial--.mzML`) and a `.converting` lock makes a + concurrent converter of the same input wait and reuse the result. Two different + conversion recipes aimed at one destination remain a documented limitation of writing + beside the input. + +| Finding | Change | Test | +|---|---|---| +| R1 | `finetune_batch = 0` is the automatic batch again; only `finetune_epochs >= 1` is required | `enabling_the_fine_tune_keeps_the_automatic_batch` | +| R2 | `portable_dir_name_problem`: trailing dots and spaces, `<>:"|?*` and control characters, and the Windows reserved device names are rejected on every platform, before anything is written | `run_names_that_alias_on_windows_are_rejected_everywhere` | +| R3 | `cancel` is intent and kill only, inert once terminal, and holds the pid lock across the kill; `publish_exit` retires the pid under that lock, sweeps, then releases; the progress thread writes only while active | `a_late_stop_after_the_run_ended_leaves_the_folder_alone`, `a_stop_still_in_flight_finishes_before_the_folder_is_released` | +| R4 | unique partial names and the conversion lock | `a_conversion_lock_is_exclusive_and_released_on_drop`, `a_stale_lock_is_broken_and_a_fresh_one_is_honoured` | +| R5 | domain checks for the remaining numeric fields, boundaries accepted | `numeric_domains_are_validated_at_load` | +| R6 | structural mismatch is an error, retained imported values are counted and reported | `rewrite_irt` summary; engine warning | +| R7 | `reported` is `rejection_reason == REPORTED`; decoys past both gates are `REMOVED_DURING_REPORTING`; a malformed present `precursor_q` is an error | `the_reported_flag_follows_the_reason_and_the_report_rules`, `a_present_but_malformed_precursor_q_column_is_an_error_not_a_fallback` | +| R8 | reservations refuse an ancestor or descendant of an active folder, by path components | `overlapping_result_folders_are_refused_in_both_orders_but_siblings_are_not` | +| R9 | the CLI runs on a thread with a 256 MiB stack reservation | `tests/cli_version.rs` runs the built binary's `--version` and `--help` | + +Of the cleanup notes: model identities carry the installed DeepLC and MS2PIP versions +(`deeplc-4.1.1-base`, `ms2pip-4.2.0-HCDch2`, `deeplc-4.1.1-finetuned`); the MBR worker +writes a full-schema empty transfer table and an unflagged augmented scored table when +there is nothing to transfer; the desktop progress thread no longer writes after +publication. Checkpoint hashes and the producer-to-consumer unit tests beyond the smoke +harness are not in E. + +## Release check on real data (2026-09-08) + +Run on doxy with a binary built from the D head (`d7874f2`), which is the code of PRs #61 +to #65; E changes no stage behaviour on these paths. Two runs, both from the shipped example +configurations with the doxy interpreters (`configs/examples/diann-library.json`, +`configs/examples/fasta-sidecars.json`), `mumdia doctor --config` passing on both. + +**Full default HYE experiment**: `run-experiment` over the six ProteoBench HYE AIF files +(A_01 to A_03, B_01 to B_03), imported DIA-NN library with `rt_im_train.library_irt = auto` +(DeepLC 4.1.1 base-model re-prediction, once, 28 min), strict `nn_torch`, per-run quant on +the pooled `q_value`, cross-run MaxLFQ, 64 threads. Exit 0, 1:54:16 wall, 39.1 GB peak RSS, +109 GB of output. + +| run | PSMs at `run_psm_q` 1% | stripped peptides | decoy fraction | `w_rt` | anchors | +|---|---|---|---|---|---| +| A_01 | 78,101 | 66,108 | 0.0100 | 375 s | 18,456 | +| A_02 | 81,129 | 67,606 | 0.0100 | 349 s | 18,650 | +| A_03 | 72,865 | 62,168 | 0.0100 | 374 s | 18,493 | +| B_01 | 68,779 | 61,745 | 0.0100 | 397 s | 18,619 | +| B_02 | 71,958 | 62,810 | 0.0100 | 367 s | 18,660 | +| B_03 | 64,347 | 57,750 | 0.0100 | 399 s | 18,682 | + +Pooled rescore: 11,897,712 PSMs in 32.7 min, 437,208 target PSMs at 1% at a 1.00% decoy +fraction, **80,803 experiment-wide stripped peptides at `peptide_q_value` 1%**, 100,885 +precursors, 10,659 protein groups. The experiment-wide `peptides.tsv` has 80,803 rows with +six `quantity_` columns and the two transfer columns, 38,059 of them identified in all +six runs on `run_psm_q`; `proteins.tsv` has 10,659 rows with six `lfq_` columns; +`lfq_maxlfq.parquet` has 86,856 rows. `experiment_manifest.json` carries `config_json` +(4,163 characters), `model_identities` (`deeplc-base-model`, `imported-library`, +`nn-torch-semisup-sidecar-v1`, feature schema id, `None` for MBR), `inputs_hashed_at: +start`, eight inputs, twenty artifacts and `quant_q_filter` configured `RunPsmQ` / effective +`PsmQ`. Reference points: the 2026-08-26 pooled run of the same six files with the raw +imported iRT reported 72,044 stripped peptides; DIA-NN 2.2.0 library-free with `--reanalyse` +reports 65.6k to 72.5k stripped peptides per run at run and global 1% in 61 minutes on 32 +threads. The per-run MuMDIA counts above are 88 to 93 percent of DIA-NN's; the comparison is +not controlled for the changes since August and is a readiness check, not a sensitivity claim. + +**E. coli FASTA single run** (A_01, the 4,401 `_ECOLI` entries of the HYE FASTA, DeepLC 4.1.1 ++ MS2PIP 4.2.0 `HCDch2`, 12 fragments, strict `nn_torch`, 32 threads): exit 0, 12:22, 17.9 GB. +1,924,656 candidates predicted in 10:51 with `candidates_dropped_unpredicted = 0` and +`pairs_dropped_unpredicted = 0`, so review A's coverage checks ran against the real workers +and found complete predictions; 101 confident seeds, 97 anchors, `w_rt` 196 s; 527 stripped +peptides, 558 precursors, 155 protein groups at 1%. The count is small because condition A +carries 5 percent E. coli and the search space is E. coli only; the run exists to exercise +the FASTA path end to end, which it did. + +Not exercised by either run: `mumdia mbr` on real data, the desktop application against the +merged code, vendor conversion, entrapment. diff --git a/docs/31_code_review_2026-09-08_full.md b/docs/31_code_review_2026-09-08_full.md new file mode 100644 index 00000000..d5ed834b --- /dev/null +++ b/docs/31_code_review_2026-09-08_full.md @@ -0,0 +1,201 @@ +# MuMDIA full code review — 2026-09-08 + +Baseline: `21b8ac7`, the head of `review/e-followup` (work packages A to E). Whole +repository rather than a diff: engine, core, IO, the Python workers and library helpers, +the desktop application, the tests, the benchmark utilities and the delivery +configuration. Twelve parallel finders and one verification sweep; the maintainer +re-verified eleven findings against the source before recording them here. + +Twenty-one findings. Six are confirmed correctness or security defects in code that +predates this review series, five are regressions introduced by packages A, D and E, and +ten are second-tier robustness, coverage and delivery problems. Package F below closes +the first eleven. + +**Verdict at the time of review: do not tag a release from this tree.** Two of the +findings produce a plausible wrong answer rather than an error, and one is a code-execution +path through an untrusted working directory. The 2026-09-08 HYE release check passing does +not contradict that: it exercised the library path with calibration available and no +match-between-runs transfer, so it could not reach any of them. + +## Findings + +Priority P1 means fix before relying on the affected workflow. *Reproduced* means an +executed probe demonstrated the failure; *source-confirmed* means the trigger and the +consequence follow from the implementation without a full run. + +| | Priority | Finding | Origin | +|---|---|---|---| +| F1 | P1 | prescan reads the no-calibration sentinel as "cannot be screened" and discards the entire library at exit 0 | pre-existing | +| F2 | P1 | a malformed `is_transferred` silently deletes every transfer from both TSV reports | pre-existing | +| F3 | P1 | `sidecar::resolve_script` tries the working directory first, executing a planted worker | pre-existing | +| F4 | P1 | `Loess::predict` indexes out of bounds when the query is NaN | pre-existing | +| F5 | P1 | three rescorer backends, two standardisations, and an unchecked fold-key companion | pre-existing | +| F6 | P1 | the output-over-input guard is wired into 2 of 18 stages | pre-existing | +| F7 | P2 | the LOESS boundary extrapolation slope is unbounded and may be negative | package D (#65) | +| F8 | P2 | a desktop stop can kill a recycled process id after the reap | package E (#66) | +| F9 | P2 | the conversion lock spins, mis-detects staleness three ways, and leaks partial files | package E (#66) | +| F10 | P2 | a position-free pair key deletes positional isomers that predicted correctly | package A (#62) | +| F11 | P2 | ten further robustness, coverage and delivery findings, listed at the end | mixed | + +**F1. prescan discards the library on the documented no-calibration path. Reproduced.** + +`rt_im_train::candidate_window` returns `(NaN, -inf, +inf)` whenever calibration is +unavailable and its doc comment calls the infinite bounds recall-safe. prescan's guard +rejected exactly that: `if !lo.is_finite() || !hi.is_finite() { return None; }`. This is the +FASTA/MS2PIP failure CLAUDE.md already records, where the seed search finds no confident +PSMs and every window row is infinite; prescan then wrote a zero-row survivors table with a +NaN target/decoy ratio and exited 0. The single-label bail could not see it, being gated on +`surv.len() > 1000`. A candidate simply absent from `run_windows` was dropped the same way, +uncounted. + +**F2. A malformed transfer flag deletes every transfer from the reports. Reproduced.** + +`mbr_worker.py` writes `is_transferred` through pandas, so a nullable boolean dtype, a null +or int8 0/1 reaches the report stage. `match t.bool(...) { Err(_) => vec![false; n] }` then +degraded acceptance to the q threshold alone: every transferred identification vanished +from `peptides.tsv` and `proteins.tsv`, indistinguishable from an MBR run that transferred +nothing, while the parquet still carried them. Four sites. The same absent-versus-malformed +mistake was removed from `quant` in package A and from `audit` in package D, and left in +the stage that decides what users read. + +**F3. Script resolution prefers the working directory. Source-confirmed.** + +`python::resolve_script_dir` was reordered in an earlier pass to config-relative, +exe-relative, working-directory-last, with a comment explaining the attack: the shipped +default is the relative `"scripts"`, which both example configurations carry, so unpacking +a dataset archive and running with an example config could execute a worker the archive +contained. That resolver only claims a directory holding `mbr_worker.py` or +`deeplc_worker.py`, so a directory containing any of the other ten workers passed through +unresolved, and `sidecar::resolve_script`, used at all eleven call sites, still tried the +working directory first. + +**F4. A NaN query indexes before the start of the grid. Reproduced.** + +With `x = NaN` both boundary comparisons are false and `partition_point` returns 0, so +`grid_x[j - 1]` underflows: a panic in debug, an out-of-bounds read in release. One library +row with a null `predicted_irt` is enough, because the parquet reader maps a null f32 to +NaN. The anchor loop in `rt_im_train` filters for finite values; the whole-library +application does not, and `align` and `search_seed` share the exposure. + +**F5. The rescorer's three backends do not agree. Reproduced.** + +In-memory TSV standardised with median/IQR while in-memory parquet and the streaming +memmap used mean/std, so the same pool scored differently depending on `rescore.handoff` +and on which side of `MUMDIA_NN_STREAM_GB` the matrix fell, with nothing logged. The +docs/28 comparison cited as identical identifications was measured on a 4.03 GB matrix, so +both arms streamed and the divergence was masked. Separately, the `MUMDIA_NN_FOLD_KEYS` +companion was sliced without a length check: numpy returns a short array rather than +raising, the tail rows land in no fold, are never scored, keep the zero initialiser and +emerge from the final rank-normalisation as a plausible tied mid-rank score, which +satisfies the caller's completeness contract so `rescore.strict` does not catch it. + +**F6. The output-over-input guard is almost unwired. Reproduced.** + +`refuse_output_over_input` had three references in the workspace: its definition and two +call sites. `mumdia compete --features features.parquet --out features.parquet` opens the +table footer-only, streams the surviving rows, and publishes by rename after the read has +finished, so nothing errors and the widest artifact of the run is replaced by the competed +subset at exit 0. Recovering it means re-running extract and features, the tallest stage. +The same shape applies to `rescore`, `features`, `quant`, `extract`, `search_seed`, +`rt-im-train` and `audit`. + +**F7 to F10** are regressions from packages A, D and E and are described in the fix table +below. + +## Package F + +One stacked pull request on `review/e-followup`, with a test per finding. + +| | Change | Test | +|---|---|---| +| F1 | an unbounded window means "screen over the whole gradient", the recall-safe reading of the sentinel; candidates with no window row are treated the same; both are counted and warned about; screening every candidate away is now an error naming the likely causes | `prescan` counts and bail | +| F2 | `transfer_columns` reads present columns in their declared type and errors on a type mismatch; only an absent column falls back | `a_malformed_transfer_column_is_an_error_not_a_silent_loss_of_every_transfer`, `a_table_that_never_saw_mbr_reports_no_transfers_without_complaint` | +| F3 | absolute directories as given, then the executable's directory, then `/scripts`, and the working directory last | `the_shipped_directory_beside_the_binary_wins_over_the_working_directory` and two more | +| F4 | `Loess::predict` returns NaN for a non-finite query; `rt_im_train` treats a non-finite library iRT as "no calibrated RT" and counts it | `a_nan_query_returns_nan_instead_of_indexing_out_of_bounds` | +| F5 | one mean/std standardisation in all three backends, which leaves the shipped parquet default and every published benchmark unchanged; the fold-key companion is length-checked; the backend-size estimate counts feature columns by name | `a_short_fold_key_file_is_refused_rather_than_leaving_rows_unfolded` | +| F6 | the guard is wired into `search_seed`, `rt-im-train`, `extract`, `features`, `compete`, `rescore`, `quant` and `audit`, for every output each writes | `writing_the_output_over_the_input_is_refused` | +| F7 | the extrapolation slope is the secant of the fitted curve over its end decile, clamped non-negative and to at most four times the global least-squares slope | `boundary_extrapolation_stays_monotone_and_bounded_under_noise`, on noisy anchors rather than a noiseless quadratic | +| F8 | the waiter retires the process id the instant `wait` returns, before it reads the output directory | `a_stop_after_the_reap_has_no_pid_to_kill` | +| F9 | bounded take-over attempts with a pause and a deadline; a token written and read back, so only the real holder owns or removes a lock; a future modification time counts as fresh; the partial-file probe matches this destination only; abandoned partials of this destination are swept under the lock | five tests in `raw::tests` | +| F10 | the direct misses and the rows dropped for sharing a pair key are counted separately, and exceeding 2% of the library is an error naming the sidecar | `a_large_unpredicted_fraction_is_a_failure_not_a_warning`, `a_positional_isomer_still_shares_its_pair_key_by_design` | + +### Two changes deliberately not made + +**F10 does not make the pair key position-aware.** `M[Oxidation]PEPTIDEMK` and +`MPEPTIDEM[Oxidation]K` share a base peptide, a charge and a modification multiset, so a +miss on either drops both. Adding position would be worse rather than better: a reverse +decoy carries its modifications at mirrored positions, so a positional key would stop +matching a target to its decoy, and a target could be dropped while its decoy stayed. That +is an FDR defect in exchange for a sensitivity one. The collateral is counted and bounded +instead, and a test pins the trade so a future change has to argue with it. + +**F9 does not add a recipe-keyed conversion cache.** Unique temporary names, an owned lock +and the staleness rules cover the reproduced failure. Two different conversion recipes +aimed at one destination name remain a documented limitation of writing beside the input. + +### F7 merge check + +HYE B01 on doxy, the same arm as the docs/29 #10 pair 2: the DeepLC 4.1.1 re-predicted +precursor table, `rt_im_train.library_irt = library` so no DeepLC draw, `native_tda`, 32 +threads, one binary changed. The comparison arm is the package-D head, whose pointwise +boundary slope this replaces. + +| | package D (pointwise slope) | package F (clamped end-decile secant) | +|---|---|---| +| stripped peptides at `peptide_q_value` 1% | 48,533 | 48,533 | +| PSM-q 1% targets | 53,127 | 53,127 | +| PSM-q 1% decoy fraction | 0.010 | 0.010 | +| protein groups at `pg_q_value` 1% | 6,519 | 6,519 | +| extract accepted rows | 1,961,800 | 1,961,800 | +| `w_rt`, in-sample residual median | 414 s, 78.3 s | 414 s, 78.3 s | +| wall (32 threads) | 25:46 | 24:38 | + +Identical on every count, which is the expected result and the reason the change is safe: +both extrapolations are continuous at the boundary and agree closely wherever the anchors +are dense, and they differ only for queries outside the anchor range, where this benchmark +has almost nothing. The guard exists for the case the counts cannot show, a sparse or noisy +boundary window producing a negative or several-times-global slope, which the unit test +covers directly on noisy anchors. + +## F11: the second tier, not in package F + +Recorded for a later pass, in rough order of value: + +- `matchers/binning.rs`: `LogBins::new` has no bound on the derived bin count while + `validate()` only requires a positive tolerance, so `frag_tol_ppm: 2e-5` asks for 547 GB + and dies on an allocation failure rather than on a message naming the setting. +- `digest.rs`: `collision_safe_decoy` XORs `fnv1a(pep)` and the identical term cancels + inside `make_decoy`, so the scramble seed has no peptide dependence and every peptide of + one length gets the same positional permutation. The shipped `reverse` default only + correlates retried decoys; `strategy = scramble` gets a structured null. +- `ci/smoke.sh`: the NaN-retention-time regression compares two reports through process + substitution with no existence check, so both files missing is a passing `diff`. +- `sbom.cdx.json`: the committed SBOM, which ships in every release archive, references two + components it does not define, so it fails CycloneDX validation and `--check` regenerates + the same document. +- `tests/python/test_predictor_workers.py`: the DeepLC import-order test asserts on two + Windows-only strings and no CI job is both Windows and DeepLC-installed, so it cannot + fail. +- `features/similarity.rs`: `rank_overlap_top3` and `frac_top3_predicted_observed` divide by + a hardcoded 3.0 rather than the number of positive predicted fragments, so they encode how + many fragments were predicted; `mass_uncertainty.rs` computes the same quantity correctly. +- `config.rs`: the `max_feature_matrix_gib` doc comment, copied into the CI-checked + configuration reference, still describes the f64 layout the ceiling no longer uses. +- `run.rs`: the audit is hardcoded at q 0.01 against the report's `quant.q_threshold`. +- `build.rs` cannot distinguish a clean tree from a failed `git status`, so a dirty tree can + be stamped clean. +- `check_doc_refs.py` skips `../..`-prefixed links, `make_fixture_mzml.py` truncates rather + than spans the m/z range when planting peptides, `cargo install tauri-cli --version "^2"` + is a range under a comment calling it pinned, and one workflow pins + `softprops/action-gh-release` at two different revisions. + +## Verified and sound + +Recorded because a review that only lists defects invites the same checks again: all +seventeen documented defaults match the code; no `HashMap` iteration feeds a float +reduction or an output ordering; every configuration struct denies unknown fields and all +five shipped configurations parse; the four DeepLC 4.1.1 enforcement points agree with the +Rust constant and both Python literals match it; neither sidecar leaks a target/decoy label +into the feature matrix; every worker's argument surface matches the arguments its Rust +caller builds; the experiment report headers match the writer in order; and +`split_by_source` runs before per-run quantification. diff --git a/docs/README.md b/docs/README.md index fe30b2dd..c577b25e 100644 --- a/docs/README.md +++ b/docs/README.md @@ -99,3 +99,6 @@ measurement locally, so the same finding cannot drift into two versions. | [26_gui_plan.md](26_gui_plan.md) | Design for a simple graphical interface: why an in-binary local web UI (`mumdia serve`) rather than a desktop toolkit or an Electron app, the four-phase path from machine-readable progress to a browser front end, and what each phase is worth on its own. | | [27_memory_footprint_audit.md](27_memory_footprint_audit.md) | Memory footprint audit of the default single-run workflow: per-stage resident-set model, ranked hotspots, reduction plan (streaming IO, incremental writes, f32 storage with f64 arithmetic), rewrite candidates, acceptance gates and the `bench/mem_profile.py` protocol. | | [28_feature_selection_analysis.md](28_feature_selection_analysis.md) | Feature-selection analysis for `nn_torch` rescoring: 43 of 387 Extended features are dead by construction, ~120 chosen multivariately reproduce the full set on two HYE runs (real worker confirmed), training time is flat in the feature count, and the parquet handoff is the larger memory lever. | +| [29_code_review_2026-09-07.md](29_code_review_2026-09-07.md) | Repository-wide code review at `ab7049e`: 21 findings with reproductions (pooled `source` type, streaming NULLs, entrapment in-sample scores, artifact publication, output ownership, MBR confidence, Met excision, LOESS endpoints, provenance, validation), the agreed four work packages A-D and the deferrals. The decision record the fixes cite by finding number. | +| [30_code_review_2026-09-08.md](30_code_review_2026-09-08.md) | Follow-up review at `d7874f2` after work packages A-D: nine remaining findings (fine-tune batch regression, Windows run-name aliases, cancellation cleanup lifetime, concurrent conversions, remaining numeric domains, library rewrite coverage, audit flag consistency, overlapping reservations, debug-binary stack), the status of the original 21, and the implementation status of package E. | +| [31_code_review_2026-09-08_full.md](31_code_review_2026-09-08_full.md) | Whole-repository review at `21b8ac7`, after work packages A-E: 21 findings (prescan's no-calibration sentinel, the malformed transfer flag, script-resolution order, a NaN LOESS query, three rescorer standardisations, the unwired output-over-input guard, and five regressions from A/D/E), work package F, and the second tier left for later. | diff --git a/rust/mumdia/crates/mumdia-core/src/config.rs b/rust/mumdia/crates/mumdia-core/src/config.rs index abfb345f..8243e7fa 100644 --- a/rust/mumdia/crates/mumdia-core/src/config.rs +++ b/rust/mumdia/crates/mumdia-core/src/config.rs @@ -1978,6 +1978,301 @@ impl Config { baseline is subtracted" ); } + // ── numeric domains (docs/29 #18) ─────────────────────────────────────── + // + // A value outside its domain used to be accepted and then either excluded every + // discovery (`quant.q_threshold = -0.1`), was clamped into a different analysis + // (`rt_im_train.rt_window_multiplier = -1.0` became a one-second window), or fed a + // training recipe a fraction above one. Each field's domain is stated here once; + // 0 keeps its documented "off" or "no cap" meaning where a field has one. + for (name, value) in [ + ("quant.q_threshold", self.quant.q_threshold), + ("rescore.train_margin_frac", self.rescore.train_margin_frac), + ("mbr.q_transfer", self.mbr.q_transfer), + ] { + if !value.is_finite() || !(0.0..=1.0).contains(&value) { + return Err(Invalid(format!( + "{name} must be finite and in [0, 1] (got {value})" + ))); + } + } + for (name, value) in [ + ("search_seed.fdr_seed", self.search_seed.fdr_seed), + ("rt_im_train.q_train", self.rt_im_train.q_train), + ("rt_im_train.p_rt", self.rt_im_train.p_rt), + ("rt_im_train.loess_span", self.rt_im_train.loess_span), + ] { + if !value.is_finite() || value <= 0.0 || value > 1.0 { + return Err(Invalid(format!( + "{name} must be finite and in (0, 1] (got {value})" + ))); + } + } + if !self.rt_im_train.window_holdout_frac.is_finite() + || !(0.0..1.0).contains(&self.rt_im_train.window_holdout_frac) + { + return Err(Invalid(format!( + "rt_im_train.window_holdout_frac must be finite and in [0, 1) (got {}); 0 \ + sizes the window in-sample", + self.rt_im_train.window_holdout_frac + ))); + } + for (name, value) in [ + ( + "rt_im_train.rt_window_multiplier", + self.rt_im_train.rt_window_multiplier, + ), + ( + "rt_im_train.fallback_rt_window_s", + self.rt_im_train.fallback_rt_window_s, + ), + ] { + if !value.is_finite() || value <= 0.0 { + return Err(Invalid(format!( + "{name} must be finite and > 0 (got {value}); a non-positive value was \ + previously clamped into a one-second window instead of being rejected" + ))); + } + } + for (name, value) in [ + ("extract.apex_rt_prior_s", self.extract.apex_rt_prior_s), + ( + "rt_im_train.rt_window_min_s", + self.rt_im_train.rt_window_min_s, + ), + ( + "extract.apex_gaussian_sigma_scans", + self.extract.apex_gaussian_sigma_scans, + ), + ( + "rescore.max_feature_matrix_gib", + self.rescore.max_feature_matrix_gib, + ), + ("rescore.train_neg_ratio", self.rescore.train_neg_ratio), + ("rescore.train_subsample", self.rescore.train_subsample), + ] { + if !value.is_finite() || value < 0.0 { + return Err(Invalid(format!( + "{name} must be finite and >= 0 (got {value}); 0 keeps its documented \ + meaning (off, or no cap)" + ))); + } + } + if self.digest.min_len == 0 || self.digest.min_len > self.digest.max_len { + return Err(Invalid(format!( + "digest.min_len must be >= 1 and <= digest.max_len (got {} and {})", + self.digest.min_len, self.digest.max_len + ))); + } + if self.peptidoforms.charge_min < 1 + || self.peptidoforms.charge_min > self.peptidoforms.charge_max + { + return Err(Invalid(format!( + "peptidoforms.charge_min must be >= 1 and <= peptidoforms.charge_max (got {} \ + and {})", + self.peptidoforms.charge_min, self.peptidoforms.charge_max + ))); + } + for (name, value) in [ + ( + "predict_frag.top_n_fragments", + self.predict_frag.top_n_fragments, + ), + ( + "search_seed.min_matched_peaks", + self.search_seed.min_matched_peaks, + ), + ("search_seed.report_psms", self.search_seed.report_psms), + ("extract.apex_count_window", self.extract.apex_count_window), + ( + "extract.presence_min_fragments", + self.extract.presence_min_fragments, + ), + ("experiment.parallel_runs", self.experiment.parallel_runs), + ("rescore.seeds", self.rescore.seeds), + ] { + if value == 0 { + return Err(Invalid(format!("{name} must be >= 1"))); + } + } + if self.predict_frag.charge2_from_precursor_charge < 1 { + return Err(Invalid( + "predict_frag.charge2_from_precursor_charge must be >= 1".into(), + )); + } + if self.rt_im_train.min_seed_for_calibration < 2 { + return Err(Invalid( + "rt_im_train.min_seed_for_calibration must be >= 2; a calibration needs at \ + least two anchors" + .into(), + )); + } + // `finetune_batch = 0` is the documented automatic batch size (see the field), so + // only the epoch count has a lower bound. Rejecting the zero batch made simply + // enabling the fine-tune invalid with its own defaults (docs/30 R1). + if self.rt_im_train.finetune_deeplc && self.rt_im_train.finetune_epochs == 0 { + return Err(Invalid( + "rt_im_train.finetune_epochs must be >= 1 when finetune_deeplc is on".into(), + )); + } + + // The remaining active numeric fields, by their documented domains (docs/30 R5): + // fractions and q-values inside their unit interval, correlations and percentiles + // inside theirs, tolerances and widths positive, counts at least one where zero + // has no documented meaning. Every default sits inside its domain, which the + // `Config::default().validate()` assertion in the tests keeps true. + for (name, value) in [ + ( + "extract.min_matched_fraction", + self.extract.min_matched_fraction, + ), + ( + "extract.alt_peak_min_area_frac", + self.extract.alt_peak_min_area_frac, + ), + ( + "extract.gate_coelution_min", + self.extract.gate_coelution_min, + ), + ("quant.baseline_quantile", self.quant.baseline_quantile), + ("mbr.consensus_corr_min", self.mbr.consensus_corr_min), + ] { + if !value.is_finite() || !(0.0..=1.0).contains(&value) { + return Err(Invalid(format!( + "{name} must be finite and in [0, 1] (got {value})" + ))); + } + } + for (name, value) in [ + ("mbr.q_anchor", self.mbr.q_anchor), + ("quant.reliable_q", self.quant.reliable_q), + ("quant.peak_fraction", self.quant.peak_fraction), + ( + "features.bound_peak_fraction", + self.features.bound_peak_fraction, + ), + ("rescore.train_fdr", self.rescore.train_fdr), + ] { + if !value.is_finite() || value <= 0.0 || value > 1.0 { + return Err(Invalid(format!( + "{name} must be finite and in (0, 1] (got {value})" + ))); + } + } + if !self.features.coelution_corr_threshold.is_finite() + || !(-1.0..=1.0).contains(&self.features.coelution_corr_threshold) + { + return Err(Invalid(format!( + "features.coelution_corr_threshold is a correlation and must be in [-1, 1] \ + (got {})", + self.features.coelution_corr_threshold + ))); + } + if !self.features.bound_confident_pct.is_finite() + || !(0.0..=100.0).contains(&self.features.bound_confident_pct) + { + return Err(Invalid(format!( + "features.bound_confident_pct is a percentile and must be in [0, 100] (got {})", + self.features.bound_confident_pct + ))); + } + for (name, value) in [ + ( + "search_seed.fragment_tol_ppm", + self.search_seed.fragment_tol_ppm, + ), + ("extract.frag_tol_ppm", self.extract.frag_tol_ppm), + ("extract.prec_tol_ppm", self.extract.prec_tol_ppm), + ("features.prec_tol_ppm", self.features.prec_tol_ppm), + ("prescan.tol_da", self.prescan.tol_da), + ("prescan.rt_bin_s", self.prescan.rt_bin_s), + ( + "extract.claim_cues.mz_close_sigma_ppm", + self.extract.claim_cues.mz_close_sigma_ppm, + ), + ( + "extract.claim_cues.rt_prior_tau_s", + self.extract.claim_cues.rt_prior_tau_s, + ), + ("extract.peak_claim_margin", self.extract.peak_claim_margin), + ("mbr.rt_window_s", self.mbr.rt_window_s), + ("rescore.entrapment_ratio", self.rescore.entrapment_ratio), + ] { + if !value.is_finite() || value <= 0.0 { + return Err(Invalid(format!( + "{name} must be finite and > 0 (got {value})" + ))); + } + } + for (name, value) in [ + ("prescan.rt_slack_s", self.prescan.rt_slack_s), + ("extract.demix_lambda", self.extract.demix_lambda), + ( + "extract.alt_peak_min_separation_s", + self.extract.alt_peak_min_separation_s, + ), + ( + "compete.apex_rt_tolerance_s", + self.compete.apex_rt_tolerance_s, + ), + ("compete.margin", self.compete.margin), + ("quant.fixed_window_s", self.quant.fixed_window_s), + ] { + if !value.is_finite() || value < 0.0 { + return Err(Invalid(format!( + "{name} must be finite and >= 0 (got {value}); 0 keeps its documented \ + meaning" + ))); + } + } + for (name, value) in [ + ("mbr.min_anchor_runs", self.mbr.min_anchor_runs), + ("rescore.num_iter", self.rescore.num_iter), + ("quant.top_n_fragments", self.quant.top_n_fragments), + ("quant.top_n_peptides", self.quant.top_n_peptides), + ( + "quant.baseline_flank_scans", + self.quant.baseline_flank_scans, + ), + ( + "extract.demix_max_candidates", + self.extract.demix_max_candidates, + ), + ("extract.demix_scan_stride", self.extract.demix_scan_stride), + ("extract.retain_top_peaks", self.extract.retain_top_peaks), + ("extract.promote_top_peaks", self.extract.promote_top_peaks), + ( + "rt_im_train.adaptive_rt_bins", + self.rt_im_train.adaptive_rt_bins, + ), + ] { + if value == 0 { + return Err(Invalid(format!("{name} must be >= 1"))); + } + } + if self.rescore.folds < 2 { + return Err(Invalid(format!( + "rescore.folds must be >= 2 for cross-validated scores (got {})", + self.rescore.folds + ))); + } + if self.extract.promote_top_peaks > self.extract.retain_top_peaks { + return Err(Invalid(format!( + "extract.promote_top_peaks ({}) must be <= extract.retain_top_peaks ({})", + self.extract.promote_top_peaks, self.extract.retain_top_peaks + ))); + } + if !self.extract.bucket_size.is_power_of_two() { + return Err(Invalid(format!( + "extract.bucket_size must be a power of two (got {})", + self.extract.bucket_size + ))); + } + if self.rt_im_train.adaptive_rt_window && self.rt_im_train.adaptive_rt_bins == 0 { + return Err(Invalid( + "rt_im_train.adaptive_rt_bins must be >= 1 when adaptive_rt_window is on".into(), + )); + } Ok(()) } @@ -2211,6 +2506,92 @@ mod tests { assert!(Config::from_json(r#"{"rescore":{"classifier":"entrapment"}}"#).is_err()); } + #[test] + fn numeric_domains_are_validated_at_load() { + // The three values docs/29 #18 reproduced as accepted: a negative q threshold + // excluded every discovery, a negative multiplier was clamped into a one-second + // window, and a fraction above one fed the training recipe out of its domain. + for bad in [ + r#"{"quant":{"q_threshold":-0.1}}"#, + r#"{"rt_im_train":{"rt_window_multiplier":-1.0}}"#, + r#"{"rescore":{"train_margin_frac":2.0}}"#, + r#"{"digest":{"min_len":10,"max_len":5}}"#, + r#"{"digest":{"min_len":0}}"#, + r#"{"peptidoforms":{"charge_min":3,"charge_max":2}}"#, + r#"{"rt_im_train":{"window_holdout_frac":1.0}}"#, + r#"{"rt_im_train":{"p_rt":0.0}}"#, + r#"{"search_seed":{"min_matched_peaks":0}}"#, + r#"{"experiment":{"parallel_runs":0}}"#, + r#"{"rescore":{"train_neg_ratio":-1.0}}"#, + r#"{"predict_frag":{"top_n_fragments":0}}"#, + // docs/30 R5: the five values the follow-up review found still accepted. + r#"{"mbr":{"q_anchor":-0.1}}"#, + r#"{"mbr":{"q_anchor":2.0,"min_anchor_runs":0}}"#, + r#"{"mbr":{"min_anchor_runs":0}}"#, + r#"{"extract":{"min_matched_fraction":2.0}}"#, + r#"{"features":{"bound_peak_fraction":-1.0}}"#, + r#"{"quant":{"reliable_q":2.0}}"#, + r#"{"features":{"coelution_corr_threshold":1.5}}"#, + r#"{"features":{"bound_confident_pct":101}}"#, + r#"{"extract":{"frag_tol_ppm":0.0}}"#, + r#"{"extract":{"bucket_size":1000}}"#, + r#"{"extract":{"retain_top_peaks":1,"promote_top_peaks":3}}"#, + r#"{"rescore":{"folds":1}}"#, + r#"{"quant":{"top_n_peptides":0}}"#, + r#"{"rescore":{"train_fdr":0.0}}"#, + ] { + assert!(Config::from_json(bad).is_err(), "{bad} must be rejected"); + } + // Documented zero semantics survive: no negative cap, in-sample window sizing, + // an uncapped subsample, a row-cap subsample above one. And the boundaries of + // every new domain are accepted, not only their violations rejected. + for ok in [ + r#"{"rescore":{"train_neg_ratio":0}}"#, + r#"{"rt_im_train":{"window_holdout_frac":0.0}}"#, + r#"{"rescore":{"train_subsample":0.0}}"#, + r#"{"rescore":{"train_subsample":2000}}"#, + r#"{"rescore":{"max_feature_matrix_gib":0}}"#, + r#"{"extract":{"apex_rt_prior_s":0.0}}"#, + r#"{"mbr":{"q_anchor":1.0}}"#, + r#"{"mbr":{"q_anchor":0.001,"min_anchor_runs":1}}"#, + r#"{"extract":{"min_matched_fraction":0.0}}"#, + r#"{"extract":{"min_matched_fraction":1.0}}"#, + r#"{"features":{"bound_peak_fraction":1.0}}"#, + r#"{"quant":{"reliable_q":0.001}}"#, + r#"{"features":{"coelution_corr_threshold":-1.0}}"#, + r#"{"features":{"bound_confident_pct":100}}"#, + r#"{"mbr":{"consensus_corr_min":0.0}}"#, + r#"{"compete":{"margin":0.0}}"#, + r#"{"quant":{"fixed_window_s":0.0}}"#, + r#"{"extract":{"bucket_size":4096}}"#, + r#"{"extract":{"retain_top_peaks":3,"promote_top_peaks":3}}"#, + r#"{"rescore":{"folds":2}}"#, + ] { + assert!(Config::from_json(ok).is_ok(), "{ok} must be accepted"); + } + assert!(Config::default().validate().is_ok()); + } + + #[test] + fn enabling_the_fine_tune_keeps_the_automatic_batch() { + // docs/30 R1: turning the fine-tune on with its own defaults was rejected because + // the documented automatic batch size is 0. Omitted, explicit zero and a positive + // override are all valid; a zero epoch count is not. + for ok in [ + r#"{"rt_im_train":{"finetune_deeplc":true}}"#, + r#"{"rt_im_train":{"finetune_deeplc":true,"finetune_batch":0}}"#, + r#"{"rt_im_train":{"finetune_deeplc":true,"finetune_batch":256}}"#, + ] { + let c = Config::from_json(ok).unwrap_or_else(|e| panic!("{ok} must be accepted: {e}")); + assert!(c.rt_im_train.finetune_deeplc); + } + let e = + Config::from_json(r#"{"rt_im_train":{"finetune_deeplc":true,"finetune_epochs":0}}"#) + .unwrap_err() + .to_string(); + assert!(e.contains("finetune_epochs"), "{e}"); + } + #[test] fn explicit_uncapped_seed_and_invalid_gate_are_distinguished() { let c = Config::from_json(r#"{"search_seed":{"top_n_peaks":0}}"#).unwrap(); diff --git a/rust/mumdia/crates/mumdia-core/src/rejection.rs b/rust/mumdia/crates/mumdia-core/src/rejection.rs index 06c19745..943e8c5e 100644 --- a/rust/mumdia/crates/mumdia-core/src/rejection.rs +++ b/rust/mumdia/crates/mumdia-core/src/rejection.rs @@ -7,8 +7,15 @@ //! DIA-NN-only precursor first lost?" without conflating later stages. //! //! The serialized spelling is SCREAMING_SNAKE_CASE and matches the reason strings -//! in the specification exactly (e.g. `NO_PEAK_GROUP`). Use [`RejectionReason::code`] -//! for the stable string written to Parquet/JSON (no serde round-trip cost). +//! in the specification exactly (e.g. `NO_FRAGMENT_TRACES`). Use +//! [`RejectionReason::code`] for the stable string written to Parquet/JSON (no serde +//! round-trip cost). +//! +//! `DID_NOT_SURVIVE_EXTRACTION` was `NO_PEAK_GROUP` until docs/29 #16. The audit +//! assigns it to every candidate that has traces in no extracted row, and `extract` +//! does not write the per-candidate table that would separate presence failures, +//! matched-fraction failures and gate rejections, so the old name claimed a cause the +//! audit cannot see. The name now says what is known. use serde::{Deserialize, Serialize}; @@ -29,7 +36,9 @@ pub enum RejectionReason { CandidateCapReached, // --- extraction + peak formation (Stages C, D) --- NoFragmentTraces, - NoPeakGroup, + /// Extracted no accepted row, for any extraction-side reason; see the module + /// documentation for why this is not called a peak-group failure. + DidNotSurviveExtraction, // --- peak / peptide ranking (Stage E) --- PeakNotSelected, // --- competition (Stage G) --- @@ -59,7 +68,7 @@ impl RejectionReason { RtPruned => "RT_PRUNED", CandidateCapReached => "CANDIDATE_CAP_REACHED", NoFragmentTraces => "NO_FRAGMENT_TRACES", - NoPeakGroup => "NO_PEAK_GROUP", + DidNotSurviveExtraction => "DID_NOT_SURVIVE_EXTRACTION", PeakNotSelected => "PEAK_NOT_SELECTED", OutcompetedByTarget => "OUTCOMPETED_BY_TARGET", OutcompetedByDecoy => "OUTCOMPETED_BY_DECOY", @@ -85,7 +94,7 @@ impl RejectionReason { RtPruned => 6, CandidateCapReached => 7, NoFragmentTraces => 8, - NoPeakGroup => 9, + DidNotSurviveExtraction => 9, PeakNotSelected => 10, OutcompetedByTarget => 11, OutcompetedByDecoy => 12, @@ -119,7 +128,7 @@ mod tests { #[test] fn codes_match_spec_strings() { - assert_eq!(NoPeakGroup.code(), "NO_PEAK_GROUP"); + assert_eq!(DidNotSurviveExtraction.code(), "DID_NOT_SURVIVE_EXTRACTION"); assert_eq!(PeakNotSelected.code(), "PEAK_NOT_SELECTED"); assert_eq!(OutcompetedByDecoy.code(), "OUTCOMPETED_BY_DECOY"); assert_eq!(Reported.code(), "REPORTED"); @@ -136,8 +145,14 @@ mod tests { #[test] fn earliest_keeps_smaller_stage() { // an extraction loss precedes an FDR loss - assert_eq!(NoPeakGroup.earliest(FailedPrecursorFdr), NoPeakGroup); - assert_eq!(FailedPrecursorFdr.earliest(NoPeakGroup), NoPeakGroup); + assert_eq!( + DidNotSurviveExtraction.earliest(FailedPrecursorFdr), + DidNotSurviveExtraction + ); + assert_eq!( + FailedPrecursorFdr.earliest(DidNotSurviveExtraction), + DidNotSurviveExtraction + ); // Reported never wins against a real rejection assert_eq!(Reported.earliest(NoFragmentTraces), NoFragmentTraces); assert_eq!(NoFragmentTraces.earliest(Reported), NoFragmentTraces); @@ -145,7 +160,7 @@ mod tests { #[test] fn is_rejection_flags_only_losses() { - assert!(NoPeakGroup.is_rejection()); + assert!(DidNotSurviveExtraction.is_rejection()); assert!(!Reported.is_rejection()); } @@ -153,8 +168,8 @@ mod tests { fn stage_order_is_monotone_ladder() { // ladder ordering across the major stages assert!(PeptideNotGenerated.stage_order() < NoFragmentTraces.stage_order()); - assert!(NoFragmentTraces.stage_order() < NoPeakGroup.stage_order()); - assert!(NoPeakGroup.stage_order() < PeakNotSelected.stage_order()); + assert!(NoFragmentTraces.stage_order() < DidNotSurviveExtraction.stage_order()); + assert!(DidNotSurviveExtraction.stage_order() < PeakNotSelected.stage_order()); assert!(PeakNotSelected.stage_order() < OutcompetedByTarget.stage_order()); assert!(OutcompetedByTarget.stage_order() < FailedPrecursorFdr.stage_order()); assert!(FailedPrecursorFdr.stage_order() < RemovedDuringReporting.stage_order()); diff --git a/rust/mumdia/crates/mumdia-io/src/table.rs b/rust/mumdia/crates/mumdia-io/src/table.rs index b951b813..8e85bae0 100644 --- a/rust/mumdia/crates/mumdia-io/src/table.rs +++ b/rust/mumdia/crates/mumdia-io/src/table.rs @@ -342,10 +342,19 @@ impl TableWriter { /// directory of unopenable artifacts is still worse than a directory of intact ones; /// - nothing distinguished "this run wrote it" from "a previous run left it". /// -/// Writing to `.tmp-` and renaming on success addresses all three: the +/// Writing to `.tmp--` and renaming on success addresses all three: the /// rename is atomic on both POSIX and Windows for a same-directory target, so a reader /// sees either the old artifact or the new one, never a partial one. A killed run -/// leaves at most a recognisable `.tmp-` file, which is inert. +/// leaves at most a recognisable `.tmp--` file, which is inert. +/// +/// Two guarantees this makes, and one it does not (docs/29 #4): the destination is never +/// removed before the rename, so a failed publication leaves the previous artifact in +/// place; and `n` is a process-wide counter, so two writers for one destination in one +/// process cannot share a temporary file. It does not promise that the rename succeeds +/// while another Windows process holds the destination open without delete sharing; +/// then the rename fails and the old file stays, which is the first guarantee at work. +static TMP_COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + pub struct AtomicPath { tmp: std::path::PathBuf, final_path: std::path::PathBuf, @@ -361,7 +370,8 @@ impl AtomicPath { .with_context(|| format!("creating output directory {}", parent.display()))?; } } - let tmp = std::path::PathBuf::from(format!("{path}.tmp-{}", std::process::id())); + let n = TMP_COUNTER.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + let tmp = std::path::PathBuf::from(format!("{path}.tmp-{}-{n}", std::process::id())); Ok(AtomicPath { tmp, final_path, @@ -375,14 +385,12 @@ impl AtomicPath { /// Move the completed temp file onto the final path. pub fn publish(mut self) -> Result<()> { - // Windows `rename` fails when the destination exists, unlike POSIX. Removing - // first opens a window in which neither file is at the final path; that is - // strictly better than the previous behaviour, where the window lasted for the - // whole write. - if self.final_path.exists() { - std::fs::remove_file(&self.final_path) - .with_context(|| format!("replacing {}", self.final_path.display()))?; - } + // `std::fs::rename` replaces an existing destination FILE on POSIX and, through + // `MoveFileExW(MOVEFILE_REPLACE_EXISTING)`, on Windows, so the destination is not + // removed first. Removing it first meant a rename that then failed had already + // destroyed the previous result, and gave every reader a window with no file at + // the final path at all (docs/29 #4). Now a failed rename is an error with the + // previous artifact still where it was. std::fs::rename(&self.tmp, &self.final_path).with_context(|| { format!( "publishing {} -> {}", @@ -748,6 +756,27 @@ fn reject_null(name: &str, row: usize) -> anyhow::Error { ) } +/// The same contract as `reject_null`, for a reader that walks Arrow batches itself. +/// +/// The streaming library loader reads fragment columns straight from record batches, and +/// `values()` on an Arrow array is the physical buffer: it ignores the validity bitmap, +/// so a NULL reads as 0 or 0.0 and a finiteness check over it proves nothing. Call this +/// on every required column of a batch before touching its values (docs/29 #2). +/// `row_offset` is the absolute row of the batch's first element, so the message names +/// the row a person can find in the file. +pub fn require_no_nulls( + array: &dyn arrow::array::Array, + name: &str, + path: &str, + row_offset: usize, +) -> Result<()> { + if array.null_count() == 0 { + return Ok(()); + } + let row = (0..array.len()).find(|&i| array.is_null(i)).unwrap_or(0); + Err(reject_null(name, row_offset + row).context(format!("in {path}"))) +} + fn push_i64(out: &mut Vec, col: &ArrayRef, name: &str) -> Result<()> { let a: &Int64Array = downcast(col, name, "i64")?; if a.null_count() == 0 { @@ -1523,3 +1552,82 @@ mod streaming_tests { std::fs::remove_file(&p).ok(); } } + +#[cfg(test)] +mod atomic_path_tests { + use super::*; + + fn dir(name: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!("mumdia_atomic_{}_{}", name, std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d + } + + #[test] + fn two_writers_for_one_destination_get_different_temporary_files() { + let d = dir("two_writers"); + let final_path = d.join("out.parquet").to_str().unwrap().to_string(); + let a = AtomicPath::new(&final_path).unwrap(); + let b = AtomicPath::new(&final_path).unwrap(); + assert_ne!( + a.tmp(), + b.tmp(), + "the suffix must not be the process id alone" + ); + drop(a); + drop(b); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn a_failed_publication_leaves_the_previous_result_in_place() { + // The destination is a non-empty directory, which a file cannot be renamed onto + // on any platform, so publication fails. Before the fix the destination was + // removed first and the failure left nothing behind. + let d = dir("failed_publish"); + let final_path = d.join("out.parquet"); + std::fs::create_dir_all(&final_path).unwrap(); + std::fs::write(final_path.join("previous"), b"previous result").unwrap(); + let ap = AtomicPath::new(final_path.to_str().unwrap()).unwrap(); + std::fs::write(ap.tmp(), b"new result").unwrap(); + assert!( + ap.publish().is_err(), + "renaming a file onto a directory must fail" + ); + assert!( + final_path.join("previous").is_file(), + "the previous result must survive a failed publication" + ); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn a_successful_publication_replaces_the_previous_file() { + let d = dir("replace"); + let final_path = d.join("out.parquet"); + let fp = final_path.to_str().unwrap().to_string(); + let first = AtomicPath::new(&fp).unwrap(); + std::fs::write(first.tmp(), b"v1").unwrap(); + first.publish().unwrap(); + let second = AtomicPath::new(&fp).unwrap(); + std::fs::write(second.tmp(), b"v2").unwrap(); + second.publish().unwrap(); + assert_eq!(std::fs::read(&final_path).unwrap(), b"v2"); + assert!( + std::fs::read_dir(&d).unwrap().count() == 1, + "no temporary file may remain after publication" + ); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn require_no_nulls_names_the_column_and_the_absolute_row() { + let a = arrow::array::Float64Array::from(vec![Some(1.0), None, Some(3.0)]); + let err = require_no_nulls(&a, "mz", "lib.parquet", 1000).unwrap_err(); + let msg = format!("{err:#}"); + assert!(msg.contains("'mz'") && msg.contains("row 1001"), "{msg}"); + let ok = arrow::array::Float64Array::from(vec![Some(1.0), Some(2.0)]); + assert!(require_no_nulls(&ok, "mz", "lib.parquet", 0).is_ok()); + } +} diff --git a/rust/mumdia/crates/mumdia/src/calibrate.rs b/rust/mumdia/crates/mumdia/src/calibrate.rs index 8594d964..b9902eaa 100644 --- a/rust/mumdia/crates/mumdia/src/calibrate.rs +++ b/rust/mumdia/crates/mumdia/src/calibrate.rs @@ -28,12 +28,38 @@ pub fn linear_fit(xs: &[f64], ys: &[f64]) -> (f64, f64) { } /// A LOESS smoother evaluated on a precomputed grid for fast bulk application. +/// +/// Outside the training range the smoother continues the local fit at the nearest +/// boundary: the value at the boundary grid point and the slope of the local line +/// fitted there. It used to switch to the global least-squares line the moment `x` +/// left the grid, and the two models need not agree at the boundary: on +/// `y = 200 + 10 x^2` over `x in [0, 9.9]` (span 0.3) the prediction jumped from +/// 193.4 at `x = 1e-6` to 38.3 at `x = 0`, and from 1173.5 to 1018.4 at the top, +/// discontinuities of about 155 s in retention-time units that misplaced +/// gradient-edge peptides relative to a narrow extraction window (docs/29 #10). pub struct Loess { grid_x: Vec, grid_y: Vec, - // linear fallback for x outside the training range + /// The global least-squares line: the fit for fewer than four points, and the + /// value a degenerate local window (no spread) falls back to. Not the + /// extrapolation model. slope: f64, intercept: f64, + /// Extrapolation slope below the first and above the last grid point; the boundary + /// grid VALUE plus this slope keeps `predict` continuous at both ends. + /// + /// The secant of the fitted curve over its end decile, not the pointwise local slope + /// at the boundary. The pointwise slope comes from the one place where the tricubic + /// window is one-sided and the anchors are sparsest, so it is the noisiest number the + /// fit produces, unbounded, and free to be negative: on 55 anchors with 120 s of + /// scatter it ranged 6.9 to 28.4 against a global slope near 24, and a peptidoform + /// one iRT decade outside the anchors landed 1077 s from where the previous global + /// line put it, against an `w_rt` near 180 s (docs/31 F7). A negative slope inverts + /// the iRT-to-RT map, which the global line it replaced could not do. The secant + /// averages the same curve over many grid points, and the clamp below keeps the + /// result monotone and within a factor of the global least-squares slope. + lo_slope: f64, + hi_slope: f64, } impl Loess { @@ -62,6 +88,8 @@ impl Loess { grid_y, slope, intercept, + lo_slope: slope, + hi_slope: slope, }; } let k = ((span * n as f64).ceil() as usize).clamp(3, n); @@ -71,26 +99,45 @@ impl Loess { let mut grid_y = Vec::with_capacity(gn); for g in 0..gn { let x = lo + (hi - lo) * g as f64 / (gn - 1) as f64; + let (y, _) = local_linear(&sx, &sy, x, k, slope, intercept); grid_x.push(x); - grid_y.push(local_linear(&sx, &sy, x, k, slope, intercept)); + grid_y.push(y); } + let (lo_slope, hi_slope) = boundary_slopes(&grid_x, &grid_y, slope); Loess { grid_x, grid_y, slope, intercept, + lo_slope, + hi_slope, } } - /// Predict at x by linear interpolation over the grid; linear extrapolation - /// outside the training range. + /// Predict at x by linear interpolation over the grid. Outside the training range + /// the local fit at the nearest boundary is continued (its value there and its + /// slope), so the prediction is continuous across the boundary; see the type's + /// documentation for what the previous global-line switch did. pub fn predict(&self, x: f64) -> f64 { + // NaN in, NaN out. With x = NaN both boundary comparisons are false and + // `partition_point` returns 0, so the interpolation below indexed `grid_x[-1]`: + // a panic in debug and an out-of-bounds read in release, reachable from one + // library row whose `predicted_irt` is null, because the parquet reader maps a + // null f32 to NaN (docs/31 F4). Callers treat a NaN prediction as "no + // calibration for this row", which is the documented recall-safe sentinel. + if !x.is_finite() { + return f64::NAN; + } let g = &self.grid_x; - if x <= g[0] || g.len() < 2 { + if g.len() < 2 { return self.slope * x + self.intercept; } - if x >= g[g.len() - 1] { - return self.slope * x + self.intercept; + if x <= g[0] { + return self.grid_y[0] + self.lo_slope * (x - g[0]); + } + let last = g.len() - 1; + if x >= g[last] { + return self.grid_y[last] + self.hi_slope * (x - g[last]); } let j = g.partition_point(|&gx| gx < x); let (x0, x1) = (g[j - 1], g[j]); @@ -102,9 +149,56 @@ impl Loess { } } +/// Extrapolation slopes for the two ends of a fitted grid. +/// +/// Each is the secant of the fitted curve over its end decile, clamped to be +/// non-negative (an inverted iRT-to-RT map is never right) and, when the global +/// least-squares slope is positive, to at most four times it. Both guards exist because +/// this number multiplies a distance that has no bound: it is applied to queries outside +/// the anchor range entirely. See the field documentation on [`Loess`]. +fn boundary_slopes(grid_x: &[f64], grid_y: &[f64], global_slope: f64) -> (f64, f64) { + let n = grid_x.len(); + if n < 2 { + return (global_slope, global_slope); + } + let step = (n / 10).max(1).min(n - 1); + let secant = |a: usize, b: usize| { + let dx = grid_x[b] - grid_x[a]; + if dx.abs() < 1e-12 { + global_slope + } else { + (grid_y[b] - grid_y[a]) / dx + } + }; + let cap = if global_slope.is_finite() && global_slope > 0.0 { + 4.0 * global_slope + } else { + f64::INFINITY + }; + let clamp = |v: f64| { + if !v.is_finite() { + return if global_slope.is_finite() { + global_slope.max(0.0) + } else { + 0.0 + }; + } + v.clamp(0.0, cap) + }; + (clamp(secant(0, step)), clamp(secant(n - 1 - step, n - 1))) +} + /// Weighted local linear regression at `x0` over the `k` nearest points -/// (tricubic weights). Falls back to the global line if degenerate. -fn local_linear(sx: &[f64], sy: &[f64], x0: f64, k: usize, slope: f64, intercept: f64) -> f64 { +/// (tricubic weights): the fitted value at `x0` and the slope of the local line. +/// Falls back to the global line (value and slope) if degenerate. +fn local_linear( + sx: &[f64], + sy: &[f64], + x0: f64, + k: usize, + slope: f64, + intercept: f64, +) -> (f64, f64) { let n = sx.len(); // find window of k nearest by walking from the insertion point let mut lo = sx.partition_point(|&x| x < x0); @@ -145,11 +239,11 @@ fn local_linear(sx: &[f64], sy: &[f64], x0: f64, k: usize, slope: f64, intercept } let denom = sw * swxx - swx * swx; if sw < 1e-12 || denom.abs() < 1e-12 { - return slope * x0 + intercept; + return (slope * x0 + intercept, slope); } let b = (sw * swxy - swx * swy) / denom; let a = (swy - b * swx) / sw; - a + b * x0 + (a + b * x0, b) } /// The p-th percentile (0..1) of the values (a copy is sorted). @@ -184,6 +278,93 @@ mod tests { assert!((lo.predict(5.0) - 25.0).abs() < 2.0, "{}", lo.predict(5.0)); } + #[test] + fn loess_is_continuous_at_both_training_boundaries() { + // The docs/29 #10 reproduction: a curved map whose global line disagrees with + // the local fit at both ends. The old predictor jumped by about 155 at each + // boundary; the continued local fit must not. + let xs: Vec = (0..100).map(|i| i as f64 / 10.0).collect(); + let ys: Vec = xs.iter().map(|x| 200.0 + 10.0 * x * x).collect(); + let lo = Loess::fit(&xs, &ys, 0.3, 50); + let eps = 1e-6; + let (a0, a1) = (lo.predict(0.0), lo.predict(eps)); + assert!((a0 - a1).abs() < 1e-3, "lower boundary jump: {a0} vs {a1}"); + let (b0, b1) = (lo.predict(9.9), lo.predict(9.9 - eps)); + assert!((b0 - b1).abs() < 1e-3, "upper boundary jump: {b0} vs {b1}"); + // Extrapolation follows the LOCAL slope, not the global line's (99 here): the + // curve is nearly flat at the low end and steep (about 200) at the high end. + let low_step = lo.predict(0.0) - lo.predict(-1.0); + assert!( + (0.0..50.0).contains(&low_step), + "low-end extrapolation should be nearly flat, got a step of {low_step}" + ); + let high_step = lo.predict(10.9) - lo.predict(9.9); + assert!( + high_step > 120.0, + "high-end extrapolation should follow the steep local slope, got {high_step}" + ); + // Far from the data the extrapolation is a line through the boundary value. + assert!( + (lo.predict(-2.0) - (lo.predict(0.0) - 2.0 * low_step)).abs() < 1e-9, + "extrapolation must be linear in x" + ); + } + + #[test] + fn a_nan_query_returns_nan_instead_of_indexing_out_of_bounds() { + // docs/31 F4: one library row with a null predicted_irt reaches this as NaN. + let xs: Vec = (0..40).map(|i| i as f64).collect(); + let ys: Vec = xs.iter().map(|x| 2.0 * x + 1.0).collect(); + let lo = Loess::fit(&xs, &ys, 0.3, 20); + assert!(lo.predict(f64::NAN).is_nan()); + assert!(lo.predict(f64::INFINITY).is_nan()); + assert!(lo.predict(f64::NEG_INFINITY).is_nan()); + // A degenerate fit takes the linear path and must not panic either. + let tiny = Loess::fit(&[1.0, 2.0], &[1.0, 2.0], 0.5, 2); + assert!(tiny.predict(f64::NAN).is_nan()); + } + + #[test] + fn boundary_extrapolation_stays_monotone_and_bounded_under_noise() { + // docs/31 F7: the pointwise boundary slope came from the sparsest, most one-sided + // local window in the fit. On noisy anchors it was free to be negative (inverting + // the iRT-to-RT map) or several times the global slope, and it multiplies a + // distance that is unbounded because it applies outside the anchor range. + // Deterministic pseudo-noise so the assertion is reproducible. + let mut seed = 0x243f_6a88_85a3_08d3u64; + let mut noise = || { + seed = seed + .wrapping_mul(6364136223846793005) + .wrapping_add(1442695040888963407); + ((seed >> 33) as f64 / (1u64 << 31) as f64 - 1.0) * 120.0 + }; + let xs: Vec = (0..55).map(|i| i as f64 * 2.0).collect(); + let ys: Vec = xs.iter().map(|x| 500.0 + 24.0 * x + noise()).collect(); + let lo = Loess::fit(&xs, &ys, 0.3, 200); + let (slope, _) = linear_fit(&xs, &ys); + for s in [lo.lo_slope, lo.hi_slope] { + assert!(s >= 0.0, "an inverted extrapolation slope: {s}"); + assert!( + s <= 4.0 * slope, + "an unbounded extrapolation slope: {s} vs {slope}" + ); + } + // Monotone outside the range, and continuous at both ends. + let x0 = xs[0]; + let xn = xs[xs.len() - 1]; + assert!(lo.predict(x0 - 30.0) <= lo.predict(x0)); + assert!(lo.predict(xn + 30.0) >= lo.predict(xn)); + assert!((lo.predict(x0) - lo.predict(x0 + 1e-6)).abs() < 1.0); + assert!((lo.predict(xn) - lo.predict(xn - 1e-6)).abs() < 1.0); + // And the error one decade outside the anchors stays within a search window. + let far = lo.predict(x0 - 30.0); + let linear = slope * (x0 - 30.0) + lo.intercept; + assert!( + (far - linear).abs() < 600.0, + "extrapolation {far} is implausibly far from the global line {linear}" + ); + } + #[test] fn percentile_basic() { let v: Vec = (0..=100).map(|i| i as f64).collect(); diff --git a/rust/mumdia/crates/mumdia/src/index.rs b/rust/mumdia/crates/mumdia/src/index.rs index 74e82373..4f26c53e 100644 --- a/rust/mumdia/crates/mumdia/src/index.rs +++ b/rust/mumdia/crates/mumdia/src/index.rs @@ -17,7 +17,7 @@ use anyhow::Result; use arrow::array::{Array, Float32Array, Float64Array, StringArray, UInt32Array}; use mumdia_core::constants::{ppm_bounds, PROTON}; -use mumdia_io::table::TableFile; +use mumdia_io::table::{require_no_nulls, TableFile}; use rayon::prelude::*; /// Fragment rows per decoded batch while streaming the fragment table (a few MB). @@ -244,6 +244,10 @@ impl Library { .as_any() .downcast_ref::() .ok_or_else(|| anyhow::anyhow!("fragment column 'candidate_id' is not u32"))?; + // `values()` is the physical buffer and ignores the validity bitmap: a NULL + // candidate_id would read as 0 and attach the fragment to candidate 0 + // (docs/29 #2). + require_no_nulls(a, "candidate_id", fragments, row)?; for &candidate_id in a.values().iter() { let c = candidate_id as usize; if c >= ncand { @@ -286,6 +290,7 @@ impl Library { ix("predicted_intensity")?, ix("name")?, ); + let mut row_base = 0usize; for b in reader { let b = b?; let a_cid = b @@ -305,6 +310,19 @@ impl Library { .ok_or_else(|| { anyhow::anyhow!("fragment column 'predicted_intensity' is not f32") })?; + let a_name = b + .column(i_name) + .as_any() + .downcast_ref::() + .ok_or_else(|| anyhow::anyhow!("fragment column 'name' is not utf8"))?; + // Every required column, before any value is read: the finiteness checks + // above ran on physical buffers, where a NULL is a perfectly finite 0.0, and + // the fill below used to turn NULLs into NaN and "" instead of refusing + // them (docs/29 #2). + require_no_nulls(a_cid, "candidate_id", fragments, row_base)?; + require_no_nulls(a_mz, "mz", fragments, row_base)?; + require_no_nulls(a_int, "predicted_intensity", fragments, row_base)?; + require_no_nulls(a_name, "name", fragments, row_base)?; // Same contract as the precursor columns above, applied batch by batch. A // non-finite fragment m/z is worse than a wrong value: `FragIndex::build` // collapses its whole m/z range when the observed min or max is not finite, @@ -331,11 +349,6 @@ impl Library { a_cid.value(k) ); } - let a_name = b - .column(i_name) - .as_any() - .downcast_ref::() - .ok_or_else(|| anyhow::anyhow!("fragment column 'name' is not utf8"))?; for k in 0..b.num_rows() { let c = a_cid.value(k) as usize; if c >= ncand { @@ -345,23 +358,10 @@ impl Library { } let pos = cursor[c] as usize; cursor[c] += 1; - // Null policy matches the typed getters (null f64/f32 -> NaN, null utf8 - // -> ""); the artifact has no nulls, this only keeps the contract exact. - frag_mz[pos] = if a_mz.is_null(k) { - f32::NAN - } else { - a_mz.value(k) as f32 - }; - frag_int[pos] = if a_int.is_null(k) { - f32::NAN - } else { - a_int.value(k) - }; - let name = if a_name.is_null(k) { - "" - } else { - a_name.value(k) - }; + // NULLs were rejected above, so the physical values are the values. + frag_mz[pos] = a_mz.value(k) as f32; + frag_int[pos] = a_int.value(k); + let name = a_name.value(k); let id = match name_lookup.get(name) { Some(&id) => id, None => { @@ -379,6 +379,7 @@ impl Library { }; frag_name_id[pos] = id; } + row_base += b.num_rows(); } } drop(name_lookup); @@ -922,3 +923,111 @@ mod tests { std::fs::remove_dir_all(&dir).ok(); } } + +#[cfg(test)] +mod null_fixture_tests { + use super::*; + use arrow::array::{Float32Array, Float64Array, Int32Array, StringArray, UInt32Array}; + use arrow::datatypes::{DataType, Field, Schema}; + use arrow::record_batch::RecordBatch; + use mumdia_io::table::{write_batches, write_table, Col}; + use std::sync::Arc; + + fn precursors(dir: &std::path::Path) -> String { + let p = dir.join("prec.parquet").to_str().unwrap().to_string(); + write_table( + &p, + vec![ + Col::U32("candidate_id".into(), vec![0, 1]), + Col::U32("peptidoform_id".into(), vec![0, 1]), + Col::U32("base_peptide_id".into(), vec![0, 1]), + Col::Str( + "peptidoform".into(), + vec!["PEPTIDEK".into(), "SAMPLER".into()], + ), + Col::I32("charge".into(), vec![2, 2]), + Col::F64("precursor_mz".into(), vec![400.0, 500.0]), + Col::F32("predicted_irt".into(), vec![10.0, 20.0]), + Col::Str("label".into(), vec!["target".into(), "decoy".into()]), + Col::Str("protein".into(), vec!["P1".into(), "P2".into()]), + Col::I32("n_fragments".into(), vec![1, 1]), + ], + ) + .unwrap(); + p + } + + /// A two-row fragment table with one NULL in `null_in`. + fn fragments_with_null(dir: &std::path::Path, null_in: &str) -> String { + let f = dir + .join(format!("frag_{null_in}.parquet")) + .to_str() + .unwrap() + .to_string(); + let cid = UInt32Array::from(if null_in == "candidate_id" { + vec![Some(0u32), None] + } else { + vec![Some(0u32), Some(1)] + }); + let mz = Float64Array::from(if null_in == "mz" { + vec![Some(200.1), None] + } else { + vec![Some(200.1), Some(250.5)] + }); + let inten = Float32Array::from(if null_in == "predicted_intensity" { + vec![None, Some(0.9f32)] + } else { + vec![Some(1.0f32), Some(0.9)] + }); + let name = StringArray::from(if null_in == "name" { + vec![Some("b2"), None] + } else { + vec![Some("b2"), Some("y3")] + }); + let schema = Arc::new(Schema::new(vec![ + Field::new("candidate_id", DataType::UInt32, true), + Field::new("mz", DataType::Float64, true), + Field::new("predicted_intensity", DataType::Float32, true), + Field::new("name", DataType::Utf8, true), + Field::new("ion_type", DataType::Utf8, false), + Field::new("ordinal", DataType::Int32, false), + Field::new("frag_charge", DataType::Int32, false), + ])); + let batch = RecordBatch::try_new( + schema.clone(), + vec![ + Arc::new(cid), + Arc::new(mz), + Arc::new(inten), + Arc::new(name), + Arc::new(StringArray::from(vec!["b", "y"])), + Arc::new(Int32Array::from(vec![2, 3])), + Arc::new(Int32Array::from(vec![1, 1])), + ], + ) + .unwrap(); + write_batches(&f, schema, &[batch]).unwrap(); + f + } + + #[test] + fn a_null_in_any_required_fragment_column_is_refused_by_name() { + // Before docs/29 #2 a NULL m/z or intensity loaded as NaN (the finiteness check + // looked at physical buffers) and a NULL candidate_id attached the fragment to + // candidate 0. + let dir = std::env::temp_dir().join(format!("mumdia_index_nulls_{}", std::process::id())); + let _ = std::fs::remove_dir_all(&dir); + std::fs::create_dir_all(&dir).unwrap(); + let p = precursors(&dir); + for col in ["candidate_id", "mz", "predicted_intensity", "name"] { + let f = fragments_with_null(&dir, col); + let err = match Library::load(&p, &f, 8) { + Ok(_) => panic!("a NULL {col} must not load"), + Err(e) => format!("{e:#}"), + }; + assert!(err.contains(&format!("'{col}'")), "{col}: {err}"); + assert!(err.contains("NULL"), "{col}: {err}"); + } + let _ = std::fs::remove_dir_all(&dir); + } +} diff --git a/rust/mumdia/crates/mumdia/src/main.rs b/rust/mumdia/crates/mumdia/src/main.rs index 9a06d39d..475d3823 100644 --- a/rust/mumdia/crates/mumdia/src/main.rs +++ b/rust/mumdia/crates/mumdia/src/main.rs @@ -1034,6 +1034,26 @@ fn load_config(path: &Option) -> Result { } fn main() -> Result<()> { + // The dispatch in `real_main` is one large function whose arms keep their locals in + // a single frame, and without optimisation that frame exceeds the 1 MiB main-thread + // stack Windows reserves: the debug binary overflowed on `--version` before printing + // anything (docs/30 R9). The CLI therefore runs on a thread with a generous + // reservation; pages are committed only as they are touched, so release builds pay + // nothing for it. `tests/cli_version.rs` runs the built binary to keep this true. + const MAIN_STACK_BYTES: usize = 256 << 20; + let handle = std::thread::Builder::new() + .name("mumdia-main".into()) + .stack_size(MAIN_STACK_BYTES) + .spawn(real_main) + .context("spawning the main thread")?; + match handle.join() { + Ok(result) => result, + // The panic hook has already printed the message and location. + Err(_) => anyhow::bail!("mumdia stopped on an internal error (see the panic above)"), + } +} + +fn real_main() -> Result<()> { // Held for the whole process: dropping the guard is what writes dhat-heap.json into // the working directory, so it must outlive the stage that is being profiled. #[cfg(feature = "dhat-heap")] diff --git a/rust/mumdia/crates/mumdia/src/raw.rs b/rust/mumdia/crates/mumdia/src/raw.rs index d7282d34..5dda7cb9 100644 --- a/rust/mumdia/crates/mumdia/src/raw.rs +++ b/rust/mumdia/crates/mumdia/src/raw.rs @@ -511,9 +511,251 @@ fn sciex_scan_hint(src: &Path) -> Option { /// 6:48 conversion of a 3.7 GB Astral run was reported as "exited successfully but wrote no /// file" and discarded (doxy, 2026-09-06). msconvert's `--outfile` has the same habit. With /// `x.partial.mzML` there is nothing for either to fix up. -fn partial_name(out_name: &str) -> String { +fn partial_name(out_name: &str, tag: &str) -> String { let stem = out_name.strip_suffix(".mzML").unwrap_or(out_name); - format!("{stem}.partial.mzML") + format!("{stem}.partial-{tag}.mzML") +} + +/// A tag no other conversion in any process shares: process id plus a per-process +/// counter. Two searches converting one acquisition at the same time used to share +/// `.partial.mzML` (docs/30 R4). +fn unique_tag() -> String { + static COUNTER: std::sync::atomic::AtomicU64 = std::sync::atomic::AtomicU64::new(0); + format!( + "{}-{}", + std::process::id(), + COUNTER.fetch_add(1, std::sync::atomic::Ordering::Relaxed) + ) +} + +/// A claim on a conversion destination, held from before the temporary file is written +/// until the result has been renamed into place or the attempt has failed. +/// +/// Two searches converting the same input concurrently wrote one temporary file: +/// converter B replaced A's partial output, A renamed B's bytes into place and reported +/// success, and B failed because its output had been renamed away (docs/30 R4). The +/// temporary name is unique now, and this lock beside the destination makes a second +/// converter wait for the first and reuse what it produced rather than convert again +/// into the same destination. Dropping the guard releases the lock. +struct ConvertLock { + path: PathBuf, + /// Written into the lock file and verified after creation, so a lock this process + /// took cannot be one another process took a moment earlier, and `Drop` cannot + /// delete a lock that is no longer ours. + token: String, +} + +impl ConvertLock { + fn path_for(out: &Path) -> PathBuf { + let mut name = out + .file_name() + .map(|n| n.to_os_string()) + .unwrap_or_default(); + name.push(".converting"); + out.with_file_name(name) + } + + /// Take the lock for `out`, waiting for a holder to finish first. `Ok(None)` means the + /// holder finished and left a usable conversion at `out`, which the caller reuses. + /// + /// Three things this has to survive, all of which the first version did not + /// (docs/31 F9). A stale lock that cannot be deleted is bounded rather than retried + /// forever with no pause. Two waiters that both judge one lock stale cannot both come + /// away holding it, because each writes a token and reads it back, and only the + /// process whose token survives owns the lock. And waiting itself has a deadline, so + /// a lock nothing will ever release fails the run with a message instead of hanging. + fn acquire(out: &Path, src: &Path, reuse: bool) -> Result> { + const RETRY: std::time::Duration = std::time::Duration::from_secs(2); + const MAX_TAKEOVERS: u32 = 8; + const MAX_WAIT: std::time::Duration = std::time::Duration::from_secs(12 * 3600); + let path = Self::path_for(out); + let token = format!("{}:{}", std::process::id(), unique_tag()); + let waiting_since = std::time::Instant::now(); + let mut announced = false; + let mut takeovers = 0u32; + loop { + match std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(&path) + { + Ok(mut f) => { + use std::io::Write; + f.write_all(token.as_bytes()) + .and_then(|()| f.sync_all()) + .with_context(|| { + format!("writing the conversion lock {}", path.display()) + })?; + drop(f); + // Read it back. If another waiter removed this file and created its + // own between the two calls, the token no longer matches and the lock + // is theirs; wait and try again rather than converting alongside them. + let held = std::fs::read_to_string(&path) + .map(|t| t.trim() == token) + .unwrap_or(false); + if held { + return Ok(Some(ConvertLock { path, token })); + } + std::thread::sleep(RETRY); + continue; + } + Err(e) if e.kind() == std::io::ErrorKind::AlreadyExists => { + if lock_is_stale(&path, out) { + takeovers += 1; + if takeovers > MAX_TAKEOVERS { + bail!( + "the conversion lock {} looks abandoned but cannot be \ + removed after {MAX_TAKEOVERS} attempts. Delete it by hand \ + once nothing is converting {}, then rerun.", + path.display(), + out.display() + ); + } + warn!( + lock = %path.display(), + attempt = takeovers, + "convert: removing a stale conversion lock; its holder stopped \ + writing" + ); + // A failed removal used to `continue` straight back into + // `create_new` with no pause, pinning a core and writing one + // warning per iteration for as long as the run lasted. + let _ = std::fs::remove_file(&path); + std::thread::sleep(RETRY); + continue; + } + if !announced { + info!( + mzml = %out.display(), + "convert: another process is converting this input; waiting for \ + it rather than converting into the same destination" + ); + announced = true; + } + if waiting_since.elapsed() > MAX_WAIT { + bail!( + "waited {} hours for the conversion lock {} and it is still \ + held and still fresh. Check for another MuMDIA converting {}.", + MAX_WAIT.as_secs() / 3600, + path.display(), + out.display() + ); + } + std::thread::sleep(RETRY); + if !path.exists() && reuse && out.is_file() && is_newer_than(out, src) { + return Ok(None); + } + } + Err(e) => { + return Err(e).with_context(|| { + format!("creating the conversion lock {}", path.display()) + }) + } + } + } + } +} + +impl Drop for ConvertLock { + fn drop(&mut self) { + // Only if it is still ours: a lock taken over after this one was judged stale + // belongs to the taker, and removing it would leave two converters writing one + // destination, which is the situation the lock exists to prevent. + let ours = std::fs::read_to_string(&self.path) + .map(|t| t.trim() == self.token) + .unwrap_or(false); + if ours { + let _ = std::fs::remove_file(&self.path); + } + } +} + +/// How long a lock and its partial output may go untouched before the holder counts as +/// gone. +const STALE_AFTER: std::time::Duration = std::time::Duration::from_secs(15 * 60); + +/// Was `p` written within [`STALE_AFTER`]? +/// +/// A modification time in the future counts as fresh. It is an unreadable clock, not +/// evidence that nobody is writing: `SystemTime::elapsed` returns `Err` for a future +/// timestamp, and treating that as "not fresh" made ordinary clock skew on a network +/// share declare a live lock stale one second after it was taken (docs/31 F9). +fn written_recently(p: &Path) -> bool { + match std::fs::metadata(p).and_then(|m| m.modified()) { + Ok(t) => match t.elapsed() { + Ok(age) => age < STALE_AFTER, + Err(_) => true, + }, + Err(_) => false, + } +} + +/// The `.partial-` prefix of the temporary files belonging to one destination. +fn partial_prefix(out: &Path) -> String { + let name = out.file_name().unwrap_or_default().to_string_lossy(); + let stem = name.strip_suffix(".mzML").unwrap_or(&name); + format!("{stem}.partial-") +} + +/// A lock whose holder has stopped: neither the lock nor a partial conversion file OF +/// THIS DESTINATION has been written for [`STALE_AFTER`]. Converters write continuously, +/// so a live conversion keeps its partial file fresh; a crashed or killed holder leaves +/// both untouched. +/// +/// Matching this destination's prefix rather than any `.partial-` name is what makes the +/// probe mean anything with several conversions in one directory: it used to accept any +/// partial file, so one live conversion kept a dead peer's lock fresh indefinitely, and +/// with `experiment.parallel_runs > 1` that is the normal case (docs/31 F9). +fn lock_is_stale(lock: &Path, out: &Path) -> bool { + if written_recently(lock) { + return false; + } + let dir = lock.parent().unwrap_or(Path::new(".")); + let prefix = partial_prefix(out); + if let Ok(rd) = std::fs::read_dir(dir) { + for e in rd.flatten() { + let name = e.file_name().to_string_lossy().into_owned(); + if name.starts_with(&prefix) && written_recently(&e.path()) { + return false; + } + } + } + true +} + +/// Remove this destination's abandoned partial conversions. +/// +/// Called with the lock held, so anything matching that is older than [`STALE_AFTER`] +/// belongs to a conversion that died. Giving every attempt a unique temporary name fixed +/// one bug and created this one: the pre-conversion `remove_file` then named a file that +/// cannot exist yet, so a killed conversion left its multi-gigabyte partial mzML beside +/// the input for ever and nothing anywhere deleted it (docs/31 F9). +fn sweep_stale_partials(out_dir: &Path, out: &Path) { + let prefix = partial_prefix(out); + let Ok(rd) = std::fs::read_dir(out_dir) else { + return; + }; + for e in rd.flatten() { + let name = e.file_name().to_string_lossy().into_owned(); + if !name.starts_with(&prefix) || !name.ends_with(".mzML") { + continue; + } + let path = e.path(); + if written_recently(&path) { + continue; + } + match std::fs::remove_file(&path) { + Ok(()) => warn!( + partial = %path.display(), + "convert: removed an abandoned partial conversion" + ), + Err(e) => warn!( + partial = %path.display(), + error = %e, + "convert: could not remove an abandoned partial conversion" + ), + } + } } pub fn ensure_mzml( @@ -627,6 +869,17 @@ msconvert was not usable either: {e}" format!("{stem}.{}.mzML", path_discriminator(src)) }; let out = out_dir.join(&out_name); + // One converter per destination at a time; a concurrent one waits and reuses. + let _lock = match ConvertLock::acquire(&out, src, cfg.reuse_converted)? { + Some(lock) => lock, + None => { + info!( + mzml = %out.display(), + "convert: reusing the mzML a concurrent conversion of this input just wrote" + ); + return Ok(out.to_string_lossy().into_owned()); + } + }; // Convert to a temporary name and rename only on success. // // Writing straight to `out` meant a killed run, a power loss or a converter crash @@ -634,8 +887,11 @@ msconvert was not usable either: {e}" // `reuse_converted` accepted it on every later run: a partial acquisition searched // silently, for ever, presenting as unexplained low identification counts with // nothing in the interface to reveal it. Neither failure path removed it. - let tmp = out_dir.join(partial_name(&out_name)); - let _ = std::fs::remove_file(&tmp); + // Under the lock, so every abandoned partial of THIS destination is dead by + // definition. The unique name below cannot collide with a live one. + sweep_stale_partials(&out_dir, &out); + let tmp_name = partial_name(&out_name, &unique_tag()); + let tmp = out_dir.join(&tmp_name); let args: Vec = if is_thermo { // `-f 2` is indexed mzML, which is what msconvert produces by default and so @@ -662,7 +918,7 @@ msconvert was not usable either: {e}" a.push("-o".into()); a.push(out_dir.to_string_lossy().into_owned()); a.push("--outfile".into()); - a.push(partial_name(&out_name)); + a.push(tmp_name.clone()); a }; @@ -822,13 +1078,160 @@ mod tests { fn the_partial_name_keeps_the_mzml_extension() { // The regression this guards: `x.mzML.partial` made ThermoRawFileParser write // `x.mzML.partial.mzML`, and the conversion was thrown away as "wrote no file". - assert_eq!(partial_name("run.mzML"), "run.partial.mzML"); + assert_eq!(partial_name("run.mzML", "7-0"), "run.partial-7-0.mzML"); assert_eq!( - partial_name("run.1a2b3c4d.mzML"), - "run.1a2b3c4d.partial.mzML" + partial_name("run.1a2b3c4d.mzML", "7-1"), + "run.1a2b3c4d.partial-7-1.mzML" + ); + assert!(partial_name("odd", "1-1").ends_with(".mzML")); + assert_ne!(partial_name("run.mzML", "1-1"), "run.mzML"); + // Two conversions in one process never share a temporary file (docs/30 R4). + assert_ne!(unique_tag(), unique_tag()); + } + + #[test] + fn a_conversion_lock_is_exclusive_and_released_on_drop() { + let d = tmp("lock"); + let out = d.join("run.mzML"); + let src = d.join("run.raw"); + std::fs::write(&src, b"raw").unwrap(); + let first = ConvertLock::acquire(&out, &src, true) + .unwrap() + .expect("first holder"); + assert!(ConvertLock::path_for(&out).is_file()); + // A second claim cannot be taken while the first is held; the probe below asks + // the primitive directly rather than waiting through `acquire`. + let taken = std::fs::OpenOptions::new() + .write(true) + .create_new(true) + .open(ConvertLock::path_for(&out)); + assert!(taken.is_err(), "the lock must be exclusive"); + drop(first); + assert!( + !ConvertLock::path_for(&out).exists(), + "dropping releases the lock" + ); + let second = ConvertLock::acquire(&out, &src, true).unwrap(); + assert!(second.is_some(), "free again once released"); + let _ = std::fs::remove_dir_all(&d); + } + + /// Backdate a file's modification time by `secs`. + fn age(path: &Path, secs: u64) { + let t = std::time::SystemTime::now() - std::time::Duration::from_secs(secs); + std::fs::OpenOptions::new() + .write(true) + .open(path) + .unwrap() + .set_modified(t) + .unwrap(); + } + + #[test] + fn a_stale_lock_is_broken_and_a_fresh_one_is_honoured() { + let d = tmp("stale"); + let out = d.join("run.mzML"); + let lock = d.join("run.mzML.converting"); + std::fs::write(&lock, b"1").unwrap(); + assert!( + !lock_is_stale(&lock, &out), + "a lock written a moment ago is live" + ); + age(&lock, 20 * 60); + assert!( + lock_is_stale(&lock, &out), + "an old lock with no live partial file is stale" + ); + // A partial file of THIS destination, still being written, keeps an old lock alive. + std::fs::write(d.join("run.partial-99-0.mzML"), b"...").unwrap(); + assert!(!lock_is_stale(&lock, &out)); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn another_destinations_partial_does_not_keep_this_lock_alive() { + // docs/31 F9: the probe accepted any `.partial-` name, so one live conversion in a + // shared directory kept a dead peer's lock fresh for ever. With + // `experiment.parallel_runs > 1` that is the ordinary case. + let d = tmp("peer"); + let out = d.join("dead.mzML"); + let lock = d.join("dead.mzML.converting"); + std::fs::write(&lock, b"1").unwrap(); + age(&lock, 20 * 60); + std::fs::write(d.join("alive.partial-1-0.mzML"), b"...").unwrap(); + assert!( + lock_is_stale(&lock, &out), + "a peer's live partial file must not vouch for this destination" + ); + std::fs::write(d.join("dead.partial-1-0.mzML"), b"...").unwrap(); + assert!(!lock_is_stale(&lock, &out), "its own partial file does"); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn a_future_modification_time_reads_as_fresh_not_stale() { + // Clock skew on a network share, not evidence that nobody is writing. + let d = tmp("skew"); + let f = d.join("run.mzML.converting"); + std::fs::write(&f, b"1").unwrap(); + let future = std::time::SystemTime::now() + std::time::Duration::from_secs(3600); + std::fs::OpenOptions::new() + .write(true) + .open(&f) + .unwrap() + .set_modified(future) + .unwrap(); + assert!(written_recently(&f)); + assert!(!lock_is_stale(&f, &d.join("run.mzML"))); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn abandoned_partials_are_swept_and_live_ones_are_not() { + let d = tmp("sweep"); + let out = d.join("run.mzML"); + let dead = d.join("run.partial-1-0.mzML"); + let live = d.join("run.partial-2-0.mzML"); + let peer = d.join("other.partial-3-0.mzML"); + let real = d.join("run.mzML"); + for f in [&dead, &live, &peer, &real] { + std::fs::write(f, b"x").unwrap(); + } + age(&dead, 20 * 60); + age(&peer, 20 * 60); + sweep_stale_partials(&d, &out); + assert!( + !dead.is_file(), + "an abandoned partial of this destination goes" + ); + assert!(live.is_file(), "a partial still being written stays"); + assert!( + peer.is_file(), + "another destination's partial is not ours to remove" + ); + assert!(real.is_file(), "the converted mzML itself is untouched"); + let _ = std::fs::remove_dir_all(&d); + } + + #[test] + fn a_lock_is_only_removed_by_the_holder_that_still_owns_it() { + // docs/31 F9: two waiters that both judged a lock stale each removed it and each + // created their own, so the second unlinked the first's and both believed they + // held it; the first's Drop then removed the second's. + let d = tmp("token"); + let out = d.join("run.mzML"); + std::fs::write(d.join("run.raw"), b"raw").unwrap(); + let first = ConvertLock::acquire(&out, &d.join("run.raw"), true) + .unwrap() + .expect("first holder"); + // Simulate a take-over: another process replaces the lock with its own token. + std::fs::write(ConvertLock::path_for(&out), b"9999:other").unwrap(); + drop(first); + assert!( + ConvertLock::path_for(&out).is_file(), + "dropping a lock we no longer own must not remove the new holder's" ); - assert!(partial_name("odd").ends_with(".mzML")); - assert_ne!(partial_name("run.mzML"), "run.mzML"); + let _ = std::fs::remove_dir_all(&d); } #[test] diff --git a/rust/mumdia/crates/mumdia/src/sidecar.rs b/rust/mumdia/crates/mumdia/src/sidecar.rs index 7f786189..da5420f3 100644 --- a/rust/mumdia/crates/mumdia/src/sidecar.rs +++ b/rust/mumdia/crates/mumdia/src/sidecar.rs @@ -8,7 +8,7 @@ use std::process::Command; use anyhow::{bail, Context, Result}; use mumdia_io::table::{write_table, Col, TableFile}; -use tracing::info; +use tracing::{info, warn}; /// Per candidate row: `(ion byte, ordinal, fragment charge)` -> linear predicted intensity. /// The charge is 1 for every series a single-charge MS2PIP model emits and 2 for the @@ -17,27 +17,43 @@ pub type FragmentIntensityMap = HashMap>; /// Resolve a sidecar worker script path so a deployed binary finds its workers /// regardless of the working directory: try the configured dir relative to the -/// CWD, then relative to the binary's own directory, then `/scripts`. -/// Falls back to the CWD-relative path (so the eventual error names it) if none -/// of those exist. +/// the binary's own directory, then `/scripts`, and the current working +/// directory LAST. Falls back to the directory-relative path (so the eventual error +/// names it) if none of those exist. +/// +/// The working directory used to be tried first. `python::resolve_script_dir` was +/// reordered away from exactly that and documents why at length: the shipped default is +/// the relative `"scripts"`, which both sidecar example configs carry, so unpacking a +/// dataset archive, `cd`-ing into it and running with an example config executed any +/// worker the archive happened to contain. That resolver only claims a directory holding +/// `mbr_worker.py` or `deeplc_worker.py`, so a directory with any of the other ten +/// workers reached this function still relative, and this function ran it (docs/31 F3). +/// An absolute directory is taken as given: naming one is how a user is unambiguous. pub fn resolve_script(dir: &str, worker: &str) -> String { - let cwd_rel = format!("{dir}/{worker}"); - if std::path::Path::new(&cwd_rel).exists() { - return cwd_rel; + let exe_dir = std::env::current_exe() + .ok() + .and_then(|e| e.parent().map(|p| p.to_path_buf())); + resolve_script_in(dir, worker, exe_dir.as_deref()) +} + +/// [`resolve_script`] with the executable's directory supplied, so the ordering can be +/// tested without a second binary. +fn resolve_script_in(dir: &str, worker: &str, exe_dir: Option<&std::path::Path>) -> String { + let dir_rel = format!("{dir}/{worker}"); + if std::path::Path::new(dir).is_absolute() { + return dir_rel; } - if let Ok(exe) = std::env::current_exe() { - if let Some(base) = exe.parent() { - for cand in [ - base.join(dir).join(worker), - base.join("scripts").join(worker), - ] { - if cand.exists() { - return cand.to_string_lossy().into_owned(); - } + if let Some(base) = exe_dir { + for cand in [ + base.join(dir).join(worker), + base.join("scripts").join(worker), + ] { + if cand.exists() { + return cand.to_string_lossy().into_owned(); } } } - cwd_rel + dir_rel } /// MS2PIP: predict singly-charged b/y intensities per (peptidoform, charge). @@ -88,13 +104,34 @@ pub fn run_ms2pip( } else { None }; + // Returned ids must be requested ones and each (ion, ordinal, charge) may appear once + // per id; coverage of the requested set is the caller's decision (docs/29 #17). + let requested: std::collections::HashSet = ids.iter().copied().collect(); let mut map: FragmentIntensityMap = HashMap::new(); for i in 0..t.nrows { + if !requested.contains(&oid[i]) { + bail!( + "MS2PIP worker returned id {}, which was not among the {} peptidoforms \ + requested", + oid[i], + ids.len() + ); + } let ib = ion[i].as_bytes().first().copied().unwrap_or(b'?'); let z = fch.as_ref().map(|c| c[i].clamp(1, 255) as u8).unwrap_or(1); - map.entry(oid[i]) + if map + .entry(oid[i]) .or_default() - .insert((ib, ord[i] as u16, z), inten[i]); + .insert((ib, ord[i] as u16, z), inten[i]) + .is_some() + { + bail!( + "MS2PIP worker returned fragment {}{} charge {z} of id {} more than once", + ion[i], + ord[i], + oid[i] + ); + } } Ok(map) } @@ -128,6 +165,40 @@ pub fn require_deeplc_version(python: &str) -> Result { } } +/// `deeplc--` when the interpreter answers, `deeplc-` when there +/// is none to ask: the manifest's RT identity should say which DeepLC release produced the +/// library, not only the recipe (docs/30, model identity). +pub fn deeplc_identity(python: Option<&str>, suffix: &str) -> String { + match python.and_then(|py| module_version(py, "deeplc")) { + Some(v) => format!("deeplc-{v}-{suffix}"), + None => format!("deeplc-{suffix}"), + } +} + +/// Read the `.summary.json` the fine-tune worker writes beside a rewritten +/// library and warn when rows kept their imported iRT, so a mixed RT source is visible +/// in the log rather than only in the file (docs/30 R6). +fn warn_on_retained_imported(lib_out: &str) { + let path = format!("{lib_out}.summary.json"); + let Ok(v) = mumdia_io::json::read_json::(&path) else { + return; + }; + let n = |k: &str| v.get(k).and_then(|x| x.as_u64()).unwrap_or(0); + let retained = n("retained_imported"); + if retained > 0 { + warn!( + rows = n("rows"), + repredicted = n("repredicted"), + retained_imported = retained, + retained_non_standard = n("retained_non_standard"), + retained_no_prediction = n("retained_no_prediction"), + summary = %path, + "sidecar: the rewritten library keeps the imported iRT on some rows, so its RT \ + source is mixed; see the summary for the counts" + ); + } +} + /// DeepLC: predict retention time per peptidoform. Returns `id -> predicted_rt`. pub fn run_deeplc( python: &str, @@ -155,7 +226,25 @@ pub fn run_deeplc( let t = TableFile::open(&outp)?; let oid = t.u32("id")?; let rt = t.f32("predicted_rt")?; - Ok(oid.into_iter().zip(rt).collect()) + // Returned ids must be a subset of the requested ones, each at most once. A repeated + // id used to overwrite silently and an unrequested one was kept; coverage (ids with + // no prediction) is the caller's to decide, and it drops those candidates rather + // than substituting a value (docs/29 #17). + let requested: std::collections::HashSet = ids.iter().copied().collect(); + let mut map: HashMap = HashMap::with_capacity(oid.len()); + for (id, value) in oid.into_iter().zip(rt) { + if !requested.contains(&id) { + bail!( + "DeepLC worker returned id {id}, which was not among the {} peptidoforms \ + requested", + ids.len() + ); + } + if map.insert(id, value).is_some() { + bail!("DeepLC worker returned id {id} more than once"); + } + } + Ok(map) } /// DeepLC multitask fine-tune: adapt the RT model to this run's confident seed @@ -222,7 +311,9 @@ pub fn run_deeplc_finetune( ], true, ) - .context("DeepLC fine-tune failed") + .context("DeepLC fine-tune failed")?; + warn_on_retained_imported(lib_out); + Ok(()) } /// DeepLC base-model re-prediction of an imported library's `predicted_irt`: the @@ -259,7 +350,9 @@ pub fn run_deeplc_repredict( ], true, ) - .context("DeepLC library re-prediction failed") + .context("DeepLC library re-prediction failed")?; + warn_on_retained_imported(lib_out); + Ok(()) } /// MBR transfer (Stage D3): match-between-runs identification transfer over the @@ -363,3 +456,67 @@ fn run_worker(python: &str, script: &str, args: &[&str], utf8: bool) -> Result<( } Ok(()) } + +#[cfg(test)] +mod resolve_tests { + use super::resolve_script_in; + use std::path::Path; + + fn scratch(name: &str) -> std::path::PathBuf { + let d = std::env::temp_dir().join(format!("mumdia_resolve_{}_{name}", std::process::id())); + let _ = std::fs::remove_dir_all(&d); + std::fs::create_dir_all(&d).unwrap(); + d + } + + #[test] + fn the_shipped_directory_beside_the_binary_wins_over_the_working_directory() { + // docs/31 F3. Both locations hold a worker of the same name; the one that ships + // with the binary must win, because the working directory can be an untrusted + // dataset the user merely unpacked and `cd`-ed into. + let exe = scratch("exe"); + std::fs::create_dir_all(exe.join("scripts")).unwrap(); + std::fs::write(exe.join("scripts").join("ms2pip_worker.py"), b"# shipped").unwrap(); + let got = resolve_script_in("scripts", "ms2pip_worker.py", Some(&exe)); + assert_eq!( + got, + exe.join("scripts") + .join("ms2pip_worker.py") + .to_string_lossy() + ); + assert_ne!(got, "scripts/ms2pip_worker.py"); + let _ = std::fs::remove_dir_all(&exe); + } + + #[test] + fn an_absolute_directory_is_taken_as_given() { + let abs = if cfg!(windows) { + "C:/opt/mumdia/scripts" + } else { + "/opt/mumdia/scripts" + }; + let exe = scratch("abs"); + std::fs::create_dir_all(exe.join("scripts")).unwrap(); + std::fs::write(exe.join("scripts").join("mbr_worker.py"), b"# shipped").unwrap(); + assert_eq!( + resolve_script_in(abs, "mbr_worker.py", Some(&exe)), + format!("{abs}/mbr_worker.py"), + "naming a directory outright is how a user is unambiguous" + ); + let _ = std::fs::remove_dir_all(&exe); + } + + #[test] + fn nothing_beside_the_binary_falls_back_to_the_relative_path_for_the_error() { + let exe = scratch("empty"); + assert_eq!( + resolve_script_in("scripts", "deeplc_worker.py", Some(&exe)), + "scripts/deeplc_worker.py" + ); + assert_eq!( + resolve_script_in("scripts", "deeplc_worker.py", None::<&Path>), + "scripts/deeplc_worker.py" + ); + let _ = std::fs::remove_dir_all(&exe); + } +} diff --git a/rust/mumdia/crates/mumdia/src/stages/audit.rs b/rust/mumdia/crates/mumdia/src/stages/audit.rs index 10b9e36b..c8aec13f 100644 --- a/rust/mumdia/crates/mumdia/src/stages/audit.rs +++ b/rust/mumdia/crates/mumdia/src/stages/audit.rs @@ -35,7 +35,9 @@ pub struct AuditParams<'a> { pub scored: &'a str, /// Output `candidate_audit.parquet`. pub out: &'a str, - /// Precursor q-value threshold for `passed_precursor_fdr` / `reported`. + /// Precursor q-value threshold for `passed_precursor_fdr` / `reported`, applied to + /// the scored table's `precursor_q` (the PSM `q_value` only on an older table + /// without that column; `.metrics.json` records which, as `q_unit`). pub q_threshold: f64, /// Run identifier stamped on every row. pub run_id: &'a str, @@ -62,6 +64,18 @@ fn load_extract_reasons(psms_path: &str) -> HashMap { pub fn run(p: AuditParams) -> Result { let t0 = Instant::now(); + // `--out` must not be one of this stage's own inputs: every input is read + // before the output is published, so writing over one replaces it and exits 0 + // (docs/31 F6). The shared guard existed and was wired into two stages. + mumdia_io::refuse_output_over_input( + p.out, + &[ + ("--lib-precursors", p.library_precursors), + ("--psms", p.psms), + ("--competed", p.competed), + ("--psms-scored", p.scored), + ], + )?; // Search space = all library precursors. let lib = TableFile::open(p.library_precursors) @@ -85,8 +99,41 @@ pub fn run(p: AuditParams) -> Result { .into_iter() .collect(); let scored_t = TableFile::open(p.scored)?; + // One run only. Every survivor set and q lookup here is keyed by candidate_id, so a + // pooled table (several `source` values) would overwrite one run's q with another's + // and attribute the last run's fate to every run (docs/29 #16). `run-experiment` + // writes per-run split tables; audit those. + if let Some(n_sources) = crate::stages::quant::pooled_source_count(&scored_t, p.scored)? { + if n_sources > 1 { + anyhow::bail!( + "audit: {} is a pooled scored table with {n_sources} sources; the audit is \ + per run and keys on candidate_id, so pass one run's split table \ + (//scored.parquet)", + p.scored + ); + } + } let scored_cid = scored_t.u32("candidate_id")?; - let scored_q = scored_t.f64("q_value")?; + // The gate is named `passed_precursor_fdr`, so it reads the precursor q. It read + // the PSM `q_value` (docs/29 #16), a different unit: a PSM can pass at 1% while + // its precursor group does not, and the other way round. Older scored tables have + // no `precursor_q`; there the PSM q is used and the metrics say so. + let (scored_q, q_unit) = if scored_t.has_column("precursor_q") { + // Present means present: a column of the wrong type is an error, not a reason to + // read another unit in its place (docs/30 R7, the absent-versus-malformed rule + // quant applies to `source`). + let v = scored_t + .f64("precursor_q") + .with_context(|| format!("audit: reading precursor_q from {}", p.scored))?; + (v, "precursor_q") + } else { + tracing::warn!( + scored = p.scored, + "audit: no `precursor_q` column; the precursor gate falls back to the PSM \ + q_value, which is not the same unit" + ); + (scored_t.f64("q_value")?, "q_value") + }; // peptide-level q is optional (only present in some scored schemas). let scored_pep_q = scored_t.f64("peptide_q_value").ok(); let mut q_by_cid: HashMap = HashMap::with_capacity(scored_cid.len()); @@ -135,15 +182,16 @@ pub fn run(p: AuditParams) -> Result { // Earliest rejection reason along the ladder. let reason: RejectionReason = if !traces { - // Extraction produced no accepted peak for this candidate. Refine with - // the in-extract audit sidecar if present; otherwise the generic bucket. + // Extraction produced no accepted row for this candidate. Refine with the + // in-extract audit sidecar if present; otherwise the generic bucket, which + // says only that the candidate did not survive extraction. match extract_reasons.get(&c).map(String::as_str) { Some("NO_FRAGMENT_TRACES") => RejectionReason::NoFragmentTraces, Some("NO_VALID_FRAGMENTS") => RejectionReason::NoValidFragments, Some("PEAK_NOT_SELECTED") => RejectionReason::PeakNotSelected, Some("RT_PRUNED") => RejectionReason::RtPruned, Some("WRONG_ISOLATION_WINDOW") => RejectionReason::WrongIsolationWindow, - _ => RejectionReason::NoPeakGroup, + _ => RejectionReason::DidNotSurviveExtraction, } } else if !variant { if is_decoy { @@ -155,6 +203,9 @@ pub fn run(p: AuditParams) -> Result { RejectionReason::FailedPrecursorFdr } else if !passed_pep { RejectionReason::FailedPeptideFdr + } else if is_decoy { + // Passed every gate, and the report never writes a decoy (docs/30 R7). + RejectionReason::RemovedDuringReporting } else { RejectionReason::Reported }; @@ -174,7 +225,11 @@ pub fn run(p: AuditParams) -> Result { f_td_winner.push(in_scored); f_prec_fdr.push(passed_prec); f_pep_fdr.push(passed_pep && passed_prec); - f_reported.push(passed_prec); + // One definition of "reported": the rejection reason. The flag used to repeat + // the precursor gate alone, so a row could read `reported = true` next to + // `FAILED_PEPTIDE_FDR`, and a decoy could be reported (docs/30 R7). The gate + // diagnostics keep their own columns above. + f_reported.push(reason == RejectionReason::Reported); reason_c.push(reason.code().to_string()); } @@ -207,6 +262,7 @@ pub fn run(p: AuditParams) -> Result { let metrics = json!({ "run_id": p.run_id, "q_threshold": p.q_threshold, + "q_unit": q_unit, "search_space": n, "extracted": n_extracted, "competed": n_competed, @@ -281,9 +337,9 @@ mod tests { // 1 target -> reported (extract+compete+scored q<=0.01) // 2 target -> failed precursor (scored q=0.5) // 3 target -> outcompeted (extract yes, compete no) - // 4 target -> no peak group (not extracted) + // 4 target -> did not survive extraction (not extracted) // 5 decoy -> outcompeted decoy (extract yes, compete no) - // 6 decoy -> no peak group (not extracted) + // 6 decoy -> did not survive extraction (not extracted) let lib = tmp("lib.parquet"); let psms = tmp("psms.parquet"); let comp = tmp("comp.parquet"); @@ -332,9 +388,181 @@ mod tests { assert_eq!(by[&2].0, "FAILED_PRECURSOR_FDR"); assert!(!by[&2].1); assert_eq!(by[&3].0, "OUTCOMPETED_BY_TARGET"); - assert_eq!(by[&4].0, "NO_PEAK_GROUP"); + assert_eq!(by[&4].0, "DID_NOT_SURVIVE_EXTRACTION"); assert_eq!(by[&5].0, "OUTCOMPETED_BY_DECOY"); - assert_eq!(by[&6].0, "NO_PEAK_GROUP"); + assert_eq!(by[&6].0, "DID_NOT_SURVIVE_EXTRACTION"); + } + + #[test] + fn the_precursor_gate_reads_precursor_q_not_psm_q() { + // Two candidates whose PSM and precursor q lie on opposite sides of 1%. The + // label `passed_precursor_fdr` has to follow `precursor_q` (docs/29 #16). + let lib = tmp("lib_q.parquet"); + let psms = tmp("psms_q.parquet"); + let comp = tmp("comp_q.parquet"); + let scored = tmp("scored_q.parquet"); + let out = tmp("audit_q.parquet"); + write_lib(&lib, &[1, 2], &["target", "target"]); + write_cid_only(&psms, &[1, 2]); + write_cid_only(&comp, &[1, 2]); + write_table( + &scored, + vec![ + Col::U32("candidate_id".into(), vec![1, 2]), + Col::F64("q_value".into(), vec![0.001, 0.5]), + Col::F64("precursor_q".into(), vec![0.5, 0.001]), + ], + ) + .unwrap(); + run(AuditParams { + library_precursors: &lib, + psms: &psms, + competed: &comp, + scored: &scored, + out: &out, + q_threshold: 0.01, + run_id: "t", + entrapment_substr: "", + }) + .unwrap(); + let a = TableFile::open(&out).unwrap(); + let cid = a.u32("precursor_id").unwrap(); + let reason = a.str("rejection_reason").unwrap(); + let passed = a.bool("passed_precursor_fdr").unwrap(); + let by: std::collections::HashMap = cid + .iter() + .cloned() + .zip(reason.into_iter().zip(passed)) + .collect(); + // PSM q 0.001 but precursor q 0.5: fails the precursor gate. + assert_eq!(by[&1], ("FAILED_PRECURSOR_FDR".to_string(), false)); + // PSM q 0.5 but precursor q 0.001: passes it. + assert_eq!(by[&2], ("REPORTED".to_string(), true)); + let m: serde_json::Value = + mumdia_io::json::read_json(&format!("{out}.metrics.json")).unwrap(); + assert_eq!(m["q_unit"], "precursor_q"); + } + + #[test] + fn the_reported_flag_follows_the_reason_and_the_report_rules() { + // docs/30 R7: a target passing the precursor gate but not the peptide gate, a + // target passing both, and a decoy passing both. Only the second is reported, + // the flag says so, and the metrics count the same row. + let lib = tmp("lib_rep.parquet"); + let psms = tmp("psms_rep.parquet"); + let comp = tmp("comp_rep.parquet"); + let scored = tmp("scored_rep.parquet"); + let out = tmp("audit_rep.parquet"); + write_lib(&lib, &[1, 2, 3], &["target", "target", "decoy"]); + write_cid_only(&psms, &[1, 2, 3]); + write_cid_only(&comp, &[1, 2, 3]); + write_table( + &scored, + vec![ + Col::U32("candidate_id".into(), vec![1, 2, 3]), + Col::F64("q_value".into(), vec![0.001, 0.001, 0.001]), + Col::F64("precursor_q".into(), vec![0.001, 0.001, 0.001]), + Col::F64("peptide_q_value".into(), vec![0.5, 0.001, 0.001]), + ], + ) + .unwrap(); + run(AuditParams { + library_precursors: &lib, + psms: &psms, + competed: &comp, + scored: &scored, + out: &out, + q_threshold: 0.01, + run_id: "t", + entrapment_substr: "", + }) + .unwrap(); + let a = TableFile::open(&out).unwrap(); + let cid = a.u32("precursor_id").unwrap(); + let reason = a.str("rejection_reason").unwrap(); + let reported = a.bool("reported").unwrap(); + let by: std::collections::HashMap = cid + .iter() + .cloned() + .zip(reason.into_iter().zip(reported)) + .collect(); + assert_eq!(by[&1], ("FAILED_PEPTIDE_FDR".to_string(), false)); + assert_eq!(by[&2], ("REPORTED".to_string(), true)); + assert_eq!(by[&3], ("REMOVED_DURING_REPORTING".to_string(), false)); + let m: serde_json::Value = + mumdia_io::json::read_json(&format!("{out}.metrics.json")).unwrap(); + assert_eq!(m["reported"], 1); + } + + #[test] + fn a_present_but_malformed_precursor_q_column_is_an_error_not_a_fallback() { + let lib = tmp("lib_bad.parquet"); + let psms = tmp("psms_bad.parquet"); + let comp = tmp("comp_bad.parquet"); + let scored = tmp("scored_bad.parquet"); + let out = tmp("audit_bad.parquet"); + write_lib(&lib, &[1], &["target"]); + write_cid_only(&psms, &[1]); + write_cid_only(&comp, &[1]); + write_table( + &scored, + vec![ + Col::U32("candidate_id".into(), vec![1]), + Col::F64("q_value".into(), vec![0.001]), + Col::I32("precursor_q".into(), vec![0]), + ], + ) + .unwrap(); + let e = run(AuditParams { + library_precursors: &lib, + psms: &psms, + competed: &comp, + scored: &scored, + out: &out, + q_threshold: 0.01, + run_id: "t", + entrapment_substr: "", + }) + .unwrap_err() + .to_string(); + assert!(e.contains("precursor_q"), "{e}"); + } + + #[test] + fn a_pooled_scored_table_is_refused() { + // Keyed by candidate_id alone, a two-source table would let the second run's q + // overwrite the first's. The experiment writes split tables; audit those. + let lib = tmp("lib_pool.parquet"); + let psms = tmp("psms_pool.parquet"); + let comp = tmp("comp_pool.parquet"); + let scored = tmp("scored_pool.parquet"); + let out = tmp("audit_pool.parquet"); + write_lib(&lib, &[1], &["target"]); + write_cid_only(&psms, &[1]); + write_cid_only(&comp, &[1]); + write_table( + &scored, + vec![ + Col::U32("candidate_id".into(), vec![1, 1]), + Col::U32("source".into(), vec![0, 1]), + Col::F64("q_value".into(), vec![0.001, 0.5]), + Col::F64("precursor_q".into(), vec![0.001, 0.5]), + ], + ) + .unwrap(); + let e = run(AuditParams { + library_precursors: &lib, + psms: &psms, + competed: &comp, + scored: &scored, + out: &out, + q_threshold: 0.01, + run_id: "t", + entrapment_substr: "", + }) + .unwrap_err() + .to_string(); + assert!(e.contains("pooled") && e.contains("2 sources"), "{e}"); } #[test] diff --git a/rust/mumdia/crates/mumdia/src/stages/compete.rs b/rust/mumdia/crates/mumdia/src/stages/compete.rs index 4c22d5f0..3441ccf4 100644 --- a/rust/mumdia/crates/mumdia/src/stages/compete.rs +++ b/rust/mumdia/crates/mumdia/src/stages/compete.rs @@ -32,6 +32,10 @@ pub struct CompeteParams<'a> { pub fn run(p: CompeteParams) -> Result { let t0 = Instant::now(); + // `--out` must not be one of this stage's own inputs: every input is read + // before the output is published, so writing over one replaces it and exits 0 + // (docs/31 F6). The shared guard existed and was wired into two stages. + mumdia_io::refuse_output_over_input(p.out, &[("--features", p.features)])?; // Footer-only open. The key columns below stream one at a time and the feature columns // (hundreds of them) are never materialised: the previous path read the whole features // table into Arrow and then copied every column into an owned Vec, so compete held two @@ -539,6 +543,44 @@ fn resolve_competition( mod tests { use super::*; + #[test] + fn writing_the_output_over_the_input_is_refused() { + // docs/31 F6: the features table is opened footer-only and streamed, so the read + // completes before `AtomicPath::publish` renames over it. Nothing errored: the + // widest artifact of the run, hundreds of columns and gigabytes on a real library, + // was replaced by the competed subset at exit 0, recoverable only by re-running + // extract and features. + let dir = std::env::temp_dir().join(format!("mumdia_compete_guard_{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let features = dir.join("features.parquet"); + mumdia_io::table::write_table( + features.to_str().unwrap(), + vec![mumdia_io::table::Col::U32( + "candidate_id".into(), + vec![1, 2], + )], + ) + .unwrap(); + let cfg = mumdia_core::config::CompeteConfig::default(); + let e = run(CompeteParams { + features: features.to_str().unwrap(), + out: features.to_str().unwrap(), + cfg: &cfg, + config_hash: "h", + }) + .unwrap_err() + .to_string(); + assert!(e.contains("its own input"), "{e}"); + assert!( + mumdia_io::table::TableFile::open(features.to_str().unwrap()) + .unwrap() + .nrows + == 2, + "the input must still be there" + ); + let _ = std::fs::remove_dir_all(&dir); + } + fn one_group(members: Vec) -> HashMap<(u32, u8, i64, i32), Vec> { let mut g = HashMap::new(); g.insert((0u32, 0u8, 0i64, 0i32), members); diff --git a/rust/mumdia/crates/mumdia/src/stages/digest.rs b/rust/mumdia/crates/mumdia/src/stages/digest.rs index 4ac3f69e..90768ee7 100644 --- a/rust/mumdia/crates/mumdia/src/stages/digest.rs +++ b/rust/mumdia/crates/mumdia/src/stages/digest.rs @@ -83,26 +83,29 @@ fn digest_protein(seq: &[u8], cfg: &DigestConfig) -> Vec<(usize, usize, String)> break; } let (start, end) = (sites[i], sites[j]); - let len = end - start; - if len < cfg.min_len || len > cfg.max_len { - continue; - } let sub = &seq[start..end]; - if !sub.iter().all(|&c| is_standard_residue(c)) { - continue; + let len = end - start; + // The ordinary form and the Met-excised form are judged independently. The + // ordinary form used to be rejected with `continue`, which also skipped the + // excision below, so an N-terminal peptide of `max_len + 1` residues produced + // nothing although its excised form was in range (docs/29 #9). + if len >= cfg.min_len + && len <= cfg.max_len + && sub.iter().all(|&c| is_standard_residue(c)) + { + out.push((start, end, String::from_utf8_lossy(sub).to_string())); } - out.push((start, end, String::from_utf8_lossy(sub).to_string())); // N-terminal methionine excision: for a peptide anchored at the // protein N-terminus whose first residue is the initiator Met, also // emit the Met-removed form (start shifted to 1). The excised peptide - // is re-checked against the length bounds and standard-residue rule. - // This mirrors DIA-NN's `--met-excision`; without it the search + // is checked against the length bounds and standard-residue rule on its + // own. This mirrors DIA-NN's `--met-excision`; without it the search // database cannot contain these (biologically dominant) peptides. if cfg.n_term_met_excision && start == 0 && seq.first() == Some(&b'M') { let ex = &seq[1..end]; let ex_len = ex.len(); - if ex_len >= cfg.min_len + if ex_len >= cfg.min_len.max(1) && ex_len <= cfg.max_len && ex.iter().all(|&c| is_standard_residue(c)) { @@ -436,6 +439,36 @@ mod tests { assert!(!peps.contains(&"DER".to_string()), "{peps:?}"); } + #[test] + fn met_excision_applies_when_only_the_excised_form_is_in_range() { + // MPEPTIDK is eight residues; with a length window of exactly seven the + // Met-retained form is too long and the excised PEPTIDK is the only valid + // peptide. The length rejection used to `continue` past the excision (docs/29 + // #9), so the digest returned nothing. + let peps = |min_len: usize, max_len: usize| -> Vec { + let cfg = DigestConfig { + missed_cleavages: 0, + min_len, + max_len, + n_term_met_excision: true, + ..Default::default() + }; + digest_protein(b"MPEPTIDK", &cfg) + .into_iter() + .map(|(_, _, p)| p) + .collect() + }; + assert_eq!(peps(7, 7), vec!["PEPTIDK".to_string()]); + // With room for both, both appear, Met-retained first. + assert_eq!( + peps(7, 8), + vec!["MPEPTIDK".to_string(), "PEPTIDK".to_string()] + ); + // The excised form is judged on its own length too: at min_len 8 only the + // ordinary form is in range. + assert_eq!(peps(8, 8), vec!["MPEPTIDK".to_string()]); + } + #[test] fn reverse_decoy_keeps_cterm() { let d = make_decoy("PEPTIDER", DecoyStrategy::Reverse, 0).unwrap(); diff --git a/rust/mumdia/crates/mumdia/src/stages/extract.rs b/rust/mumdia/crates/mumdia/src/stages/extract.rs index f57bdef4..abef7bd5 100644 --- a/rust/mumdia/crates/mumdia/src/stages/extract.rs +++ b/rust/mumdia/crates/mumdia/src/stages/extract.rs @@ -1430,6 +1430,14 @@ fn extract_twopass_windows( pub fn run(p: ExtractParams) -> Result<(u64, u64)> { let t0 = Instant::now(); + // Neither output may be one of the inputs (docs/31 F6). + let inputs = [ + ("--ms2", p.ms2), + ("--lib-precursors", p.library_precursors), + ("--lib-fragments", p.library_fragments), + ]; + mumdia_io::refuse_output_over_input(p.out_psms, &inputs)?; + mumdia_io::refuse_output_over_input(p.out_chrom, &inputs)?; // Skip the bucketed page_search index when the fragindex backend is selected (the // default): it is never read on that path and costs a full sort plus several full // copies of every library fragment. diff --git a/rust/mumdia/crates/mumdia/src/stages/features.rs b/rust/mumdia/crates/mumdia/src/stages/features.rs index 40f0d2e0..7a678ed8 100644 --- a/rust/mumdia/crates/mumdia/src/stages/features.rs +++ b/rust/mumdia/crates/mumdia/src/stages/features.rs @@ -1217,6 +1217,15 @@ fn confident_global_bounds( } pub fn run(p: FeaturesParams) -> Result { + // Neither output may be one of the inputs (docs/31 F6). + let mut inputs = vec![("--psms", p.psms), ("--chromatograms", p.chromatograms)]; + if let Some(seed) = p.seed { + inputs.push(("--seed-psms", seed)); + } + mumdia_io::refuse_output_over_input(p.out, &inputs)?; + if !p.out_pin.is_empty() { + mumdia_io::refuse_output_over_input(p.out_pin, &inputs)?; + } run_with_chunk_rows(p, CHUNK_CHROM_ROWS) } diff --git a/rust/mumdia/crates/mumdia/src/stages/predict_frag.rs b/rust/mumdia/crates/mumdia/src/stages/predict_frag.rs index 431f2b16..1ec1253d 100644 --- a/rust/mumdia/crates/mumdia/src/stages/predict_frag.rs +++ b/rust/mumdia/crates/mumdia/src/stages/predict_frag.rs @@ -129,10 +129,63 @@ pub fn run(p: PredictFragParams) -> Result<(u64, u64)> { "predict-frag: parsed" ); - let rt_model_id = assign_rt(&p, &mut raws)?; - let frag_model_id = assign_intensities(&p, &mut raws)?; + let (rt_model_id, rt_missing) = assign_rt(&p, &mut raws)?; + let (frag_model_id, frag_missing) = assign_intensities(&p, &mut raws)?; let model_identity = format!("{rt_model_id}; {frag_model_id}"); + // Coverage. A candidate a predictor returned nothing for is dropped, together with + // every candidate sharing its pair key (base peptide, charge, modification set), so + // a target and its paired decoy leave the library together and the exchangeability + // the FDR rests on is kept. Substituting a value was the previous behaviour, iRT 0.0 + // for DeepLC and the native heuristic for MS2PIP, and it produced a plausible, finite + // library whose rows came from an unrecorded mixture of predictors (docs/29 #17). + let n_before_drop = raws.len() as u64; + let (n_dropped_rows, n_dropped_pairs) = drop_unpredicted(&mut raws, &rt_missing, &frag_missing); + // The rows the predictors actually missed, and the rows that went with them for + // sharing a pair key. Reported separately because the second number is the cost of + // the position-free key (see `rows_to_drop`) and the first is the sidecar's own miss + // rate; one warning conflating them hid a worker returning nothing for a large batch. + let n_direct = { + let mut d: Vec = rt_missing + .iter() + .chain(frag_missing.iter()) + .copied() + .collect(); + d.sort_unstable(); + d.dedup(); + d.len() as u64 + }; + let n_collateral = n_dropped_rows.saturating_sub(n_direct); + if n_dropped_rows > 0 { + tracing::warn!( + candidates_dropped = n_dropped_rows, + unpredicted = n_direct, + dropped_with_their_pair = n_collateral, + pairs_dropped = n_dropped_pairs, + without_irt = rt_missing.len(), + without_intensities = frag_missing.len(), + "predict-frag: candidates without a prediction were dropped with their pairs \ + instead of receiving a substitute value; the counts are in the library report" + ); + } + if let Some(frac) = dropped_fraction_exceeded(n_dropped_rows, n_before_drop) { + bail!( + "predict-frag: {n_dropped_rows} of {n_before_drop} candidates ({:.2}%) have no \ + prediction ({n_direct} the predictor missed, {n_collateral} dropped with their \ + pair) and would silently leave the library. That is a failing sidecar rather \ + than a few unsupported peptidoforms; the ceiling is {:.0}%. Check the worker's \ + output above, then rerun.", + frac * 100.0, + MAX_DROPPED_FRACTION * 100.0 + ); + } + if raws.is_empty() && (n_dropped_rows > 0 || n_parse_err > 0) { + bail!( + "predict-frag: no candidate received a prediction ({n_dropped_rows} dropped for \ + missing predictions); the predictor sidecar produced nothing usable" + ); + } + // Finite guard at the prediction sidecar boundary. A NaN/Inf predicted iRT or // fragment intensity from a misbehaving MS2PIP/DeepLC run would silently // corrupt the library and every downstream spectral-similarity feature (and @@ -271,6 +324,18 @@ pub fn run(p: PredictFragParams) -> Result<(u64, u64)> { stats.insert("candidates".to_string(), json!(n_prec)); stats.insert("fragments".to_string(), json!(n_frag)); stats.insert("parse_errors".to_string(), json!(n_parse_err)); + stats.insert( + "candidates_dropped_with_their_pair".to_string(), + json!(n_collateral), + ); + stats.insert( + "candidates_dropped_unpredicted".to_string(), + json!(n_dropped_rows), + ); + stats.insert( + "pairs_dropped_unpredicted".to_string(), + json!(n_dropped_pairs), + ); for (path, schema) in [ (p.out_precursors, artifact::FRAGMENT_LIBRARY_PRECURSORS), (p.out_fragments, artifact::FRAGMENT_LIBRARY_FRAGMENTS), @@ -305,8 +370,9 @@ pub fn run(p: PredictFragParams) -> Result<(u64, u64)> { Ok((n_prec, n_frag)) } -/// Assign predicted iRT to every candidate. Returns the model id. -fn assign_rt(p: &PredictFragParams, raws: &mut [Raw]) -> Result { +/// Assign predicted iRT to every candidate. Returns the model id and the indices of the +/// candidates the predictor returned nothing for; the caller drops those. +fn assign_rt(p: &PredictFragParams, raws: &mut [Raw]) -> Result<(String, Vec)> { match p.cfg.rt_predictor { RtPredictorKind::Native => { let m = NativeRt; @@ -314,7 +380,7 @@ fn assign_rt(p: &PredictFragParams, raws: &mut [Raw]) -> Result { raws.par_iter_mut().for_each(|r| { r.irt = m.predict_irt(&r.parsed); }); - Ok(m.identity()) + Ok((m.identity(), Vec::new())) } RtPredictorKind::Deeplc => { let python = p.cfg.deeplc_python.as_deref().ok_or_else(|| { @@ -334,30 +400,25 @@ fn assign_rt(p: &PredictFragParams, raws: &mut [Raw]) -> Result { } } let out = sidecar::run_deeplc(python, &script, p.work_dir, &ids, &peps)?; - let mut n_irt_missing = 0u64; - for r in raws.iter_mut() { + let mut missing = Vec::new(); + for (i, r) in raws.iter_mut().enumerate() { let uid = uniq[&r.peptidoform]; match out.get(&uid) { Some(&v) => r.irt = v, - None => { - r.irt = 0.0; - n_irt_missing += 1; - } + None => missing.push(i), } } - if n_irt_missing > 0 { - tracing::warn!( - n_irt_missing, - "predict-frag: DeepLC returned no iRT for some peptidoforms; anchored at iRT 0.0" - ); - } - Ok("deeplc-4.0-mt".to_string()) + // The installed DeepLC version, not a family label: two libraries predicted by + // different DeepLC releases are different libraries (docs/30, model identity). + let version = sidecar::require_deeplc_version(python)?; + Ok((format!("deeplc-{version}-base"), missing)) } } } -/// Assign a predicted intensity to every fragment. Returns the model id. -fn assign_intensities(p: &PredictFragParams, raws: &mut [Raw]) -> Result { +/// Assign a predicted intensity to every fragment. Returns the model id and the indices +/// of the candidates the predictor returned nothing for; the caller drops those. +fn assign_intensities(p: &PredictFragParams, raws: &mut [Raw]) -> Result<(String, Vec)> { match p.cfg.predictor { FragPredictorKind::Native => { let m = NativeFrag; @@ -367,7 +428,7 @@ fn assign_intensities(p: &PredictFragParams, raws: &mut [Raw]) -> Result raws.par_iter_mut().for_each(|r| { r.frag_int = m.predict_intensities(&r.parsed, &r.frags); }); - Ok(m.identity()) + Ok((m.identity(), Vec::new())) } FragPredictorKind::Ms2pip => { let python = p.cfg.ms2pip_python.as_deref().ok_or_else(|| { @@ -398,34 +459,147 @@ fn assign_intensities(p: &PredictFragParams, raws: &mut [Raw]) -> Result // self-contained, so this is bit-identical to the serial loop. The MS2PIP sidecar // call already happened above -- what is parallelized here is the per-row native // prediction and normalization, which is real CPU work, not sidecar wait. - raws.par_iter_mut().enumerate().for_each(|(i, r)| { - let per = map.get(&(i as u32)); - match per { - Some(per) if !per.is_empty() => { - let keys: Vec<(u8, u16, u8)> = r - .frags - .iter() - .map(|fr| { - ( - fr.ion_type.symbol() as u8, - fr.ordinal as u16, - fr.charge.clamp(1, 255) as u8, - ) - }) - .collect(); - let nat = native.predict_intensities(&r.parsed, &r.frags); - r.frag_int = ms2pip_values(&keys, per, &nat); + // + // A candidate MS2PIP returned nothing for is reported to the caller, which + // drops it with its pair; it used to receive the native heuristic silently, + // under a library-wide MS2PIP model identity (docs/29 #17). + let covered: Vec = raws + .par_iter_mut() + .enumerate() + .map(|(i, r)| { + let per = map.get(&(i as u32)); + match per { + Some(per) if !per.is_empty() => { + let keys: Vec<(u8, u16, u8)> = r + .frags + .iter() + .map(|fr| { + ( + fr.ion_type.symbol() as u8, + fr.ordinal as u16, + fr.charge.clamp(1, 255) as u8, + ) + }) + .collect(); + let nat = native.predict_intensities(&r.parsed, &r.frags); + r.frag_int = ms2pip_values(&keys, per, &nat); + true + } + _ => { + r.frag_int = vec![0.0; r.frags.len()]; + false + } } - _ => { - r.frag_int = native.predict_intensities(&r.parsed, &r.frags); - } - } - }); - Ok(format!("ms2pip-{}", p.cfg.ms2pip_model)) + }) + .collect(); + let missing: Vec = covered + .iter() + .enumerate() + .filter(|(_, &c)| !c) + .map(|(i, _)| i) + .collect(); + let version = + sidecar::module_version(python, "ms2pip").unwrap_or_else(|| "unknown".into()); + Ok((format!("ms2pip-{version}-{}", p.cfg.ms2pip_model), missing)) } } } +/// The modification content of a ProForma peptidoform, order-free: every bracketed +/// token, sorted and joined. Two peptidoforms with the same base peptide, charge and +/// signature are the same precursor on the target and decoy side. +fn mod_signature(peptidoform: &str) -> String { + let mut mods: Vec<&str> = Vec::new(); + let mut rest = peptidoform; + while let Some(open) = rest.find('[') { + match rest[open..].find(']') { + Some(close) => { + mods.push(&rest[open + 1..open + close]); + rest = &rest[open + close + 1..]; + } + None => break, + } + } + mods.sort_unstable(); + mods.join(",") +} + +/// The key under which a target and its paired decoy are one precursor. +fn pair_key(base_peptide_id: u32, charge: i32, peptidoform: &str) -> (u32, i32, String) { + (base_peptide_id, charge, mod_signature(peptidoform)) +} + +/// The largest fraction of the library that may be dropped for want of a prediction +/// before the build is treated as a failed sidecar rather than a few unsupported +/// peptidoforms. +/// +/// The only previous guard fired when the library was emptied entirely, so a worker that +/// returned nothing for 5% of a 9.8M-peptidoform batch removed those rows, plus every +/// positional isomer sharing their signature, behind one warning (docs/31 F10). +const MAX_DROPPED_FRACTION: f64 = 0.02; + +/// The dropped fraction when it exceeds [`MAX_DROPPED_FRACTION`], else `None`. +fn dropped_fraction_exceeded(dropped: u64, total: u64) -> Option { + if total == 0 || dropped == 0 { + return None; + } + let frac = dropped as f64 / total as f64; + (frac > MAX_DROPPED_FRACTION).then_some(frac) +} + +/// Which rows to drop so that every row sharing a pair key with an unpredicted row goes +/// with it. `keys[i]` is the pair key of row `i`; `missing` lists unpredicted rows. +/// +/// The key is deliberately position-free, and that over-deletes: `M[Oxidation]PEPTIDEMK` +/// and `MPEPTIDEM[Oxidation]K` share a base peptide, a charge and a modification +/// multiset, so a miss on either removes both. Making the key position-aware would be +/// worse, not better: a reverse decoy carries its modifications at mirrored positions, so +/// a positional key would stop matching a target to its decoy and a target could be +/// dropped while its decoy stayed, which is an FDR defect rather than a sensitivity one. +/// The collateral is counted separately and bounded by [`MAX_DROPPED_FRACTION`] instead. +fn rows_to_drop(keys: &[(u32, i32, String)], missing: &[usize]) -> Vec { + let doomed: std::collections::HashSet<&(u32, i32, String)> = + missing.iter().map(|&i| &keys[i]).collect(); + keys.iter().map(|k| doomed.contains(k)).collect() +} + +/// Drop every candidate without a prediction together with its pair. Returns the +/// number of rows and of distinct pair keys removed. +fn drop_unpredicted( + raws: &mut Vec, + rt_missing: &[usize], + frag_missing: &[usize], +) -> (u64, u64) { + if rt_missing.is_empty() && frag_missing.is_empty() { + return (0, 0); + } + let keys: Vec<(u32, i32, String)> = raws + .iter() + .map(|r| pair_key(r.base_peptide_id, r.charge, &r.peptidoform)) + .collect(); + let mut missing: Vec = rt_missing + .iter() + .chain(frag_missing.iter()) + .copied() + .collect(); + missing.sort_unstable(); + missing.dedup(); + let drop = rows_to_drop(&keys, &missing); + let n_pairs = missing + .iter() + .map(|&i| &keys[i]) + .collect::>() + .len() as u64; + let before = raws.len(); + let mut i = 0usize; + raws.retain(|_| { + let keep = !drop[i]; + i += 1; + keep + }); + ((before - raws.len()) as u64, n_pairs) +} + /// One candidate's fragment intensities from its MS2PIP predictions. /// /// `keys` are the candidate's fragments as `(ion byte, ordinal, charge)`, `per` the @@ -538,9 +712,40 @@ fn fragment_cardinality(cid: &[u32], mz: &[f64]) -> Vec { #[cfg(test)] mod cardinality_tests { - use super::{fragment_cardinality, ms2pip_values}; + use super::{ + dropped_fraction_exceeded, fragment_cardinality, mod_signature, ms2pip_values, pair_key, + rows_to_drop, MAX_DROPPED_FRACTION, + }; use std::collections::HashMap; + #[test] + fn unpredicted_candidates_are_dropped_with_their_pairs() { + // Pair key = (base peptide, charge, modification set): the target PEPTIDEK/2 and + // its reversed decoy share base 7 and charge 2 with no modifications; the oxidised + // form is a different precursor and stays. + assert_eq!(mod_signature("PEPTM[Oxidation]IDEK"), "Oxidation"); + assert_eq!( + mod_signature("C[Carbamidomethyl]M[Oxidation]C[Carbamidomethyl]K"), + "Carbamidomethyl,Carbamidomethyl,Oxidation" + ); + assert_eq!(mod_signature("[Acetyl]-PEPTIDEK"), "Acetyl"); + assert_eq!(mod_signature("PEPTIDEK"), ""); + let keys = vec![ + pair_key(7, 2, "PEPTIDEK"), + pair_key(7, 2, "KEDITPEP"), + pair_key(7, 2, "PEPTM[Oxidation]IDEK"), + pair_key(7, 3, "PEPTIDEK"), + pair_key(8, 2, "SAMPLER"), + ]; + // Only the target PEPTIDEK/2 lacked a prediction: it and its decoy go, the + // oxidised form, the charge-3 form and the other peptide stay. + assert_eq!( + rows_to_drop(&keys, &[0]), + vec![true, true, false, false, false] + ); + assert_eq!(rows_to_drop(&keys, &[]), vec![false; 5]); + } + #[test] fn ms2pip_values_keep_the_two_group_regime_for_single_charge_models() { // b2(1), y3(1), y3(2), y4(2): the model emitted charge 1 only. @@ -559,6 +764,42 @@ mod cardinality_tests { assert_eq!(v2, vec![1.0, 0.0]); } + #[test] + fn a_large_unpredicted_fraction_is_a_failure_not_a_warning() { + // docs/31 F10: the only previous guard fired when the library was emptied, so a + // worker returning nothing for a large batch removed those rows plus every + // positional isomer sharing their signature behind one warning. + assert_eq!(dropped_fraction_exceeded(0, 1000), None); + assert_eq!(dropped_fraction_exceeded(0, 0), None); + assert_eq!( + dropped_fraction_exceeded(5, 1000), + None, + "a few misses are fine" + ); + let frac = dropped_fraction_exceeded(500_000, 9_800_000).expect("5% must be refused"); + assert!(frac > MAX_DROPPED_FRACTION); + // The boundary itself is accepted; only exceeding it is refused. + assert_eq!(dropped_fraction_exceeded(20, 1000), None); + assert!(dropped_fraction_exceeded(21, 1000).is_some()); + } + + #[test] + fn a_positional_isomer_still_shares_its_pair_key_by_design() { + // Deliberate over-deletion, documented on `rows_to_drop`: a positional key would + // stop matching a reverse decoy to its target, which is an FDR defect. The test + // pins the trade so a future change has to argue with it. + let a = pair_key(7, 2, "M[Oxidation]PEPTIDEMK"); + let b = pair_key(7, 2, "MPEPTIDEM[Oxidation]K"); + assert_eq!(a, b); + let keys = vec![a.clone(), b, pair_key(7, 3, "M[Oxidation]PEPTIDEMK")]; + let drop = rows_to_drop(&keys, &[0]); + assert_eq!( + drop, + vec![true, true, false], + "the other charge is a different precursor" + ); + } + #[test] fn ms2pip_values_use_one_scale_when_the_model_emitted_charge_2() { let keys = [(b'b', 2, 1), (b'y', 3, 1), (b'y', 3, 2), (b'y', 4, 2)]; diff --git a/rust/mumdia/crates/mumdia/src/stages/prescan.rs b/rust/mumdia/crates/mumdia/src/stages/prescan.rs index b24e9e23..ae42387f 100644 --- a/rust/mumdia/crates/mumdia/src/stages/prescan.rs +++ b/rust/mumdia/crates/mumdia/src/stages/prescan.rs @@ -32,7 +32,7 @@ use mumdia_io::report::ArtifactReport; use mumdia_io::table::{write_table, Col, TableFile}; use rayon::prelude::*; use serde_json::json; -use tracing::info; +use tracing::{info, warn}; pub struct PrescanParams<'a> { /// `spectra_ms2` for this run. @@ -290,6 +290,12 @@ pub fn run(p: PrescanParams) -> Result { for (k, v) in per_spectrum { obs.entry(k).or_default().extend(v); } + // The full observed RT-bin range, for candidates whose RT window is unbounded (see + // the screening loop). Empty when no spectrum produced a tag, in which case nothing + // can survive on any path. + let (bin_min, bin_max) = obs.keys().fold((i64::MAX, i64::MIN), |(lo, hi), &(_, b)| { + (lo.min(b), hi.max(b)) + }); info!( cells = obs.len(), spectra = ms2.nrows, @@ -330,6 +336,8 @@ pub fn run(p: PrescanParams) -> Result { let slack = p.cfg.rt_slack_s; let t1 = Instant::now(); + // Candidates screened over the whole gradient because their RT window is unbounded. + let unbounded = std::sync::atomic::AtomicU64::new(0); // Screening is independent per candidate, which is what makes this worth doing in Rust: the // equivalent Python loop was single-threaded and ~40% of the per-file wall clock. let mut surv: Vec<(u32, &str)> = (0..lib.nrows) @@ -345,12 +353,26 @@ pub fn run(p: PrescanParams) -> Result { return None; } let c = cid[i] as usize; - let (lo, hi) = (*lo_by.get(c)?, *hi_by.get(c)?); - if !lo.is_finite() || !hi.is_finite() { - return None; - } - let b0 = ((lo - slack) / bin).floor() as i64; - let b1 = ((hi + slack) / bin).floor() as i64; + // An unbounded RT window is "search the whole gradient", not "cannot be + // screened". `rt_im_train::candidate_window` writes (NaN, -inf, +inf) whenever + // calibration is unavailable and documents the infinite bounds as recall-safe; + // this guard used to read them as a screening failure and drop the candidate, + // so a run with no confident seeds -- the documented FASTA/MS2PIP failure -- + // discarded the ENTIRE library and exited 0 (docs/31 F1). A candidate with no + // run_windows row at all is the same case: unknown RT, not absent evidence. + let (lo, hi) = match (lo_by.get(c), hi_by.get(c)) { + (Some(&l), Some(&h)) => (l, h), + _ => (f64::NEG_INFINITY, f64::INFINITY), + }; + let (b0, b1) = if lo.is_finite() && hi.is_finite() { + ( + ((lo - slack) / bin).floor() as i64, + ((hi + slack) / bin).floor() as i64, + ) + } else { + unbounded.fetch_add(1, std::sync::atomic::Ordering::Relaxed); + (bin_min, bin_max) + }; let m = pmz[i]; for w in 0..win.nrows { if !(w_lo[w] <= m && m < w_hi[w]) { @@ -378,15 +400,39 @@ pub fn run(p: PrescanParams) -> Result { } else { f64::NAN }; + let n_unbounded = unbounded.load(std::sync::atomic::Ordering::Relaxed); info!( screened = lib.nrows, survivors = surv.len(), targets = n_t, decoys = n_d, target_decoy_ratio = ratio, + rt_unbounded = n_unbounded, elapsed_ms = t1.elapsed().as_millis() as u64, "prescan: screened candidates" ); + if n_unbounded > 0 { + warn!( + candidates = n_unbounded, + of = lib.nrows, + "prescan: these candidates have no bounded RT window (no calibration, or no \ + run_windows row) and were screened over the whole gradient, which is the \ + recall-safe reading of the sentinel but a weaker screen" + ); + } + // Screening everything away is not a result. Before docs/31 F1 this was the silent + // outcome of an uncalibrated run: every candidate dropped, a zero-row survivors table, + // exit 0. The single-label bail below cannot see it, because it is gated on a + // survivor count. + if lib.nrows > 0 && surv.is_empty() { + anyhow::bail!( + "prescan screened {} candidates and none survived; a search would have no \ + candidates at all. Check that the observed tag index is not empty (spectra, \ + prescan.tol_da, prescan.top_peaks) and that prescan.anchor_mods matches the \ + modifications in this library", + lib.nrows + ); + } // A one-sided survival means the screen has become label-dependent and the modification's // null is gone. Fail loudly rather than emit a library whose modified q-values cannot be // estimated. diff --git a/rust/mumdia/crates/mumdia/src/stages/quant.rs b/rust/mumdia/crates/mumdia/src/stages/quant.rs index ca587e0e..c99823d3 100644 --- a/rust/mumdia/crates/mumdia/src/stages/quant.rs +++ b/rust/mumdia/crates/mumdia/src/stages/quant.rs @@ -454,6 +454,17 @@ fn rollup_protein_bases( pub fn run(p: QuantParams) -> Result<(u64, u64)> { let t0 = Instant::now(); + // No output may be one of the inputs (docs/31 F6). + let inputs = [ + ("--psms-scored", p.psms_scored), + ("--chromatograms", p.chromatograms), + ]; + for out in [Some(p.out_peptide), Some(p.out_protein), p.out_fragment] + .into_iter() + .flatten() + { + mumdia_io::refuse_output_over_input(out, &inputs)?; + } // Identified target PSMs below the peptide q threshold. let ps = TableFile::open(p.psms_scored)?; @@ -519,11 +530,7 @@ pub fn run(p: QuantParams) -> Result<(u64, u64)> { // `run-experiment` splits by `source` before calling quant, and the docs say to do // the same by hand, but nothing enforced it -- and the pooled table is precisely what // the recorded multi-run recipe produces. Refuse instead, naming the fix. - if let Ok(source) = ps.i32("source") { - let n_sources = source - .iter() - .collect::>() - .len(); + if let Some(n_sources) = pooled_source_count(&ps, p.psms_scored)? { if n_sources > 1 { anyhow::bail!( "quant: {} names a pooled scored table covering {n_sources} runs \ @@ -2193,3 +2200,91 @@ mod tests { assert_eq!(q, Some(20.0)); } } + +/// Distinct values of the scored table's `source` column, or `None` when the column is +/// absent (a single-run table from before pooled rescoring stamped it). +/// +/// `rescore` writes `source` as u32. The previous guard read it as i32 and treated the +/// resulting type error like an absent column, so the pooled-table refusal above never +/// ran on the engine's own output: a pooled table quantified against one run's +/// chromatograms produced one identical row per run without a word (docs/29 #1). A +/// present column that is neither u32 nor i32 is an error now, not a shrug. +pub(crate) fn pooled_source_count(ps: &TableFile, path: &str) -> Result> { + if !ps.has_column("source") { + return Ok(None); + } + let distinct = match ps.u32("source") { + Ok(v) => v + .into_iter() + .collect::>() + .len(), + Err(u32_err) => match ps.i32("source") { + Ok(v) => v + .into_iter() + .collect::>() + .len(), + Err(_) => anyhow::bail!( + "column `source` in {path} is present but neither u32 (what rescore writes) \ + nor i32: {u32_err:#}" + ), + }, + }; + Ok(Some(distinct)) +} + +#[cfg(test)] +mod source_guard_tests { + use super::*; + + fn table(name: &str, cols: Vec) -> String { + let dir = std::env::temp_dir().join(format!("mumdia_quant_source_{}", std::process::id())); + std::fs::create_dir_all(&dir).unwrap(); + let p = dir + .join(format!("{name}.parquet")) + .to_str() + .unwrap() + .to_string(); + write_table(&p, cols).unwrap(); + p + } + + #[test] + fn the_pooled_guard_reads_the_u32_source_rescore_writes() { + // The exact type rescore emits (`Col::U32("source", ...)`, rescore.rs): two runs. + let p = table( + "u32", + vec![ + Col::U32("candidate_id".into(), vec![0, 0]), + Col::U32("source".into(), vec![0, 1]), + ], + ); + let t = TableFile::open(&p).unwrap(); + assert_eq!(pooled_source_count(&t, &p).unwrap(), Some(2)); + + // The signed spelling still counts, an absent column is None, and a present + // column of the wrong type is an error rather than "absent". + let p = table( + "i32", + vec![ + Col::U32("candidate_id".into(), vec![0, 0]), + Col::I32("source".into(), vec![3, 3]), + ], + ); + let t = TableFile::open(&p).unwrap(); + assert_eq!(pooled_source_count(&t, &p).unwrap(), Some(1)); + + let p = table("none", vec![Col::U32("candidate_id".into(), vec![0, 1])]); + let t = TableFile::open(&p).unwrap(); + assert_eq!(pooled_source_count(&t, &p).unwrap(), None); + + let p = table( + "f64", + vec![ + Col::U32("candidate_id".into(), vec![0, 1]), + Col::F64("source".into(), vec![0.0, 1.0]), + ], + ); + let t = TableFile::open(&p).unwrap(); + assert!(pooled_source_count(&t, &p).is_err()); + } +} diff --git a/rust/mumdia/crates/mumdia/src/stages/report.rs b/rust/mumdia/crates/mumdia/src/stages/report.rs index b705abb1..1eff48ee 100644 --- a/rust/mumdia/crates/mumdia/src/stages/report.rs +++ b/rust/mumdia/crates/mumdia/src/stages/report.rs @@ -10,7 +10,7 @@ use std::collections::{HashMap, HashSet}; use std::io::Write; -use anyhow::Result; +use anyhow::{Context, Result}; use mumdia_io::table::TableFile; pub struct ReportParams<'a> { @@ -47,6 +47,41 @@ fn qcell(q: f64) -> String { } } +/// The match-between-runs acceptance basis of every row: the flag and the q it was +/// accepted at, or all-false and all-NaN when the table has never been through +/// `mumdia mbr`. +/// +/// Absent means absent; present means readable. A present column of the wrong type used +/// to be swallowed by an `Err(_) => vec![false; n]` arm, so a boolean written as a +/// nullable dtype or as int8 (which `mbr_worker.py` writes through pandas) silently +/// removed EVERY transfer from both TSVs at exit 0, indistinguishable from an MBR run +/// that transferred nothing while the parquet still showed them (docs/31 F2). This is the +/// absent-versus-malformed rule `quant` and `audit` already follow. +fn transfer_columns(t: &TableFile, n: usize) -> Result<(Vec, Vec)> { + let flag = if t.has_column("is_transferred") { + t.bool("is_transferred").with_context(|| { + "report: `is_transferred` is present but not a boolean column; it is written by \ + `mumdia mbr` and decides which rows are reported" + })? + } else { + vec![false; n] + }; + let q = if t.has_column("transfer_q") { + t.f64("transfer_q") + .with_context(|| "report: `transfer_q` is present but not a float column")? + } else { + vec![f64::NAN; n] + }; + if flag.len() != n || q.len() != n { + anyhow::bail!( + "report: the transfer columns have {} and {} rows against {n} scored rows", + flag.len(), + q.len() + ); + } + Ok((flag, q)) +} + /// Write peptides.tsv + proteins.tsv from a scored PSM table. Returns /// (n_peptides, n_protein_groups) at the FDR threshold. pub fn run(p: ReportParams) -> Result<(u64, u64)> { @@ -65,12 +100,27 @@ pub fn run(p: ReportParams) -> Result<(u64, u64)> { // this stage filters on, so without this a `mumdia mbr` followed by `mumdia report` // showed no transfers at all. Same contract as `quant`: a transfer has already // passed `mbr.q_transfer`, and a decoy is still never reported. - let is_transferred: Vec = match t.bool("is_transferred") { - Ok(v) => v, - Err(_) => vec![false; n], - }; + let (is_transferred, transfer_q) = transfer_columns(&t, n)?; let accepted = |i: usize, q: &[f64]| label[i] == "target" && (is_transferred[i] || q[i] <= p.q_threshold); + // The acceptance basis is exported with every row (`is_transferred`, `transfer_q`), + // because the rule above is otherwise invisible in the TSV: a transferred row keeps + // its grouped q, usually 1.0, and a tighter threshold does not revoke a transfer that + // already passed `mbr.q_transfer` (docs/29 #19). A transfer is a peptide-level + // acceptance and not protein-group confidence; a group admitted through one carries + // the flag so a reader can tell. + let transfer_cells = |i: usize| -> String { + if is_transferred[i] { + let q = if transfer_q[i].is_nan() { + String::new() + } else { + format!("{:.6}", transfer_q[i]) + }; + format!("\ttrue\t{q}") + } else { + "\tfalse\t".to_string() + } + }; let pep_quant: HashMap<(String, i32), f64> = match p.peptide_quant { Some(path) => { @@ -111,7 +161,8 @@ pub fn run(p: ReportParams) -> Result<(u64, u64)> { // sequence; the header and the returned count reflect that. writeln!( w, - "precursor\tstripped_sequence\tcharge\tprotein\tq_value\tscore\tquantity" + "precursor\tstripped_sequence\tcharge\tprotein\tq_value\tscore\tquantity\t\ + is_transferred\ttransfer_q" )?; let mut npep = 0u64; for &i in &order { @@ -126,14 +177,15 @@ pub fn run(p: ReportParams) -> Result<(u64, u64)> { let qv = pep_quant.get(&key).copied().unwrap_or(f64::NAN); writeln!( w, - "{}\t{}\t{}\t{}\t{:.6}\t{:.4}\t{}", + "{}\t{}\t{}\t{}\t{:.6}\t{:.4}\t{}{}", pform[i], strip(&pform[i]), charge[i], protein[i], pep_q[i], score[i], - qcell(qv) + qcell(qv), + transfer_cells(i) )?; npep += 1; } @@ -147,7 +199,10 @@ pub fn run(p: ReportParams) -> Result<(u64, u64)> { let mut pseen: HashSet = HashSet::new(); let prot_target = mumdia_io::table::AtomicPath::new(p.out_proteins)?; let mut w2 = std::io::BufWriter::new(std::fs::File::create(prot_target.tmp())?); - writeln!(w2, "protein_group\tq_value\tquantity")?; + writeln!( + w2, + "protein_group\tq_value\tquantity\tis_transferred\ttransfer_q" + )?; let mut nprot = 0u64; for &i in &porder { if !accepted(i, &pg_q) || pg[i].is_empty() { @@ -157,7 +212,14 @@ pub fn run(p: ReportParams) -> Result<(u64, u64)> { continue; } let qv = prot_quant.get(&pg[i]).copied().unwrap_or(f64::NAN); - writeln!(w2, "{}\t{:.6}\t{}", pg[i], pg_q[i], qcell(qv))?; + writeln!( + w2, + "{}\t{:.6}\t{}{}", + pg[i], + pg_q[i], + qcell(qv), + transfer_cells(i) + )?; nprot += 1; } w2.flush()?; @@ -223,12 +285,27 @@ pub fn run_experiment(p: ExperimentReportParams) -> Result<(u64, u64)> { let source = t.u32("source")?; let run_q = t.f64("run_psm_q")?; let n = t.nrows; - let is_transferred: Vec = match t.bool("is_transferred") { - Ok(v) => v, - Err(_) => vec![false; n], - }; + let (is_transferred, transfer_q) = transfer_columns(&t, n)?; let accepted = |i: usize, q: &[f64]| label[i] == "target" && (is_transferred[i] || q[i] <= p.q_threshold); + // The acceptance basis is exported with every row (`is_transferred`, `transfer_q`), + // because the rule above is otherwise invisible in the TSV: a transferred row keeps + // its grouped q, usually 1.0, and a tighter threshold does not revoke a transfer that + // already passed `mbr.q_transfer` (docs/29 #19). A transfer is a peptide-level + // acceptance and not protein-group confidence; a group admitted through one carries + // the flag so a reader can tell. + let transfer_cells = |i: usize| -> String { + if is_transferred[i] { + let q = if transfer_q[i].is_nan() { + String::new() + } else { + format!("{:.6}", transfer_q[i]) + }; + format!("\ttrue\t{q}") + } else { + "\tfalse\t".to_string() + } + }; // Runs in which each precursor / protein group was identified on its own per-run FDR. let mut pep_runs: HashMap<(String, i32), Vec> = HashMap::new(); @@ -305,6 +382,7 @@ pub fn run_experiment(p: ExperimentReportParams) -> Result<(u64, u64)> { write!(w, "\tquantity_{name}")?; } } + write!(w, "\tis_transferred\ttransfer_q")?; writeln!(w)?; let mut npep = 0u64; for &i in &order { @@ -330,6 +408,7 @@ pub fn run_experiment(p: ExperimentReportParams) -> Result<(u64, u64)> { for m in &pep_quant { write!(w, "\t{}", qcell(m.get(&key).copied().unwrap_or(f64::NAN)))?; } + write!(w, "{}", transfer_cells(i))?; writeln!(w)?; npep += 1; } @@ -349,6 +428,7 @@ pub fn run_experiment(p: ExperimentReportParams) -> Result<(u64, u64)> { write!(w2, "\tlfq_{name}")?; } } + write!(w2, "\tis_transferred\ttransfer_q")?; writeln!(w2)?; let mut nprot = 0u64; for &i in &porder { @@ -371,6 +451,7 @@ pub fn run_experiment(p: ExperimentReportParams) -> Result<(u64, u64)> { write!(w2, "\t{}", qcell(*x))?; } } + write!(w2, "{}", transfer_cells(i))?; writeln!(w2)?; nprot += 1; } @@ -510,7 +591,7 @@ mod tests { let lines: Vec<&str> = text.lines().collect(); assert_eq!( lines[0], - "precursor\tstripped_sequence\tcharge\tprotein\tq_value\tscore\tn_runs\tquantity_a\tquantity_b" + "precursor\tstripped_sequence\tcharge\tprotein\tq_value\tscore\tn_runs\tquantity_a\tquantity_b\tis_transferred\ttransfer_q" ); assert_eq!(lines.len(), 3, "header plus two precursors:\n{text}"); assert!( @@ -531,7 +612,10 @@ mod tests { let prot = std::fs::read_to_string(&proteins).unwrap(); let plines: Vec<&str> = prot.lines().collect(); - assert_eq!(plines[0], "protein_group\tq_value\tn_runs\tlfq_a\tlfq_b"); + assert_eq!( + plines[0], + "protein_group\tq_value\tn_runs\tlfq_a\tlfq_b\tis_transferred\ttransfer_q" + ); assert_eq!( plines.len(), 2, @@ -642,11 +726,13 @@ mod tests { "stripped sequence missing:\n{text}" ); // Quantity is empty when no quant table was supplied, not zero: absence of - // a measurement is not a measurement of zero. + // a measurement is not a measurement of zero. Nothing here is a transfer. let data_line = text.lines().nth(1).unwrap(); - assert!( - data_line.ends_with('\t'), - "expected an empty quantity cell: {data_line:?}" + let cells: Vec<&str> = data_line.split('\t').collect(); + assert_eq!( + (cells[6], cells[7], cells[8]), + ("", "false", ""), + "expected an empty quantity cell and no transfer: {data_line:?}" ); let prot = std::fs::read_to_string(&proteins).unwrap(); @@ -656,6 +742,184 @@ mod tests { assert_eq!(prot.lines().count(), 2, "header plus one group:\n{prot}"); } + /// The same table after `mumdia mbr`: the above-threshold target is an accepted + /// transfer (`is_transferred`, with the transfer q the worker accepted it at). + fn scored_table_with_transfer() -> String { + let path = tmp("scored_mbr.parquet"); + write_table( + &path, + vec![ + Col::Str( + "peptidoform".into(), + vec![ + "PEPTIDEK".into(), + "M[Oxidation]EGVDGHK".into(), + "DECOY_KEDITPEP".into(), + "LATEPEPTIDEK".into(), + ], + ), + Col::I32("charge".into(), vec![2, 3, 2, 2]), + Col::Str( + "protein".into(), + vec!["P1".into(), "P2".into(), "DECOY_P1".into(), "P1".into()], + ), + Col::Str( + "label".into(), + vec![ + "target".into(), + "target".into(), + "decoy".into(), + "target".into(), + ], + ), + Col::F64("peptide_q_value".into(), vec![0.001, 0.005, 0.0001, 0.5]), + Col::Str( + "protein_group".into(), + vec!["PG1".into(), "PG2".into(), "DECOY_PG1".into(), "".into()], + ), + Col::F64("pg_q_value".into(), vec![0.002, 0.9, 0.0001, 1.0]), + Col::F64("score".into(), vec![3.5, 2.5, 9.9, 0.1]), + Col::Bool("is_transferred".into(), vec![false, false, false, true]), + Col::F64( + "transfer_q".into(), + vec![f64::NAN, f64::NAN, f64::NAN, 0.004], + ), + ], + ) + .unwrap(); + path + } + + #[test] + fn a_malformed_transfer_column_is_an_error_not_a_silent_loss_of_every_transfer() { + // docs/31 F2: `mbr_worker.py` writes `is_transferred` through pandas, so a + // nullable dtype or int8 0/1 reaches this stage. The old `Err(_) => vec![false; n]` + // arm turned that into "no transfers", removing every match-between-runs + // identification from both TSVs at exit 0 while the parquet still showed them. + let path = tmp("scored_int_flag.parquet"); + write_table( + &path, + vec![ + Col::Str("peptidoform".into(), vec!["PEPTIDEK".into()]), + Col::I32("charge".into(), vec![2]), + Col::Str("protein".into(), vec!["P1".into()]), + Col::Str("label".into(), vec!["target".into()]), + Col::F64("peptide_q_value".into(), vec![0.5]), + Col::Str("protein_group".into(), vec!["PG1".into()]), + Col::F64("pg_q_value".into(), vec![0.5]), + Col::F64("score".into(), vec![1.0]), + // The flag as an integer, which is what a pandas round-trip can produce. + Col::I32("is_transferred".into(), vec![1]), + Col::F64("transfer_q".into(), vec![0.004]), + ], + ) + .unwrap(); + let out = tmp("report_badflag"); + std::fs::create_dir_all(&out).unwrap(); + let e = run(ReportParams { + scored: &path, + peptide_quant: None, + protein_quant: None, + out_peptides: &format!("{out}/peptides.tsv"), + out_proteins: &format!("{out}/proteins.tsv"), + q_threshold: 0.01, + }) + .unwrap_err() + .to_string(); + assert!(e.contains("is_transferred"), "{e}"); + } + + #[test] + fn a_table_that_never_saw_mbr_reports_no_transfers_without_complaint() { + // The absent case stays absent: `mumdia report` on a plain scored table must not + // start demanding columns only `mumdia mbr` writes. + let scored = scored_table(); + let out = tmp("report_nombr"); + std::fs::create_dir_all(&out).unwrap(); + let (n_pep, _) = run(ReportParams { + scored: &scored, + peptide_quant: None, + protein_quant: None, + out_peptides: &format!("{out}/peptides.tsv"), + out_proteins: &format!("{out}/proteins.tsv"), + q_threshold: 0.01, + }) + .unwrap(); + assert_eq!(n_pep, 2); + let text = std::fs::read_to_string(format!("{out}/peptides.tsv")).unwrap(); + for line in text.lines().skip(1) { + let c: Vec<&str> = line.split('\t').collect(); + assert_eq!((c[7], c[8]), ("false", "")); + } + } + + #[test] + fn report_exports_the_transfer_basis_and_a_tighter_threshold_does_not_revoke_it() { + // docs/29 #19: the acceptance rule admits any target flagged `is_transferred` + // whatever its grouped q, and the TSV has to show that basis rather than a + // `q_value` of 0.5 next to a 1% threshold with no explanation. + let scored = scored_table_with_transfer(); + let out = tmp("report_mbr"); + std::fs::create_dir_all(&out).unwrap(); + let peptides = format!("{out}/peptides.tsv"); + let proteins = format!("{out}/proteins.tsv"); + let report = |q: f64| { + run(ReportParams { + scored: &scored, + peptide_quant: None, + protein_quant: None, + out_peptides: &peptides, + out_proteins: &proteins, + q_threshold: q, + }) + .unwrap() + }; + + let (n_pep, _) = report(0.01); + assert_eq!(n_pep, 3, "two confident targets plus the transfer"); + let text = std::fs::read_to_string(&peptides).unwrap(); + let header = text.lines().next().unwrap(); + assert!( + header.ends_with("\tquantity\tis_transferred\ttransfer_q"), + "{header}" + ); + let late = text + .lines() + .find(|l| l.starts_with("LATEPEPTIDEK")) + .expect("the transfer is reported"); + let cells: Vec<&str> = late.split('\t').collect(); + // The grouped q is preserved as it is (0.5), and the basis is next to it. + assert_eq!( + (cells[4], cells[7], cells[8]), + ("0.500000", "true", "0.004000") + ); + let first = text.lines().nth(1).unwrap(); + let c: Vec<&str> = first.split('\t').collect(); + assert_eq!( + (c[7], c[8]), + ("false", ""), + "a scored row is not a transfer: {first}" + ); + + // Protein rows carry the same two columns; PG1 was admitted on its own q. + let prot = std::fs::read_to_string(&proteins).unwrap(); + let plines: Vec<&str> = prot.lines().collect(); + assert_eq!( + plines[0], + "protein_group\tq_value\tquantity\tis_transferred\ttransfer_q" + ); + let pg: Vec<&str> = plines[1].split('\t').collect(); + assert_eq!((pg[0], pg[3], pg[4]), ("PG1", "false", "")); + + // Tighter threshold: the 0.005 target drops, the transfer stays. This is the + // rule the columns exist to make visible, not a new one. + let (n_pep, _) = report(0.001); + assert_eq!(n_pep, 2); + let text = std::fs::read_to_string(&peptides).unwrap(); + assert!(text.contains("LATEPEPTIDEK\t"), "{text}"); + assert!(!text.contains("M[Oxidation]EGVDGHK"), "{text}"); + } + #[test] fn report_writes_header_only_when_nothing_passes() { // A run that identifies nothing at the threshold still has to produce the @@ -715,7 +979,8 @@ mod tests { let mut quantified = 0; let mut empty = 0; for line in text.lines().skip(1) { - let cell = line.rsplit('\t').next().unwrap(); + // The quantity column, by position: the transfer columns follow it. + let cell = line.split('\t').nth(6).unwrap(); if cell.is_empty() { empty += 1; } else { diff --git a/rust/mumdia/crates/mumdia/src/stages/rescore.rs b/rust/mumdia/crates/mumdia/src/stages/rescore.rs index 9fbd0181..351b8762 100644 --- a/rust/mumdia/crates/mumdia/src/stages/rescore.rs +++ b/rust/mumdia/crates/mumdia/src/stages/rescore.rs @@ -57,6 +57,24 @@ pub struct RescoreParams<'a> { /// The matrix spans six orders of magnitude between the smoke fixture and a 40-run /// experiment, so a fixed unit is unhelpful at one end or the other: `0.00 GiB` says /// nothing, and `270336.0 MiB` says it badly. +/// Bytes of the flat f32 feature matrix for `rows` PSMs and `features` columns, or +/// `None` when the product overflows. +fn feature_matrix_bytes(rows: usize, features: usize) -> Option { + (rows as u64) + .checked_mul(features as u64)? + .checked_mul(std::mem::size_of::() as u64) +} + +/// `Some(ceiling)` when a configured `rescore.max_feature_matrix_gib` (0 = off) is +/// exceeded by `matrix_bytes`. +fn matrix_ceiling_exceeded(matrix_bytes: u64, max_gib: f64) -> Option { + if max_gib <= 0.0 { + return None; + } + let gib = matrix_bytes as f64 / (1024.0 * 1024.0 * 1024.0); + (gib > max_gib).then_some(max_gib) +} + fn human_bytes(bytes: f64) -> String { const KIB: f64 = 1024.0; const MIB: f64 = KIB * 1024.0; @@ -94,6 +112,14 @@ pub fn run(p: RescoreParams) -> Result { if p.competed.is_empty() { anyhow::bail!("rescore requires at least one competed input"); } + // `--out` must not be one of the competed tables: they are read before the scored + // table is published, so writing over one replaces it and exits 0 (docs/31 F6). + let competed_inputs: Vec<(&str, &str)> = p + .competed + .iter() + .map(|c| ("--competed", c.as_str())) + .collect(); + mumdia_io::refuse_output_over_input(p.out, &competed_inputs)?; if p.cfg.folds < 2 { anyhow::bail!("rescore.folds must be >= 2 for out-of-fold scoring"); } @@ -143,6 +169,38 @@ pub fn run(p: RescoreParams) -> Result { for path in p.competed.iter() { total_rows += TableFile::open(path)?.nrows; } + // The matrix is one contiguous f32 buffer (`rescoring::FeatureMatrix`), so its size + // follows from the parquet footers and the selected feature count before a byte is + // allocated. The ceiling used to be applied after the matrix had been filled, against + // an estimate of the old `Vec>` layout (8 bytes per value plus a 24-byte + // spine per PSM), so it could neither prevent the allocation it described nor + // describe the one that happened (docs/29 #11). It is a limit on the matrix alone. + let matrix_bytes = feature_matrix_bytes(total_rows, feat_names.len()).ok_or_else(|| { + anyhow::anyhow!( + "rescore feature matrix size overflows: {total_rows} PSMs x {} features", + feat_names.len() + ) + })?; + info!( + psms = total_rows, + features = feat_names.len(), + feature_matrix = %human_bytes(matrix_bytes as f64), + folds = p.cfg.folds, + "rescore: feature matrix size before allocation" + ); + if let Some(ceiling) = matrix_ceiling_exceeded(matrix_bytes, p.cfg.max_feature_matrix_gib) { + anyhow::bail!( + "rescore feature matrix would be {} ({total_rows} PSMs x {} features x 4 bytes, \ + f32), over the configured rescore.max_feature_matrix_gib of {ceiling:.2}. This \ + is the matrix alone: per-PSM metadata, the per-fold standardised training \ + copies of native_tda (roughly (1 + folds) times this at peak) and the Python \ + worker's own copy come on top. Either raise the ceiling, or rescore fewer runs \ + per invocation -- `run_psm_q` is computed per source, so sub-batching costs no \ + per-run FDR, though it does change which PSMs share the pooled q_value.", + human_bytes(matrix_bytes as f64), + feat_names.len(), + ); + } let mut matrix = FeatureMatrix::with_capacity(total_rows, feat_names.len()); for (src, path) in p.competed.iter().enumerate() { let actual_schema = FeatureSchema::read(path)?; @@ -257,37 +315,12 @@ pub fn run(p: RescoreParams) -> Result { } } } - // Say how big the feature matrix is, and refuse it if a ceiling is configured. - // - // `feats` is `Vec>`, so eight bytes per value plus a heap allocation and a - // 24-byte spine entry per PSM -- twice the width CLAUDE.md documented, because that - // figure describes the Python worker's f32 matrix. On an experiment-wide pool this is - // the peak-RSS wall (the code's own comment says ~27 GB), `native_tda` runs all folds - // in parallel each holding an owned standardised copy of its training slice, and - // nothing here estimates available memory. So the failure mode was an OS kill after - // however long the run took to reach it, with no number to plan against. - let matrix_bytes = (n as f64) * (feat_names.len() as f64) * 8.0 + (n as f64) * 24.0; - let matrix_gib = matrix_bytes / (1024.0 * 1024.0 * 1024.0); info!( psms = n, features = feat_names.len(), - feature_matrix = %human_bytes(matrix_bytes), folds = p.cfg.folds, "rescore: loaded competed PSMs" ); - if p.cfg.max_feature_matrix_gib > 0.0 && matrix_gib > p.cfg.max_feature_matrix_gib { - anyhow::bail!( - "rescore feature matrix would be {} ({n} PSMs x {} features x 8 bytes), over \ - the configured rescore.max_feature_matrix_gib of {:.2}. The native rescorer \ - holds roughly (1 + folds) times this at peak. Either raise the ceiling, or \ - rescore fewer runs per invocation -- `run_psm_q` is computed per source, so \ - sub-batching costs no per-run FDR, though it does change which PSMs share \ - the pooled q_value.", - human_bytes(matrix_bytes), - feat_names.len(), - p.cfg.max_feature_matrix_gib - ); - } // Track the path actually taken so the report reflects reality rather than a // hardcoded label, and pick the null the q-values are computed against. @@ -1642,3 +1675,22 @@ b assert!(qd[1] <= 1.0); } } + +#[cfg(test)] +mod matrix_ceiling_tests { + use super::*; + + #[test] + fn the_ceiling_is_judged_on_the_f32_matrix_before_allocation() { + // 1,000,000 PSMs x 387 features: 1.548 GB as f32. The old estimate + // (8 bytes + a 24-byte spine per PSM) was 3.12 GB, so a 2 GiB ceiling used to + // refuse a matrix that fits with room to spare. + let bytes = feature_matrix_bytes(1_000_000, 387).unwrap(); + assert_eq!(bytes, 1_548_000_000); + assert_eq!(matrix_ceiling_exceeded(bytes, 2.0), None); + assert_eq!(matrix_ceiling_exceeded(bytes, 1.0), Some(1.0)); + // 0 disables the ceiling; an overflowing product is refused rather than wrapped. + assert_eq!(matrix_ceiling_exceeded(bytes, 0.0), None); + assert_eq!(feature_matrix_bytes(usize::MAX, 2), None); + } +} diff --git a/rust/mumdia/crates/mumdia/src/stages/rt_im_train.rs b/rust/mumdia/crates/mumdia/src/stages/rt_im_train.rs index f248bff3..250cd011 100644 --- a/rust/mumdia/crates/mumdia/src/stages/rt_im_train.rs +++ b/rust/mumdia/crates/mumdia/src/stages/rt_im_train.rs @@ -99,6 +99,23 @@ fn candidate_window(calibrated_rt: Option, width: Option) -> (f64, f64 pub fn run(p: RtImTrainParams) -> Result { let t0 = Instant::now(); + // `--out` must not be one of this stage's own inputs: every input is read + // before the output is published, so writing over one replaces it and exits 0 + // (docs/31 F6). The shared guard existed and was wired into two stages. + mumdia_io::refuse_output_over_input( + p.out_windows, + &[ + ("--seed-psms", p.seed_psms), + ("--lib-precursors", p.library_precursors), + ], + )?; + mumdia_io::refuse_output_over_input( + p.out_cal, + &[ + ("--seed-psms", p.seed_psms), + ("--lib-precursors", p.library_precursors), + ], + )?; let holdout_frac = p.cfg.window_holdout_frac; if !(0.0..=0.9).contains(&holdout_frac) { @@ -357,8 +374,17 @@ pub fn run(p: RtImTrainParams) -> Result { Vec::with_capacity(n), Vec::with_capacity(n), ); + let mut n_nonfinite_irt = 0u64; for i in 0..n { - let calibrated_rt = calibration_available.then(|| predict(irt[i] as f64)); + // A row whose library iRT is not finite has no calibrated RT, which is the + // documented "search the whole gradient" sentinel rather than an arithmetic + // accident. The parquet reader maps a null f32 to NaN, so one failed prediction + // in an imported library reaches here (docs/31 F4). + let usable_irt = (irt[i] as f64).is_finite(); + if !usable_irt { + n_nonfinite_irt += 1; + } + let calibrated_rt = (calibration_available && usable_irt).then(|| predict(irt[i] as f64)); let width = calibrated_rt.map(|cal| match &adaptive { Some((rt_min, span, widths)) => { let nb = widths.len(); @@ -457,6 +483,10 @@ pub fn run(p: RtImTrainParams) -> Result { stats.insert("n_train".to_string(), json!(n_train)); stats.insert("w_rt".to_string(), json!(w_rt)); stats.insert("calibration_status".to_string(), json!(status)); + stats.insert( + "candidates_without_finite_irt".to_string(), + json!(n_nonfinite_irt), + ); ArtifactReport { logical_name: artifact::RUN_WINDOWS.0.to_string(), schema_name: artifact::RUN_WINDOWS.0.to_string(), @@ -471,10 +501,18 @@ pub fn run(p: RtImTrainParams) -> Result { } .write_for(p.out_windows)?; + if n_nonfinite_irt > 0 { + tracing::warn!( + candidates = n_nonfinite_irt, + of = rows, + "rt-im-train: these candidates have no finite library iRT, so they get the unbounded RT window rather than a calibrated one; a null predicted_irt reads as NaN (docs/31 F4)" + ); + } info!( rows, w_rt = ?w_rt, status, + without_finite_irt = n_nonfinite_irt, elapsed_ms = elapsed, "rt-im-train: done" ); diff --git a/rust/mumdia/crates/mumdia/src/stages/run.rs b/rust/mumdia/crates/mumdia/src/stages/run.rs index 160fec06..46571d07 100644 --- a/rust/mumdia/crates/mumdia/src/stages/run.rs +++ b/rust/mumdia/crates/mumdia/src/stages/run.rs @@ -615,13 +615,14 @@ pub fn run(p: RunParams) -> Result<()> { // Model identities reflect the path that produced the downstream artifacts, // including imported libraries and per-run RT fine-tuning. let library_input = p.lib_precursors.is_some(); + let deeplc_py = cfg.predict_frag.deeplc_python.as_deref(); let rt_identity = if cfg.rt_im_train.finetune_deeplc { - "deeplc-finetuned".to_string() + crate::sidecar::deeplc_identity(deeplc_py, "finetuned") } else if cfg .rt_im_train - .repredicts_library_irt(library_input, cfg.predict_frag.deeplc_python.is_some()) + .repredicts_library_irt(library_input, deeplc_py.is_some()) { - "deeplc-base-model".to_string() + crate::sidecar::deeplc_identity(deeplc_py, "base") } else if library_input { "imported-library".to_string() } else { diff --git a/rust/mumdia/crates/mumdia/src/stages/run_experiment.rs b/rust/mumdia/crates/mumdia/src/stages/run_experiment.rs index 3cc2c0f1..aa659429 100644 --- a/rust/mumdia/crates/mumdia/src/stages/run_experiment.rs +++ b/rust/mumdia/crates/mumdia/src/stages/run_experiment.rs @@ -320,6 +320,72 @@ fn split_by_source(scored: &str, out_paths: &[String]) -> Result<()> { Ok(()) } +/// Reject run names that would share a per-run output directory. +/// +/// Compared without regard to case on every platform, not only where the filesystem +/// is known to fold case: `RunA` and `runa` are two directories on ext4 and one +/// directory on NTFS, APFS and most network shares, and an experiment's output may be +/// written to any of them. Sequential runs overwrite each other's artifacts there and +/// parallel ones interleave, with no error from either (docs/29 #5). Refusing the pair +/// everywhere costs nothing anyone would want. +fn check_run_names_distinct(ns: &[String]) -> Result<()> { + let mut seen: std::collections::BTreeMap = std::collections::BTreeMap::new(); + for n in ns { + if let Some(prev) = seen.insert(n.to_lowercase(), n.as_str()) { + if prev == n { + anyhow::bail!( + "--run-names must be unique: {n:?} is given twice; each name is a per-run \ + output subdirectory, so a repeat makes two runs write the same artifacts \ + into one directory and interleave their results with no error" + ); + } + anyhow::bail!( + "--run-names {prev:?} and {n:?} differ only in case; on a case-insensitive \ + filesystem (Windows, macOS, most network shares) they are one per-run output \ + directory, so the names must be distinct without regard to case" + ); + } + } + Ok(()) +} + +/// Why `name` cannot be a per-run directory name on every platform, or `None`. +/// +/// Syntactic, not probed: Windows's rules are applied everywhere, because an experiment's +/// output may be written to any filesystem and a name that is one directory on NTFS must +/// not be two on ext4. `a` and `a.` passed the old check (empty, separators, `.`, `..`) and +/// were one directory on Windows: the second run overwrote the first and the experiment +/// exited 0 with both split tables holding `source = 1` (docs/30 R2). +fn portable_dir_name_problem(name: &str) -> Option<&'static str> { + const RESERVED: [&str; 22] = [ + "CON", "PRN", "AUX", "NUL", "COM1", "COM2", "COM3", "COM4", "COM5", "COM6", "COM7", "COM8", + "COM9", "LPT1", "LPT2", "LPT3", "LPT4", "LPT5", "LPT6", "LPT7", "LPT8", "LPT9", + ]; + if name.is_empty() { + return Some("it is empty"); + } + if name == "." || name == ".." { + return Some("`.` and `..` are not names"); + } + if name.contains('/') || name.contains('\\') { + return Some("it contains a path separator"); + } + if name + .chars() + .any(|c| matches!(c, '<' | '>' | ':' | '"' | '|' | '?' | '*') || (c as u32) < 0x20) + { + return Some("it contains a character Windows forbids in a file name (<>:\"|?* or a control character)"); + } + if name.ends_with('.') || name.ends_with(' ') { + return Some("it ends with a dot or a space, which Windows strips, so it names the same directory as the trimmed form"); + } + let stem = name.split('.').next().unwrap_or(name).to_ascii_uppercase(); + if RESERVED.contains(&stem.as_str()) { + return Some("it is a Windows reserved device name (CON, PRN, AUX, NUL, COM1-9, LPT1-9), with or without an extension"); + } + None +} + pub fn run(p: RunExperimentParams) -> Result<()> { let t0 = Instant::now(); // Same contract as the single-run orchestrator, and it matters more here: an @@ -339,6 +405,34 @@ pub fn run(p: RunExperimentParams) -> Result<()> { std::fs::create_dir_all(p.out_dir).ok(); let d = |name: &str| format!("{}/{}", p.out_dir, name); let n_runs = p.mzmls.len(); + + // Provenance: the identity of the code, of the configuration and of the INPUTS, + // hashed now, before anything reads them for compute (docs/29 #15). Hashing at the + // end recorded whatever bytes were on disk after a multi-hour experiment, which is + // not necessarily what the search read; the single-run orchestrator has always + // hashed first, and the two now agree. + let mut prov = Manifest::new(cfg.canonical_json(), ch.clone()); + for (i, m) in p.mzmls.iter().enumerate() { + if let (Ok(bytes), Ok(hash)) = ( + std::fs::metadata(m).map(|x| x.len()), + mumdia_io::hash::blake3_file(m), + ) { + prov.record_input(&format!("mzml[{i}]"), m, bytes, hash); + } + } + for (role, path) in [ + ("fasta", p.fasta), + ("lib_precursors", p.lib_precursors), + ("lib_fragments", p.lib_fragments), + ] { + let Some(path) = path else { continue }; + if let (Ok(bytes), Ok(hash)) = ( + std::fs::metadata(path).map(|x| x.len()), + mumdia_io::hash::blake3_file(path), + ) { + prov.record_input(role, path, bytes, hash); + } + } // Reject a bad --run-names rather than silently substituting r0..rN-1. // // The old `_ =>` arm swallowed any count mismatch with no warning, and accepted @@ -358,22 +452,14 @@ pub fn run(p: RunExperimentParams) -> Result<()> { n_runs - 1 ); } - let mut sorted = ns.to_vec(); - sorted.sort(); - sorted.dedup(); - if sorted.len() != ns.len() { - anyhow::bail!( - "--run-names must be unique: each name is a per-run output \ - subdirectory, so a repeat makes two runs write the same artifacts \ - into one directory and interleave their results with no error" - ); - } - if let Some(bad) = ns.iter().find(|n| { - n.is_empty() || n.contains('/') || n.contains('\\') || *n == "." || *n == ".." - }) { + check_run_names_distinct(ns)?; + if let Some((bad, why)) = ns + .iter() + .find_map(|n| portable_dir_name_problem(n).map(|why| (n, why))) + { anyhow::bail!( - "--run-names entry {bad:?} is not usable as a directory name; each \ - becomes a subdirectory of --out-dir" + "--run-names entry {bad:?} is not usable as a directory name: {why}; each \ + name becomes a subdirectory of --out-dir" ); } ns.to_vec() @@ -583,6 +669,20 @@ pub fn run(p: RunExperimentParams) -> Result<()> { cfg: &cfg.rescore, config_hash: &ch, })?; + // The classifier that actually ran, from the rescore artifact report: the source of + // truth, since the configured enum can differ from it under a compatibility path. + let rescore_report: mumdia_io::report::ArtifactReport = + mumdia_io::json::read_json(&format!("{scored_combined}.report.json"))?; + let actual_rescorer = rescore_report + .params + .get("classifier") + .and_then(|v| v.as_str()) + .unwrap_or("unknown") + .to_string(); + let actual_rescorer_model = rescore_report + .model_identity + .clone() + .unwrap_or_else(|| actual_rescorer.clone()); // --- optional rescuable-tier MBR transfer --- let scored_for_quant = if cfg.mbr.strategy != mumdia_core::config::MbrStrategy::None { @@ -721,7 +821,15 @@ pub fn run(p: RunExperimentParams) -> Result<()> { "runs": names, "scored_combined": scored_combined, "scored_for_quant": scored_for_quant, + "rescorer": actual_rescorer, "mbr": format!("{:?}", cfg.mbr.strategy), + // Per-run quant gates on the pooled q_value whatever the configuration says + // (see the warning above). The configuration hash is of the configuration as + // given, so the substitution has to be recorded here or it is recorded nowhere. + "quant_q_filter": { + "configured": format!("{:?}", cfg.quant.q_filter), + "effective": format!("{:?}", qcfg.q_filter), + }, "lfq": lfq, "peptide_quants": peptide_quants, "report": { @@ -734,7 +842,8 @@ pub fn run(p: RunExperimentParams) -> Result<()> { }, }); // Provenance parity with the single-run manifest: the identity of the code, of - // the inputs, and of every artifact this stage produced. + // the inputs (hashed at the start of the run, see above), and of every artifact + // this stage produced. // // The per-artifact records were the gap. The experiment manifest listed output // PATHS in its `experiment` block and nothing else, so an experiment result had @@ -743,28 +852,45 @@ pub fn run(p: RunExperimentParams) -> Result<()> { // needed to tell whether two experiment outputs are the same data. Files written // by the per-run chains are not covered here (those chains do not thread a shared // manifest); what is covered is everything `run-experiment` itself writes. - let mut prov = Manifest::new(cfg.canonical_json(), ch.clone()); - for (i, m) in p.mzmls.iter().enumerate() { - if let (Ok(bytes), Ok(hash)) = ( - std::fs::metadata(m).map(|x| x.len()), - mumdia_io::hash::blake3_file(m), - ) { - prov.record_input(&format!("mzml[{i}]"), m, bytes, hash); - } - } - for (role, path) in [ - ("fasta", p.fasta), - ("lib_precursors", p.lib_precursors), - ("lib_fragments", p.lib_fragments), - ] { - let Some(path) = path else { continue }; - if let (Ok(bytes), Ok(hash)) = ( - std::fs::metadata(path).map(|x| x.len()), - mumdia_io::hash::blake3_file(path), - ) { - prov.record_input(role, path, bytes, hash); + // + // Model identities reflect the path that produced the downstream artifacts, as + // in the single-run manifest (docs/29 #15): which RT source the library carried, + // which fragment predictor, and the classifier that actually ran. + let library_input = p.lib_precursors.is_some(); + let deeplc_py = cfg.predict_frag.deeplc_python.as_deref(); + let rt_identity = if cfg.rt_im_train.finetune_deeplc { + if matches!(cfg.experiment.finetune_scope, FinetuneScope::FirstRunOnly) { + crate::sidecar::deeplc_identity(deeplc_py, "finetuned-first-run") + } else { + crate::sidecar::deeplc_identity(deeplc_py, "finetuned-per-run") } - } + } else if cfg + .rt_im_train + .repredicts_library_irt(library_input, deeplc_py.is_some()) + { + crate::sidecar::deeplc_identity(deeplc_py, "base") + } else if library_input { + "imported-library".to_string() + } else { + format!("{:?}", cfg.predict_frag.rt_predictor) + }; + let fragment_identity = if library_input { + "imported-library".to_string() + } else { + format!("{:?}", cfg.predict_frag.predictor) + }; + prov.model_identities + .insert("rt_predictor".into(), rt_identity); + prov.model_identities + .insert("fragment_predictor".into(), fragment_identity); + prov.model_identities + .insert("rescorer".into(), actual_rescorer_model); + prov.model_identities.insert( + "feature_schema_id".into(), + features::feature_schema_id(&features::active_features(cfg.features.set)), + ); + prov.model_identities + .insert("mbr".into(), format!("{:?}", cfg.mbr.strategy)); // Recorded in a fixed order, and every record hashes its file. `Manifest` // stores them in a BTreeMap, so the serialised order is by logical name and // does not depend on this sequence. @@ -821,12 +947,18 @@ pub fn run(p: RunExperimentParams) -> Result<()> { &ch, )?); + // The resolved configuration itself, not only its hash: a hash identifies a + // configuration but cannot replay one (docs/29 #15). let manifest = serde_json::json!({ "mumdia_version": prov.mumdia_version, "git_sha": prov.git_sha, "commit_date": prov.commit_date, "cli_args": prov.cli_args, + "config_hash": prov.config_hash, + "config_json": prov.config_json, + "model_identities": prov.model_identities, "inputs": prov.inputs, + "inputs_hashed_at": "start", "artifacts": prov.artifacts, "experiment": manifest, }); @@ -846,6 +978,59 @@ mod tests { use super::*; use mumdia_io::table::{write_table, Col, Table}; + fn names(xs: &[&str]) -> Vec { + xs.iter().map(|s| s.to_string()).collect() + } + + #[test] + fn run_names_that_differ_only_in_case_are_rejected() { + // One directory on Windows, macOS and most network shares (docs/29 #5): the + // check refuses it everywhere rather than probing the destination filesystem. + let e = check_run_names_distinct(&names(&["RunA", "runa"])) + .unwrap_err() + .to_string(); + assert!(e.contains("differ only in case"), "{e}"); + assert!(e.contains("RunA") && e.contains("runa"), "{e}"); + } + + #[test] + fn run_names_that_alias_on_windows_are_rejected_everywhere() { + // docs/30 R2: `a` and `a.` are one directory on Windows and the experiment ran + // to completion with one run overwriting the other. The syntactic rules apply on + // every platform so the output is portable. + for (bad, why) in [ + ("a.", "dot or a space"), + ("a ", "dot or a space"), + ("NUL", "reserved"), + ("com1.log", "reserved"), + ("run:1", "forbids"), + ("run?", "forbids"), + ("a/b", "separator"), + ("", "empty"), + ("..", "not names"), + ] { + let why_got = portable_dir_name_problem(bad) + .unwrap_or_else(|| panic!("{bad:?} must be rejected")); + assert!(why_got.contains(why), "{bad:?}: {why_got}"); + } + for ok in ["r0", "run.1", "A-b_c", "B_01", "sample 3", "com10", "conx"] { + assert_eq!( + portable_dir_name_problem(ok), + None, + "{ok:?} must be accepted" + ); + } + } + + #[test] + fn repeated_run_names_are_rejected_and_distinct_ones_pass() { + let e = check_run_names_distinct(&names(&["a", "b", "a"])) + .unwrap_err() + .to_string(); + assert!(e.contains("given twice"), "{e}"); + check_run_names_distinct(&names(&["a", "b", "c_1"])).unwrap(); + } + fn tmp(name: &str) -> String { use std::sync::atomic::{AtomicU64, Ordering}; static CTR: AtomicU64 = AtomicU64::new(0); diff --git a/rust/mumdia/crates/mumdia/src/stages/search_seed.rs b/rust/mumdia/crates/mumdia/src/stages/search_seed.rs index 710ef789..1c3880c2 100644 --- a/rust/mumdia/crates/mumdia/src/stages/search_seed.rs +++ b/rust/mumdia/crates/mumdia/src/stages/search_seed.rs @@ -44,6 +44,17 @@ struct Best { pub fn run(p: SearchSeedParams) -> Result { let t0 = Instant::now(); + // `--out` must not be one of this stage's own inputs: every input is read + // before the output is published, so writing over one replaces it and exits 0 + // (docs/31 F6). The shared guard existed and was wired into two stages. + mumdia_io::refuse_output_over_input( + p.out, + &[ + ("--ms2", p.ms2), + ("--lib-precursors", p.library_precursors), + ("--lib-fragments", p.library_fragments), + ], + )?; // See extract: the bucketed index is dead weight on the fragindex path. let build_bucketed = !matches!(p.cfg.matcher, MatcherKind::Fragindex); let lib = Library::load_with( diff --git a/rust/mumdia/crates/mumdia/tests/cli_version.rs b/rust/mumdia/crates/mumdia/tests/cli_version.rs new file mode 100644 index 00000000..cb024067 --- /dev/null +++ b/rust/mumdia/crates/mumdia/tests/cli_version.rs @@ -0,0 +1,43 @@ +//! The built binary, run as a subprocess. +//! +//! The debug build overflowed its 1 MiB Windows main-thread stack on `--version`, before +//! printing anything (docs/30 R9). No library test could see that: it is a property of the +//! binary's entry point under the developer profile, so it is checked here on whichever +//! profile `cargo test` builds. + +use std::process::Command; + +fn mumdia() -> Command { + Command::new(env!("CARGO_BIN_EXE_mumdia")) +} + +#[test] +fn the_built_binary_prints_its_version() { + let out = mumdia() + .arg("--version") + .output() + .expect("run mumdia --version"); + assert!( + out.status.success(), + "status {:?}\nstderr:\n{}", + out.status, + String::from_utf8_lossy(&out.stderr) + ); + let text = String::from_utf8_lossy(&out.stdout); + assert!( + text.contains(env!("CARGO_PKG_VERSION")), + "expected the crate version in {text:?}" + ); +} + +#[test] +fn the_built_binary_prints_its_help() { + let out = mumdia().arg("--help").output().expect("run mumdia --help"); + assert!( + out.status.success(), + "{}", + String::from_utf8_lossy(&out.stderr) + ); + let text = String::from_utf8_lossy(&out.stdout); + assert!(text.contains("run-experiment"), "{text}"); +} diff --git a/scripts/augment_library.py b/scripts/augment_library.py index 4a3ef2d3..23a3e8ab 100644 --- a/scripts/augment_library.py +++ b/scripts/augment_library.py @@ -57,6 +57,30 @@ def stripped(peptidoform: str) -> str: return STRIP_MODS.sub("", s) +def assign_base_ids(imported_base_by_seq, added_seqs, next_id): + """`base_peptide_id` for each added target, by stripped sequence. + + A sequence the imported library already carries keeps that sequence's id, so an + added charge state or modform stays in its peptide's competition group and CV fold + (`--match-level peptidoform_charge` adds exactly those). A sequence new to the + library gets a fresh id above the imported range, one per sequence, so its own forms + share it. Offsetting every added id, the previous rule, split one stripped peptide + across two base ids whenever it already existed (docs/29 #12). The shift-decoy + helper copies `base_peptide_id` onto each decoy, so pairs stay paired. + """ + fresh = {} + out = [] + for seq in added_seqs: + if seq in imported_base_by_seq: + out.append(int(imported_base_by_seq[seq])) + continue + if seq not in fresh: + fresh[seq] = next_id + next_id += 1 + out.append(fresh[seq]) + return out + + def run_stage(mumdia_bin, subcmd, args, config): cmd = [mumdia_bin, subcmd] + args if config: @@ -136,13 +160,24 @@ def main(): mprec = pd.read_parquet(W("aug_missing_prec.parquet")) mfrag = pd.read_parquet(W("aug_missing_frag.parquet")) - # 5. Make ids disjoint from the imported library; keep sibling/base linkage. + # 5. Make ids disjoint from the imported library where they must be (peptidoform + # and candidate ids), and keep base-peptide linkage where it exists: an added + # form of a sequence the library already has reuses that sequence's base id. print("[4/6] merge imported targets + missing targets", flush=True) pfid_off = int(imp_t.peptidoform_id.max()) + 1 - base_off = int(imp_t.base_peptide_id.max()) + 1 cid_off = int(imp_t.candidate_id.max()) + 1 mprec["peptidoform_id"] = mprec["peptidoform_id"].astype(np.int64) + pfid_off - mprec["base_peptide_id"] = mprec["base_peptide_id"].astype(np.int64) + base_off + imported_base_by_seq = {} + for seq, bid in zip(imp_t.peptidoform.map(stripped), imp_t.base_peptide_id.astype(np.int64)): + imported_base_by_seq.setdefault(seq, int(bid)) + mprec["base_peptide_id"] = np.asarray( + assign_base_ids(imported_base_by_seq, list(mprec.peptidoform.map(stripped)), + int(imp_t.base_peptide_id.max()) + 1), + dtype=np.int64, + ) + n_reused = int(mprec.peptidoform.map(stripped).isin(imported_base_by_seq).sum()) + print(f" base ids: {n_reused} added forms keep an imported sequence's id, " + f"{len(mprec) - n_reused} rows belong to new sequences", flush=True) # candidate_id only needs to be unique before make_shift_decoys re-densifies. mprec["candidate_id"] = mprec["candidate_id"].astype(np.int64) + cid_off mfrag["candidate_id"] = mfrag["candidate_id"].astype(np.int64) + cid_off diff --git a/scripts/deeplc_finetune.py b/scripts/deeplc_finetune.py index 5f26f5de..8edaebbd 100644 --- a/scripts/deeplc_finetune.py +++ b/scripts/deeplc_finetune.py @@ -32,6 +32,8 @@ os.environ["NUMEXPR_NUM_THREADS"] = "1" import argparse +import json +import math import re import time import deeplc # import before numpy (OpenMP load order) @@ -271,6 +273,13 @@ def main(): t0 = time.time() batch = uniq[s:s + chunk] p = agg(deeplc.predict(batch) if ft_model is None else deeplc.predict(batch, model=ft_model)) + # A structurally short or long answer is a broken predictor, not a set of + # unsupported peptidoforms: zipping it silently paired predictions with the wrong + # peptidoforms and left the tail on its imported value (docs/30 R6). + if len(p) != len(batch): + raise SystemExit( + f"DeepLC returned {len(p)} predictions for {len(batch)} peptidoforms in one " + f"batch; refusing to rewrite the library from a malformed response") for pf, v in zip(batch, p): preds[pf] = float(v) done = min(s + chunk, len(uniq)) @@ -281,14 +290,61 @@ def main(): f"({rate:.0f} peptidoforms/s, ETA {eta / 60:.1f} min)", flush=True) print(f"prediction phase: {time.time() - t_pred0:.1f}s total", flush=True) - # `base_pf` is recomputed here rather than cached from the pass above on purpose: - # caching it would retain one extra string per library row (hundreds of MB at - # library scale) to avoid a `startswith` and a slice. - new = np.array([preds.get(base_pf(pf), orig[i]) for i, pf in enumerate(pform)], dtype=np.float32) + new, summary = rewrite_irt(pform, orig, preds) idx = lib.schema.get_field_index("predicted_irt") lib = lib.set_column(idx, "predicted_irt", pa.array(new, pa.float32())) pq.write_table(lib, args.lib_out) + summary["model"] = which + summary["lib_in"] = args.lib_in + summary["lib_out"] = args.lib_out + with open(args.lib_out + ".summary.json", "w", encoding="utf-8") as fh: + json.dump(summary, fh, indent=2) print(f"wrote library with re-predicted iRT ({which}): {args.lib_out}") + print(f" rows={summary['rows']} repredicted={summary['repredicted']} " + f"retained_imported={summary['retained_imported']} " + f"(non-standard residues {summary['retained_non_standard']}, " + f"no finite prediction {summary['retained_no_prediction']})") + if summary["retained_imported"]: + print(f"WARNING: {summary['retained_imported']} of {summary['rows']} rows " + f"({100.0 * summary['retained_imported'] / max(1, summary['rows']):.2f}%) keep " + f"their imported iRT, which is on the imported model's scale, not {which}'s; " + f"the counts are in {args.lib_out}.summary.json", flush=True) + + +def rewrite_irt(pform, orig, preds): + """The new `predicted_irt` column and a count of where each value came from. + + A peptidoform with a finite prediction for its DECOY_-stripped sequence takes it. A + peptidoform without one keeps its imported value: rows with non-standard residues are + never sent to DeepLC (`is_std`), and a prediction that came back non-finite is an + unsupported input rather than a number. Both are counted so the mixture of RT sources + in the written library is explicit instead of silent (docs/30 R6). `base_pf` is + recomputed here rather than cached from the pass above on purpose: caching it would + retain one extra string per library row (hundreds of MB at library scale). + """ + n = len(pform) + new = np.empty(n, dtype=np.float32) + repredicted = 0 + no_prediction = 0 + non_standard = 0 + for i, pf in enumerate(pform): + v = preds.get(base_pf(pf)) + if v is None: + new[i] = orig[i] + non_standard += 1 + elif not math.isfinite(v): + new[i] = orig[i] + no_prediction += 1 + else: + new[i] = v + repredicted += 1 + return new, { + "rows": n, + "repredicted": repredicted, + "retained_imported": non_standard + no_prediction, + "retained_non_standard": non_standard, + "retained_no_prediction": no_prediction, + } if __name__ == "__main__": diff --git a/scripts/entrapment_worker.py b/scripts/entrapment_worker.py index 00c7ffb8..3f68d438 100644 --- a/scripts/entrapment_worker.py +++ b/scripts/entrapment_worker.py @@ -11,6 +11,11 @@ and the entrapment negatives are scored by a model that never saw them. Decoy rows are scored by a final model fit on all non-decoy PSMs. +A training fold that holds a single class is an error, not a gap to fill: the +previous behaviour skipped such a fold and then scored its held-out rows with the +final model, which had been trained on those very rows, so the confidence estimate +downstream rested on in-sample scores (docs/29 #3). + Input columns: candidate_id, base_peptide_id, is_entrapment (0/1), is_decoy (0/1), and one column per feature. Output columns: candidate_id, score. Run with an env that has scikit-learn + pyarrow (py312_mumdia). @@ -91,21 +96,37 @@ def main(): scores = np.full(len(t), np.nan, dtype=np.float64) gkf = GroupKFold(n_splits=k) - for tr, te in gkf.split(Xt, yt, gt): - # A fold whose training side is single-class cannot fit; leave NaN, the - # final full model fills it below. - if len(np.unique(yt[tr])) < 2: - continue + for fold_no, (tr, te) in enumerate(gkf.split(Xt, yt, gt)): + classes = np.unique(yt[tr]) + if len(classes) < 2: + raise SystemExit( + f"entrapment_worker: training fold {fold_no + 1} of {k} holds a single " + f"class ({'real targets' if classes[0] == 1 else 'entrapment'} only). Every " + "training fold needs both real-target and entrapment PSMs: use more base " + "peptides, fewer folds, or a larger spike-in library. Scoring the held-out " + "rows with a model trained on them would inflate the entrapment FDR, so the " + "worker refuses instead of filling the gap." + ) m = _new_model() m.fit(Xt[tr], yt[tr]) scores[idx[te]] = m.predict_proba(Xt[te])[:, 1] - # Final model on all non-decoy PSMs: scores decoys and any out-of-fold gaps. + # Every non-decoy row now carries an out-of-fold score; anything else is a fold + # construction bug, not a data condition. + gap = np.isnan(scores) & train + if gap.any(): + raise SystemExit( + f"entrapment_worker: {int(gap.sum())} non-decoy rows received no out-of-fold " + "score; GroupKFold did not cover the training set" + ) + + # Final model on all non-decoy PSMs scores the decoys, which took no part in + # training and have no fold. mf = _new_model() mf.fit(Xt, yt) - gap = np.isnan(scores) - if gap.any(): - scores[gap] = mf.predict_proba(X[gap])[:, 1] + dec_idx = np.where(is_dec)[0] + if len(dec_idx): + scores[dec_idx] = mf.predict_proba(X[dec_idx])[:, 1] out = pa.table({ "row_id": pa.array(rid.astype("uint32")), diff --git a/scripts/mbr_worker.py b/scripts/mbr_worker.py index 036643a0..bfc5d848 100644 --- a/scripts/mbr_worker.py +++ b/scripts/mbr_worker.py @@ -20,7 +20,10 @@ Output .parquet: one row per ACCEPTED transfer (candidate_id, source, peptidoform, charge, protein_group, label, expected_rt, observed_rt, rt_delta, transfer_q). -Also prints a validation summary (accepted counts per run, empirical decoy fraction). +Also prints a validation summary: accepted counts per run and how many permuted-RT +null draws fall inside the accepted RT window. Transfer candidates are confident +targets of other runs by construction, so a decoy-label count among them is +structurally zero and is not printed as validation (docs/29 #6). """ import argparse import numpy as np @@ -50,6 +53,34 @@ def binned_map(x, y, nb=80): return lambda q: np.interp(q, cx, cy) +def selected_apex_map(psms_path, source, selected): + """candidate_id -> apex_rt for one run, one peak per candidate. + + A competed table carries several peaks per candidate when `extract.retain_top_peaks` + is above 1. Rescore chose one of them and wrote its rank as `selected_peak_rank`; the + transfer's RT residual is measured on that peak, because that is the apex the scored + row carries and quant integrates (docs/29 #8). For a candidate rescore did not select + a peak for, the highest `prelim_score` peak stands in; a table without `peak_rank` + has one row per candidate and its value is taken as is. + """ + names = set(pq.read_schema(psms_path).names) + cols = ["candidate_id", "apex_rt"] + [c for c in ("peak_rank", "prelim_score") if c in names] + d = pq.read_table(psms_path, columns=cols).to_pandas() + if "peak_rank" not in d.columns or not d.candidate_id.duplicated().any(): + return dict(zip(d.candidate_id.astype(int), d.apex_rt.astype(float))) + want = np.array([selected.get((source, int(c)), -1) for c in d.candidate_id], dtype=np.int64) + match = d[(want >= 0) & (d.peak_rank.to_numpy() == want)] + out = dict(zip(match.candidate_id.astype(int), match.apex_rt.astype(float))) + rest = d[~d.candidate_id.isin(list(out))] + if len(rest): + if "prelim_score" in rest.columns: + best = rest.loc[rest.groupby("candidate_id")["prelim_score"].idxmax()] + else: + best = rest.drop_duplicates("candidate_id", keep="last") + out.update(dict(zip(best.candidate_id.astype(int), best.apex_rt.astype(float)))) + return out + + def main(): ap = argparse.ArgumentParser() ap.add_argument("scored") @@ -85,8 +116,14 @@ def main(): psms_paths = a.psms_csv.split(",") n_runs = len(psms_paths) - sc = pq.read_table(a.scored, columns=["candidate_id", "source", "label", "q_value", - "peptidoform", "charge", "protein_group"]).to_pandas() + sc_cols = ["candidate_id", "source", "label", "q_value", "peptidoform", "charge", "protein_group"] + has_selected = "selected_peak_rank" in set(pq.read_schema(a.scored).names) + sc = pq.read_table(a.scored, columns=sc_cols + (["selected_peak_rank"] if has_selected else [])).to_pandas() + # rescore's chosen peak per (source, candidate), for the per-run apex lookup below. + selected = {} + if has_selected: + selected = {(int(s_), int(c)): int(r) for c, s_, r in + zip(sc.candidate_id, sc.source, sc.selected_peak_rank)} # meta per candidate_id (peptidoform/charge/protein_group/label) from any row meta = sc.drop_duplicates("candidate_id").set_index("candidate_id")[ ["peptidoform", "charge", "protein_group", "label"]] @@ -97,11 +134,11 @@ def main(): conf_t = {i: set(sc[(sc.source == i) & (sc.label == "target") & (sc.q_value <= a.q_anchor)].candidate_id) for i in range(n_runs)} - # per-run apex RT (all extracted candidates) + confident-target apex (for maps) + # per-run apex RT (all extracted candidates, the rescore-selected peak where a + # candidate has several) + confident-target apex (for maps) rt_all, rt_anchor = {}, {} for i, p in enumerate(psms_paths): - d = pq.read_table(p, columns=["candidate_id", "apex_rt"]).to_pandas() - m = dict(zip(d.candidate_id, d.apex_rt)) + m = selected_apex_map(p, i, selected) rt_all[i] = m rt_anchor[i] = {c: m[c] for c in conf_t[i] if c in m} @@ -192,16 +229,26 @@ def expected_rt(c, i): target_delta = np.array(rows["rt_delta"]) decoy_delta = np.array(decoy_delta) if len(target_delta) == 0: - print("MBR: no transfer candidates"); pa_write_empty(a.out); return + # The same output contract as a run with transfers: a full-schema empty transfer + # table, and the requested augmented scored table with every row unflagged, so a + # downstream stage never sees a missing file or a one-column placeholder (docs/30). + print("MBR: no transfer candidates") + write_empty_transfers(a.out) + if a.out_scored: + write_unflagged_scored(a.scored, a.out_scored) + return # transfer q via target/decoy competition on rt_delta (smaller = better). At a - # threshold delta, FDR = (#decoy <= delta) / (#target <= delta). q = running min. + # threshold delta, FDR = (#null <= delta + 1) / (#target <= delta), the same +1 + # pseudocount as the engine's `fdr.rs`: without it a pool no permuted residual + # undercuts gets q exactly 0 however small it is, and a three-candidate pool was + # accepted whole at 1% (docs/29 #7). q = running min. order = np.argsort(target_delta) dt = np.sort(target_delta) dd = np.sort(decoy_delta) - dec_cum = np.searchsorted(dd, dt, side="right") # decoys within each delta + dec_cum = np.searchsorted(dd, dt, side="right") # null draws within each delta tgt_cum = np.arange(1, len(dt) + 1) - fdr = dec_cum / tgt_cum + fdr = (dec_cum + 1) / tgt_cum q_sorted = np.minimum.accumulate(fdr[::-1])[::-1] # monotone q from the tail q = np.empty_like(q_sorted); q[order] = q_sorted # map back to row order @@ -248,11 +295,14 @@ def cos(pa_, pb): f"{int((q <= a.q_transfer).sum())} FDR-passing transfers") n_acc = int(accept.sum()) - acc_dec = int(((lab == "decoy") & accept).sum()) - print(f"MBR transfer: candidates={len(cid)} accepted@q<={a.q_transfer}={n_acc} " - f"(target={n_acc-acc_dec}, decoy={acc_dec}, empirical decoy-frac=" - f"{acc_dec/max(1,n_acc)*100:.2f}%)") delta_star = dt[q_sorted <= a.q_transfer].max() if (q_sorted <= a.q_transfer).any() else 0.0 + # The validation statistic is the permuted-RT null, not the decoy label: every + # transfer candidate is a confident target somewhere else, so decoys cannot enter + # this population and a "decoy fraction" over it was structurally zero (docs/29 #6). + null_within = int(np.searchsorted(dd, delta_star, side="right")) if n_acc else 0 + print(f"MBR transfer: candidates={len(cid)} accepted@q<={a.q_transfer}={n_acc}; " + f"permuted-RT null: {len(dd)} draws, {null_within} within the accepted window " + f"(transfer q = (null + 1) / targets, running minimum)") print(f" RT window at q<={a.q_transfer}: {delta_star:.1f}s") for i in range(n_runs): m = accept & (src == i) & (lab == "target") @@ -294,12 +344,38 @@ def cos(pa_, pb): if col in full.columns: full[col] = np.minimum(full[col].to_numpy(dtype=float), tq) full["is_transferred"] = is_tr + # The q the transfer was accepted at, on the transferred rows only, so the + # report can print the acceptance basis next to the untouched grouped q + # (docs/29 #19). NaN on every other row: no transfer, no transfer q. + full["transfer_q"] = np.where(is_tr, tq, np.nan) write_engine_parquet(full, a.out_scored) print(f"wrote {a.out_scored} (augmented scored; {int(is_tr.sum())} rows flagged transferred)") -def pa_write_empty(path): - write_engine_table(pa.table({"candidate_id": pa.array([], pa.uint32())}), path) +def write_empty_transfers(path): + """A zero-row transfer table with the same ten columns a run with transfers writes.""" + write_engine_table(pa.table({ + "candidate_id": pa.array([], pa.uint32()), + "source": pa.array([], pa.uint32()), + "peptidoform": pa.array([], pa.string()), + "charge": pa.array([], pa.int32()), + "protein_group": pa.array([], pa.string()), + "label": pa.array([], pa.string()), + "expected_rt": pa.array([], pa.float64()), + "observed_rt": pa.array([], pa.float64()), + "rt_delta": pa.array([], pa.float64()), + "transfer_q": pa.array([], pa.float64()), + }), path) + + +def write_unflagged_scored(scored_in, scored_out): + """The scored table unchanged, with `is_transferred` false and `transfer_q` NaN on + every row: the augmented schema with no transfer in it.""" + full = pq.read_table(scored_in).to_pandas() + full["is_transferred"] = np.zeros(len(full), dtype=bool) + full["transfer_q"] = np.full(len(full), np.nan) + write_engine_parquet(full, scored_out) + print(f"wrote {scored_out} (augmented scored; 0 rows flagged transferred)") if __name__ == "__main__": diff --git a/scripts/nn_rescore_worker.py b/scripts/nn_rescore_worker.py index d8650b6d..fa9cce19 100644 --- a/scripts/nn_rescore_worker.py +++ b/scripts/nn_rescore_worker.py @@ -18,10 +18,12 @@ MEMORY (multi-run / large PINs): two feature backends behind one accessor. - in-memory (default for PINs <= MUMDIA_NN_STREAM_GB, 4 GB): the full standardised - feature matrix is held in RAM (median/IQR standardisation). + feature matrix is held in RAM. Mean/std standardisation, the same as the streaming + backend: one transform for every backend and every handoff, so a score does not + depend on which one ran (docs/31 F5). - streaming memmap (large PINs, or MUMDIA_NN_STREAM=1): the PIN is read ONCE in - chunks into a disk-backed float32 memmap (mean/std standardisation accumulated - in the same pass); training and scoring then draw MINIBATCHES indexed into the + chunks into a disk-backed float32 memmap (the same mean/std, accumulated in the + same pass); training and scoring then draw MINIBATCHES indexed into the memmap, so peak RAM is one batch + per-row metadata, NOT the whole matrix. This is what makes combining many runs into one rescoring tractable: the full PIN never lives in RAM at once. @@ -165,7 +167,21 @@ def folds_for(peptides, fold_keys, folds, off=0): serve the chunked streaming backend as well as the two whole-table ones. """ if fold_keys is not None: - return (fold_keys[off:off + len(peptides)] % folds).astype(np.int16) + want = len(peptides) + got = fold_keys[off:off + want] + # Numpy slicing past the end returns a SHORT array rather than raising, and a + # short `fold` puts the tail rows in no fold at all: `np.where(fold == f)` cannot + # index them, they are never scored, and they keep the zero initialiser, which the + # final rank-normalisation turns into a plausible tied mid-rank score. The Rust + # caller's completeness contract is satisfied by that, so `rescore.strict` does not + # catch it either (docs/31 F5). + if len(got) != want: + raise SystemExit( + "MUMDIA_NN_FOLD_KEYS has %d rows but the PIN needs at least %d " + "(rows %d..%d); the companion table does not belong to this PIN, and " + "folding on a truncated one would leave the tail unscored." + % (len(fold_keys), off + want, off, off + want)) + return (got % folds).astype(np.int16) return np.array([peptide_fold(x, folds) for x in peptides], np.int16) @@ -333,7 +349,10 @@ def main(): # times smaller than the equivalent text, so the raw file size would understate the # memory a full read actually needs. _md = pq.read_metadata(pin_path) - _nf_guess = max(1, _md.num_columns - 3) # minus SpecId / Label / Peptide + # Count the feature columns by name. Subtracting a hardcoded 3 undercounted the + # non-feature columns, of which NON_FEATURE lists 7, so the estimate that picks the + # backend was biased upward and could stream a matrix that fits (docs/31 F5). + _nf_guess = max(1, sum(1 for c in pq.read_schema(pin_path).names if c not in NON_FEATURE)) filesize = int(_md.num_rows) * _nf_guess * 4 stream_gb = env_f("MUMDIA_NN_STREAM_GB", 4) stream = stream_env in ("1", "on", "true") or ( @@ -475,10 +494,18 @@ def main(): pin[feat_cols].to_numpy(np.float32), nan=0.0, posinf=0.0, neginf=0.0 ) del pin - med = np.median(X, axis=0) - iqr = np.subtract(*np.percentile(X, [75, 25], axis=0)) - iqr[iqr == 0] = 1.0 - Xs = np.clip((X - med) / iqr, -8, 8).astype(np.float32) + # Mean/std, the same transform as the parquet and streaming backends. + # + # This path used median/IQR while the other two used mean/std, so the same PSM + # pool produced different scores depending on `rescore.handoff` and on which + # side of MUMDIA_NN_STREAM_GB the matrix fell: a 3.99 GB PIN was standardised + # one way and a 4.01 GB PIN the other, with nothing in the log to say which + # (docs/31 F5). Converging on mean/std leaves the shipped default (parquet) + # and every published benchmark unchanged, and only moves this legacy path. + mean = X.mean(axis=0, dtype=np.float64).astype(np.float32) + std = X.std(axis=0, dtype=np.float64).astype(np.float32) + std[std == 0] = 1.0 + Xs = np.clip((X - mean) / std, -8, 8).astype(np.float32) del X n = len(y) get = lambda idx: Xs[idx] diff --git a/tests/python/conftest.py b/tests/python/conftest.py index bbd349d8..9924692c 100644 --- a/tests/python/conftest.py +++ b/tests/python/conftest.py @@ -433,13 +433,14 @@ def write_features_parquet(path, psms): # --------------------------------------------------------------------------- -def write_scored_table(path, rows, extra_q=None): +def write_scored_table(path, rows, extra_q=None, extra_int=None): """Write an experiment-wide scored_combined table for `mbr_worker.py`. `rows` holds equal-length sequences for the columns MBR reads (`mbr_worker.py:81-82`): candidate_id, source, label, q_value, peptidoform, charge, protein_group. `extra_q` adds further PSM-level q columns - (`run_psm_q`, `experiment_psm_q`) that the M5 augmentation must also lower. + (`run_psm_q`, `experiment_psm_q`) that the M5 augmentation must also lower; + `extra_int` adds int32 columns such as `selected_peak_rank`. """ cols = { "candidate_id": pa.array( @@ -454,26 +455,29 @@ def write_scored_table(path, rows, extra_q=None): } for name, values in (extra_q or {}).items(): cols[name] = pa.array(np.asarray(values, dtype=np.float64), pa.float64()) + for name, values in (extra_int or {}).items(): + cols[name] = pa.array(np.asarray(values, dtype=np.int32), pa.int32()) pq.write_table(pa.table(cols), str(path), compression="snappy") return path -def write_psms_table(path, candidate_ids, apex_rts): - """Write a per-run psms.parquet: the two columns MBR reads (`mbr_worker.py:96`).""" - pq.write_table( - pa.table( - { - "candidate_id": pa.array( - np.asarray(candidate_ids, dtype=np.uint32), pa.uint32() - ), - "apex_rt": pa.array( - np.asarray(apex_rts, dtype=np.float64), pa.float64() - ), - } - ), - str(path), - compression="snappy", - ) +def write_psms_table(path, candidate_ids, apex_rts, extra_cols=None): + """Write a per-run psms.parquet: the columns MBR reads (`mbr_worker.py`). + + `extra_cols` adds the optional top-K columns (`peak_rank` int32, `prelim_score` + float64) a competed table carries when `extract.retain_top_peaks` is above 1. + """ + cols = { + "candidate_id": pa.array(np.asarray(candidate_ids, dtype=np.uint32), pa.uint32()), + "apex_rt": pa.array(np.asarray(apex_rts, dtype=np.float64), pa.float64()), + } + for name, values in (extra_cols or {}).items(): + arr = np.asarray(values) + if np.issubdtype(arr.dtype, np.integer): + cols[name] = pa.array(arr.astype(np.int32), pa.int32()) + else: + cols[name] = pa.array(arr.astype(np.float64), pa.float64()) + pq.write_table(pa.table(cols), str(path), compression="snappy") return path diff --git a/tests/python/test_augment_library.py b/tests/python/test_augment_library.py new file mode 100644 index 00000000..4467995d --- /dev/null +++ b/tests/python/test_augment_library.py @@ -0,0 +1,40 @@ +"""Contract tests for the id rules of `scripts/augment_library.py` (no engine needed). + +The helper's pipeline runs the engine's digest and predict-frag, which these tests do +not; `assign_base_ids` is the pure rule that decides whether an added form joins an +existing peptide or founds a new one (docs/29 #12). +""" + +from __future__ import annotations + +import importlib.util +import sys + +from conftest import SCRIPTS + + +def _load(): + # The helper imports the shared writer `_lib_io` from its own directory. + if str(SCRIPTS) not in sys.path: + sys.path.insert(0, str(SCRIPTS)) + spec = importlib.util.spec_from_file_location( + "mumdia_augment_library", SCRIPTS / "augment_library.py" + ) + module = importlib.util.module_from_spec(spec) + spec.loader.exec_module(module) + return module + + +def test_an_added_form_of_an_existing_peptide_keeps_that_peptides_base_id(): + """`--match-level peptidoform_charge` adds missing charge states and modforms of + sequences the imported library already has. Offsetting every added base id split + such a peptide across two competition groups and two CV folds; the added forms + must reuse the imported id, and only new sequences get fresh ids, one per sequence. + """ + m = _load() + imported = {"PEPTIDEK": 5, "SAMPLER": 9} + added = ["PEPTIDEK", "NEWSEQK", "SAMPLER", "NEWSEQK", "OTHERK", "PEPTIDEK"] + ids = m.assign_base_ids(imported, added, next_id=12) + assert ids == [5, 12, 9, 12, 13, 5] + # Nothing added: nothing allocated. + assert m.assign_base_ids(imported, [], next_id=12) == [] diff --git a/tests/python/test_entrapment_worker.py b/tests/python/test_entrapment_worker.py index 9d0ed501..b09d99d6 100644 --- a/tests/python/test_entrapment_worker.py +++ b/tests/python/test_entrapment_worker.py @@ -179,3 +179,39 @@ def test_single_class_input_fails_instead_of_scoring(sklearn_available, tmp_path assert rc != 0 assert "entrapment" in err.lower() assert not out.exists() + + +def test_a_single_class_training_fold_is_an_error_not_an_in_sample_fill(sklearn_available, tmp_path): + """Two groups, one all real and one all entrapment: every training fold is + single-class. The worker used to skip such folds and score their held-out rows + with the final model trained on those very rows (docs/29 #3). It must refuse, + naming the condition, rather than hand in-sample scores to the FDR estimate. + """ + n_real, n_ent, n_dec = 40, 40, 10 + n_rows = n_real + n_ent + n_dec + rng = np.random.default_rng(1) + feats = rng.normal(0.0, 1.0, size=(n_rows, N_FEATURES)) + feats[:n_real, 0] += 3.0 + is_entrapment = np.zeros(n_rows, dtype=np.int32) + is_entrapment[n_real:n_real + n_ent] = 1 + is_decoy = np.zeros(n_rows, dtype=np.int32) + is_decoy[n_real + n_ent:] = 1 + # group 0 = every real target, group 1 = every entrapment PSM, group 2 = decoys. + groups = np.concatenate([np.zeros(n_real), np.ones(n_ent), np.full(n_dec, 2)]).astype(np.uint32) + cols = { + "row_id": pa.array(np.arange(n_rows, dtype=np.uint32), pa.uint32()), + "candidate_id": pa.array(np.arange(n_rows, dtype=np.uint32), pa.uint32()), + "base_peptide_id": pa.array(groups, pa.uint32()), + "is_entrapment": pa.array(is_entrapment, pa.int32()), + "is_decoy": pa.array(is_decoy, pa.int32()), + } + for j in range(N_FEATURES): + cols["feat_{}".format(j)] = pa.array(feats[:, j], pa.float64()) + inp = tmp_path / "entrapment_in.parquet" + pq.write_table(pa.table(cols), str(inp), compression="snappy") + out = tmp_path / "entrapment_out.parquet" + + rc, _, err = run_worker("entrapment_worker.py", inp, out, 2) + assert rc != 0, "a single-class training fold must fail the worker" + assert "single" in err and "class" in err, err + assert not out.exists(), "no output may be written for a refused run" diff --git a/tests/python/test_mbr_worker.py b/tests/python/test_mbr_worker.py index d9bce3ca..3b3ea1cf 100644 --- a/tests/python/test_mbr_worker.py +++ b/tests/python/test_mbr_worker.py @@ -29,6 +29,7 @@ from __future__ import annotations import numpy as np +import pyarrow as pa import pyarrow.parquet as pq import pytest @@ -333,6 +334,18 @@ def test_m5_sets_is_transferred_on_exactly_the_accepted_rows(mbr_dataset, mbr_re got = np.asarray(after["is_transferred"], dtype=bool) assert np.array_equal(got, expect) + # The acceptance basis travels with the flag: `transfer_q` is the accepted q on + # exactly the flagged rows and NaN everywhere else (docs/29 #19). + tq = np.asarray(after["transfer_q"], dtype=float) + assert np.array_equal(np.isfinite(tq), expect) + tr = read_columns(mbr_result["transferred"]) + accepted_q = {k: q for k, q in zip( + zip((int(c) for c in tr["candidate_id"]), (int(s) for s in tr["source"])), + (float(q) for q in tr["transfer_q"]))} + for k, flagged, q in zip(key, expect, tq): + if flagged: + assert q == accepted_q[k] + def test_m5_touches_only_the_matching_candidate_id_and_source(mbr_dataset, mbr_result): """Only the `(candidate_id, source)` row of an accepted transfer may change. @@ -512,13 +525,15 @@ def test_binned_map_removes_a_systematic_inter_run_rt_offset( ) -def test_no_transfer_candidates_writes_an_empty_table_and_no_scored_table(tmp_path): - """With nothing to transfer the worker exits 0, and `--out-scored` is skipped. +def test_no_transfer_candidates_writes_the_full_schema_and_an_unflagged_scored_table(tmp_path): + """With nothing to transfer the worker exits 0 and keeps its output contract. - `mbr_worker.py:187-188` returns before the M5 block, so a caller that - passed `--out-scored` gets no file. A downstream quant pointed at that path - fails on a missing input rather than on a nonzero MBR exit, so the true - cause is not in the MBR log; pinning the behaviour keeps that documented. + It used to return before the M5 block: a caller that passed `--out-scored` got no + file, and the transfer table had a single `candidate_id` column. Downstream stages + then failed on a missing input or an unexpected schema instead of reading an empty + result (docs/30). Now the transfer table carries all ten columns with zero rows, and + the augmented scored table is the input with `is_transferred` false and `transfer_q` + NaN on every row. """ ids = list(range(20)) cols = {k: [] for k in ("candidate_id", "source", "label", "q_value", @@ -545,8 +560,19 @@ def test_no_transfer_candidates_writes_an_empty_table_and_no_scored_table(tmp_pa "--out-scored", scored_out, ) assert "no transfer candidates" in stdout - assert pq.read_table(str(out)).num_rows == 0 - assert not scored_out.exists() + transfers = pq.read_table(str(out)) + assert transfers.num_rows == 0 + assert transfers.column_names == [ + "candidate_id", "source", "peptidoform", "charge", "protein_group", "label", + "expected_rt", "observed_rt", "rt_delta", "transfer_q", + ] + assert scored_out.exists(), "--out-scored is honoured even with nothing to transfer" + after = read_columns(scored_out) + assert len(after["candidate_id"]) == len(cols["candidate_id"]) + assert not np.asarray(after["is_transferred"], dtype=bool).any() + assert np.isnan(np.asarray(after["transfer_q"], dtype=float)).all() + for col in ("q_value", "peptidoform"): + assert list(after[col]) == list(cols[col]), "{} must be unchanged".format(col) def test_missing_psms_path_fails_loudly(mbr_dataset, tmp_path): @@ -565,3 +591,115 @@ def test_missing_psms_path_fails_loudly(mbr_dataset, tmp_path): ) assert rc != 0 assert "does_not_exist" in err or "No such file" in err or "FileNotFound" in err + + +# --------------------------------------------------------------------------- +# small pools and the selected peak (docs/29 #7, #8) +# --------------------------------------------------------------------------- + + +def _small_dataset(d, n_cand, run0_apex, run0_extra=None, selected_peak_rank=None): + """`n_cand` candidates confident in runs 1 and 2, extracted sub-threshold in run 0. + + `run0_apex(cid)` gives run 0's observed apex; `run0_extra` appends rows to run 0's + table (candidate_id, apex_rt, peak_rank, prelim_score) to model a second retained + peak; `selected_peak_rank` maps candidate_id -> rescore's chosen rank for source 0. + """ + ids = list(range(n_cand)) + rows = {k: [] for k in ("candidate_id", "source", "label", "q_value", + "peptidoform", "charge", "protein_group")} + sel = [] + for cid in ids: + for src in range(3): + rows["candidate_id"].append(cid) + rows["source"].append(src) + rows["label"].append("target") + rows["q_value"].append(0.5 if src == 0 else 0.001) + rows["peptidoform"].append("PEP{}K".format(cid)) + rows["charge"].append(2) + rows["protein_group"].append("PG{}".format(cid % 3)) + sel.append(selected_peak_rank.get(cid, 0) if (selected_peak_rank and src == 0) else 0) + extra_int = {"selected_peak_rank": sel} if selected_peak_rank is not None else None + scored = write_scored_table(d / "scored.parquet", rows, extra_int=extra_int) + base = lambda cid: 200.0 + 5.0 * cid + run0_ids = list(ids) + run0_rt = [run0_apex(c) for c in ids] + extra = None + if run0_extra: + extra = {"peak_rank": [0] * len(ids), "prelim_score": [10.0] * len(ids)} + for cid, rt, rank, score in run0_extra: + run0_ids.append(cid) + run0_rt.append(rt) + extra["peak_rank"].append(rank) + extra["prelim_score"].append(score) + psms = [ + write_psms_table(d / "psms_0.parquet", run0_ids, run0_rt, extra_cols=extra), + write_psms_table(d / "psms_1.parquet", ids, [base(c) + 0.02 for c in ids]), + write_psms_table(d / "psms_2.parquet", ids, [base(c) - 0.02 for c in ids]), + ] + return {"scored": scored, "psms_csv": ",".join(str(p) for p in psms)} + + +def _derangement_seed(n): + """A seed whose first `permutation(n)` has no fixed point, so no permuted + residual coincides with a real one and the null count is exactly zero.""" + for seed in range(1, 10_000): + perm = np.random.default_rng(seed).permutation(n) + if not np.any(perm == np.arange(n)): + return seed + raise AssertionError("no derangement seed found") + + +def test_a_tiny_concordant_pool_is_not_accepted_at_one_percent(tmp_path): + """Three candidates, every observed apex within 0.01 s of its prediction and no + permuted residual as small: the ratio (null <= delta) / targets is exactly 0 for + all of them, and the worker used to accept all three at 1% (docs/29 #7). With the + engine's +1 pseudocount the best q a three-candidate pool can reach is 1/3. + """ + d = tmp_path / "tiny" + d.mkdir() + ds = _small_dataset(d, 3, lambda cid: 200.0 + 5.0 * cid + 0.01) + seed = _derangement_seed(3) + strict = d / "strict.parquet" + run_worker_ok("mbr_worker.py", ds["scored"], ds["psms_csv"], strict, + "--q-anchor", 0.01, "--min-anchor-runs", 2, "--q-transfer", 0.01, "--seed", seed) + assert pq.read_table(strict).num_rows == 0, "1/3 is not <= 0.01" + + loose = d / "loose.parquet" + run_worker_ok("mbr_worker.py", ds["scored"], ds["psms_csv"], loose, + "--q-anchor", 0.01, "--min-anchor-runs", 2, "--q-transfer", 0.5, "--seed", seed) + t = read_columns(loose) + assert len(t["candidate_id"]) == 3 + assert all(abs(float(q) - 1.0 / 3.0) < 1e-9 for q in t["transfer_q"]), t["transfer_q"] + + +def test_the_transfer_is_measured_on_the_rescore_selected_peak(tmp_path): + """Candidate 0 has two retained peaks in run 0: rank 1 at the concordant RT and, + listed last, rank 0 five hundred seconds away. Rescore selected rank 1. The worker + used to keep the last row per candidate (docs/29 #8), measured the transfer on the + wrong peak and rejected it; it must use the selected peak and accept. + """ + d = tmp_path / "selected" + d.mkdir() + n = 40 + concordant = lambda cid: 200.0 + 5.0 * cid + 0.01 + ds = _small_dataset( + d, n, concordant, + run0_extra=[(0, concordant(0) + 500.0, 0, 99.0)], + selected_peak_rank={0: 1}, + ) + # The concordant row of candidate 0 is rank 1 (the base rows carry rank 0), so + # rewrite run 0's peak_rank for that first row. + p0 = d / "psms_0.parquet" + tbl = pq.read_table(p0).to_pandas() + tbl.loc[(tbl.candidate_id == 0) & (tbl.apex_rt < 400.0), "peak_rank"] = 1 + pq.write_table(pa.Table.from_pandas(tbl, preserve_index=False), str(p0), compression="snappy") + + out = d / "transferred.parquet" + run_worker_ok("mbr_worker.py", ds["scored"], ds["psms_csv"], out, + "--q-anchor", 0.01, "--min-anchor-runs", 2, "--q-transfer", 0.05, + "--seed", _derangement_seed(n)) + t = read_columns(out) + accepted = {int(c): float(rt) for c, rt in zip(t["candidate_id"], t["observed_rt"])} + assert 0 in accepted, "the selected concordant peak must carry the transfer" + assert abs(accepted[0] - concordant(0)) < 1e-6, accepted[0] diff --git a/tests/python/test_nn_rescore_worker.py b/tests/python/test_nn_rescore_worker.py index 79afd07a..5fa31304 100644 --- a/tests/python/test_nn_rescore_worker.py +++ b/tests/python/test_nn_rescore_worker.py @@ -237,6 +237,32 @@ def test_explicit_fold_keys_pair_a_target_with_its_decoy(): assert hashed[0] != hashed[1] +def test_a_short_fold_key_file_is_refused_rather_than_leaving_rows_unfolded(): + """A fold-key companion shorter than the PIN must stop the run. + + Numpy slicing past the end returns a SHORT array instead of raising, and a short + `fold` puts the tail rows in no fold at all: `np.where(fold == f)` cannot reach + them, they are never scored, and they keep the zero initialiser, which the final + rank-normalisation turns into a plausible tied mid-rank score. The Rust caller's + completeness contract is satisfied by that, so `rescore.strict` does not catch it + either (docs/31 F5). + """ + w = _import_worker() + np = pytest.importorskip("numpy") + + keys = np.arange(6, dtype=np.uint32) + peptides = [f"p{i}" for i in range(10)] + with pytest.raises(SystemExit) as exc: + w.folds_for(peptides, keys, 3) + assert "MUMDIA_NN_FOLD_KEYS" in str(exc.value) + # A companion that covers the rows is unaffected, at an offset too. + full = np.arange(10, dtype=np.uint32) + assert len(w.folds_for(peptides, full, 3)) == 10 + assert len(w.folds_for(peptides[7:], full, 3, off=7)) == 3 + with pytest.raises(SystemExit): + w.folds_for(peptides[7:], keys, 3, off=7) + + def test_fold_keys_respect_the_streaming_row_offset(): """The chunked backend passes a flat row offset; the keys must be sliced by it.