Evaluate prediction postprocessing during cross-validation

[gbeane]

KLAUS-588

Adds an option to measure how prediction postprocessing (stitching, duration filtering, gap interpolation) affects classifier performance, by reporting postprocessed metrics alongside the raw ones in cross-validation.

Draft: the design decisions below are worth agreeing on before review effort goes into the details.

Where the setting lives

A per-behavior Cross-Validation Evaluation group at the bottom of the Tools → Prediction Postprocessing dialog, stored as behavior[<name>].evaluate_postprocessing_in_cv.

It went there rather than into Project Settings because the postprocessing stage configuration is itself behavior-scoped, the toggle is meaningless without seeing which stages are enabled next to it, and postprocessing is binary-only while Project Settings is shared with multi-class mode. No new controls on the main window.

Why the held-out tracks are re-predicted

This is the part most worth scrutinising.

Cross-validation normally scores a fold using only the labeled frames of the held-out group. Those rows are a sparse, gap-collapsed subset of the video: collect_binary_labeled_features keeps only labeled frames, and _extract_identity_features returns a DataFrame with a fresh RangeIndex, so frame numbers are gone and two labeled bouts thousands of frames apart become adjacent rows. Under Video or Filename Pattern grouping a fold also concatenates several identities back to back.

Running the pipeline on that array would stitch gaps that do not exist and measure bout durations in "labeled frames" rather than frames. The numbers would look entirely plausible and be wrong, which is worse than not shipping the feature.

So each fold instead re-predicts the held-out group's full tracks the way the classify path does, applies the pipeline to the full-length vectors, and only then restricts to the labeled frames where ground truth exists. That also means the -1 (no-pose) runs GapInterpolationStage fills actually exist — on the labeled-rows-only path they never do, so that stage would silently be a no-op.

tests/classifier/test_cv_postprocessing.py::test_postprocessing_uses_full_track_not_just_labeled_frames pins this: two 1-frame gaps that stitch over the full track would form one unstitchable 2-frame gap on the labeled rows alone, so the test fails if the implementation ever regresses to the sparse approach.

Implementation

• jabs/classifier/inference.py — predict_identity(), extracted from ClassifyThread so cross-validation and classification share one implementation and cannot drift. This is what makes the reported numbers reflect what classification actually produces.
• jabs/classifier/cv_postprocessing.py — loads ground truth, applies the pipeline to full-length prediction vectors, masks to labeled frames. Members are processed grouped by video so each pose file opens once, and one identity's features are released before the next loads (a full-video feature matrix for a long video runs to roughly a gigabyte, so a whole group's worth at once is not viable).
• Project._assign_cv_group_ids now records a members list of (video, identity) pairs per group. Video and Filename Pattern entries carried identity: None, so there was previously no way to know which identities to re-predict.
• Postprocessed metrics use an explicit labels=[0, 1]. Postprocessing can in principle leave a frame with no prediction, and letting sklearn infer the label set would make -1 its own class and silently shift which array element belongs to which metric.
• Report: optional PostprocessedMetrics on BinaryCVResult, rendered as summary means plus a second iteration table, with the evaluated stage configuration recorded so a saved report is self-describing. Raw metrics are untouched so the two are directly comparable, and the session tracker still receives raw accuracy.
• jabs-cli cross-validation honours the same behavior setting, with --postprocessing/--no-postprocessing to override, and two comparison columns in its console table.

Cost

Each group is the held-out group in exactly one fold, so the extra prediction work totals one pass over the labeled identities per training run — not one pass per fold. Features come from the cache the preceding labeled-features pass already wrote, so the per-fold cost is a disk read plus predict_proba rather than a recompute. A cold or invalidated feature cache makes the first run slower; that is called out in the dialog help text and the user guide. Progress still advances once per fold, with per-identity status text.

Guard rails

• Skipped in multi-class mode, where the pipeline has no semantics.
• Skipped when no stages are enabled, since the pipeline would be a no-op not worth the cost.
• A consistency check compares the full-sequence pass's raw accuracy against the fold's own raw accuracy and warns on disagreement, rather than quietly producing incomparable numbers.

Drive-by fix

SettingsManager.save_behavior did merged_data = all_behavior_data.get(behavior, defaults) and then mutated the result, so saving settings for a behavior with no entry yet wrote them straight into the shared project defaults dict. Verified against a real project file: enabling this toggle on a new behavior would have made it the default for every behavior created afterwards. Fixed by copying, with a regression test. The bug already affected postprocessing and every other behavior setting, so this is broader than the feature needed.

Testing

New: tests/classifier/test_inference.py, tests/classifier/test_cv_postprocessing.py. Extended: the cross-validation, training-report, settings-manager, CV-grouping, settings-dialog, training-thread and CLI test modules.

Rebased onto 93991bcc and re-verified in the full environment (uv sync --all-packages --all-extras --all-groups): root 1105, jabs-io 299, jabs-vision 177, jabs-core 88, jabs-behavior 113 — all passing. ruff check and ruff format --check clean. The new setting was also round-tripped through the real SettingsManager rather than only a fake.

Docs

The in-app user guide page (src/jabs/resources/docs/user_guide/postprocessing.md) documents the option. The published site has no postprocessing page at all, tracked separately in KLAUS-587.