diff --git a/docs/development/adr/0002-unified-pose-file-format.md b/docs/development/adr/0002-unified-pose-file-format.md new file mode 100644 index 00000000..8c27e0bc --- /dev/null +++ b/docs/development/adr/0002-unified-pose-file-format.md @@ -0,0 +1,1239 @@ +# ADR 0002: Unified Pose File Format + +## Status + +Proposed + +## Context + +The pose format has been extended seven times. Each extension was reasonable in isolation; the +accumulated result is that every consumer must know which version it is reading, and the version is +encoded in the *filename*. Reading a pose file today means dispatching to one of seven classes +(`pose_est_v2.py` … `pose_est_v8.py`), each of which knows a slightly different dialect of the same +idea. + +The dialect differences are not cosmetic: + +- **Coordinate order is per-dataset.** `poseest/points` is (y, x); `static_objects/corners` is + (x, y); `poseest/seg_data` is (x, y); `dynamic_objects/fecal_boli/points` is (y, x). The + producer's own specification says of dynamic objects that "x,y or y,x sorting may be different + per-static object." +- **Padding collides with data.** `poseest/points` pads with `0`, which is a valid pixel + coordinate. Validity is therefore recovered from `confidence > 0`, and then re-derived a second + time by each consumer applying its own threshold — JABS uses `MINIMUM_CONFIDENCE = 0.3`. +- **One integer carries three meanings.** In `poseest/instance_embed_id`, `0` means invalid, + `1…N` means identity, and `> N` means "valid instance we could not assign to an identity." +- **The coordinate space is not recorded.** No version stores frame width or height. A pose file + cannot state the pixel space its own coordinates live in, so it cannot be validated, rendered or + sanity-checked without the video beside it. + +### The tell: one transform, written three times + +Because slots in `points` are detector order rather than identity, every consumer that wants +per-animal data performs the same scatter-and-flip. That code exists independently in three +repositories: + +| Location | What it does | +|---|---| +| `src/jabs/pose_estimation/pose_est_v4.py` | scatters by `instance_embed_id`, flips (y,x)→(x,y), NaNs sub-threshold points | +| `src/jabs/pose_estimation/pose_est_v4.py` (`_cache_poses`) | writes the result to a second HDF5 file per video | +| `JABS-postprocess/.../analysis_utils/clip_utils.py` | reimplements the same scatter and flip | + +JABS additionally maintains an entire per-video cache file whose only purpose is to memoize that +transform, plus `jabs/cache/pose_attribute_cache.json` to memoize the small attributes that a +project-load scan would otherwise reread from every pose file. + +### What the files actually contain + +Measured from a one-hour single-mouse `pose_est_v6.h5` (108,150 frames, 33 MB on disk): + +| dataset | stored | uncompressed | read by | +|---|---:|---:|---| +| `poseest/points` | 5.19 MB | 5.19 MB | everything | +| `poseest/confidence` | 5.19 MB | 5.19 MB | everything | +| `poseest/id_mask` | 0.11 MB | 0.11 MB | everything (v4+) | +| `poseest/instance_embed_id` | 0.43 MB | 0.43 MB | everything (v4+) | +| `poseest/instance_id_center` | <0.01 MB | <0.01 MB | JABS (shape only); JABS-postprocess (data) | +| `poseest/identity_embeds` | 0.43 MB | 0.43 MB | JABS-postprocess (`attrs["network"]`) | +| `poseest/instance_track_id` | 0.43 MB | 0.43 MB | clip utilities (v3 path) | +| `poseest/instance_embedding` | 5.19 MB | 5.19 MB | nothing outside producer tests | +| `poseest/seg_data` | 16.52 MB | **1,184 MB** | segmentation features | + +Two observations shaped this design. `instance_embedding` is 5.19 MB per mouse-hour that no +consumer reads — but `identity_embeds` and `instance_id_center`, which look equally vestigial from +inside JABS, are read by `JABS-postprocess` for cross-video identity linking. A redesign that +trusted one repository's usage would have deleted them. + +And `seg_data` expands 72× on read, because the array's shape must accommodate the single longest +contour in the whole video: + +| file | real data | dense uncompressed | stored | +|---|---:|---:|---:| +| v6 · 1 mouse · 1 hour | 7.3% | 1,184 MB | 16.5 MB | +| v8 · 3 mice · 60 seconds | 0.82% | 1,123 MB | 3.4 MB | + +### Identity slots are already aligned — the format just never promised it + +| file | version | slots | valid instances in the wrong slot | +|---|---|---:|---| +| 1-mouse 1 hr (JABS-pose) | v6 | 1 | 0 / 108,150 | +| 3-mouse clips ×3 (JABS-pose) | v8 | 3 | 0 / ~3,600 each | +| multi-mouse test file | v6 | 5 | 1,102 / 2,444 | +| multi-mouse samples ×2 | v5 | 5 | 10,914 / 14,393 | + +Current inference recycles instance ids (`"recycle_instance_ids": true` in v8 +`model_metadata_json`), so slot *k* already holds identity *k*. Older multi-animal files do not. +Because the format never guaranteed it, every consumer scatters defensively — and against v5/v6 +files it genuinely must. + +### Requirements + +Hard requirements, agreed with the JABS tech lead: + +1. Readable on HPC with `h5py` alone — no service, no special runtime. +2. Partial / range reads over cloud object storage, for the in-browser pose overlay. +3. One file per video. +4. Efficient whole-file sequential reads for training and feature extraction. + +Nice to have: a community-standard archival form (satisfied by an NWB converter, lossy accepted — +see `docs/development/jabs-nwb-format.md`), and enough self-description that a collaborator +without JABS can find the keypoints. + +Requirements 2 and 4 pull in opposite directions and together decide the chunking policy. +Requirement 1 plus "one file" rules out a directory-based store such as Zarr. + +## Decision + +**One format, one reader, forever additive.** A file declares its contents in a manifest; a reader +asks what a file *contains*, never how old it is. Data JABS does not understand is a first-class +citizen with declared axes, not an escape hatch. + +### Design goals + +1. **Scope is the per-video pose file.** Inference output only, read-only once written. Labels, + features, predictions and classifiers stay where they are. +2. **One HDF5 file**, readable with `h5py` and nothing else. +3. **Identified by root attributes, never by filename.** `