Add common media file definitions for BEP044/BEP047 #2367
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,184 @@ | ||
| # Media Files | ||
|
|
||
| ## Introduction | ||
|
|
||
| Several BIDS datatypes make use of media files — audio recordings, video recordings, | ||
| combined audio-video recordings, and still images. | ||
| This appendix defines the common file formats, metadata conventions, | ||
| and codec identification schemes shared across all datatypes that use media files. | ||
|
|
||
| Datatypes that incorporate media files (for example, behavioral recordings or stimuli) | ||
| define their own file-naming rules, directory placement, and datatype-specific metadata. | ||
| The conventions described here apply uniformly to all such datatypes. | ||
|
|
||
| ### Relationship to the `photo` suffix | ||
|
|
||
| The media file definitions introduced here generalize the concept of all media in BIDS. | ||
| The existing `photo` suffix (used for photographs of anatomical landmarks, | ||
| head localization coils, and tissue samples) predates this framework and covers | ||
| a narrower use case — still images in specific electrophysiology and microscopy datatypes. | ||
|
|
||
| The media suffixes (`audio`, `video`, `audiovideo`, `image`) are intended as the | ||
| general-purpose mechanism for all media content in BIDS. | ||
| In practice, a "photo" could equally be a video of an experimental setup with verbal | ||
| narration, an audio recording describing electrode placement, or a drawing rather than | ||
| a photograph. | ||
| The media file framework should be generally adopted for new datatypes, | ||
| and a future proposal may deprecate the `photo` suffix in favor of the broader `image` | ||
| suffix with appropriate migration tooling | ||
| (see [bids-utils](https://github.com/bids-standard/bids-utils)). | ||
|
|
||
| ## Supported Formats | ||
|
|
||
| ### Audio formats | ||
|
|
||
| | Format | Extension | Description | | ||
| | ---------------------- | --------- | --------------------------------------------- | | ||
| | Waveform Audio (WAV) | `.wav` | Uncompressed PCM audio; lossless, large files | | ||
| | MP3 | `.mp3` | Lossy compressed audio; widely supported | | ||
| | Advanced Audio Coding | `.aac` | Lossy compressed audio; successor to MP3 | | ||
| | Ogg Vorbis | `.ogg` | Open lossy compressed audio format | | ||
|
yarikoptic marked this conversation as resolved.
Outdated
|
||
|
|
||
| ### Video container formats | ||
|
|
||
| | Format | Extension | Description | | ||
| | ---------------------- | --------- | ---------------------------------------- | | ||
| | MPEG-4 Part 14 | `.mp4` | Widely supported multimedia container | | ||
| | Audio Video Interleave | `.avi` | Legacy multimedia container | | ||
| | Matroska | `.mkv` | Open, flexible multimedia container | | ||
| | WebM | `.webm` | Open format optimized for web delivery | | ||
|
|
||
| ### Image formats | ||
|
|
||
| | Format | Extension | Description | | ||
| | ------------------------- | --------------- | -------------------------------------------- | | ||
| | JPEG | `.jpg` | Lossy compressed photographic images | | ||
| | Portable Network Graphics | `.png` | Lossless compressed images with transparency | | ||
| | Scalable Vector Graphics | `.svg` | XML-based vector image format | | ||
| | WebP | `.webp` | Modern format supporting lossy and lossless | | ||
| | Tag Image File Format | `.tif`, `.tiff` | Lossless format common in scientific imaging | | ||
|
|
||
| When choosing a format, consider the trade-off between file size and data fidelity. | ||
|
yarikoptic marked this conversation as resolved.
Outdated
|
||
| Uncompressed or lossless formats (WAV, PNG, TIFF) preserve full quality | ||
| but produce larger files. | ||
| Lossy formats (MP3, AAC, JPEG) significantly reduce file size | ||
| at the cost of some data loss. | ||
|
|
||
| ## Media Stream Metadata | ||
|
|
||
| Media files SHOULD be accompanied by a JSON sidecar file | ||
| containing technical metadata about the media streams. | ||
| The following metadata fields are defined for media files: | ||
|
|
||
| ### Duration | ||
|
|
||
| | Field | Suffix | Requirement Level | | ||
| | ------------------- | ------------------------------ | ----------------- | | ||
| | `RecordingDuration` | `audio`, `video`, `audiovideo` | RECOMMENDED | | ||
|
|
||
| `RecordingDuration` is the total duration of the media file in seconds. | ||
| This reuses the existing BIDS metadata field already defined for | ||
| electrophysiology recordings (EEG, iEEG, MEG, and others). | ||
|
|
||
| ### Audio stream properties | ||
|
|
||
| | Field | Suffix | Requirement Level | | ||
| | ------------------- | --------------------- | ----------------- | | ||
| | `AudioCodec` | `audio`, `audiovideo` | RECOMMENDED | | ||
| | `AudioSampleRate` | `audio`, `audiovideo` | RECOMMENDED | | ||
| | `AudioChannelCount` | `audio`, `audiovideo` | RECOMMENDED | | ||
| | `AudioCodecRFC6381` | `audio`, `audiovideo` | OPTIONAL | | ||
|
yarikoptic marked this conversation as resolved.
Outdated
|
||
|
|
||
| Note: `AudioSampleRate` is used instead of the existing `SamplingFrequency` field | ||
| because audio-video files require distinguishing the audio sampling rate from the | ||
| video frame rate. The `Audio` prefix makes this unambiguous in multi-stream containers. | ||
|
|
||
| ### Visual properties | ||
|
yarikoptic marked this conversation as resolved.
Outdated
|
||
|
|
||
| | Field | Suffix | Requirement Level | | ||
| | -------- | ----------------------------------- | ----------------- | | ||
| | `Width` | `video`, `audiovideo`, `image` | RECOMMENDED | | ||
| | `Height` | `video`, `audiovideo`, `image` | RECOMMENDED | | ||
|
|
||
| ### Video stream properties | ||
|
yarikoptic marked this conversation as resolved.
|
||
|
|
||
|
yarikoptic marked this conversation as resolved.
|
||
| | Field | Suffix | Requirement Level | | ||
| | ------------------- | --------------------- | ----------------- | | ||
| | `VideoCodec` | `video`, `audiovideo` | RECOMMENDED | | ||
| | `FrameRate` | `video`, `audiovideo` | RECOMMENDED | | ||
|
There was a problem hiding this comment. The proposal includes The spec should indicate whether There was a problem hiding this comment. Partial progress: the field is now
Contributor
There was a problem hiding this comment. I think it's nice to have both the approximate framerate and the |
||
| | `VideoCodecRFC6381` | `video`, `audiovideo` | OPTIONAL | | ||
|
|
||
|
yarikoptic marked this conversation as resolved.
|
||
| ## Codec Identification | ||
|
|
||
| Codec identification uses two complementary naming systems: | ||
|
|
||
| ### FFmpeg codec names (RECOMMENDED) | ||
|
|
||
| The `AudioCodec` and `VideoCodec` fields use | ||
| [FFmpeg codec names](https://www.ffmpeg.org/ffmpeg-codecs.html) as the RECOMMENDED | ||
| convention. These names are the de facto standard in scientific computing and can be | ||
| auto-extracted from media files using: | ||
|
|
||
| ```bash | ||
| ffprobe -v quiet -print_format json -show_streams <file> | ||
| ``` | ||
|
|
||
| ### RFC 6381 codec strings (OPTIONAL) | ||
|
|
||
| The `AudioCodecRFC6381` and `VideoCodecRFC6381` fields use | ||
| [RFC 6381](https://datatracker.ietf.org/doc/html/rfc6381) codec strings. | ||
| These provide precise codec profile and level information useful for | ||
| web and broadcast interoperability. | ||
|
|
||
| ### Common codec reference | ||
|
|
||
| | Codec | FFmpeg Name | RFC 6381 String | Notes | | ||
| | -------------- | ----------- | ------------------ | ----------------------- | | ||
| | H.264 / AVC | `h264` | `avc1.640028` | Most widely supported | | ||
| | H.265 / HEVC | `hevc` | `hev1.1.6.L93.B0` | High efficiency | | ||
| | VP9 | `vp9` | `vp09.00.10.08` | Open, royalty-free | | ||
| | AV1 | `av1` | `av01.0.01M.08` | Next-gen open codec | | ||
| | AAC-LC | `aac` | `mp4a.40.2` | Default audio for MP4 | | ||
| | MP3 | `mp3` | `mp4a.6B` | Legacy lossy audio | | ||
| | Opus | `opus` | `Opus` | Open, low-latency audio | | ||
| | FLAC | `flac` | `fLaC` | Open lossless audio | | ||
| | PCM 16-bit LE | `pcm_s16le` | — | Uncompressed (WAV) | | ||
|
|
||
| The FFmpeg name column shows the value to use for `VideoCodec` or `AudioCodec`. | ||
| The RFC 6381 column shows the value for `VideoCodecRFC6381` or `AudioCodecRFC6381`. | ||
| RFC 6381 strings vary by profile and level; | ||
| the values shown are representative examples. | ||
|
|
||
| ## Privacy Considerations | ||
|
|
||
| Media files — particularly audio and video recordings — may contain | ||
| personally identifiable information (PII), including but not limited to: | ||
|
|
||
| - Voices and speech content | ||
| - Facial features and other physical characteristics | ||
| - Background environments that could identify locations | ||
| - Metadata embedded in file headers (for example, GPS coordinates, device identifiers) | ||
|
|
||
| Researchers MUST ensure that sharing of media files complies with the | ||
| informed consent obtained from participants and with applicable privacy regulations. | ||
| De-identification techniques (for example, voice distortion, face blurring, | ||
| metadata stripping) SHOULD be applied where appropriate before data sharing. | ||
|
bendichter marked this conversation as resolved.
|
||
|
|
||
| ## Example | ||
|
|
||
| A complete sidecar JSON file for an audio-video recording: | ||
|
|
||
| ```json | ||
| { | ||
| "RecordingDuration": 312.5, | ||
| "VideoCodec": "h264", | ||
| "VideoCodecRFC6381": "avc1.640028", | ||
| "FrameRate": 30, | ||
| "Width": 1920, | ||
| "Height": 1080, | ||
| "AudioCodec": "aac", | ||
| "AudioCodecRFC6381": "mp4a.40.2", | ||
| "AudioSampleRate": 48000, | ||
| "AudioChannelCount": 2 | ||
| } | ||
| ``` | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.