[ENH] BEP047 - Add audio/video recordings to behavioral experiments

src/modality-specific-files/behavioral-experiments.md

@@ -1,4 +1,4 @@
-# Behavioral experiments (with no neural recordings)
+# Behavioral experiments
 
 !!! example "Example datasets"
 
@@ -15,16 +15,12 @@ and a guide for using macros can be found at
 -->
 {{ MACROS___make_filename_template("raw", datatypes=["beh"]) }}
 
-In addition to logs from behavioral experiments
-performed alongside imaging data acquisitions,
-one MAY also include data from experiments
-performed with no neural recordings.
-The results of those experiments MAY be stored in the `beh` directory
-using the same formats for event timing (`_events.tsv`),
-metadata (`_events.json`),
-physiological (`_physio.tsv.gz`, `_physio.json`)
-and other continuous recordings (`_stim.tsv.gz`, `_stim.json`)
-as for tasks performed during MRI, electrophysiological or other neural recordings.
+Behavioral data MAY be stored in the `beh` directory,
+The `beh` directory uses standard formats for event timing (`_events.tsv`),
+metadata (`_events.json`), video recordings (`_video.*`), audio recordings (`_audio.*`),
+physiological recordings (`_physio.tsv.gz`, `_physio.json`),
+and other continuous recordings (`_stim.tsv.gz`, `_stim.json`).
+
 Additionally, events files
 that do not include the mandatory `onset` and `duration` columns
 MAY be included,
@@ -76,6 +72,141 @@ A guide for using macros can be found at
 -->
 {{ MACROS___make_sidecar_table("beh.BEHInstitutionInformation") }}
 
+## Audio and video recordings
+
+Audio and video recordings of behaving subjects MAY be stored in the `beh` directory
+using the `_audio` and `_video` suffixes respectively.
+These recordings are typically used to capture vocalizations, speech, facial expressions,
+body movements, or other behavioral aspects during experimental tasks or rest periods.
+
+!!! warning "Privacy and personally identifiable information"
+
+    Audio and video recordings of human subjects often contain personally identifiable
+    information (PII) such as faces, voices, and other identifying features.
+    Data curators MUST take special care to ensure compliance with applicable privacy
+    regulations (such as HIPAA in the United States, GDPR in the European Union, or other
+    local data protection laws) when handling these recordings.
+
+    These recordings are generally more suitable for internal use or for sharing
+    non-human subject data, unless appropriate privacy protections are implemented.
+
+### File formats
+
+Audio recordings MUST use one of the following extensions:
+
+-   `.mp3` - MPEG Audio Layer III
+-   `.wav` - Waveform Audio File Format
+
+Video recordings MUST use one of the following extensions:
+
+-   `.mp4` - MPEG-4 Part 14
+-   `.mkv` - Matroska video container
+-   `.avi` - Audio Video Interleave
+
+### Entities
+
+Audio and video files MAY use the following entities:
+
+-   `task` - OPTIONAL for audio and video recordings
+-   `acquisition` - OPTIONAL, can distinguish different recording setups
+-   `run` - OPTIONAL, for multiple recordings with identical parameters
+-   `recording` - OPTIONAL, to differentiate simultaneous recordings from different angles, locations, or devices
+-   `split` - OPTIONAL, for continuous recordings split into multiple files
+
+### Examples
+
+```Text
+sub-01/
+    beh/
+        sub-01_task-rest_video.mp4
+        sub-01_task-rest_video.json
+        sub-01_task-stroop_recording-face_video.mp4
+        sub-01_task-stroop_recording-face_video.json
+        sub-01_task-stroop_recording-room_video.mp4
+        sub-01_task-stroop_recording-room_video.json
+        sub-01_task-vocalization_audio.wav
+        sub-01_task-vocalization_audio.json
+```
+
+For continuous recordings split into multiple files:
+
+```Text
+sub-01/
+    ses-01/
+        beh/
+            sub-01_ses-01_task-freeplay_run-01_video/
+                split-001.mp4
+                split-002.mp4
+                split-003.mp4
+            sub-01_ses-01_task-freeplay_run-01_video.json
+```
+
+### Sidecar JSON for audio and video recordings
+
+The following metadata fields are available for audio and video recordings:
+
+<!-- This block generates a metadata table.
+These tables are defined in
+  src/schema/rules/sidecars
+The definitions of the fields specified in these tables may be found in
+  src/schema/objects/metadata.yaml
+A guide for using macros can be found at
+ https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md
+-->
+{{ MACROS___make_sidecar_table("beh.AudioVideoDevice") }}
+
+{{ MACROS___make_sidecar_table("beh.AudioVideoStreams") }}
+
+### Example video sidecar JSON
+
+For a video file containing both video and audio streams:
+
+```JSON
+{
+  "TaskName": "RestingState",
+  "Device": "Sony FDR-AX53",
+  "AudioChannelCount": 2,
+  "AudioSampleRate": 48000,
+  "FrameRate": 30.0,
+  "Height": 1920,
+  "Width": 1080,
+  "Duration": 600.5
+}
+```
+
+### Example audio sidecar JSON
+
+For an audio-only recording:
+
+```JSON
+{
+  "TaskName": "Vocalization",
+  "Device": "Zoom H6 Handy Recorder",
+  "AudioChannelCount": 2,
+  "AudioSampleRate": 44100,
+  "Duration": 300.2
+}
+```
+
+### Annotations and events
+
+Behavioral annotations or event markers for audio and video recordings
+SHOULD be stored in accompanying `_events.tsv` files following the standard
+[events file format](../modality-agnostic-files/events.md).
+These events files use the same filename entities as the audio/video file they describe,
+but with the `_events` suffix.
+
+For example:
+
+```Text
+sub-01/
+    beh/
+        sub-01_task-speech_audio.wav
+        sub-01_task-speech_audio.json
+        sub-01_task-speech_events.tsv
+        sub-01_task-speech_events.json
+```
+
 ## Example `_beh.tsv`
 
 ```tsv

src/schema/objects/extensions.yaml

@@ -1,5 +1,11 @@
 ---
 # This file describes valid file extensions in the specification.
+avi:
+  value: .avi
+  display_name: Audio Video Interleave
+  description: |
+    An [Audio Video Interleave](https://en.wikipedia.org/wiki/Audio_Video_Interleave) video file.
+    This format is commonly used for behavioral video recordings.
 ave:
   value: .ave
   display_name: AVE # not sure what ave stands for
@@ -153,6 +159,24 @@ md:
   display_name: Markdown
   description: |
     A Markdown file.
+mkv:
+  value: .mkv
+  display_name: Matroska Video
+  description: |
+    A [Matroska](https://www.matroska.org/) video container file.
+    This format is commonly used for behavioral video recordings.
+mp3:
+  value: .mp3
+  display_name: MPEG Audio Layer III
+  description: |
+    An [MP3](https://en.wikipedia.org/wiki/MP3) audio file.
+    This format is commonly used for behavioral audio recordings.
+mp4:
+  value: .mp4
+  display_name: MPEG-4 Part 14
+  description: |
+    An [MPEG-4 Part 14](https://en.wikipedia.org/wiki/MPEG-4_Part_14) video container file.
+    This format is commonly used for behavioral video recordings and may contain both video and audio streams.
 mefd:
   value: .mefd/
   display_name: Multiscale Electrophysiology File Format Version 3.0
@@ -286,6 +310,12 @@ tsv_gz:
     A gzipped tab-delimited file.
     This file extension is only used for very large tabular data, such as physiological recordings.
     For smaller data, the unzipped `.tsv` extension is preferred.
+wav:
+  value: .wav
+  display_name: Waveform Audio File Format
+  description: |
+    A [Waveform Audio File Format](https://en.wikipedia.org/wiki/WAV) audio file.
+    This format is commonly used for behavioral audio recordings.
 txt:
   value: .txt
   display_name: Text

src/schema/objects/metadata.yaml

@@ -807,6 +807,13 @@ Descriptors:
     - type: array
       items:
         type: string
+Device:
+  name: Device
+  display_name: Device
+  description: |
+    Free-form description of the device used to record the data
+    (for example, `"iPhone 12"`, `"Canon EOS R5"`).
+  type: string
 DeviceSerialNumber:
   name: DeviceSerialNumber
   display_name: Device Serial Number
@@ -2990,6 +2997,12 @@ RecordingDuration:
     Length of the recording in seconds (for example, `3600`).
   type: number
   unit: s
+RecordingSetupName:
+  name: RecordingSetupName
+  display_name: Recording Setup Name
+  description: |
+    Custom name of the recording setup.
+  type: string
 RecordingType:
   name: RecordingType
   display_name: Recording Type
@@ -3789,6 +3802,13 @@ SubjectArtefactDescription:
     If this field is set to `"n/a"`, it will be interpreted as absence of major
     source of artifacts except cardiac and blinks.
   type: string
+SupplementarySignals:
+  name: SupplementarySignals
+  display_name: Supplementary Signals
+  description: |
+    Description of the supplementary signal (additional modalities) recorded
+    in parallel and are also stored in the data file.
+  type: string
 TablePosition:
   name: TablePosition
   display_name: Table Position
@@ -4197,3 +4217,63 @@ iEEGReference:
     this field should have a general description and the channel specific
     reference should be defined in the `channels.tsv` file.
   type: string
+
+AudioChannelCount:
+  name: AudioChannelCount
+  display_name: Audio Channel Count
+  description: |
+    Number of audio channels in the recording (for example, `2` for stereo).
+  type: integer
+  minimum: 1
+
+AudioDuration:
+  name: AudioDuration
+  display_name: Audio Duration
+  description: |
+    Duration of the audio recording in seconds.
+  type: number
+  exclusiveMinimum: 0
+  unit: s
+
+AudioSampleRate:
+  name: AudioSampleRate
+  display_name: Audio Sample Rate
+  description: |
+    Sample rate of the audio recording in Hertz (for example, `44100`).
+  type: number
+  exclusiveMinimum: 0
+  unit: Hz
+
+Duration:
+  name: Duration
+  display_name: Duration
+  description: |
+    Total duration of the audio or video recording in seconds.
+  type: number
+  exclusiveMinimum: 0
+  unit: s
+
+FrameRate:
+  name: FrameRate
+  display_name: Frame Rate
+  description: |
+    Frame rate of the video recording in frames per second (for example, `30.0`).
+  type: number
+  exclusiveMinimum: 0
+  unit: Hz
+
+Height:
+  name: Height
+  display_name: Video Height
+  description: |
+    Height of the video in pixels (for example, `1920`).
+  type: integer
+  minimum: 1
+
+Width:
+  name: Width
+  display_name: Video Width
+  description: |
+    Width of the video in pixels (for example, `1080`).
+  type: integer
+  minimum: 1

src/schema/objects/suffixes.yaml

@@ -528,6 +528,13 @@ asllabeling:
     A deidentified screenshot of the planning of the labeling slab/plane
     with respect to the imaging slab or slices.
     This screenshot is based on DICOM macro C.8.13.5.14.
+audio:
+  value: audio
+  display_name: Audio Recording
+  description: |
+    Audio recording of a behaving subject.
+    This may include vocalizations, speech, or environmental sounds.
+    The audio stream may be from a standalone audio recording or extracted from a video file.
 beh:
   value: beh
   display_name: Behavioral recording
@@ -877,3 +884,11 @@ unloc:
   description: |
     MRS acquisitions run without localization.
     This includes signals detected using coil sensitivity only.
+video:
+  value: video
+  display_name: Video Recording
+  description: |
+    Video recording of a behaving subject.
+    This may include both video and audio streams.
+    Video recordings are typically used to capture behavior, facial expressions,
+    or body movements during experimental tasks or rest.

src/schema/rules/files/raw/beh.yaml

@@ -14,3 +14,42 @@ noncontinuous:
     task: required
     acquisition: optional
     run: optional
+
+# Audio recordings
+audio:
+  suffixes:
+    - audio
+  extensions:
+    - .mp3
+    - .wav
+    - .json
+  datatypes:
+    - beh
+  entities:
+    subject: required
+    session: optional
+    task: optional
+    acquisition: optional
+    run: optional
+    recording: optional
+    split: optional
+
+# Video recordings
+video:
+  suffixes:
+    - video
+  extensions:
+    - .mp4
+    - .mkv
+    - .avi
+    - .json
+  datatypes:
+    - beh
+  entities:
+    subject: required
+    session: optional
+    task: optional
+    acquisition: optional
+    run: optional
+    recording: optional
+    split: optional