From ee69b4cabdb1794fbbcd4d3f553f508d5ac91b9c Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 25 Oct 2025 11:41:03 -0400 Subject: [PATCH 01/28] feat(beh): add audio and video recording support Add comprehensive support for audio and video recordings in behavioral experiments: - Add audio file extensions (mp3, wav) and video file extensions (mp4, mkv, avi) with corresponding _audio and _video suffixes - Document usage of audio/video recordings in beh directory for capturing vocalizations, speech, facial expressions, and body movements - Add metadata schema for audio/video device information and stream properties - Include privacy warnings about personally identifiable information in human subject recordings - Update behavioral experiments title to remove "with no neural recordings" restriction, clarifying data can be stored with or without neural recordings - Add examples for file organization including multi-angle recordings and split files - Define optional entities: task, acquisition, run, recording, split --- .../behavioral-experiments.md | 153 +++++- src/schema/objects/extensions.yaml | 30 + src/schema/objects/metadata.yaml | 516 ++++++++++++++---- src/schema/objects/suffixes.yaml | 15 + src/schema/rules/files/raw/beh.yaml | 39 ++ src/schema/rules/sidecars/beh.yaml | 21 + 6 files changed, 650 insertions(+), 124 deletions(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index cfb8264ce3..32bc07a7cd 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -1,4 +1,4 @@ -# Behavioral experiments (with no neural recordings) +# Behavioral experiments !!! example "Example datasets" @@ -15,16 +15,12 @@ and a guide for using macros can be found at --> {{ MACROS___make_filename_template("raw", datatypes=["beh"]) }} -In addition to logs from behavioral experiments -performed alongside imaging data acquisitions, -one MAY also include data from experiments -performed with no neural recordings. -The results of those experiments MAY be stored in the `beh` directory -using the same formats for event timing (`_events.tsv`), -metadata (`_events.json`), -physiological (`_physio.tsv.gz`, `_physio.json`) -and other continuous recordings (`_stim.tsv.gz`, `_stim.json`) -as for tasks performed during MRI, electrophysiological or other neural recordings. +Behavioral data MAY be stored in the `beh` directory, +The `beh` directory uses standard formats for event timing (`_events.tsv`), +metadata (`_events.json`), video recordings (`_video.*`), audio recordings (`_audio.*`), +physiological recordings (`_physio.tsv.gz`, `_physio.json`), +and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). + Additionally, events files that do not include the mandatory `onset` and `duration` columns MAY be included, @@ -76,6 +72,141 @@ A guide for using macros can be found at --> {{ MACROS___make_sidecar_table("beh.BEHInstitutionInformation") }} +## Audio and video recordings + +Audio and video recordings of behaving subjects MAY be stored in the `beh` directory +using the `_audio` and `_video` suffixes respectively. +These recordings are typically used to capture vocalizations, speech, facial expressions, +body movements, or other behavioral aspects during experimental tasks or rest periods. + +!!! warning "Privacy and personally identifiable information" + + Audio and video recordings of human subjects often contain personally identifiable + information (PII) such as faces, voices, and other identifying features. + Data curators MUST take special care to ensure compliance with applicable privacy + regulations (such as HIPAA in the United States, GDPR in the European Union, or other + local data protection laws) when handling these recordings. + + These recordings are generally more suitable for internal use or for sharing + non-human subject data, unless appropriate privacy protections are implemented. + +### File formats + +Audio recordings MUST use one of the following extensions: + +- `.mp3` - MPEG Audio Layer III +- `.wav` - Waveform Audio File Format + +Video recordings MUST use one of the following extensions: + +- `.mp4` - MPEG-4 Part 14 +- `.mkv` - Matroska video container +- `.avi` - Audio Video Interleave + +### Entities + +Audio and video files MAY use the following entities: + +- `task` - OPTIONAL for audio and video recordings +- `acquisition` - OPTIONAL, can distinguish different recording setups +- `run` - OPTIONAL, for multiple recordings with identical parameters +- `recording` - OPTIONAL, to differentiate simultaneous recordings from different angles, locations, or devices +- `split` - OPTIONAL, for continuous recordings split into multiple files + +### Examples + +```Text +sub-01/ + beh/ + sub-01_task-rest_video.mp4 + sub-01_task-rest_video.json + sub-01_task-stroop_recording-face_video.mp4 + sub-01_task-stroop_recording-face_video.json + sub-01_task-stroop_recording-room_video.mp4 + sub-01_task-stroop_recording-room_video.json + sub-01_task-vocalization_audio.wav + sub-01_task-vocalization_audio.json +``` + +For continuous recordings split into multiple files: + +```Text +sub-01/ + ses-01/ + beh/ + sub-01_ses-01_task-freeplay_run-01_video/ + split-001.mp4 + split-002.mp4 + split-003.mp4 + sub-01_ses-01_task-freeplay_run-01_video.json +``` + +### Sidecar JSON for audio and video recordings + +The following metadata fields are available for audio and video recordings: + + +{{ MACROS___make_sidecar_table("beh.AudioVideoDevice") }} + +{{ MACROS___make_sidecar_table("beh.AudioVideoStreams") }} + +### Example video sidecar JSON + +For a video file containing both video and audio streams: + +```JSON +{ + "TaskName": "RestingState", + "Device": "Sony FDR-AX53", + "AudioChannelCount": 2, + "AudioSampleRate": 48000, + "FrameRate": 30.0, + "Height": 1920, + "Width": 1080, + "Duration": 600.5 +} +``` + +### Example audio sidecar JSON + +For an audio-only recording: + +```JSON +{ + "TaskName": "Vocalization", + "Device": "Zoom H6 Handy Recorder", + "AudioChannelCount": 2, + "AudioSampleRate": 44100, + "Duration": 300.2 +} +``` + +### Annotations and events + +Behavioral annotations or event markers for audio and video recordings +SHOULD be stored in accompanying `_events.tsv` files following the standard +[events file format](../modality-agnostic-files/events.md). +These events files use the same filename entities as the audio/video file they describe, +but with the `_events` suffix. + +For example: + +```Text +sub-01/ + beh/ + sub-01_task-speech_audio.wav + sub-01_task-speech_audio.json + sub-01_task-speech_events.tsv + sub-01_task-speech_events.json +``` + ## Example `_beh.tsv` ```tsv diff --git a/src/schema/objects/extensions.yaml b/src/schema/objects/extensions.yaml index d482f8849d..64c44bcd91 100644 --- a/src/schema/objects/extensions.yaml +++ b/src/schema/objects/extensions.yaml @@ -1,5 +1,11 @@ --- # This file describes valid file extensions in the specification. +avi: + value: .avi + display_name: Audio Video Interleave + description: | + An [Audio Video Interleave](https://en.wikipedia.org/wiki/Audio_Video_Interleave) video file. + This format is commonly used for behavioral video recordings. ave: value: .ave display_name: AVE # not sure what ave stands for @@ -153,6 +159,24 @@ md: display_name: Markdown description: | A Markdown file. +mkv: + value: .mkv + display_name: Matroska Video + description: | + A [Matroska](https://www.matroska.org/) video container file. + This format is commonly used for behavioral video recordings. +mp3: + value: .mp3 + display_name: MPEG Audio Layer III + description: | + An [MP3](https://en.wikipedia.org/wiki/MP3) audio file. + This format is commonly used for behavioral audio recordings. +mp4: + value: .mp4 + display_name: MPEG-4 Part 14 + description: | + An [MPEG-4 Part 14](https://en.wikipedia.org/wiki/MPEG-4_Part_14) video container file. + This format is commonly used for behavioral video recordings and may contain both video and audio streams. mefd: value: .mefd/ display_name: Multiscale Electrophysiology File Format Version 3.0 @@ -286,6 +310,12 @@ tsv_gz: A gzipped tab-delimited file. This file extension is only used for very large tabular data, such as physiological recordings. For smaller data, the unzipped `.tsv` extension is preferred. +wav: + value: .wav + display_name: Waveform Audio File Format + description: | + A [Waveform Audio File Format](https://en.wikipedia.org/wiki/WAV) audio file. + This format is commonly used for behavioral audio recordings. txt: value: .txt display_name: Text diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index acb7fa4447..85e00930ed 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -20,9 +20,8 @@ AcquisitionDuration: name: AcquisitionDuration display_name: Acquisition Duration description: | - Duration (in seconds) of volume acquisition. + Duration (in seconds) of scan acquisition, including all volumes for multi-volume scans. Corresponds to [DICOM Tag 0018, 9073](https://dicomlookup.com/dicomtags/(0018,9073)) `Acquisition Duration`. - This field is mutually exclusive with `"RepetitionTime"`. type: number exclusiveMinimum: 0 unit: s @@ -484,6 +483,42 @@ Code: revision-specific URIs are recommended. type: string format: uri +CodeValue: + name: CodeValue + display_name: Code Value + description: | + An identifier that is unambiguous within the Coding Scheme + denoted by Coding Scheme Designator and Coding Scheme Version. + Corresponds to [DICOM Tag 0008, 0100](https://dicomlookup.com/dicomtags/(0008,0100)) `Code Value`. + type: string +CodeMeaning: + name: CodeMeaning + display_name: Code Meaning + description: | + Text that has meaning to a human and conveys the meaning of the term + Corresponds to [DICOM Tag 0008, 0104](https://dicomlookup.com/dicomtags/(0008,0104)) `Code Meaning`. + type: string +CodingSchemeDesignator: + name: CodingSchemeDesignator + display_name: Coding Scheme Designator + description: | + The identifier of the coding scheme in which the coded entry is defined. + Corresponds to [DICOM Tag 0008, 0102](https://dicomlookup.com/dicomtags/(0008,0102)) + `Coding Scheme Designator`. + type: string +CodingSchemeVersion: + name: CodingSchemeVersion + display_name: Coding Scheme version + description: | + An identifier of the version of the coding scheme if necessary to resolve ambiguity. + Corresponds to [DICOM Tag 0008, 0103](https://dicomlookup.com/dicomtags/(0008,0103)) `Coding Scheme Version`. + type: string +CodeURL: + name: CodeURL + display_name: Code URL + description: URL where the code used to generate the dataset may be found. + type: string + format: uri CogAtlasID: name: CogAtlasID display_name: Cognitive Atlas ID @@ -519,6 +554,44 @@ Columns: type: array items: type: string +Container: + name: Container + display_name: Container + description: | + Used to specify the location and relevant attributes of software container image + used to produce the dataset. Valid keys in this object include `Type`, `Tag` and + [`URI`][uri] with [string][] values. + type: object + recommended: + - Type + - Tag + - URI + properties: + ContainerType: + $ref: objects.metadata.ContainerType + ContainerTag: + $ref: objects.metadata.ContainerTag + ContainerURI: + $ref: objects.metadata.ContainerURI +ContainerTag: + name: ContainerTag + display_name: Container Tag + description: | + Tag of software container image used to produce the dataset + type: string +ContainerType: + name: ContainerType + display_name: Container Type + description: | + Type of software container image used to produce the dataset + type: string +ContainerURI: + name: ContainerURI + display_name: Container URI + description: | + URI for software container image used to produce the dataset + type: string + format: uri ContinuousHeadLocalization: name: ContinuousHeadLocalization display_name: Continuous Head Localization @@ -550,6 +623,22 @@ DCOffsetCorrection: If the method used was subtracting the mean value for each channel, use "mean". type: string +Database: + name: Database + display_name: Database + description: | + [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator) + of database where the dataset is hosted. + type: string + format: uri +Dataset: + name: Dataset + display_name: Dataset + description: | + [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator) + where data can be retrieved. + type: string + format: uri DatasetDOI: name: DatasetDOI display_name: DatasetDOI @@ -583,6 +672,7 @@ DatasetType: enum: - $ref: objects.enums.raw.value - $ref: objects.enums.derivative.value + - $ref: objects.enums.study__datasettype.value DecayCorrectionFactor: name: DecayCorrectionFactor display_name: Decay Correction Factor @@ -610,38 +700,20 @@ DeidentificationMethodCodeSequence: type: array items: type: object - recommended_fields: + recommended: - CodeValue - CodeMeaning - CodingSchemeDesignator - CodingSchemeVersion properties: CodeValue: - name: CodeValue - type: string - description: | - An identifier that is unambiguous within the Coding Scheme - denoted by Coding Scheme Designator and Coding Scheme Version. - Corresponds to [DICOM Tag 0008, 0100](https://dicomlookup.com/dicomtags/(0008,0100)) `Code Value`. + $ref: objects.metadata.CodeValue CodeMeaning: - name: CodeMeaning - type: string - description: | - Text that has meaning to a human and conveys the meaning of the term - Corresponds to [DICOM Tag 0008, 0104](https://dicomlookup.com/dicomtags/(0008,0104)) `Code Meaning`. + $ref: objects.metadata.CodeMeaning CodingSchemeDesignator: - name: CodingSchemeDesignator - type: string - description: | - The identifier of the coding scheme in which the coded entry is defined. - Corresponds to [DICOM Tag 0008, 0102](https://dicomlookup.com/dicomtags/(0008,0102)) - `Coding Scheme Designator`. + $ref: objects.metadata.CodingSchemeDesignator CodingSchemeVersion: - name: CodingSchemeVersion - type: string - description: | - An identifier of the version of the coding scheme if necessary to resolve ambiguity. - Corresponds to [DICOM Tag 0008, 0103](https://dicomlookup.com/dicomtags/(0008,0103)) `Coding Scheme Version`. + $ref: objects.metadata.CodingSchemeVersion DelayAfterTrigger: name: DelayAfterTrigger display_name: Delay After Trigger @@ -700,6 +772,13 @@ Description: description: | Free-form natural language description. type: string +Description__GeneratedBy: + name: Description + display_name: Pipeline Description + description: | + Plain-text description of the pipeline or process that generated the outputs. + RECOMMENDED if `Name` is `"Manual"`. + type: string DetectorType: name: DetectorType display_name: Detector Type @@ -715,6 +794,26 @@ DetectorType: - type: string enum: - $ref: objects.enums.mixed.value +Descriptors: + name: Descriptors + display_name: Descriptors + description: | + List of relevant descriptors (for example, journal articles) for dataset + using a valid + [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator) + when possible. + anyOf: + - type: string + - type: array + items: + type: string +Device: + name: Device + display_name: Device + description: | + Free-form description of the device used to record the data + (for example, `"iPhone 12"`, `"Canon EOS R5"`). + type: string DeviceSerialNumber: name: DeviceSerialNumber display_name: Device Serial Number @@ -805,6 +904,13 @@ DispersionCorrected: Boolean flag specifying whether the blood data have been dispersion-corrected. NOTE: not customary for manual samples, and hence should be set to `false`. type: boolean +DOI: + name: DOI + display_name: DOI + description: | + A Digital Object Identifier (DOI) is a persistent, resolvable name for a digital resource. + (see [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator)) + type: string DoseCalibrationFactor: name: DoseCalibrationFactor display_name: Dose Calibration Factor @@ -1001,19 +1107,12 @@ EditPulse: type: object additionalProperties: type: object - recommended_fields: [FrequencyOffset, PulseDuration] + recommended: [FrequencyOffset, PulseDuration] properties: FrequencyOffset: - anyOf: - - type: number - unit: ppm - - type: array - items: - type: number - unit: ppm + $ref: objects.metadata.FrequencyOffset PulseDuration: - type: number - unit: ms + $ref: objects.metadata.PulseDuration EditTarget: name: EditTarget display_name: Editing Target @@ -1200,6 +1299,48 @@ FlipAngle: unit: degree exclusiveMinimum: 0 maximum: 360 +Format: + name: Format + display_name: Value format + description: | + Permitted formats for values in the described column. + type: string + # Keep synced with objects.formats + enum: + # Type formats + - string + - number + - integer + - boolean + # Numeric/alphanumeric strings + - index + - label + # Dates/times + - date + - datetime + - time + # Units + - unit + # URIs + - uri + - rrid + - bids_uri + # Paths + - dataset_relative + - file_relative + - participant_relative + - stimuli_relative + # Miscellaneous + - hed_version +FrameAcquisitionDuration: + name: FrameAcquisitionDuration + display_name: Frame Acquisition Duration + description: | + Duration (in seconds) of volume acquisition. + Corresponds to DICOM Tag 0018, 9220 `Frame Acquisition Duration`. + type: number + exclusiveMinimum: 0 + unit: s FrameDuration: name: FrameDuration display_name: Frame Duration @@ -1220,6 +1361,18 @@ FrameTimesStart: items: type: number unit: s +FrequencyOffset: + name: FrequencyOffset + display_name: Frequency Offset + description: | + List or value for frequency offset for spectral editing in ppm. See EditPulse + anyOf: + - type: number + unit: ppm + - type: array + items: + type: number + unit: ppm Funding: name: Funding display_name: Funding @@ -1241,46 +1394,15 @@ GeneratedBy: recommended: [Version] properties: Name: - name: Name - description: | - Name of the pipeline or process that generated the outputs. Use `"Manual"` to - indicate the derivatives were generated by hand, or adjusted manually after an - initial run of an automated pipeline. - type: string + $ref: objects.metadata.Name__GeneratedBy Version: - name: Version - description: Version of the pipeline - type: string + $ref: objects.metadata.Version__GeneratedBy Description: - name: Description - description: | - Plain-text description of the pipeline or process that generated the outputs. - RECOMMENDED if `Name` is `"Manual"`. - type: string + $ref: objects.metadata.Description__GeneratedBy CodeURL: - name: CodeURL - description: URL where the code used to generate the dataset may be found. - type: string - format: uri + $ref: objects.metadata.CodeURL Container: - name: Container - description: | - Used to specify the location and relevant attributes of software container image - used to produce the dataset. Valid keys in this object include `Type`, `Tag` and - [`URI`][uri] with [string][] values. - type: object - recommended: - - Type - - Tag - - URI - properties: - Type: - type: string - Tag: - type: string - URI: - type: string - format: uri + $ref: objects.metadata.Container GeneticLevel: name: GeneticLevel display_name: Genetic Level @@ -1306,31 +1428,11 @@ Genetics: required: [Dataset] properties: Dataset: - name: Dataset - description: | - [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator) - where data can be retrieved. - type: string - format: uri + $ref: objects.metadata.Dataset Database: - name: Database - description: | - [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator) - of database where the dataset is hosted. - type: string - format: uri + $ref: objects.metadata.Database Descriptors: - name: Descriptors - description: | - List of relevant descriptors (for example, journal articles) for dataset - using a valid - [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator) - when possible. - anyOf: - - type: string - - type: array - items: - type: string + $ref: objects.metadata.Descriptors GradientSetType: name: GradientSetType display_name: Gradient Set Type @@ -1728,6 +1830,15 @@ JNTANGChannelCount: Number of joint angle channels. type: integer minimum: 0 +Keywords: + name: Keywords + display_name: Keywords + description: | + A list of keywords that describe the content or subject matter of the dataset. + These may be used to enhance the findability of the dataset on public archives. + type: array + items: + type: string LabelingDistance: name: LabelingDistance display_name: Labeling Distance @@ -1878,10 +1989,9 @@ Levels: - type: object properties: TermURL: - type: string - format: uri + $ref: objects.metadata.TermURL Description: - type: string + $ref: objects.metadata.Description License: name: License display_name: License @@ -1987,6 +2097,54 @@ MEGREFChannelCount: `MEGREFChannelCount` should be set to `0`. type: integer minimum: 0 +# TODO: Change if needs to be broken down into ecephys and icephys +# TODO: Add links to the 2D coordinate system, Pixels and glossary entry. +# TODO: Add enums for the valid values specific to microephys +MicroephysCoordinateSystem: + name: MicroephysCoordinateSystem + display_name: Microephys Coordinate System + description: | + Defines the coordinate system for the microelectrode probes. + See the + [Coordinate Systems Appendix](SPEC_ROOT/appendices/coordinate-systems.md) + for a list of restricted keywords for coordinate systems. + If `"Other"`, provide definition of the coordinate system in + `"MicroephysCoordinateSystemDescription"`. If positions correspond to + pixel indices in a 2D image (of either a volume-rendering, + surface-rendering, operative photo, or operative drawing), + this MUST be "Pixels". For more information, see the section + on 2D coordinate systems. + For a list of valid values for this field, see the associated glossary entry. + anyOf: + # - $ref: objects.enums._MicroephysCoordSys + - $ref: objects.enums._StandardTemplateCoordSys +MicroephysCoordinateSystemDescription: + name: MicroephysCoordinateSystemDescription + display_name: Microephys Coordinate System Description + description: | + Free-form text description of the coordinate system. + May also include a link to a documentation page or paper describing the + system in greater detail. + type: string +MicroephysCoordinateUnits: + name: MicroephysCoordinateUnits + display_name: Microephys Coordinate Units + description: | + Units of the coordinates of `"MicroephysCoordinateSystem"`. + type: string +# TODO: Clarify if units need to be used or enums, if units, define pixels + enum: + - m + - mm + - cm + - pixels +MicroephysCoordinateSystemPhoto: + name: MicroephysCoordinateSystemPhoto + display_name: Microephys Coordinate System Photo + description: | + A link to a photo or drawing of the microelectrode probe system. + type: string + format: uri MISCChannelCount: name: MISCChannelCount display_name: Miscellaneous channel count @@ -2105,6 +2263,12 @@ ManufacturersModelName: description: | Manufacturer's model name of the equipment that produced the measurements. type: string +ManufacturersModelVersion: + name: ManufacturersModelVersion + display_name: Manufacturers Model Version + description: | + Manufacturer's model version of the equipment that produced the measurements. + type: string MatrixCoilMode: name: MatrixCoilMode display_name: Matrix Coil Mode @@ -2136,6 +2300,12 @@ MaxMovement: as measured by the head localization coils (for example, `4.8`). type: number unit: mm +Maximum: + name: Maximum + display_name: Maximum value + description: | + Maximum value a column entry is permitted to have. + type: number MeasurementToolMetadata: name: MeasurementToolMetadata display_name: Measurement Tool Metadata @@ -2147,10 +2317,9 @@ MeasurementToolMetadata: type: object properties: TermURL: - type: string - format: uri + $ref: objects.metadata.TermURL Description: - type: string + $ref: objects.metadata.Description MetaboliteAvail: name: MetaboliteAvail display_name: Metabolite Available @@ -2174,6 +2343,12 @@ MetaboliteRecoveryCorrectionApplied: If `true`, the `hplc_recovery_fractions` column MUST be present in the corresponding `*_blood.tsv` file. type: boolean +Minimum: + name: Minimum + display_name: Minimum value + description: | + Minimum value a column entry is permitted to have. + type: number MiscChannelCount: name: MiscChannelCount display_name: Misc Channel Count @@ -2249,6 +2424,14 @@ Name: description: | Name of the dataset. type: string +Name__GeneratedBy: + name: Name + display_name: Pipeline Name + description: | + Name of the pipeline or process that generated the outputs. Use `"Manual"` to + indicate the derivatives were generated by hand, or adjusted manually after an + initial run of an automated pipeline. + type: string NegativeContrast: name: NegativeContrast display_name: Negative Contrast @@ -2697,6 +2880,13 @@ PromptRate: type: array items: type: number +PulseDuration: + name: PulseDuration + display_name: Pulse Duration + description: | + Pulse duration if spectral editing was applied, in milliseconds + type: number + unit: ms PulseSequenceDetails: name: PulseSequenceDetails display_name: Pulse Sequence Details @@ -2861,6 +3051,12 @@ RecordingDuration: Length of the recording in seconds (for example, `3600`). type: number unit: s +RecordingSetupName: + name: RecordingSetupName + display_name: Recording Setup Name + description: | + Custom name of the recording setup. + type: string RecordingType: name: RecordingType display_name: Recording Type @@ -2983,9 +3179,13 @@ ResonantNucleus: For multi-nuclei experiments such as 1H-[13C] MR, an array can be used: `["1H", "13C"]`. Corresponds to [DICOM Tag 0018, 9100](https://dicomlookup.com/dicomtags/(0018,9100)) `Resonant Nucleus`. - type: array - items: - type: string + anyOf: + - type: string + - type: array + items: + type: string + minItems: 1 + maxItems: 2 RotationOrder: name: RotationOrder display_name: RotationOrder @@ -3410,7 +3610,7 @@ SourceDatasets: name: SourceDatasets display_name: Source Datasets description: | - Used to specify the locations and relevant attributes of all source datasets. + Used to specify the locations and relevant attributes of all source datasets (BIDS or not). Valid keys in each object include `"URL"`, `"DOI"` (see [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator)), and `"Version"` with @@ -3421,12 +3621,11 @@ SourceDatasets: type: object properties: URL: - type: string - format: uri + $ref: objects.metadata.URL DOI: - type: string + $ref: objects.metadata.DOI Version: - type: string + $ref: objects.metadata.Version__Dataset Sources: name: Sources display_name: Sources @@ -3543,10 +3742,15 @@ SpectrometerFrequency: to the resonant frequency of 1H. For multi-nuclei experiments such as 1H-[13C] MR at 3T, an array can be used: `[127.731, 32.125]`. - type: array - items: - type: number - unit: MHz + anyOf: + - type: number + unit: MHz + - type: array + items: + type: number + unit: MHz + minItems: 1 + maxItems: 2 SpoilingGradientDuration: name: SpoilingGradientDuration display_name: Spoiling Gradient Duration @@ -3652,6 +3856,13 @@ SubjectArtefactDescription: If this field is set to `"n/a"`, it will be interpreted as absence of major source of artifacts except cardiac and blinks. type: string +SupplementarySignals: + name: SupplementarySignals + display_name: Supplementary Signals + description: | + Description of the supplementary signal (additional modalities) recorded + in parallel and are also stored in the data file. + type: string TablePosition: name: TablePosition display_name: Table Position @@ -3681,7 +3892,7 @@ TaskName: description: | Name of the task. No two tasks should have the same name. - The task label included in the filename is derived from this `"TaskName"` field + The task label included in the filename MAY be derived from this `"TaskName"` field by removing all non-alphanumeric or `+` characters (that is, all except those matching `[0-9a-zA-Z+]`), and potentially replacing spaces with `+` to ease readability. For example `"TaskName"` `"faces n-back"` or `"head nodding"` could correspond to task labels @@ -3844,6 +4055,15 @@ Units: (see [Units](SPEC_ROOT/common-principles.md#units)). type: string format: unit +URL: + name: URL + display_name: URL + description: | + A Uniform Resource Locator (URL) is a structured reference to an object that + includes a resource transfer scheme and a location. + (see [URI](SPEC_ROOT/common-principles.md#uniform-resource-indicator)) + type: string + format: uri VascularCrushing: name: VascularCrushing display_name: Vascular Crushing @@ -3867,6 +4087,16 @@ VascularCrushingVENC: items: type: number unit: cm/s +Version__Dataset: + name: Version + display_name: Dataset Version + description: Version of the dataset. + type: string +Version__GeneratedBy: + name: Version + display_name: Pipeline Version + description: Version of the pipeline + type: string VolumeAffineMatrix: name: VolumeAffineMatrix display_name: Volume Affine Matrix @@ -4041,3 +4271,63 @@ iEEGReference: this field should have a general description and the channel specific reference should be defined in the `channels.tsv` file. type: string + +AudioChannelCount: + name: AudioChannelCount + display_name: Audio Channel Count + description: | + Number of audio channels in the recording (for example, `2` for stereo). + type: integer + minimum: 1 + +AudioDuration: + name: AudioDuration + display_name: Audio Duration + description: | + Duration of the audio recording in seconds. + type: number + exclusiveMinimum: 0 + unit: s + +AudioSampleRate: + name: AudioSampleRate + display_name: Audio Sample Rate + description: | + Sample rate of the audio recording in Hertz (for example, `44100`). + type: number + exclusiveMinimum: 0 + unit: Hz + +Duration: + name: Duration + display_name: Duration + description: | + Total duration of the audio or video recording in seconds. + type: number + exclusiveMinimum: 0 + unit: s + +FrameRate: + name: FrameRate + display_name: Frame Rate + description: | + Frame rate of the video recording in frames per second (for example, `30.0`). + type: number + exclusiveMinimum: 0 + unit: Hz + +Height: + name: Height + display_name: Video Height + description: | + Height of the video in pixels (for example, `1920`). + type: integer + minimum: 1 + +Width: + name: Width + display_name: Video Width + description: | + Width of the video in pixels (for example, `1080`). + type: integer + minimum: 1 diff --git a/src/schema/objects/suffixes.yaml b/src/schema/objects/suffixes.yaml index 1e2825ced2..6e6efb66b6 100644 --- a/src/schema/objects/suffixes.yaml +++ b/src/schema/objects/suffixes.yaml @@ -528,6 +528,13 @@ asllabeling: A deidentified screenshot of the planning of the labeling slab/plane with respect to the imaging slab or slices. This screenshot is based on DICOM macro C.8.13.5.14. +audio: + value: audio + display_name: Audio Recording + description: | + Audio recording of a behaving subject. + This may include vocalizations, speech, or environmental sounds. + The audio stream may be from a standalone audio recording or extracted from a video file. beh: value: beh display_name: Behavioral recording @@ -877,3 +884,11 @@ unloc: description: | MRS acquisitions run without localization. This includes signals detected using coil sensitivity only. +video: + value: video + display_name: Video Recording + description: | + Video recording of a behaving subject. + This may include both video and audio streams. + Video recordings are typically used to capture behavior, facial expressions, + or body movements during experimental tasks or rest. diff --git a/src/schema/rules/files/raw/beh.yaml b/src/schema/rules/files/raw/beh.yaml index 6193015295..ea17b69018 100644 --- a/src/schema/rules/files/raw/beh.yaml +++ b/src/schema/rules/files/raw/beh.yaml @@ -14,3 +14,42 @@ noncontinuous: task: required acquisition: optional run: optional + +# Audio recordings +audio: + suffixes: + - audio + extensions: + - .mp3 + - .wav + - .json + datatypes: + - beh + entities: + subject: required + session: optional + task: optional + acquisition: optional + run: optional + recording: optional + split: optional + +# Video recordings +video: + suffixes: + - video + extensions: + - .mp4 + - .mkv + - .avi + - .json + datatypes: + - beh + entities: + subject: required + session: optional + task: optional + acquisition: optional + run: optional + recording: optional + split: optional diff --git a/src/schema/rules/sidecars/beh.yaml b/src/schema/rules/sidecars/beh.yaml index f2d8410914..27fc43e12f 100644 --- a/src/schema/rules/sidecars/beh.yaml +++ b/src/schema/rules/sidecars/beh.yaml @@ -25,3 +25,24 @@ BEHInstitutionInformation: InstitutionName: recommended InstitutionAddress: recommended InstitutionalDepartmentName: recommended + +# Audio and Video metadata +AudioVideoDevice: + selectors: + - datatype == "beh" + - intersects([suffix], ["audio", "video"]) + fields: + Device: optional + DeviceSerialNumber: optional + +AudioVideoStreams: + selectors: + - datatype == "beh" + - intersects([suffix], ["audio", "video"]) + fields: + AudioChannelCount: optional + AudioSampleRate: optional + FrameRate: optional + Height: optional + Width: optional + Duration: optional From ea2e5d15ae841892a4b85b1a1ea888ec45b337f0 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 25 Oct 2025 12:38:27 -0400 Subject: [PATCH 02/28] fix(metadata): remove invalid enum reference for DatasetType --- src/schema/objects/metadata.yaml | 1 - 1 file changed, 1 deletion(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index 85e00930ed..5ce7ab51d4 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -672,7 +672,6 @@ DatasetType: enum: - $ref: objects.enums.raw.value - $ref: objects.enums.derivative.value - - $ref: objects.enums.study__datasettype.value DecayCorrectionFactor: name: DecayCorrectionFactor display_name: Decay Correction Factor From ffccafc98a5c01a5ffb35959c51a3f2559e88f49 Mon Sep 17 00:00:00 2001 From: "pre-commit-ci[bot]" <66853113+pre-commit-ci[bot]@users.noreply.github.com> Date: Sat, 25 Oct 2025 16:59:36 +0000 Subject: [PATCH 03/28] [pre-commit.ci] auto fixes from pre-commit.com hooks for more information, see https://pre-commit.ci --- src/modality-specific-files/behavioral-experiments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index d1a1cad65f..327e50347a 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -86,7 +86,7 @@ body movements, or other behavioral aspects during experimental tasks or rest pe Data curators MUST take special care to ensure compliance with applicable privacy regulations (such as HIPAA in the United States, GDPR in the European Union, or other local data protection laws) when handling these recordings. - + These recordings are generally more suitable for internal use or for sharing non-human subject data, unless appropriate privacy protections are implemented. From 4707a3347511c06a1744fc957cd79c1f9295e7d0 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 25 Oct 2025 13:54:54 -0400 Subject: [PATCH 04/28] Apply suggestion from @bendichter --- src/schema/objects/metadata.yaml | 48 -------------------------------- 1 file changed, 48 deletions(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index 85e00930ed..da3660a884 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -2097,54 +2097,6 @@ MEGREFChannelCount: `MEGREFChannelCount` should be set to `0`. type: integer minimum: 0 -# TODO: Change if needs to be broken down into ecephys and icephys -# TODO: Add links to the 2D coordinate system, Pixels and glossary entry. -# TODO: Add enums for the valid values specific to microephys -MicroephysCoordinateSystem: - name: MicroephysCoordinateSystem - display_name: Microephys Coordinate System - description: | - Defines the coordinate system for the microelectrode probes. - See the - [Coordinate Systems Appendix](SPEC_ROOT/appendices/coordinate-systems.md) - for a list of restricted keywords for coordinate systems. - If `"Other"`, provide definition of the coordinate system in - `"MicroephysCoordinateSystemDescription"`. If positions correspond to - pixel indices in a 2D image (of either a volume-rendering, - surface-rendering, operative photo, or operative drawing), - this MUST be "Pixels". For more information, see the section - on 2D coordinate systems. - For a list of valid values for this field, see the associated glossary entry. - anyOf: - # - $ref: objects.enums._MicroephysCoordSys - - $ref: objects.enums._StandardTemplateCoordSys -MicroephysCoordinateSystemDescription: - name: MicroephysCoordinateSystemDescription - display_name: Microephys Coordinate System Description - description: | - Free-form text description of the coordinate system. - May also include a link to a documentation page or paper describing the - system in greater detail. - type: string -MicroephysCoordinateUnits: - name: MicroephysCoordinateUnits - display_name: Microephys Coordinate Units - description: | - Units of the coordinates of `"MicroephysCoordinateSystem"`. - type: string -# TODO: Clarify if units need to be used or enums, if units, define pixels - enum: - - m - - mm - - cm - - pixels -MicroephysCoordinateSystemPhoto: - name: MicroephysCoordinateSystemPhoto - display_name: Microephys Coordinate System Photo - description: | - A link to a photo or drawing of the microelectrode probe system. - type: string - format: uri MISCChannelCount: name: MISCChannelCount display_name: Miscellaneous channel count From 38310aacf6f19a46b344ca91604b81dbbb2307e7 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 25 Oct 2025 13:55:22 -0400 Subject: [PATCH 05/28] Apply suggestion from @bendichter --- src/schema/objects/metadata.yaml | 6 ------ 1 file changed, 6 deletions(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index da3660a884..fa13f47da7 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -2215,12 +2215,6 @@ ManufacturersModelName: description: | Manufacturer's model name of the equipment that produced the measurements. type: string -ManufacturersModelVersion: - name: ManufacturersModelVersion - display_name: Manufacturers Model Version - description: | - Manufacturer's model version of the equipment that produced the measurements. - type: string MatrixCoilMode: name: MatrixCoilMode display_name: Matrix Coil Mode From fa98d09a9452fc6c3753c610dbf62269be0d8c9c Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 25 Oct 2025 13:55:37 -0400 Subject: [PATCH 06/28] Apply suggestion from @bendichter --- src/schema/objects/metadata.yaml | 7 ------- 1 file changed, 7 deletions(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index fa13f47da7..ad89182431 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -3802,13 +3802,6 @@ SubjectArtefactDescription: If this field is set to `"n/a"`, it will be interpreted as absence of major source of artifacts except cardiac and blinks. type: string -SupplementarySignals: - name: SupplementarySignals - display_name: Supplementary Signals - description: | - Description of the supplementary signal (additional modalities) recorded - in parallel and are also stored in the data file. - type: string TablePosition: name: TablePosition display_name: Table Position From 88849858ac71271e41a6961bf88b2f78b87d66da Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 25 Oct 2025 13:55:57 -0400 Subject: [PATCH 07/28] Apply suggestion from @bendichter --- src/schema/objects/metadata.yaml | 6 ------ 1 file changed, 6 deletions(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index ad89182431..eba58d3809 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -2997,12 +2997,6 @@ RecordingDuration: Length of the recording in seconds (for example, `3600`). type: number unit: s -RecordingSetupName: - name: RecordingSetupName - display_name: Recording Setup Name - description: | - Custom name of the recording setup. - type: string RecordingType: name: RecordingType display_name: Recording Type From 827e7d90df0ca23166f3bdc20d714b3ea5a7bf88 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sun, 26 Oct 2025 12:55:37 -0400 Subject: [PATCH 08/28] refactor(beh): update title to 'Behavioral recordings' and use filetree macros - Change section title from 'Behavioral experiments' to 'Behavioral recordings' - Convert file tree examples to use MACROS___make_filetree_example for consistent rendering - Address review comments from @yarikoptic in PR #2231 --- .../behavioral-experiments.md | 88 ++++++++++++------- 1 file changed, 57 insertions(+), 31 deletions(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index 327e50347a..52c2805cec 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -1,4 +1,4 @@ -# Behavioral experiments +# Behavioral recordings !!! example "Example datasets" @@ -115,31 +115,49 @@ Audio and video files MAY use the following entities: ### Examples -```Text -sub-01/ - beh/ - sub-01_task-rest_video.mp4 - sub-01_task-rest_video.json - sub-01_task-stroop_recording-face_video.mp4 - sub-01_task-stroop_recording-face_video.json - sub-01_task-stroop_recording-room_video.mp4 - sub-01_task-stroop_recording-room_video.json - sub-01_task-vocalization_audio.wav - sub-01_task-vocalization_audio.json -``` + +{{ MACROS___make_filetree_example( + { + "sub-01": { + "beh": { + "sub-01_task-rest_video.mp4": "", + "sub-01_task-rest_video.json": "", + "sub-01_task-stroop_recording-face_video.mp4": "", + "sub-01_task-stroop_recording-face_video.json": "", + "sub-01_task-stroop_recording-room_video.mp4": "", + "sub-01_task-stroop_recording-room_video.json": "", + "sub-01_task-vocalization_audio.wav": "", + "sub-01_task-vocalization_audio.json": "", + }, + }, + } +) }} For continuous recordings split into multiple files: -```Text -sub-01/ - ses-01/ - beh/ - sub-01_ses-01_task-freeplay_run-01_video/ - split-001.mp4 - split-002.mp4 - split-003.mp4 - sub-01_ses-01_task-freeplay_run-01_video.json -``` + +{{ MACROS___make_filetree_example( + { + "sub-01": { + "ses-01": { + "beh": { + "sub-01_ses-01_task-freeplay_run-01_video": { + "split-001.mp4": "", + "split-002.mp4": "", + "split-003.mp4": "", + }, + "sub-01_ses-01_task-freeplay_run-01_video.json": "", + }, + }, + }, + } +) }} ### Sidecar JSON for audio and video recordings @@ -198,14 +216,22 @@ but with the `_events` suffix. For example: -```Text -sub-01/ - beh/ - sub-01_task-speech_audio.wav - sub-01_task-speech_audio.json - sub-01_task-speech_events.tsv - sub-01_task-speech_events.json -``` + +{{ MACROS___make_filetree_example( + { + "sub-01": { + "beh": { + "sub-01_task-speech_audio.wav": "", + "sub-01_task-speech_audio.json": "", + "sub-01_task-speech_events.tsv": "", + "sub-01_task-speech_events.json": "", + }, + }, + } +) }} ## Example `_beh.tsv` From 7cacffdf82a25aad24134a367aed901d440b2590 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sun, 26 Oct 2025 12:56:46 -0400 Subject: [PATCH 09/28] fix(docs): update link to BIDS examples repository for behavioral datasets --- src/modality-specific-files/behavioral-experiments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index 52c2805cec..547ebfaa8a 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -3,7 +3,7 @@ !!! example "Example datasets" Datasets containing behavioral data can be found - in the [BIDS examples repository](https://bids-standard.github.io/bids-examples/#behavioral) + in the [BIDS examples repository](https://bids-website.readthedocs.io/en/latest/datasets/examples.html#behavioral) and can be used as helpful guidance when curating new datasets. {{ MACROS___make_filename_template("raw", datatypes=["beh"]) }} -Behavioral data MAY be stored in the `beh` directory, -The `beh` directory uses standard formats for event timing (`_events.tsv`), -metadata (`_events.json`), video recordings (`_video.*`), audio recordings (`_audio.*`), -physiological recordings (`_physio.tsv.gz`, `_physio.json`), -and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). - -Additionally, events files -that do not include the mandatory `onset` and `duration` columns -MAY be included, +The `beh` directory MAY store behavioral recordings such as audio (`_audio.*`) and video (`_video.*`) recordings, physiological (`_physio.*`) recordings, and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). +Audio and video recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. +The `beh` directory MAY also contain event timing files (`_events.tsv`) and their associated metadata (`_events.json`) for behavioral experiments that do not have corresponding neuroimaging or functional data. + +Additionally, events files that do not include the mandatory `onset` and `duration` columns MAY be included, but MUST be labeled `_beh.tsv` rather than `_events.tsv`. The following OPTIONAL columns are pre-defined for behavioral data files: From 3ee2c6c4fb6a938e613cbaf020b1b02ac92fff13 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 28 Oct 2025 09:39:44 -0400 Subject: [PATCH 11/28] Update src/modality-specific-files/behavioral-experiments.md Co-authored-by: Chris Markiewicz --- src/modality-specific-files/behavioral-experiments.md | 8 +++----- 1 file changed, 3 insertions(+), 5 deletions(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index f50fa70c88..7c98daca0a 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -143,11 +143,9 @@ A guide for using macros can be found at "sub-01": { "ses-01": { "beh": { - "sub-01_ses-01_task-freeplay_run-01_video": { - "split-001.mp4": "", - "split-002.mp4": "", - "split-003.mp4": "", - }, + "sub-01_ses-01_task-freeplay_run-01_split-001_video.mp4": "", + "sub-01_ses-01_task-freeplay_run-01_split-002_video.mp4": "", + "sub-01_ses-01_task-freeplay_run-01_split-003_video.mp4": "", "sub-01_ses-01_task-freeplay_run-01_video.json": "", }, }, From 826e25a7c740ae110ef5657c169448673bc0fd0a Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 28 Oct 2025 09:44:32 -0400 Subject: [PATCH 12/28] Apply suggestion from @bendichter --- src/modality-specific-files/behavioral-experiments.md | 1 + 1 file changed, 1 insertion(+) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index 7c98daca0a..2f2c3cd834 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -90,6 +90,7 @@ body movements, or other behavioral aspects during experimental tasks or rest pe Audio recordings MUST use one of the following extensions: +- `.flac` - Free Lossless Audio Codec - `.mp3` - MPEG Audio Layer III - `.wav` - Waveform Audio File Format From 00e3e265732bc82849271c1a5e05f004e5b76416 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 28 Oct 2025 10:16:13 -0400 Subject: [PATCH 13/28] Apply suggestion from @bendichter --- src/schema/rules/files/raw/beh.yaml | 1 + 1 file changed, 1 insertion(+) diff --git a/src/schema/rules/files/raw/beh.yaml b/src/schema/rules/files/raw/beh.yaml index ea17b69018..4d5324d199 100644 --- a/src/schema/rules/files/raw/beh.yaml +++ b/src/schema/rules/files/raw/beh.yaml @@ -22,6 +22,7 @@ audio: extensions: - .mp3 - .wav + - .flac - .json datatypes: - beh From 25f65e1c3ecca2a99efcebd90c5d6ee1a7abc730 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 28 Oct 2025 10:34:46 -0400 Subject: [PATCH 14/28] Add FLAC audio file extension support for behavioral recordings --- src/schema/objects/extensions.yaml | 6 ++++++ 1 file changed, 6 insertions(+) diff --git a/src/schema/objects/extensions.yaml b/src/schema/objects/extensions.yaml index 64c44bcd91..22ea426324 100644 --- a/src/schema/objects/extensions.yaml +++ b/src/schema/objects/extensions.yaml @@ -120,6 +120,12 @@ fif: display_name: Functional Imaging File Format description: | An MEG file format used by Neuromag, Elekta, and MEGIN. +flac: + value: .flac + display_name: Free Lossless Audio Codec + description: | + A [FLAC](https://en.wikipedia.org/wiki/FLAC) audio file. + This format is commonly used for behavioral audio recordings. jpg: value: .jpg display_name: Joint Photographic Experts Group Format From bf00bc64e3b9cbbcf5bd71482cf8f36086aa08d3 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 28 Oct 2025 10:47:00 -0400 Subject: [PATCH 15/28] Add StartTime field for audio/video metadata in BIDS schema --- src/schema/objects/metadata.yaml | 16 ++++++++++++++++ src/schema/rules/sidecars/beh.yaml | 1 + 2 files changed, 17 insertions(+) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index eba58d3809..920a8a7517 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -3744,6 +3744,22 @@ StartTime: where `s` reflects whole seconds, and `.000000` reflects OPTIONAL fractional seconds. type: number unit: s +StartTime__audiovideo: + name: StartTime + display_name: Start Time + description: | + Start time in seconds in relation to the start of acquisition of the first + data sample in the corresponding dataset (negative values are allowed). + This data MAY be specified with sub-second precision using the syntax `s[.000000]`, + where `s` reflects whole seconds, and `.000000` reflects OPTIONAL fractional seconds. + For audio/video files where precise temporal alignment is not possible or applicable, + this field MAY be set to `"n/a"`. + anyOf: + - type: number + unit: s + - type: string + enum: + - n/a StationName: name: StationName display_name: Station Name diff --git a/src/schema/rules/sidecars/beh.yaml b/src/schema/rules/sidecars/beh.yaml index 27fc43e12f..c283e70b73 100644 --- a/src/schema/rules/sidecars/beh.yaml +++ b/src/schema/rules/sidecars/beh.yaml @@ -40,6 +40,7 @@ AudioVideoStreams: - datatype == "beh" - intersects([suffix], ["audio", "video"]) fields: + StartTime__audiovideo: required AudioChannelCount: optional AudioSampleRate: optional FrameRate: optional From 540a1e79e69ca157844e597ac27b106b2d43bbd4 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 28 Oct 2025 12:59:10 -0400 Subject: [PATCH 16/28] Revert "Add StartTime field for audio/video metadata in BIDS schema" This reverts commit bf00bc64e3b9cbbcf5bd71482cf8f36086aa08d3. --- src/schema/objects/metadata.yaml | 16 ---------------- src/schema/rules/sidecars/beh.yaml | 1 - 2 files changed, 17 deletions(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index 920a8a7517..eba58d3809 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -3744,22 +3744,6 @@ StartTime: where `s` reflects whole seconds, and `.000000` reflects OPTIONAL fractional seconds. type: number unit: s -StartTime__audiovideo: - name: StartTime - display_name: Start Time - description: | - Start time in seconds in relation to the start of acquisition of the first - data sample in the corresponding dataset (negative values are allowed). - This data MAY be specified with sub-second precision using the syntax `s[.000000]`, - where `s` reflects whole seconds, and `.000000` reflects OPTIONAL fractional seconds. - For audio/video files where precise temporal alignment is not possible or applicable, - this field MAY be set to `"n/a"`. - anyOf: - - type: number - unit: s - - type: string - enum: - - n/a StationName: name: StationName display_name: Station Name diff --git a/src/schema/rules/sidecars/beh.yaml b/src/schema/rules/sidecars/beh.yaml index c283e70b73..27fc43e12f 100644 --- a/src/schema/rules/sidecars/beh.yaml +++ b/src/schema/rules/sidecars/beh.yaml @@ -40,7 +40,6 @@ AudioVideoStreams: - datatype == "beh" - intersects([suffix], ["audio", "video"]) fields: - StartTime__audiovideo: required AudioChannelCount: optional AudioSampleRate: optional FrameRate: optional From fc019ae17b254b9b1a9113f8a42ac51706b4021a Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Tue, 28 Oct 2025 13:01:41 -0400 Subject: [PATCH 17/28] Clarify behavioral recordings section by adding details on simultaneous recordings and timing alignment --- src/modality-specific-files/behavioral-experiments.md | 3 +++ 1 file changed, 3 insertions(+) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index 2f2c3cd834..acf565dfe2 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -17,6 +17,9 @@ and a guide for using macros can be found at The `beh` directory MAY store behavioral recordings such as audio (`_audio.*`) and video (`_video.*`) recordings, physiological (`_physio.*`) recordings, and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). Audio and video recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. +Audio/video recordings MAY occur simultaneously with other recordings, such as BOLD or EEG. +Relative timing between files may be determined by consulting the `scans.tsv` file. +If no `scans.tsv` file is present, the alignment is undefined. The `beh` directory MAY also contain event timing files (`_events.tsv`) and their associated metadata (`_events.json`) for behavioral experiments that do not have corresponding neuroimaging or functional data. Additionally, events files that do not include the mandatory `onset` and `duration` columns MAY be included, From d7b127032085600a0a08d8ef2e7d8bdb7c304421 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Wed, 29 Oct 2025 14:21:40 -0400 Subject: [PATCH 18/28] correct acquisition entity --- src/modality-specific-files/behavioral-experiments.md | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index acf565dfe2..f4f096746b 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -108,7 +108,7 @@ Video recordings MUST use one of the following extensions: Audio and video files MAY use the following entities: - `task` - OPTIONAL for audio and video recordings -- `acquisition` - OPTIONAL, can distinguish different recording setups +- `acq` - OPTIONAL, can distinguish different recording setups - `run` - OPTIONAL, for multiple recordings with identical parameters - `recording` - OPTIONAL, to differentiate simultaneous recordings from different angles, locations, or devices - `split` - OPTIONAL, for continuous recordings split into multiple files From 4245b5d1b11cd9fbd671e4e66c1c10e0e7261a0f Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 3 Nov 2025 11:50:25 -0500 Subject: [PATCH 19/28] feat: add support for Ogg Vorbis audio format in behavioral recordings --- src/modality-specific-files/behavioral-experiments.md | 1 + src/schema/objects/extensions.yaml | 6 ++++++ src/schema/rules/files/raw/beh.yaml | 3 ++- 3 files changed, 9 insertions(+), 1 deletion(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index f4f096746b..7f869ae852 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -95,6 +95,7 @@ Audio recordings MUST use one of the following extensions: - `.flac` - Free Lossless Audio Codec - `.mp3` - MPEG Audio Layer III +- `.ogg` - Ogg Vorbis - `.wav` - Waveform Audio File Format Video recordings MUST use one of the following extensions: diff --git a/src/schema/objects/extensions.yaml b/src/schema/objects/extensions.yaml index 22ea426324..6824adf224 100644 --- a/src/schema/objects/extensions.yaml +++ b/src/schema/objects/extensions.yaml @@ -231,6 +231,12 @@ nwb: A [Neurodata Without Borders](https://nwb-schema.readthedocs.io/en/latest/) file. Each recording consists of a single `.nwb` file. +ogg: + value: .ogg + display_name: Ogg Vorbis + description: | + An [Ogg Vorbis](https://en.wikipedia.org/wiki/Vorbis) audio file. + This format is commonly used for behavioral audio recordings. OMEBigTiff: value: .ome.btf display_name: Open Microscopy Environment BigTIFF diff --git a/src/schema/rules/files/raw/beh.yaml b/src/schema/rules/files/raw/beh.yaml index 1e143b1b70..cd9e681f4a 100644 --- a/src/schema/rules/files/raw/beh.yaml +++ b/src/schema/rules/files/raw/beh.yaml @@ -21,9 +21,10 @@ audio: suffixes: - audio extensions: + - .flac - .mp3 + - .ogg - .wav - - .flac - .json datatypes: - beh From aead0a455fdf9e7caa77e8ea08bfd404482980ee Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 3 Nov 2025 11:53:01 -0500 Subject: [PATCH 20/28] Apply suggestion from @bendichter --- src/schema/objects/extensions.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schema/objects/extensions.yaml b/src/schema/objects/extensions.yaml index 6824adf224..b7fc5621bb 100644 --- a/src/schema/objects/extensions.yaml +++ b/src/schema/objects/extensions.yaml @@ -176,7 +176,7 @@ mp3: display_name: MPEG Audio Layer III description: | An [MP3](https://en.wikipedia.org/wiki/MP3) audio file. - This format is commonly used for behavioral audio recordings. + This format is commonly used for behavioral audio recordings or auditory stimuli. mp4: value: .mp4 display_name: MPEG-4 Part 14 From 3756616f6144aea195f6f9c2699a75811c7f615d Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Wed, 3 Dec 2025 14:18:08 -0500 Subject: [PATCH 21/28] change example height value --- src/schema/objects/metadata.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index eba58d3809..a6e54de941 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -4253,7 +4253,7 @@ Height: name: Height display_name: Video Height description: | - Height of the video in pixels (for example, `1920`). + Height of the video in pixels (for example, `1080`). type: integer minimum: 1 From 272516c9172ff1fe799c4dffe084c5031b9e891f Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Wed, 3 Dec 2025 14:18:22 -0500 Subject: [PATCH 22/28] change example width value --- src/schema/objects/metadata.yaml | 2 +- 1 file changed, 1 insertion(+), 1 deletion(-) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index a6e54de941..14715cf07b 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -4261,6 +4261,6 @@ Width: name: Width display_name: Video Width description: | - Width of the video in pixels (for example, `1080`). + Width of the video in pixels (for example, `1920`). type: integer minimum: 1 From 0704e4aa53227d68045d20533461da0dcf6aaf38 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Wed, 3 Dec 2025 14:33:28 -0500 Subject: [PATCH 23/28] swap example height and width --- src/modality-specific-files/behavioral-experiments.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index 7f869ae852..86c405c5a1 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -185,8 +185,8 @@ For a video file containing both video and audio streams: "AudioChannelCount": 2, "AudioSampleRate": 48000, "FrameRate": 30.0, - "Height": 1920, - "Width": 1080, + "Height": 1080, + "Width": 1920, "Duration": 600.5 } ``` From b11d981580ae2493ac964ab3d2bdb457f3bb695a Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 15 Dec 2025 10:09:30 -0500 Subject: [PATCH 24/28] feat(schema): add audio bit depth and camera position metadata Introduce `AudioBitDepth` and `CameraPosition` object definitions, and allow them as optional fields in BEH sidecar `AudioVideoStreams` to capture richer audio recording parameters and camera setup context. --- src/schema/objects/metadata.yaml | 19 +++++++++++++++++++ src/schema/rules/sidecars/beh.yaml | 2 ++ 2 files changed, 21 insertions(+) diff --git a/src/schema/objects/metadata.yaml b/src/schema/objects/metadata.yaml index abe5469c97..73f5102e60 100644 --- a/src/schema/objects/metadata.yaml +++ b/src/schema/objects/metadata.yaml @@ -4400,6 +4400,25 @@ AudioSampleRate: exclusiveMinimum: 0 unit: Hz +AudioBitDepth: + name: AudioBitDepth + display_name: Audio Bit Depth + description: | + Number of bits per sample in the audio recording. + + Common values include `16`, `24`, or `32`. + type: integer + minimum: 1 + +CameraPosition: + name: CameraPosition + display_name: Camera Position + description: | + Free-form description of the camera placement relative to the subject or scene. + + Examples include "front", "profile-left", "ceiling", "room-corner", or "overhead". + type: string + Duration: name: Duration display_name: Duration diff --git a/src/schema/rules/sidecars/beh.yaml b/src/schema/rules/sidecars/beh.yaml index 27fc43e12f..c4abf6be49 100644 --- a/src/schema/rules/sidecars/beh.yaml +++ b/src/schema/rules/sidecars/beh.yaml @@ -42,7 +42,9 @@ AudioVideoStreams: fields: AudioChannelCount: optional AudioSampleRate: optional + AudioBitDepth: optional FrameRate: optional Height: optional Width: optional + CameraPosition: optional Duration: optional From bddeb17054d8b1012cf83b454fb788ec2d688340 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 10 Jan 2026 13:24:37 -0500 Subject: [PATCH 25/28] feat(beh): add `_audiovideo` suffix for combined audio-video recordings - Add new `_audiovideo` suffix for files containing both audio and video streams - Update documentation to distinguish between audio-only, video-only, and combined recordings - Split AudioVideoStreams sidecar table into separate AudioStreams and VideoStreams tables - Add example files and JSON sidecars for audiovideo recordings - Update schema suffixes to include audiovideo definition --- .../behavioral-experiments.md | 45 ++++++++++++++----- src/schema/objects/suffixes.yaml | 6 +++ src/schema/rules/files/raw/beh.yaml | 20 +++++++++ src/schema/rules/sidecars/beh.yaml | 14 ++++-- 4 files changed, 71 insertions(+), 14 deletions(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index 86c405c5a1..e328a78270 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -15,8 +15,8 @@ and a guide for using macros can be found at --> {{ MACROS___make_filename_template("raw", datatypes=["beh"]) }} -The `beh` directory MAY store behavioral recordings such as audio (`_audio.*`) and video (`_video.*`) recordings, physiological (`_physio.*`) recordings, and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). -Audio and video recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. +The `beh` directory MAY store behavioral recordings such as audio (`_audio.*`), video (`_video.*`), and combined audio-video (`_audiovideo.*`) recordings, physiological (`_physio.*`) recordings, and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). +Audio, video, and audio-video recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. Audio/video recordings MAY occur simultaneously with other recordings, such as BOLD or EEG. Relative timing between files may be determined by consulting the `scans.tsv` file. If no `scans.tsv` file is present, the alignment is undefined. @@ -71,10 +71,12 @@ A guide for using macros can be found at --> {{ MACROS___make_sidecar_table("beh.BEHInstitutionInformation") }} -## Audio and video recordings +## Audio, video, and audio-video recordings Audio and video recordings of behaving subjects MAY be stored in the `beh` directory -using the `_audio` and `_video` suffixes respectively. +using the `_audio`, `_video`, and `_audiovideo` suffixes. +The `_audio` suffix is for audio-only recordings, `_video` for video-only recordings, +and `_audiovideo` for recordings that contain both audio and video streams. These recordings are typically used to capture vocalizations, speech, facial expressions, body movements, or other behavioral aspects during experimental tasks or rest periods. @@ -98,7 +100,7 @@ Audio recordings MUST use one of the following extensions: - `.ogg` - Ogg Vorbis - `.wav` - Waveform Audio File Format -Video recordings MUST use one of the following extensions: +Video (and audio-video) recordings MUST use one of the following extensions: - `.mp4` - MPEG-4 Part 14 - `.mkv` - Matroska video container @@ -126,6 +128,8 @@ A guide for using macros can be found at "beh": { "sub-01_task-rest_video.mp4": "", "sub-01_task-rest_video.json": "", + "sub-01_task-interview_audiovideo.mp4": "", + "sub-01_task-interview_audiovideo.json": "", "sub-01_task-stroop_recording-face_video.mp4": "", "sub-01_task-stroop_recording-face_video.json": "", "sub-01_task-stroop_recording-room_video.mp4": "", @@ -158,9 +162,9 @@ A guide for using macros can be found at } ) }} -### Sidecar JSON for audio and video recordings +### Sidecar JSON for audio, video, and audio-video recordings -The following metadata fields are available for audio and video recordings: +The following metadata fields are available for audio, video, and audio-video recordings: {{ MACROS___make_sidecar_table("beh.AudioVideoDevice") }} -{{ MACROS___make_sidecar_table("beh.AudioVideoStreams") }} +The following fields are available for audio recordings (`_audio`) and audio-video recordings (`_audiovideo`): -### Example video sidecar JSON +{{ MACROS___make_sidecar_table("beh.AudioStreams") }} + +The following fields are available for video recordings (`_video`) and audio-video recordings (`_audiovideo`): + +{{ MACROS___make_sidecar_table("beh.VideoStreams") }} -For a video file containing both video and audio streams: +### Example audio-video sidecar JSON + +For an audio-video file containing both video and audio streams: ```JSON { @@ -191,6 +201,21 @@ For a video file containing both video and audio streams: } ``` +### Example video sidecar JSON + +For a video-only recording: + +```JSON +{ + "TaskName": "RestingState", + "Device": "Sony FDR-AX53", + "FrameRate": 30.0, + "Height": 1080, + "Width": 1920, + "Duration": 600.5 +} +``` + ### Example audio sidecar JSON For an audio-only recording: diff --git a/src/schema/objects/suffixes.yaml b/src/schema/objects/suffixes.yaml index 75f3b6b41b..099bd95c04 100644 --- a/src/schema/objects/suffixes.yaml +++ b/src/schema/objects/suffixes.yaml @@ -535,6 +535,12 @@ audio: Audio recording of a behaving subject. This may include vocalizations, speech, or environmental sounds. The audio stream may be from a standalone audio recording or extracted from a video file. +audiovideo: + value: audiovideo + display_name: Audio-Video Recording + description: | + Combined audio and video recording of a behaving subject. + This suffix is used for files that contain both audio and video streams. beh: value: beh display_name: Behavioral recording diff --git a/src/schema/rules/files/raw/beh.yaml b/src/schema/rules/files/raw/beh.yaml index cd9e681f4a..a033cc47b3 100644 --- a/src/schema/rules/files/raw/beh.yaml +++ b/src/schema/rules/files/raw/beh.yaml @@ -56,3 +56,23 @@ video: run: optional recording: optional split: optional + +# Combined audio-video recordings +audiovideo: + suffixes: + - audiovideo + extensions: + - .mp4 + - .mkv + - .avi + - .json + datatypes: + - beh + entities: + subject: required + session: optional + task: optional + acquisition: optional + run: optional + recording: optional + split: optional diff --git a/src/schema/rules/sidecars/beh.yaml b/src/schema/rules/sidecars/beh.yaml index c4abf6be49..ba9b9993eb 100644 --- a/src/schema/rules/sidecars/beh.yaml +++ b/src/schema/rules/sidecars/beh.yaml @@ -30,21 +30,27 @@ BEHInstitutionInformation: AudioVideoDevice: selectors: - datatype == "beh" - - intersects([suffix], ["audio", "video"]) + - intersects([suffix], ["audio", "video", "audiovideo"]) fields: Device: optional DeviceSerialNumber: optional + Duration: optional -AudioVideoStreams: +AudioStreams: selectors: - datatype == "beh" - - intersects([suffix], ["audio", "video"]) + - intersects([suffix], ["audio", "audiovideo"]) fields: AudioChannelCount: optional AudioSampleRate: optional AudioBitDepth: optional + +VideoStreams: + selectors: + - datatype == "beh" + - intersects([suffix], ["video", "audiovideo"]) + fields: FrameRate: optional Height: optional Width: optional CameraPosition: optional - Duration: optional From 7bb98a9439d4d1e6c491a37018bca3b89e890f9f Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 10 Jan 2026 17:09:15 -0500 Subject: [PATCH 26/28] feat(beh): add support for still image recordings in behavioral experiments Add `_image` suffix for storing still images captured during behavioral experiments in the `beh` directory. Changes include: - Add `.jpg` and `.png` as supported image file extensions - Document use cases: pose estimation training frames, behavioral setup snapshots, and extracted video frames - Update privacy/PII warnings to include images alongside audio/video - Add ImageProperties sidecar table and example files - Update AudioVideoDevice macro to AudioVideoImageDevice --- .../behavioral-experiments.md | 46 ++++++++++++++++--- src/schema/objects/suffixes.yaml | 7 +++ src/schema/rules/files/raw/beh.yaml | 18 ++++++++ src/schema/rules/sidecars/beh.yaml | 21 +++++++-- 4 files changed, 83 insertions(+), 9 deletions(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index e328a78270..5ad2801c5a 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -15,8 +15,8 @@ and a guide for using macros can be found at --> {{ MACROS___make_filename_template("raw", datatypes=["beh"]) }} -The `beh` directory MAY store behavioral recordings such as audio (`_audio.*`), video (`_video.*`), and combined audio-video (`_audiovideo.*`) recordings, physiological (`_physio.*`) recordings, and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). -Audio, video, and audio-video recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. +The `beh` directory MAY store behavioral recordings such as audio (`_audio.*`), video (`_video.*`), combined audio-video (`_audiovideo.*`), and still image (`_image.*`) recordings, physiological (`_physio.*`) recordings, and other continuous recordings (`_stim.tsv.gz`, `_stim.json`). +Audio, video, audio-video, and image recordings MAY be of subjects performing tasks, resting-state behavior, or recordings of stimuli being presented to the subject. Audio/video recordings MAY occur simultaneously with other recordings, such as BOLD or EEG. Relative timing between files may be determined by consulting the `scans.tsv` file. If no `scans.tsv` file is present, the alignment is undefined. @@ -71,7 +71,7 @@ A guide for using macros can be found at --> {{ MACROS___make_sidecar_table("beh.BEHInstitutionInformation") }} -## Audio, video, and audio-video recordings +## Audio, video, and audio-video recordings and images Audio and video recordings of behaving subjects MAY be stored in the `beh` directory using the `_audio`, `_video`, and `_audiovideo` suffixes. @@ -80,9 +80,14 @@ and `_audiovideo` for recordings that contain both audio and video streams. These recordings are typically used to capture vocalizations, speech, facial expressions, body movements, or other behavioral aspects during experimental tasks or rest periods. +Still images captured during behavioral experiments MAY be stored in the `beh` directory +using the `_image` suffix. +These images are typically used for training frames for pose estimation, +snapshots of behavioral setups, or individual frames extracted from video recordings. + !!! warning "Privacy and personally identifiable information" - Audio and video recordings of human subjects often contain personally identifiable + Audio and video recordings and images of human subjects often contain personally identifiable information (PII) such as faces, voices, and other identifying features. Data curators MUST take special care to ensure compliance with applicable privacy regulations (such as HIPAA in the United States, GDPR in the European Union, or other @@ -100,12 +105,17 @@ Audio recordings MUST use one of the following extensions: - `.ogg` - Ogg Vorbis - `.wav` - Waveform Audio File Format -Video (and audio-video) recordings MUST use one of the following extensions: +Video and audio-video recordings MUST use one of the following extensions: - `.mp4` - MPEG-4 Part 14 - `.mkv` - Matroska video container - `.avi` - Audio Video Interleave +Image files MUST use one of the following extensions: + +- `.jpg` - JPEG image +- `.png` - Portable Network Graphics + ### Entities Audio and video files MAY use the following entities: @@ -134,6 +144,8 @@ A guide for using macros can be found at "sub-01_task-stroop_recording-face_video.json": "", "sub-01_task-stroop_recording-room_video.mp4": "", "sub-01_task-stroop_recording-room_video.json": "", + "sub-01_task-rest_image.jpg": "", + "sub-01_task-rest_image.json": "", "sub-01_task-vocalization_audio.wav": "", "sub-01_task-vocalization_audio.json": "", }, @@ -174,7 +186,9 @@ The definitions of the fields specified in these tables may be found in A guide for using macros can be found at https://github.com/bids-standard/bids-specification/blob/master/macros_doc.md --> -{{ MACROS___make_sidecar_table("beh.AudioVideoDevice") }} +{{ MACROS___make_sidecar_table("beh.AudioVideoImageDevice") }} + +{{ MACROS___make_sidecar_table("beh.AudioVideoDuration") }} The following fields are available for audio recordings (`_audio`) and audio-video recordings (`_audiovideo`): @@ -184,6 +198,12 @@ The following fields are available for video recordings (`_video`) and audio-vid {{ MACROS___make_sidecar_table("beh.VideoStreams") }} +The following fields are available for image files (`_image`): + +{{ MACROS___make_sidecar_table("beh.AudioVideoImageDevice") }} + +{{ MACROS___make_sidecar_table("beh.ImageProperties") }} + ### Example audio-video sidecar JSON For an audio-video file containing both video and audio streams: @@ -230,6 +250,20 @@ For an audio-only recording: } ``` +### Example image sidecar JSON + +For a still image: + +```JSON +{ + "TaskName": "Reaching", + "Device": "GoPro Hero 10", + "Height": 1080, + "Width": 1920, + "CameraPosition": "overhead" +} +``` + ### Annotations and events Behavioral annotations or event markers for audio and video recordings diff --git a/src/schema/objects/suffixes.yaml b/src/schema/objects/suffixes.yaml index 099bd95c04..174de3da17 100644 --- a/src/schema/objects/suffixes.yaml +++ b/src/schema/objects/suffixes.yaml @@ -541,6 +541,13 @@ audiovideo: description: | Combined audio and video recording of a behaving subject. This suffix is used for files that contain both audio and video streams. +behimage: + value: image + display_name: Behavioral Image + description: | + Still image captured during behavioral experiments. + This may include training frames for pose estimation, snapshots of behavioral setups, + or individual frames extracted from video recordings. beh: value: beh display_name: Behavioral recording diff --git a/src/schema/rules/files/raw/beh.yaml b/src/schema/rules/files/raw/beh.yaml index a033cc47b3..cd645255ed 100644 --- a/src/schema/rules/files/raw/beh.yaml +++ b/src/schema/rules/files/raw/beh.yaml @@ -76,3 +76,21 @@ audiovideo: run: optional recording: optional split: optional + +# Still images +image: + suffixes: + - behimage + extensions: + - .jpg + - .png + - .json + datatypes: + - beh + entities: + subject: required + session: optional + task: optional + acquisition: optional + run: optional + recording: optional diff --git a/src/schema/rules/sidecars/beh.yaml b/src/schema/rules/sidecars/beh.yaml index ba9b9993eb..e20efe5fa1 100644 --- a/src/schema/rules/sidecars/beh.yaml +++ b/src/schema/rules/sidecars/beh.yaml @@ -26,14 +26,20 @@ BEHInstitutionInformation: InstitutionAddress: recommended InstitutionalDepartmentName: recommended -# Audio and Video metadata -AudioVideoDevice: +# Audio, Video, and Image metadata +AudioVideoImageDevice: selectors: - datatype == "beh" - - intersects([suffix], ["audio", "video", "audiovideo"]) + - intersects([suffix], ["audio", "video", "audiovideo", "behimage"]) fields: Device: optional DeviceSerialNumber: optional + +AudioVideoDuration: + selectors: + - datatype == "beh" + - intersects([suffix], ["audio", "video", "audiovideo"]) + fields: Duration: optional AudioStreams: @@ -54,3 +60,12 @@ VideoStreams: Height: optional Width: optional CameraPosition: optional + +ImageProperties: + selectors: + - datatype == "beh" + - suffix == "behimage" + fields: + Height: optional + Width: optional + CameraPosition: optional From 877beebe913e4d8aba89e37de76d4098cf87ab3c Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Sat, 10 Jan 2026 17:16:07 -0500 Subject: [PATCH 27/28] feat(beh): add optional License field for audio/video/image recordings - Add License field to AudioVideoImageDevice sidecar schema - Update documentation to include images in audio/video section headings - Add note explaining licensing considerations for recordings containing identifiable participant data --- .../behavioral-experiments.md | 12 ++++++++++-- src/schema/rules/sidecars/beh.yaml | 1 + 2 files changed, 11 insertions(+), 2 deletions(-) diff --git a/src/modality-specific-files/behavioral-experiments.md b/src/modality-specific-files/behavioral-experiments.md index 5ad2801c5a..cf8cea09e5 100644 --- a/src/modality-specific-files/behavioral-experiments.md +++ b/src/modality-specific-files/behavioral-experiments.md @@ -174,9 +174,9 @@ A guide for using macros can be found at } ) }} -### Sidecar JSON for audio, video, and audio-video recordings +### Sidecar JSON for audio, video, audio-video recordings, and images -The following metadata fields are available for audio, video, and audio-video recordings: +The following metadata fields are available for audio, video, audio-video recordings, and images: {{ MACROS___make_sidecar_table("beh.AudioVideoImageDevice") }} +!!! note "Licensing for recordings containing participants" + + Audio, video, and image recordings of participants may have different licensing + restrictions than the main dataset due to privacy considerations. The optional + `License` field can be used to specify different terms for individual recordings + that contain identifiable participant data. If not specified, the recording + inherits the license from `dataset_description.json`. + {{ MACROS___make_sidecar_table("beh.AudioVideoDuration") }} The following fields are available for audio recordings (`_audio`) and audio-video recordings (`_audiovideo`): diff --git a/src/schema/rules/sidecars/beh.yaml b/src/schema/rules/sidecars/beh.yaml index e20efe5fa1..545e7646f5 100644 --- a/src/schema/rules/sidecars/beh.yaml +++ b/src/schema/rules/sidecars/beh.yaml @@ -34,6 +34,7 @@ AudioVideoImageDevice: fields: Device: optional DeviceSerialNumber: optional + License: optional AudioVideoDuration: selectors: From cc41d49a1de0f4408381dd4903dd926b9d3cfd02 Mon Sep 17 00:00:00 2001 From: Ben Dichter Date: Mon, 12 Jan 2026 13:22:23 -0500 Subject: [PATCH 28/28] fix(schema): rename behimage suffix to image for consistency --- src/schema/objects/suffixes.yaml | 2 +- src/schema/rules/files/raw/beh.yaml | 2 +- 2 files changed, 2 insertions(+), 2 deletions(-) diff --git a/src/schema/objects/suffixes.yaml b/src/schema/objects/suffixes.yaml index 174de3da17..e282331079 100644 --- a/src/schema/objects/suffixes.yaml +++ b/src/schema/objects/suffixes.yaml @@ -541,7 +541,7 @@ audiovideo: description: | Combined audio and video recording of a behaving subject. This suffix is used for files that contain both audio and video streams. -behimage: +image: value: image display_name: Behavioral Image description: | diff --git a/src/schema/rules/files/raw/beh.yaml b/src/schema/rules/files/raw/beh.yaml index cd645255ed..4d96c62b41 100644 --- a/src/schema/rules/files/raw/beh.yaml +++ b/src/schema/rules/files/raw/beh.yaml @@ -80,7 +80,7 @@ audiovideo: # Still images image: suffixes: - - behimage + - image extensions: - .jpg - .png