MM-PSYCHE

Multimodal Multitask Psychological Characteristic Estimation through Cross-Domain Semi-Supervised Learning

Official implementation of MM-PSYCHE, accepted for publication in IEEE Access. The framework recognizes emotions, apparent personality traits, and ambivalence from heterogeneous multimodal corpora using cross-domain pseudo-labels for missing task annotations.

Authors

Elena Ryumina [1,4], Alexandr Axyonov [1,4], Darya Koryakovskaya [2], Timur Abdulkadirov [1], Angelina Egorova [1], Sergey Fedchin [2], Alexander Zaburdaev [2], Iana Salova [2], Andrey Savchenko [1,3], and Dmitry Ryumin [1,4]

1. HSE University, 109028 Moscow, Russia
2. HSE University, 190121 Saint Petersburg, Russia
3. Sber AI Laboratory, 117997 Moscow, Russia
4. St. Petersburg Federal Research Center of the Russian Academy of Sciences, 199178 Saint Petersburg, Russia

Framework overview

Method

MM-PSYCHE combines four modalities with a Multimodal Cross-Domain Model (MCDM):

• Task-Specific Projectors create task-conditioned prediction embeddings and project them back into the shared hidden space.
• Task-Specific Query-Based Cross-Attention Fusion uses task embeddings as queries and shared modality embeddings as keys and values.
• Guide Banks learn task-specific prototypes that anchor the final predictions.
• Cross-domain semi-supervised learning uses confident predictions as pseudo-labels for tasks that are not annotated in a sample's source corpus.
• Dual-branch GradNorm independently balances supervised and pseudo-labeled task losses.

Cross-domain training

Tasks and corpora

Corpus	Role in the paper	Target	Output
CMU-MOSEI	Training and target-corpus evaluation	Emotion recognition	Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise
First Impressions V2	Training and target-corpus evaluation	Apparent personality trait recognition	Big Five scores in [0, 1]
BAH	Training and target-corpus evaluation	Ambivalence recognition	Absence / Presence
MELD	Zero-shot emotion testing only	Emotion recognition	Seven emotion classes

MELD is not used for training, pseudo-label generation, or hyperparameter selection in the paper.

Main paper results

The tables reproduce the paper's comparison with single-task state-of-the-art methods. Values are percentages; square brackets show bootstrap confidence intervals. SL denotes supervised learning and SSL semi-supervised learning. The best results are bold and the second-best are italicized.

CMU-MOSEI

Method	Learning type	Learning domain	mWACC	mMF1
Arumugam et al. (2025)	SL	Single-domain	62.1	--
Peng et al. (2024)	SL	Single-domain	66.4	--
Chen et al. (2026)	SL	Single-domain	67.4	66.1
MM-PSYCHE with single-task SL	SL	Single-domain	63.6 [62.9, 64.3]	63.3 [62.7, 64.0]
MM-PSYCHE with single-task SSL	SSL	Cross-domain	68.9 [68.2, 69.6]	69.3 [68.5, 70.0]
MM-PSYCHE with multitask SSL	SSL	Cross-domain	63.6 [62.9, 64.3]	63.1 [62.4, 63.9]

MELD (testing only)

Method	Learning type	Learning domain	WF1	MF1
MM-PSYCHE with single-task SL	SL	Single-domain	27.0 [25.1, 28.9]	22.8 [21.1, 24.7]
MM-PSYCHE with single-task SSL	SSL	Cross-domain	30.4 [28.7, 32.4]	27.5 [25.6, 29.9]
MM-PSYCHE with multitask SSL	SSL	Cross-domain	44.1 [42.1, 46.1]	35.0 [33.0, 37.3]

First Impressions V2

Method	Learning type	Learning domain	mACC	CCC
Ning et al. (2026)	SL	Single-domain	91.8	--
Zhang et al. (2025)	SL	Single-domain	92.0	--
Wang et al. (2025)	SL	Single-domain	92.1	--
MM-PSYCHE with single-task SL	SL	Single-domain	91.8 [91.7, 92.0]	74.0 [72.6, 75.2]
MM-PSYCHE with single-task SSL	SSL	Cross-domain	92.6 [92.5, 92.8]	77.2 [75.8, 78.5]
MM-PSYCHE with multitask SSL	SSL	Cross-domain	91.9 [91.8, 92.1]	72.0 [70.6, 73.3]

BAH

Method	Learning type	Learning domain	WF1	MF1
Kollias et al. (2025)	SL	Single-domain	70.0	--
Hallmen et al. (2025)	SL	Single-domain	70.2	--
Savchenko and Savchenko (2025)	SL	Single-domain	71.0	--
MM-PSYCHE with single-task SL	SL	Single-domain	72.9 [68.5, 77.2]	71.5 [66.6, 76.0]
MM-PSYCHE with single-task SSL	SSL	Cross-domain	73.2 [68.9, 77.8]	72.1 [67.6, 76.4]
MM-PSYCHE with multitask SSL	SSL	Cross-domain	73.3 [68.6, 77.8]	71.7 [66.7, 76.2]

Single-task SSL gives the strongest target-corpus results. Multitask SSL gives the clearest gain under domain shift: on unseen MELD it raises macro-F1 from 27.5% to 35.0% relative to single-task SSL.

Implementation notes

The code supports training and evaluation on cmu_mosei, fiv2, and bah, including feature caching, three MCDM variants, pseudo-labeling with dual-branch GradNorm, ablations, and hyperparameter search. MELD evaluation and the full candidate-encoder sweep are maintained separately.

Behavior descriptions are generated with Qwen3-VL-4B-Instruct using pytorch_Qwen3-VL.py. The provided CSV files already contain these descriptions.

Repository structure

.
|-- main.py                         # Training/search entry point
|-- config.toml                     # Dataset, model, SSL, and ablation settings
|-- search_params.toml              # Hyperparameter-search space
|-- requirements.txt                # CUDA 11.8 environment
|-- pytorch_Qwen3-VL.py             # Qwen3-VL behavior-description utility
|-- figures/                        # Figures from the paper
|-- data/                           # Prepared annotation CSV files
`-- src/
    |-- data_loading/               # Indexing, preprocessing, extractors, caching
    |-- models/                     # MCDM variants and attention modules
    |-- train.py                    # Training, evaluation, checkpointing
    `-- utils/                      # Losses, GradNorm, metrics, logging, search

Installation

The pinned environment targets Python 3.10+ and PyTorch 2.6 with CUDA 11.8:

python -m venv .venv
# Windows PowerShell:
.venv\Scripts\Activate.ps1
# Linux:
source .venv/bin/activate
python -m pip install --upgrade pip
pip install -r requirements.txt

Pre-trained Hugging Face models are downloaded on first use. For CPU-only execution, install a compatible CPU PyTorch stack separately and set device = "cpu".

Telegram notifications are optional. To use them, install python-dotenv, create .env, and define TELEGRAM_BOT_TOKEN and TELEGRAM_CHAT_ID. Otherwise, set use_telegram = false in config.toml.

The Qwen3-VL helper has additional optional dependencies that are not part of the training environment:

pip install polars qwen-vl-utils flash-attn

Data preparation

This repository includes prepared CSV annotation tables under data/, but it does not redistribute the source video or audio. Obtain each corpus from its official provider and comply with its license, consent terms, and access policy.

For each configured dataset, the loader expects:

DATASET_ROOT/
|-- train_full_with_description.csv
|-- dev_full_with_description.csv
|-- test_full_with_description.csv
|-- video/
|   |-- train/
|   |-- dev/
|   `-- test/
`-- audio/
    |-- train/
    |-- dev/
    `-- test/

The value in video_name is matched against the basename of a media file. The required CSV columns are:

Dataset	Required labels	Additional modality columns
cmu_mosei	Neutral, Anger, Disgust, Fear, Happiness, Sadness, Surprise	video_name, text, text_llm
fiv2	openness, conscientiousness, extraversion, agreeableness, non-neuroticism	video_name, text, text_llm
bah	absence_full, presence_full	video_name, text, text_llm

The behavior column is controlled by dataloader.text_description_column; text_llm is the default.

The complete behavior-description prompt and examples from the paper are available in figures/prompts.jpg.

Running an experiment

1. Edit the three base_dir paths in config.toml.
2. Disable Telegram unless it is configured.
3. For a normal training run, set:

[general]
use_telegram = false

[train.general]
search_type = "none"

4. Start training:

python main.py

The committed configuration contains the paper's selected core hyperparameters: hidden and output dimensions of 512, eight attention heads, dropout 0.15, Adam with a learning rate of 1e-4, a plateau scheduler, and the selected SSL/GradNorm values.

Feature preparation only

To extract and cache features without training, set:

[dataloader]
prepare_only = true

Then run python main.py. Cached embeddings are written to train.general.save_feature_path (by default ./features/). Use cache.force_reextract to refresh selected modalities.

Hyperparameter search

Set train.general.search_type to greedy or exhaustive. Candidate values are read from search_params.toml.

The committed value is exhaustive, so change it to none unless a full search is intentional.

Ablations

The [ablation] section of config.toml can disable:

• graph layers (use_graph);
• query-based attention (use_attention);
• Guide Banks (use_guidebank);
• Task-Specific Projectors (use_task_projectors);
• any of face, audio, text, or behavior;
• any of emotion, personality, or ah.

Run artifacts are saved under results/results_<model>_<timestamp>/, including the configuration snapshot, logs, epoch metrics, search overrides, and checkpoints.

Citation

Until the final DOI, volume, and page numbers are assigned, use:

@article{ryumina2026mmpsyche,
  title   = {MM-PSYCHE: Multimodal Multitask Psychological Characteristic
             Estimation through Cross-Domain Semi-Supervised Learning},
  author  = {Ryumina, Elena and Axyonov, Alexandr and Koryakovskaya, Darya and
             Abdulkadirov, Timur and Egorova, Angelina and Fedchin, Sergey and
             Zaburdaev, Alexander and Salova, Iana and Savchenko, Andrey and
             Ryumin, Dmitry},
  journal = {IEEE Access},
  year    = {2026},
  note    = {Accepted for publication}
}

Ethics and responsible use

MM-PSYCHE outputs are probabilistic research estimates, not clinical diagnoses. Use requires informed consent, domain-specific validation, bias assessment, uncertainty reporting, and human oversight; the system should not be used alone for high-stakes decisions.

License

The code is released under the MIT License. Dataset files and pre-trained models remain subject to their providers' licenses and terms.