Skip to content

Add MaskedVisionTransformerDecoder#1978

Open
lorinczszabolcs wants to merge 6 commits into
lightly-ai:masterfrom
lorinczszabolcs:add-masked-vision-transformer-decoder
Open

Add MaskedVisionTransformerDecoder#1978
lorinczszabolcs wants to merge 6 commits into
lightly-ai:masterfrom
lorinczszabolcs:add-masked-vision-transformer-decoder

Conversation

@lorinczszabolcs

Copy link
Copy Markdown
Contributor

This PR adds a reusable MaskedVisionTransformerDecoder abstraction, resolving #1615.

The ViT-decoder logic was implemented independently for every method (MAEDecoderTIMM, PixioDecoderTIMM, IJEPAPredictorTIMM), even though these decoders share the same structure: a mask token, a fixed 2D sin-cos positional embedding, a stack of transformer blocks, and a final norm. This PR consolidates that structure into a single MaskedVisionTransformerDecoder class, mirroring the existing MaskedVisionTransformer encoder abstraction. Unlike the encoder it takes patch tokens (not images) as input, and the input embedding and prediction head are kept in the model so the decoder stays modular and reusable.

Closes #1615.

What's included

  • lightly/models/modules/masked_vision_transformer_decoder.py — new abstract MaskedVisionTransformerDecoder (mirrors MaskedVisionTransformer, but for tokens): a concrete preprocess that adds the positional embedding and applies
    masking via the idx_mask / idx_keep / mask convention, plus abstract forward / add_pos_embed / decode / sequence_length.
  • lightly/models/modules/masked_vision_transformer_decoder_timm.pyMaskedVisionTransformerDecoderTIMM, the TIMM backend (timm Blocks, fixed 2D sin-cos positional embedding, mask token, final norm), following the interface
    proposed in the issue.
  • lightly/models/modules/ijepa_timm.pyIJEPAPredictorTIMM now builds on MaskedVisionTransformerDecoderTIMM, reusing the shared positional embedding, transformer blocks, norm, and mask token, and keeping its own input embedding,
    prediction projection, and multi-mask forward. The public constructor and forward(x, masks_x, masks) signatures are unchanged and the outputs are numerically identical.
  • lightly/models/modules/__init__.py — exports MaskedVisionTransformerDecoderTIMM.
  • benchmarks/imagenet/vitb16/{mae,pixio}.py and examples/{pytorch,pytorch_lightning,pytorch_lightning_distributed}/{mae,pixio}.py (+ regenerated notebooks) — migrated to the new decoder using the encoder → embed → decoder → prediction_head pattern from the issue. The change is behaviour-preserving.
  • tests/models/modules/test_masked_vision_transformer_decoder_timm.py — new tests for the decoder.

Testing

  • New tests cover: forward output shapes (with and without prefix tokens, CPU + CUDA), the masking semantics (idx_mask, mask, idx_keep), the frozen 2D sin-cos positional embedding, the idx_mask + mask error path, equivalence of the decoder core to MAEDecoderTIMM.decode, and a full-flow test proving the MAE migration is bit-identical to the previous flow.
  • The existing MAE / Pixio / I-JEPA decoder tests remain green (the migration is non-breaking).
  • make format, mypy, and ruff pass locally, make generate-example-notebooks produces no diff, and the test suite passes.

Notes

  • Backward compatibility: MAEDecoder, MAEDecoderTIMM, PixioDecoderTIMM, IJEPAPredictor, and IJEPAPredictorTIMM remain exported and usable. First-party code (benchmarks and examples) now uses the new abstraction, mirroring how MaskedVisionTransformer was introduced alongside MAEEncoder / MAEDecoder.
  • I-JEPA state_dict: the refactor renames IJEPAPredictorTIMM's internal submodules (predictor_blocksblocks, predictor_normnorm, predictor_pos_embedpos_embed). The public API and numerics are unchanged, but existing I-JEPA checkpoints no longer load with strict=True. This is within IJEPAPredictorTIMM's documented experimental status.
  • I-JEPA masking: as anticipated in the issue, I-JEPA's multi-target masking does not generalize cleanly into the shared idx_mask / idx_keep preprocess, so IJEPAPredictorTIMM keeps its own forward and reuses the shared building blocks (positional embedding, transformer blocks, norm) via decode().
  • Backends: the issue and the shared building blocks are TIMM-based, so only a TIMM backend ships. The abstract base leaves the door open for a torchvision backend once a method needs one.

@lorinczszabolcs

Copy link
Copy Markdown
Contributor Author

@gabrielfruet this is for #1615 (open since 2024). Came up during the Pixio work: it added a third decoder with the same structure as MAEDecoderTIMM and IJEPAPredictorTIMM, so this consolidates them into one MaskedVisionTransformerDecoder. MAE/Pixio migrated, I-JEPA reuses it, non-breaking.

@mrpositron

Copy link
Copy Markdown

/review

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Add MaskedVisionTransformerDecoder

2 participants