Add MaskedVisionTransformerDecoder#1978
Open
lorinczszabolcs wants to merge 6 commits into
Open
Conversation
Contributor
Author
|
@gabrielfruet this is for #1615 (open since 2024). Came up during the Pixio work: it added a third decoder with the same structure as MAEDecoderTIMM and IJEPAPredictorTIMM, so this consolidates them into one MaskedVisionTransformerDecoder. MAE/Pixio migrated, I-JEPA reuses it, non-breaking. |
|
/review |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
This PR adds a reusable
MaskedVisionTransformerDecoderabstraction, resolving #1615.The ViT-decoder logic was implemented independently for every method (
MAEDecoderTIMM,PixioDecoderTIMM,IJEPAPredictorTIMM), even though these decoders share the same structure: a mask token, a fixed 2D sin-cos positional embedding, a stack of transformer blocks, and a final norm. This PR consolidates that structure into a singleMaskedVisionTransformerDecoderclass, mirroring the existingMaskedVisionTransformerencoder abstraction. Unlike the encoder it takes patch tokens (not images) as input, and the input embedding and prediction head are kept in the model so the decoder stays modular and reusable.Closes #1615.
What's included
lightly/models/modules/masked_vision_transformer_decoder.py— new abstractMaskedVisionTransformerDecoder(mirrorsMaskedVisionTransformer, but for tokens): a concretepreprocessthat adds the positional embedding and appliesmasking via the
idx_mask/idx_keep/maskconvention, plus abstractforward/add_pos_embed/decode/sequence_length.lightly/models/modules/masked_vision_transformer_decoder_timm.py—MaskedVisionTransformerDecoderTIMM, the TIMM backend (timmBlocks, fixed 2D sin-cos positional embedding, mask token, final norm), following the interfaceproposed in the issue.
lightly/models/modules/ijepa_timm.py—IJEPAPredictorTIMMnow builds onMaskedVisionTransformerDecoderTIMM, reusing the shared positional embedding, transformer blocks, norm, and mask token, and keeping its own input embedding,prediction projection, and multi-mask
forward. The public constructor andforward(x, masks_x, masks)signatures are unchanged and the outputs are numerically identical.lightly/models/modules/__init__.py— exportsMaskedVisionTransformerDecoderTIMM.benchmarks/imagenet/vitb16/{mae,pixio}.pyandexamples/{pytorch,pytorch_lightning,pytorch_lightning_distributed}/{mae,pixio}.py(+ regenerated notebooks) — migrated to the new decoder using theencoder → embed → decoder → prediction_headpattern from the issue. The change is behaviour-preserving.tests/models/modules/test_masked_vision_transformer_decoder_timm.py— new tests for the decoder.Testing
idx_mask,mask,idx_keep), the frozen 2D sin-cos positional embedding, theidx_mask+maskerror path, equivalence of the decoder core toMAEDecoderTIMM.decode, and a full-flow test proving the MAE migration is bit-identical to the previous flow.make format,mypy, andruffpass locally,make generate-example-notebooksproduces no diff, and the test suite passes.Notes
MAEDecoder,MAEDecoderTIMM,PixioDecoderTIMM,IJEPAPredictor, andIJEPAPredictorTIMMremain exported and usable. First-party code (benchmarks and examples) now uses the new abstraction, mirroring howMaskedVisionTransformerwas introduced alongsideMAEEncoder/MAEDecoder.state_dict: the refactor renamesIJEPAPredictorTIMM's internal submodules (predictor_blocks→blocks,predictor_norm→norm,predictor_pos_embed→pos_embed). The public API and numerics are unchanged, but existing I-JEPA checkpoints no longer load withstrict=True. This is withinIJEPAPredictorTIMM's documented experimental status.idx_mask/idx_keeppreprocess, soIJEPAPredictorTIMMkeeps its ownforwardand reuses the shared building blocks (positional embedding, transformer blocks, norm) viadecode().