Skip to content

[Qwen3.5] Fix MoE expert weight layouts in state dict adapter - #4373

Open
taking-lying-flat wants to merge 1 commit into
pytorch:mainfrom
taking-lying-flat:fix/qwen35-moe-state-dict-layout
Open

[Qwen3.5] Fix MoE expert weight layouts in state dict adapter#4373
taking-lying-flat wants to merge 1 commit into
pytorch:mainfrom
taking-lying-flat:fix/qwen35-moe-state-dict-layout

Conversation

@taking-lying-flat

Copy link
Copy Markdown
Contributor

Summary

  • align Qwen3.5 MoE grouped expert conversion with the current Transformers state-dict layouts
  • split and fuse gate_up_proj along the intermediate dimension without transposing
  • preserve the shared [E, D, F] layout for HF down_proj and TorchTitan w2_EDF

Transformers stores gate_up_proj as [E, 2F, D] and down_proj as [E, D, F]. TorchTitan stores w1_EFD and w3_EFD as [E, F, D], and w2_EDF as [E, D, F].

Testing

  • pre-commit formatting, AST, flake8, pydoclint, codespell, and repository hygiene hooks
  • targeted Pyrefly check: 0 errors
  • shape/value round-trip with asymmetric expert dimensions
  • asymmetric expert forward parity between HF and TorchTitan layouts

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 29, 2026
@tianyu-l
tianyu-l requested a review from ivy-zhou August 29, 2026 00:08
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants