Skip to content

[DO NOT REVIEW] Handle unevenly sharded expert weights in FSDP compute weights - #4371

Draft
anijain2305 wants to merge 1 commit into
mxfp8-grouped-expertsfrom
mxfp8-uneven-shard-padding
Draft

[DO NOT REVIEW] Handle unevenly sharded expert weights in FSDP compute weights#4371
anijain2305 wants to merge 1 commit into
mxfp8-grouped-expertsfrom
mxfp8-uneven-shard-padding

Conversation

@anijain2305

Copy link
Copy Markdown
Contributor

An expert count that does not divide the FSDP degree leaves the last rank
with a short shard. FSDP's contract is that fsdp_pre_all_gather returns the
padded sharded size and carries the logical size in metadata, so that
fsdp_post_all_gather can drop the padding. The compute-weight hook did
neither: it rejected the layout outright with a message about dimension 0
divisibility.

Pad the communication tensor to ceil(outer_size[0] / mesh.size()) and return
outer_size as metadata; narrow back to it before quantizing. Without the
narrow, padding rows would occupy real 32x32 scale tiles and, for a grouped
weight, appear to the grouped GEMM as extra experts.

Reject Shard(1) explicitly rather than by implication. TorchTitan selects it
for grouped experts when the FSDP degree exceeds the expert count, which any
job with more ranks than experts hits -- gpt_oss has 8. The all-gather then
concatenates dim-1 shards along dim 0, so the hook would have to rebuild the
logical weight before quantizing; that is left for later, and the error now
names the condition and the remedies. The shard dimension is recovered by
comparing the local shard against the logical size, since FSDP does not pass
it.

Add coverage for both: uneven dim 0 asserts finite gradients of the right
shape rather than mere completion, and Shard(1) is a strict xfail so the
marker has to be removed when support lands.

Note the expert count must stay a power of two independently of any of this:
TorchAO's scale-rearrange kernel does tl.arange over the token groups, so
E=3, 5, 6 and 12 fail to compile the kernel with or without FSDP.


Stack created with GitHub Stacks CLIGive Feedback 💬

@meta-cla meta-cla Bot added the CLA Signed This label is managed by the Meta Open Source bot. label Aug 28, 2026
@anijain2305
anijain2305 marked this pull request as draft August 28, 2026 22:44
@anijain2305 anijain2305 changed the title Handle unevenly sharded expert weights in FSDP compute weights [DO NOT REVIEW] Handle unevenly sharded expert weights in FSDP compute weights Aug 28, 2026
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 2211ec9 to 7a5bc02 Compare August 31, 2026 06:00
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 7a5bc02 to 606e17d Compare August 31, 2026 06:08
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 606e17d to 18a26d2 Compare August 31, 2026 18:23
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 18a26d2 to 7781e15 Compare August 31, 2026 18:38
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 7781e15 to 6bad387 Compare August 31, 2026 21:17
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 6bad387 to 1206932 Compare August 31, 2026 21:44
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 1206932 to 5995c6e Compare August 31, 2026 22:15
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from 5995c6e to b5ac523 Compare September 1, 2026 05:43
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from b5ac523 to b3fd02e Compare September 1, 2026 05:53
An expert count that does not divide the FSDP degree leaves the last rank
with a short shard. FSDP's contract is that fsdp_pre_all_gather returns the
*padded* sharded size and carries the logical size in metadata, so that
fsdp_post_all_gather can drop the padding. The compute-weight hook did
neither: it rejected the layout outright with a message about dimension 0
divisibility.

Pad the communication tensor to ceil(outer_size[0] / mesh.size()) and return
outer_size as metadata; narrow back to it before quantizing. Without the
narrow, padding rows would occupy real 32x32 scale tiles and, for a grouped
weight, appear to the grouped GEMM as extra experts.

Reject Shard(1) explicitly rather than by implication. TorchTitan selects it
for grouped experts when the FSDP degree exceeds the expert count, which any
job with more ranks than experts hits -- gpt_oss has 8. The all-gather then
concatenates dim-1 shards along dim 0, so the hook would have to rebuild the
logical weight before quantizing; that is left for later, and the error now
names the condition and the remedies. The shard dimension is recovered by
comparing the local shard against the logical size, since FSDP does not pass
it.

Add coverage for both: uneven dim 0 asserts finite gradients of the right
shape rather than mere completion, and Shard(1) is a strict xfail so the
marker has to be removed when support lands.

Note the expert count must stay a power of two independently of any of this:
TorchAO's scale-rearrange kernel does tl.arange over the token groups, so
E=3, 5, 6 and 12 fail to compile the kernel with or without FSDP.
@anijain2305
anijain2305 force-pushed the mxfp8-uneven-shard-padding branch from b3fd02e to 844ac8c Compare September 1, 2026 06:13
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

CLA Signed This label is managed by the Meta Open Source bot.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant