[DO NOT REVIEW] Handle unevenly sharded expert weights in FSDP compute weights - #4371
Draft
anijain2305 wants to merge 1 commit into
Draft
[DO NOT REVIEW] Handle unevenly sharded expert weights in FSDP compute weights#4371anijain2305 wants to merge 1 commit into
anijain2305 wants to merge 1 commit into
Conversation
anijain2305
requested review from
fegin,
tianyu-l,
wconstab and
wwwjn
as code owners
August 28, 2026 22:44
anijain2305
marked this pull request as draft
August 28, 2026 22:44
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
August 31, 2026 06:00
2211ec9 to
7a5bc02
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
August 31, 2026 06:08
7a5bc02 to
606e17d
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
August 31, 2026 18:23
606e17d to
18a26d2
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
August 31, 2026 18:38
18a26d2 to
7781e15
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
August 31, 2026 21:17
7781e15 to
6bad387
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
August 31, 2026 21:44
6bad387 to
1206932
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
August 31, 2026 22:15
1206932 to
5995c6e
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
September 1, 2026 05:43
5995c6e to
b5ac523
Compare
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
September 1, 2026 05:53
b5ac523 to
b3fd02e
Compare
An expert count that does not divide the FSDP degree leaves the last rank with a short shard. FSDP's contract is that fsdp_pre_all_gather returns the *padded* sharded size and carries the logical size in metadata, so that fsdp_post_all_gather can drop the padding. The compute-weight hook did neither: it rejected the layout outright with a message about dimension 0 divisibility. Pad the communication tensor to ceil(outer_size[0] / mesh.size()) and return outer_size as metadata; narrow back to it before quantizing. Without the narrow, padding rows would occupy real 32x32 scale tiles and, for a grouped weight, appear to the grouped GEMM as extra experts. Reject Shard(1) explicitly rather than by implication. TorchTitan selects it for grouped experts when the FSDP degree exceeds the expert count, which any job with more ranks than experts hits -- gpt_oss has 8. The all-gather then concatenates dim-1 shards along dim 0, so the hook would have to rebuild the logical weight before quantizing; that is left for later, and the error now names the condition and the remedies. The shard dimension is recovered by comparing the local shard against the logical size, since FSDP does not pass it. Add coverage for both: uneven dim 0 asserts finite gradients of the right shape rather than mere completion, and Shard(1) is a strict xfail so the marker has to be removed when support lands. Note the expert count must stay a power of two independently of any of this: TorchAO's scale-rearrange kernel does tl.arange over the token groups, so E=3, 5, 6 and 12 fail to compile the kernel with or without FSDP.
anijain2305
force-pushed
the
mxfp8-uneven-shard-padding
branch
from
September 1, 2026 06:13
b3fd02e to
844ac8c
Compare
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
An expert count that does not divide the FSDP degree leaves the last rank
with a short shard. FSDP's contract is that fsdp_pre_all_gather returns the
padded sharded size and carries the logical size in metadata, so that
fsdp_post_all_gather can drop the padding. The compute-weight hook did
neither: it rejected the layout outright with a message about dimension 0
divisibility.
Pad the communication tensor to ceil(outer_size[0] / mesh.size()) and return
outer_size as metadata; narrow back to it before quantizing. Without the
narrow, padding rows would occupy real 32x32 scale tiles and, for a grouped
weight, appear to the grouped GEMM as extra experts.
Reject Shard(1) explicitly rather than by implication. TorchTitan selects it
for grouped experts when the FSDP degree exceeds the expert count, which any
job with more ranks than experts hits -- gpt_oss has 8. The all-gather then
concatenates dim-1 shards along dim 0, so the hook would have to rebuild the
logical weight before quantizing; that is left for later, and the error now
names the condition and the remedies. The shard dimension is recovered by
comparing the local shard against the logical size, since FSDP does not pass
it.
Add coverage for both: uneven dim 0 asserts finite gradients of the right
shape rather than mere completion, and Shard(1) is a strict xfail so the
marker has to be removed when support lands.
Note the expert count must stay a power of two independently of any of this:
TorchAO's scale-rearrange kernel does tl.arange over the token groups, so
E=3, 5, 6 and 12 fail to compile the kernel with or without FSDP.
Stack created with GitHub Stacks CLI • Give Feedback 💬