Skip to content

fix(sm90): sanitize pruned FP8 MegaMoE weight blocks - #68

Open
ercaoeatting wants to merge 1 commit into
sgl-project:devfrom
ercaoeatting:agent/sm90-megamoe-tiny-scale-fix
Open

fix(sm90): sanitize pruned FP8 MegaMoE weight blocks#68
ercaoeatting wants to merge 1 commit into
sgl-project:devfrom
ercaoeatting:agent/sm90-megamoe-tiny-scale-fix

Conversation

@ercaoeatting

@ercaoeatting ercaoeatting commented Aug 1, 2026

Copy link
Copy Markdown

Summary

SM90 MegaMoE can produce incorrect output when a pruned 128x128 FP8 weight block retains arbitrary FP8 bytes while its weight scale is tiny but nonzero.

This change sanitizes those blocks during transform_weights_for_mega_moe_sm90:

  • Treat scales with abs(scale) < 1e-12 as pruned.
  • Zero the corresponding FP8 weight block before the SM90 fused kernel sees it.
  • Replace the tiny scale with the expert's median normal scale, or a finite fallback when the expert is entirely pruned.
  • Clone only when sanitization is needed, so caller-owned tensors are not modified and the existing no-tiny-scale path keeps its identity/behavior.

The same handling is applied to both W13 and W2 weights.

Root cause

The SM90 MegaMoE split-MN path rescales partial results with reciprocals of the weight scales. A tiny nonzero scale can therefore amplify residual FP8 bytes in a block into a large incorrect result. A truly pruned block must be represented as zero data with a safe finite scale.

Validation

On an H20D (SM90) in the container:

  • test_mega_moe_sm90_weight_transform.py --num-processes 1 — PASS
  • test_mega_moe_pre_dispatch_sm90.py --num-processes 1 — 8 cases PASS
  • SM90 fused-only MegaMoE smoke (tokens=1, experts=1, topk=1) — kernel launched successfully (52.5 us, non-fatal symmetric-memory multicast warning)
  • The prior minimal repro produced max_abs=1870659584.0 with the unsanitized tiny-scale block and max_abs=0.0 after sanitization.

The SGLang-side workaround can be removed once this fix is available in a released DeepGEMM package.

@ercaoeatting
ercaoeatting marked this pull request as ready for review August 1, 2026 09:39
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant