Add main_grad - #1140

Open

jianyuh wants to merge 4 commits into

ngoyal_changes_for_pp_fp8from

ngoyal_changes_for_pp_fp8_fix_handle_grad_main

Commits on Oct 2, 2023

Fix fsdp+pp+te WPS decreasing issue
jianyuh
committed
81ee78d
Address comment; remove unused stuff
jianyuh
committed
71495ba
split into wps fix P841842878 only and main_grad fix
jianyuh
committed
f3ae46e
Add main_grad
jianyuh
committed
ad54660