Add main grad before fwd pass - #1142

Draft

vedanuj wants to merge 5 commits into

ngoyal_changes_for_pp_fp8from

add_main_grad_before_fwd

Commits on Oct 10, 2023

changes for main_grad before fwd
vedanuj
committed
88589c1
move changes after orig_grad_data
vedanuj
committed
f9083cf
address comments
vedanuj
committed
c116927
ensure grads are not downcasted to bf16
vedanuj
committed
e8f54b6
guard main_grad None
vedanuj
committed
8cf28fa