Integrate RAD's LPT recipes - #37

Open

nfrumkin wants to merge 174 commits into

mainfrom

Commits on Apr 16, 2026

expr: mi308 config
hann-wang
committed
2bdd509

Commits on Apr 20, 2026

fix: fsdp sharded param without gradient
hann-wang
committed
40fd074

Commits on Apr 27, 2026

Merge branch 'han/static_clip' into han/gpt_oss_mi308
hann-wang
committed
a0a1056

Commits on Apr 28, 2026

Merge remote-tracking branch 'origin/han/static_clip' into han/gpt_oss_mi308
hann-wang
committed
f1a8f49
expr: mxfp4 with mbs
hann-wang
committed
a03db10
chore: refactor mbs to use the same grid for both 1D/2D blocks
hann-wang
committed
b6e082f
chore: skip unnecessary init
hann-wang
committed
27fa9e4
fix: rescale gradients only when clip_mode=static
hann-wang
committed
86f0f28
Merge branch 'han/static_clip' into han/gpt_oss_mi308
hann-wang
committed
fdc2de0

Commits on May 11, 2026

expr: mixed-precision
hann-wang
committed
145e68d
Merge remote-tracking branch 'origin/main' into han/gpt_oss_mi308
hann-wang
committed
c585096
fix: llama3_8b_lpt config
hann-wang
committed
f5f68ae

Commits on May 18, 2026

expr: decomp linear
hann-wang
committed
856cbb2
expr: k8s entrypoint
hann-wang
committed
8532ab2

Commits on May 25, 2026

Merge remote-tracking branch 'origin/main' into han/gpt_oss_mi308
hann-wang
committed
5337b61
hotfix: init failure on meta device
hann-wang
committed
2cf03c6

Commits on May 27, 2026

mounted submodule, smoke scripts
YBouquet
committed
4748662
adahop internal packaging
YBouquet
committed
3513023
calibration wrapper, swap_params extension
YBouquet
committed
f9499c7
logging
YBouquet
committed
9875451
fixture TrainingOpConfig
YBouquet
committed
931d22c

Commits on May 28, 2026

Merge branch 'han/weight-deosc' into han/gpt_oss_mi308
hann-wang
committed
e4331b8
feat: enable de-osc by env var
hann-wang
committed
2c6667b
hotfix: de-osc with FSDP for GPT-OSS 20B model
hann-wang
committed
d6dc693

Commits on May 29, 2026

Commits on Jun 1, 2026

Merge branch 'fix/mxfp4-backward-dtype' into yann_adahop
YBouquet
committed
1e46eac
Merge branch 'yann_adahop' of github.com:AMD-AGI/ALTO into yann_adahop
YBouquet
committed
e05ab9b
unwrap DTensor in discovery and preserve sharding in re-swap
YBouquet
committed
19e102e
Merge branch 'yann_adahop' of github.com:AMD-AGI/ALTO into yann_adahop
YBouquet
committed
f0364fe
update dispatch to detect callback on the source wrapper
YBouquet
committed
187b65a
Merge branch 'yann_adahop' of github.com:AMD-AGI/ALTO into yann_adahop
YBouquet
committed
acf6f0b
fsdp_post_all_gather in MXFP4CalibrationWrapper
YBouquet
committed
20fdbbb
Merge branch 'yann_adahop' of github.com:AMD-AGI/ALTO into yann_adahop
YBouquet
committed
ac8b92e
use pre-hooks for inputs and weights
YBouquet
committed
1bba2e1
Merge branch 'yann_adahop' of github.com:AMD-AGI/ALTO into yann_adahop
YBouquet
committed
e78d2ff
add log for torch_function *mxfp4* path
YBouquet
committed
8b99288
update logs
YBouquet
committed
2298c7f
update log
YBouquet
committed
f75bceb
remove pre-hook, try wrapper again by solving the id mismatch issue
YBouquet
committed
083737c
fix: attach calibration callback to canonical wrapper, not weight.data
YBouquet
committed
2a3addd
log for ngpu=2 debugging
YBouquet
committed
e137966
debug fsdp state
YBouquet
committed
0df3c96
fix: attach calibration callback via fsdpparam._shared_local_tensor
YBouquet
committed
66e4948
fix cdna3
YBouquet
committed
9733642
lpt_adahop recipe
YBouquet
committed
f22e06c
port adahop strategies (inner_outlier_extract_[right/left])
YBouquet
committed
c010697
fix: iht_quantization.py imports from torchtitan.experiments.kernels.hadamard_transform.hadamard
YBouquet
committed
18aa82a
fix: explicit importation for foid
YBouquet
committed
bfb17ab
increase number of iteration steps
YBouquet
committed
ce5418b

Commits on Jun 2, 2026

Commits on Jun 3, 2026

expr: decrease lbs of llama3-8b
hann-wang
committed
2b06b23

Commits on Jun 7, 2026

solve llama parallelization issue
YBouquet
committed
6969041
llama 3 registry
YBouquet
committed
e179f5e

Commits on Jun 8, 2026

fix: nvfp4 de-osc padding
hann-wang
committed
f93288b
config registry for gpt oss pretrain
YBouquet
committed
262631d

Commits on Jun 9, 2026

Commits on Jun 11, 2026

add midmax implementation
nfrumkin
committed
80aca3f

Commits on Jun 12, 2026

Add new c4 dset paths
nfrumkin
committed
dfbbd6d

Commits on Jun 16, 2026

Merge branch 'main' into yann_adahop
YBouquet
committed
3fa55e3
debug observers
YBouquet
committed
883a7c8
update config registry
YBouquet
committed
1d50932
Merge branch 'yann_adahop' of github.com:AMD-AGI/ALTO into yann_adahop
YBouquet
committed
74b90e9

Commits on Jun 17, 2026

fix viz
YBouquet
committed
f10f283
Merge branch 'yann_adahop' of github.com:AMD-AGI/ALTO into yann_adahop
YBouquet
committed
5f40c74
debug viz
YBouquet
committed
dcc2a25
gradient clipping:
YBouquet
committed
d638528
fix viz
YBouquet
committed
649fca1
correct debug observer
YBouquet
committed
d913056
disable TP
YBouquet
committed
9268860
bf16 training mode when debugging
YBouquet
committed
918c865
update lpt mode
YBouquet
committed
3d43990
correct histograms
YBouquet
committed
60363c1
fsdp drops the module_id which affects the clipping
YBouquet
committed
311a3ec
prepend clipping hook
YBouquet
committed
1c9eb58
remove fsdp
YBouquet
committed
c331eac
debug clipping
YBouquet
committed
69a4d69
check absmax
YBouquet
committed
9a99fa8
clip before quantization, observe after clipping
YBouquet
committed
26b85bf

Commits on Jun 23, 2026

Apply working LPT fixes on yann_adahop: bias-skip in swap_params, decomposed-linear device fix, gpt_oss_20b_lpt run config
arkhodamoradi
committed
1c4e801

Commits on Jun 24, 2026

Add gpt_oss AdaHOP recipe and wire gpt_oss_20b_lpt to lpt_adahop.yaml
arkhodamoradi
committed
40c0063
checkpoint correction
arkhodamoradi
committed
8261e20

Commits on Jun 26, 2026

AdaHOP: make it run + checkpoint on MI300 (gfx942/CDNA3) under FSDP
arkhodamoradi
committed
551a2dc

Commits on Jul 6, 2026

moe patterns
YBouquet
committed
6a49876

Commits on Jul 8, 2026

Fix MXFP4 dw grouped-GEMM kernel compile on MI350 (gfx950/CDNA4)
YBouquet
committed
0b0af3a
correct adahop: router-gate ignored
YBouquet
committed
de658ba
add multinode dockerfile for distributed training
nfrumkin
committed
112dec0

Commits on Jul 15, 2026

Commits on Jul 22, 2026

working training script
nfrumkin
committed
abc0b9c
update c4 with pre-tokenized dset
nfrumkin
committed
16e97ef
Merge remote-tracking branch 'origin/main' into rad/lpt
nfrumkin
committed
e817426

Commits on Jul 23, 2026

checkpoint fix and dataloader for pre-tokenized c4
Alireza Khodamoradi
committed
bd4047c

Commits on Jul 24, 2026

add gptoss20b training script
nfrumkin
committed
ed7f35c
fix mount of data directory into docker
nfrumkin
committed
6d268ca
clean up script
nfrumkin
committed
532be0b
add faster docker cleanup after ctrl-c
nfrumkin
committed
2ba2e16
add more description to run script
nfrumkin
committed
b155dce
add notes to run script about ALTO dir
nfrumkin
committed
1a51591
add description on how to view loss curves
nfrumkin
committed
9892cdd
rename container and data dir
nfrumkin
committed
7aed68a
rename container
nfrumkin
committed
2d02ae3

Commits on Jul 25, 2026

rename container and data dir
nfrumkin
committed
5dc525e

Commits on Jul 28, 2026

fix printout so it's not always saying "bf16"
nfrumkin
committed
8ad51a3
add experiment plotting script
nfrumkin
committed
3420846
make sure node's python env is not leaked to container, make checkpointing dir unique for each run
nfrumkin
committed
447a406
update plotting structure and add training grad norm & loss
nfrumkin
committed
478b632
add M+Adam implementation and configs for M+Adam + Deosc
nfrumkin
committed
72edc28
add checkpointing files to .gitignore
nfrumkin
committed
c454c3d
Ali's checkpointing fix
nfrumkin
committed
9bd0283

Commits on Jul 29, 2026

standardize config names
nfrumkin
committed
b1e6f2a
fix name of config in madam_stable
nfrumkin
committed
5aa8694
fix __all__ with correct configs
nfrumkin
committed
94a28df

Commits on Jul 30, 2026

add compatibility with several out files
nfrumkin
committed
b6e2c8c
fix multinode dockerfile so it's compatible with OCI
nfrumkin
committed
9b6e455
add info about testing RDMA
nfrumkin
committed
d90c26f
add rdma testing procedures
nfrumkin
committed
9ff72ca
docker command to pass relevant RDMA devices to image
nfrumkin
committed
83925cf
add automatic detection of libibverbs and RDMA provide modules
nfrumkin
committed
f7e8a1c
add update to test rdma backend
nfrumkin
committed
9a166bc
add info about testing RDMA
nfrumkin
committed
9ef9864
add multinode sbatch script
nfrumkin
committed
ffacea0

Commits on Jul 31, 2026

add mxfp4 base to config registry
nfrumkin
committed
b727ad5
add optional details arg and tensorboard support
nfrumkin
committed
57de119

Commits on Aug 4, 2026

Commits on Aug 5, 2026

Merge branch 'main' into yann_moe
YBouquet
committed
5769da6
add correct torchao version and slurm job id into run_id
nfrumkin
committed
4d0d3d3
add midmax compatibility w/ bw and grouped gemm
nfrumkin
committed
c49fc62
instead of dedicated "midmax" path, create more general scale selection constexpr flag
nfrumkin
committed
bfdff95
make checkpoint dir have only slrum name so it auto loads when pre-empted
nfrumkin
committed
d8e36b1
add plotting end step for viz
nfrumkin
committed
3bc7530

Commits on Aug 6, 2026

add more scale-selection options
nfrumkin
committed
866202d
add new scale selection configs
nfrumkin
committed
294f0f4
update blockscale selection choices
nfrumkin
committed
16ca725
assume ALTO dir is pwd
nfrumkin
committed
eba7bee

Commits on Aug 7, 2026

fix dtype mxfp_linear, forward only quant on llama
YBouquet
committed
e31673e
make plot only have one loss value per step
nfrumkin
committed
e979e83
Merge branch 'rad/midmax' into rad/lpt
nfrumkin
committed
9025df8
Merge remote-tracking branch 'origin/yann_moe' into rad/lpt
nfrumkin
committed
19db92b
Merge branch 'rad/scale-selection' into rad/lpt
nfrumkin
committed
9a97bec
fix bug with run_id
nfrumkin
committed
a387724
fix configs to point to original rad/lpt baseline lpt recipes
nfrumkin
committed
8ddb7a6
make adahop configs point to original rad/lpt config files
nfrumkin
committed
88c0bb5
fix bug in plotting script
nfrumkin
committed
5d611db
adjustments to run script and config registery
Alireza Khodamoradi
committed
fa88c9f

Commits on Aug 10, 2026

Commits on Aug 13, 2026

Update 3RHT implementation
Alireza Khodamoradi
committed
4d07a9f

Commits on Aug 14, 2026

Commits on Aug 16, 2026

some optimizations
arkhodamoradi
committed
912cad8

Commits on Aug 17, 2026

Merge pull request #42 from AMD-AGI/origin/rad/lpt_temp_ali
nfrumkin
authored
49a7057

Commits on Aug 27, 2026

Update for fw and dgrad projection support and GPT-OSS config
arkhodamoradi
committed
4501d4e
Restore hadamard transform weights
arkhodamoradi
committed
88d1dbb

Commits on Aug 31, 2026

Merge pull request #44 from AMD-AGI/rad/lpt_temp_ali
nfrumkin
authored
7fb56cc