Skip to content

fix: keep pretrain validate lockstep on empty ranks (#2319) - #2321

Open
ShamikOfficial wants to merge 2 commits into
Lightning-AI:mainfrom
ShamikOfficial:fix/pretrain-validate-empty-rank-2319
Open

fix: keep pretrain validate lockstep on empty ranks (#2319)#2321
ShamikOfficial wants to merge 2 commits into
Lightning-AI:mainfrom
ShamikOfficial:fix/pretrain-validate-empty-rank-2319

Conversation

@ShamikOfficial

Copy link
Copy Markdown

Summary

  • Fixes multi-GPU FSDP deadlock / crash in pretrain.validate when some ranks get zero validation batches (sharded streaming), as reported in litgpt pretrain crashes or deadlocks when a rank's val_dataloader yields zero batches (multi-GPU FSDP) #2319.
  • Before each validation step, ranks MIN-reduce a has_batch flag so they stop together; if no rank had data, return NaN instead of torch.stack([]).
  • Applies the same lockstep pattern in extensions/thunder/pretrain.py and adds regression tests (empty loader, lockstep protocol, threaded uneven-shard deadlock coverage).

Test plan

  • pytest tests/test_pretrain_validate_empty_rank.py -v — 6 passed locally (CPU)
  • Empty dataloader returns NaN (no stack([]) crash)
  • Uneven shards (rank0 has data, rank1 empty) finish without hanging under a collective-in-forward mock
  • Partial overlap (e.g. 3 vs 1 batches) stops after the shared steps only
  • Control test: naive empty-skip without MIN-reduce deadlocks as expected
  • CI on this PR (GPU / multi-process jobs if available)

Avoid FSDP deadlock and torch.stack([]) crashes when some validation shards are empty by MIN-reducing has_batch before each step.

Co-authored-by: Cursor <cursoragent@cursor.com>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant