Skip to content

fix(streaming): reset wrapper sample and cycle counters on reset_state_dict - #928

Merged
tchaton merged 3 commits into
Lightning-AI:mainfrom
truecallerabreham:fix/reset-wrapper-state-dict
Sep 29, 2026
Merged

tchaton merged 3 commits into
Lightning-AI:mainfrom
truecallerabreham:fix/reset-wrapper-state-dict

Conversation

@truecallerabreham

Copy link
Copy Markdown
Contributor
Before submitting
  • Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
  • Did you read the contributor guideline, Pull Request section?
  • Did you make sure to update the docs?
  • Did you write any new necessary tests?

What does this PR do?

Fixes #927.

Resuming CombinedStreamingDataset or ParallelStreamingDataset from a mid-epoch checkpoint no longer leaks num_samples_yielded or num_cycles into subsequent epochs.

When StreamingDataLoader.load_state_dict() restored a mid-epoch checkpoint and finished the resumed epoch, the next epoch called self.dataset.reset_state_dict(), which reset child StreamingDataset instances but left self._num_samples_yielded on _BaseStreamingDatasetWrapper and self._num_cycles on ParallelStreamingDataset populated from the checkpoint. In addition, load_state_dict() and the dataset iterators held direct references to the restored counter lists, so in-place increments mutated the loaded state dict in single-worker mode (num_workers=0).

  • Clear self._num_samples_yielded = None in _BaseStreamingDatasetWrapper.reset_state_dict() and self._num_cycles = None in ParallelStreamingDataset.reset_state_dict().
  • Deep-copy num_samples_yielded and num_cycles in load_state_dict() and _CombinedDatasetIterator / _ParallelDatasetIterator so in-place counter updates do not mutate the caller's state dict.
  • Add regression tests in tests/streaming/test_combined.py and tests/streaming/test_parallel.py verifying multi-epoch iteration and subsequent checkpoint resumption after a mid-epoch restore.

PR review

Anyone in the community is free to review the PR once the tests have passed.

Did you have fun?

Make sure you had fun coding 🙃

…te_dict`

Clear `_num_samples_yielded` in `_BaseStreamingDatasetWrapper.reset_state_dict()` and `_num_cycles` in `ParallelStreamingDataset.reset_state_dict()`, and deep-copy restored counter dicts so in-place iterator increments do not mutate checkpoint state across epochs.

Fixes Lightning-AI#927
Comment thread src/litdata/utilities/base.py Outdated

@deependujha deependujha left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

thanks :)

@codecov-commenter

codecov-commenter commented Sep 28, 2026 •

Copy link
Copy Markdown

⚠️ Please install the 'codecov app svg image' to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

❌ Patch coverage is 91.66667% with 1 line in your changes missing coverage. Please review.
✅ Project coverage is 82%. Comparing base (bb96b0e) to head (5843d9b).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files
@@          Coverage Diff          @@
##            main    #928   +/-   ##
=====================================
  Coverage     82%     82%           
=====================================
  Files         70      70           
  Lines      16153   16160    +7     
=====================================
+ Hits       13258   13264    +6     
- Misses      2895    2896    +1     
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.

@tchaton
tchaton merged commit 28e0394 into Lightning-AI:main Sep 29, 2026
36 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

reset_state_dict() on CombinedStreamingDataset and ParallelStreamingDataset does not clear _num_samples_yielded or _num_cycles after checkpoint resume

5 participants