Repository navigation
fix(streaming): reset wrapper sample and cycle counters on reset_state_dict - #928
Merged
tchaton merged 3 commits intoSep 29, 2026
Merged
Conversation
…te_dict` Clear `_num_samples_yielded` in `_BaseStreamingDatasetWrapper.reset_state_dict()` and `_num_cycles` in `ParallelStreamingDataset.reset_state_dict()`, and deep-copy restored counter dicts so in-place iterator increments do not mutate checkpoint state across epochs. Fixes Lightning-AI#927
truecallerabreham
requested review from
bhimrazy,
deependujha,
dhedey,
pwgardipee and
tchaton
as code owners
September 27, 2026 18:36
for more information, see https://pre-commit.ci
deependujha
reviewed
Sep 28, 2026
|
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #928 +/- ##
=====================================
Coverage 82% 82%
=====================================
Files 70 70
Lines 16153 16160 +7
=====================================
+ Hits 13258 13264 +6
- Misses 2895 2896 +1 🚀 New features to boost your workflow:
|
tchaton
approved these changes
Sep 29, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Before submitting
What does this PR do?
Fixes #927.
Resuming
CombinedStreamingDatasetorParallelStreamingDatasetfrom a mid-epoch checkpoint no longer leaksnum_samples_yieldedornum_cyclesinto subsequent epochs.When
StreamingDataLoader.load_state_dict()restored a mid-epoch checkpoint and finished the resumed epoch, the next epoch calledself.dataset.reset_state_dict(), which reset childStreamingDatasetinstances but leftself._num_samples_yieldedon_BaseStreamingDatasetWrapperandself._num_cyclesonParallelStreamingDatasetpopulated from the checkpoint. In addition,load_state_dict()and the dataset iterators held direct references to the restored counter lists, so in-place increments mutated the loaded state dict in single-worker mode (num_workers=0).self._num_samples_yielded = Nonein_BaseStreamingDatasetWrapper.reset_state_dict()andself._num_cycles = NoneinParallelStreamingDataset.reset_state_dict().num_samples_yieldedandnum_cyclesinload_state_dict()and_CombinedDatasetIterator/_ParallelDatasetIteratorso in-place counter updates do not mutate the caller's state dict.tests/streaming/test_combined.pyandtests/streaming/test_parallel.pyverifying multi-epoch iteration and subsequent checkpoint resumption after a mid-epoch restore.PR review
Anyone in the community is free to review the PR once the tests have passed.
Did you have fun?
Make sure you had fun coding 🙃