[hoobnn]

Before submitting

• Was this discussed/agreed via a Github issue? (no need for typos and docs improvements)
• Did you read the contributor guideline, Pull Request section?
• Did you make sure to update the docs?
• Did you write any new necessary tests?

What does this PR do?

Fixes #402.

The original report (shuffle=False) was resolved by #449, which aligns per-worker item counts to batch_size. #876's WindowShuffle assigns whole chunks to workers, so each worker can again end with its own partial batch and len(StreamingDataLoader) undercounts. For example, 10 samples with chunk_size=5, batch_size=4 and 2 workers yield 4 batches, but len(loader) reports 3. Progress bars or loops that stop after len(loader) batches silently drop samples.

For StreamingDataset, compute the length from each worker's assigned intervals, counting its trailing partial batch unless drop_last is set. CombinedStreamingDataset and ParallelStreamingDataset are unchanged.

Validation

Added a regression test over num_workers (0, 2), drop_last and shuffle mode (none, window) that compares len(loader) with the batches actually yielded; it fails on main for WindowShuffle with 2 workers and passes with this change.

I used OpenAI Codex and Claude Code to investigate the issue, write the fix and tests, and run the checks. I reviewed the code and this description myself.

PR review

Anyone in the community is free to review the PR once the tests have passed.
If we didn't discuss your PR in GitHub issues there's a high chance it will not be merged.

Did you have fun?

Make sure you had fun coding 🙃

Assisted by Claude Code

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82%. Comparing base (02aea00) to head (8f1727c).
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #939   +/-   ##
=====================================
  Coverage     82%     82%           
=====================================
  Files         72      72           
  Lines      16432   16445   +13     
=====================================
+ Hits       13491   13505   +14     
+ Misses      2941    2940    -1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.