[Yi-111-a]

Summary

optimize() and map() run DataProcessor.run in the caller's process, and run seeded the global random, NumPy and torch generators with random_seed (default 42) and never restored them. That threw away the seed the caller had set before the call, so everything random afterwards (model init, shuffling, sampling) was the same whatever seed the caller picked. It also changed depending on whether an optimize call had happened earlier in the run, for example only when a cache didn't exist yet.

litdata only needs randomness for one thing: shuffling the items assigned to each worker in the weighted / file-size mapping. That shuffle now uses a local np.random.RandomState(random_seed), and run no longer calls random.seed, np.random.seed or torch.manual_seed.

Fixes #944

Changes

• DataProcessor.run: dropped the three global seeding calls. Builds a local np.random.RandomState(self.random_seed) and passes it to both _map_items_to_workers_weighted call sites (explicit weights and file-size reordering).
• _map_items_to_workers_weighted: new optional rng argument. None keeps the old np.random behaviour for direct callers.
• optimize() / map(): expose random_seed: int | None = 42 and forward it to DataProcessor. None gives a non-deterministic shuffle.
• Docstrings for random_seed now say it only seeds the worker shuffle.

Behaviour notes

• Chunk layout doesn't change. RandomState(42) produces the same sequence as np.random.seed(42), and nothing between the old seeding point and the shuffle draws from the global NumPy generator. So optimize / map assign and order items exactly as before.
• Global generators are no longer seeded. Before, random_seed also seeded the parent process, so a recipe's prepare_structure and, under start_method="fork" (the default in notebooks), the forked workers' fn inherited the seeded state. Under the default spawn start method, fn never inherited it. Code that relied on litdata seeding its randomness should seed inside fn / the recipe.

Tests

In tests/processing/test_data_processor.py:

• test_optimize_preserves_the_callers_random_state[0|1]: seeds the caller, runs optimize, and checks that the next draw from random, NumPy and torch is the one the caller would have got without the call. On the unpatched tree both seeds return random=0.639427, exactly as reported in the issue.
• test_optimize_does_not_seed_the_global_generators: fails if random.seed, numpy.random.seed or torch.manual_seed is called during optimize. This also guards against torch queueing lazy CUDA / MPS / XPU seeds.
• test_weighted_shuffle_uses_local_rng[weights|file_sizes]: covers both weighted branches. Checks that the caller's NumPy state is preserved, that a RandomState is forwarded, and that the shuffle is the same for the same random_seed whatever the caller seeded. It fails if rng= is dropped from either call site.

tests/processing: 150 passed, 4 skipped.

[codecov-commenter]

⚠️ Please install the to ensure uploads and comments are reliably processed by Codecov.

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ Project coverage is 82%. Comparing base (c6f075c) to head (2953581).
⚠️ Report is 1 commits behind head on main.
❗ Your organization needs to install the Codecov GitHub app to enable full functionality.

Additional details and impacted files

@@          Coverage Diff          @@
##            main    #945   +/-   ##
=====================================
  Coverage     82%     82%           
=====================================
  Files         72      72           
  Lines      16445   16442    -3     
=====================================
- Hits       13505   13503    -2     
+ Misses      2940    2939    -1     

🚀 New features to boost your workflow:

• ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
• 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

[deependujha]

Thanks for the PR and the clear write-up!

One gap: torch.manual_seed also affects GPU generators (CUDA, MPS, and XPU). If CUDA hasn't been initialized yet, torch applies the seed on first use. Since users typically call optimize before model.cuda(), the seed gets overwritten and isn't restored. I also reproduced this with MPS.

I think there's a simpler fix than restoring every generator: avoid changing global random state altogether.

• Expose the seed. Add random_seed: int | None = 42 to optimize() and map() and pass it to DataProcessor. A single seed is sufficient since NumPy is the only RNG needed internally. It seems to have been missed.

• Use a local RNG. The only randomness needed inside litdata is the worker shuffle in _map_items_to_workers_weighted, which uses np.random.permutation. Create rng = np.random.RandomState(self.random_seed) in DataProcessor and use rng.permutation(...) instead. The global seed calls and save/restore logic can then be removed.

• Document the behavior change. With forked workers or num_workers=0, fn could inherit the seeded global state and was effectively getting seed 42. After this change, fn won't get that seed automatically. That's reasonable, callers should seed their own randomness if needed.

[Yi-111-a]

Thanks, that's a cleaner fix. Done in 3a976bf:

• Exposed the seed. optimize() and map() now take random_seed: int | None = 42 and pass it to DataProcessor.
• Local RNG. DataProcessor builds np.random.RandomState(self.random_seed) and hands it to _map_items_to_workers_weighted, which uses rng.permutation(...) (it falls back to np.random when no rng is passed, so direct callers keep working). The global random / np.random / torch.manual_seed calls and the save/restore block are gone, along with the now-unused random and torch imports in data_processor.py. With the same seed the shuffle is unchanged, since RandomState(42).permutation gives the same draws as np.random.seed(42) followed by np.random.permutation.
• Documented the behavior change. The random_seed docstrings on optimize, map and DataProcessor say the global generators aren't seeded any more, so fn has to seed its own randomness if it depends on it. Previously, with forked workers or num_workers=0, it effectively inherited seed 42.

I also added test_optimize_does_not_seed_the_global_generators. It makes random.seed, numpy.random.seed and torch.manual_seed raise, which covers the CUDA/MPS/XPU case you reproduced without needing a GPU. The earlier state-preservation test is still there. I only had ruff available locally (no torch install), so I'm relying on CI for the test run.

[Copilot]

🟡 Changes recommended

Weighted-shuffle RNG isolation lacks regression coverage, and the change leaves prepare_structure randomness uncontrolled by random_seed without documenting the altered determinism contract.

2 open issues

• Align random_seed behavior with the documented determinism contract · New
• Add weighted shuffle RNG state and repeatability coverage · New

What changed in this PR

This PR addresses #944 by removing processing-time global RNG seeding and using a local NumPy generator for worker-item shuffling.

Changes:

• Replace global seeding with a local shuffle generator.
• Expose and document random_seed on map() and optimize().
• Add regression tests for caller RNG preservation and avoidance of global seeders.

File	Description
tests/​processing/​test_data_processor.py	Adds caller RNG-state and global-seeder regression tests.
src/​litdata/​processing/​functions.py	Exposes and forwards the shuffle-only random_seed option.
src/​litdata/​processing/​data_processor.py	Removes global seeding and passes a local generator to weighted shuffling.

_{Review effort: Balanced}

[Copilot]

🟢 Approval recommended

No active review issues are reported for the change to isolate worker-shuffle randomness from the caller’s global random state.

1 open issue

• Add weighted shuffle RNG state and repeatability coverage

1 resolved since last review

• Align random_seed behavior with the documented determinism contract

_{Review effort: Balanced}

[deependujha]

thanks :)

[Copilot]

🟢 Approval recommended

No active review issues were identified for the change isolating shuffle randomness from caller-managed random state.

No open issues 🎉

1 resolved since last review

• Add weighted shuffle RNG state and repeatability coverage

_{Review effort: Balanced}