Repository navigation
parquet tests fails inconsistently in CI聽#813
Description
馃悰 Bug
Parquet tests fail inconsistently in CI, mostly due to timeout, waiting for index file to be available. Possibly, multiple parquet tests leaking the environment.
To Reproduce
Steps to reproduce the behavior...
Code sample
# Ideally attach a minimal code sample to reproduce the decried issue.
# Minimal means having the shortest code but still preserving the bug.Alternatively, you can share a fully reproducible Lightning Studio environment:
A simple guide on how to create such a studio can be found here.
- Create a Studio.
- Reproduce the issue in the Studio.
- Publish the Studio.
- Paste the Studio link here.
Expected behavior
Additional context
Environment detail
- PyTorch Version (e.g., 1.0):
- OS (e.g., Linux):
- How you installed PyTorch (
conda,pip, source): - Build command you used (if compiling from source):
- Python version:
- CUDA/cuDNN version:
- GPU models and configuration:
- Any other relevant information:
Activity
snakefood3232 commentedon Apr 29, 2026
I've noticed your parquet tests are failing inconsistently due to timeouts related to index file availability. I'd handle this by isolating the test environment or ensuring proper teardown to prevent leaks causing these issues. I can have a PR ready for review within 48 hours. I've previously optimized data pipeline tests in CI environments.
kostasrigatos commentedon Oct 9, 2026
I looked into this. #810 already fixed the cause described here for the tests that share
~/.lightning/chunks, but threeoptimize_hftests intests/streaming/test_parquet.pystill use theshared
$TMPDIR/chunksand$TMPDIR/datascratch dirs, and two of them aren't in the xdist group.Every
optimize()call deletes and recreates those dirs (_cleanup_cache), so the tests can race whenthey land on different workers. A macOS py3.10 CI job failed this way after #810 (
FileNotFoundErrorinside
shutil.rmtree). The other recent parquet failure I found (assert 3 == 2) was on a branchbefore #897 and is already fixed on main.
I'll open a PR that applies #810's pattern (per-test
DATA_OPTIMIZER_CACHE_FOLDER/DATA_OPTIMIZER_DATA_CACHE_FOLDER) to those three tests.