Repository navigation
[Bug]: ephemeral lock holder sidecars race directory walks and match *.json globs, causing Windows-only test flakes #6128
Description
Preflight
- I searched existing issues and discussions for this behavior.
- This report is not an unpatched security vulnerability.
- The report and attachments contain no credentials or private/internal material.
Issue origin
Observed or reproduced in a real environment
LoopX version or commit
Observed on trees near 641855d9c and c2b17b3e (October main); the sidecar mechanism itself is older (loopx/file_lock.py, LOCK_HOLDER_SCHEMA_VERSION = "file_lock_holder_v0").
Host or runtime surface
Not host-specific (reproduced on native Windows 10, zh-CN, cp936)
LoopX area
Build, packaging, installer, or CI
Problem
exclusive_cross_runtime_file_lock writes an ephemeral holder sidecar next to the locked file (lock_holder_path: <locked>.lock.holder.json). Two properties of that sidecar combine into a class of Windows-only test flakes in suites that walk runtime state directories:
- The name ends in
.json, soglob("*.json")and directory walks that collect JSON state files match it even though it is not state. - The lifecycle is ephemeral and racy against walkers: it is created around the locked operation and removed on release. On Windows the create/delete window is wide enough that a walker can list the sidecar and then fail
read_bytes()withFileNotFoundError, or assert against a directory snapshot that still contains it.
Two concrete reproductions from reviewing current PRs (both native Windows, cp936):
tests/test_loopx_turn_executor.py::test_reserved_managed_start_recovers_after_death_before_the_first_journal_write- the "no journal written" assertion at the death point usesglob("*.json")and matches the persisted...turns/<hash>.json.lock.holder.json; after a (simulated) crash the holder legitimately survives, so the assertion trips even though no Turn journal was written.tests/test_peer_collaboration.py::test_canonical_stopped_goal_rejects_peer_request_before_any_write- a helper that walks.local/manager-context/entries/**collecting{relative_path: read_bytes()}listed a....lock.lock.holder.jsonand raisedFileNotFoundErrorwhen the sidecar was deleted before the read. Flaky across runs (passes on rerun).
Both are test-side failures, but the same walk pattern exists in product-adjacent code paths that enumerate runtime directories, so the class is broader than these two rows.
Minimal reproduction
- On Windows, run a suite that walks a runtime directory containing locked files (e.g.
python -m pytest -q tests/test_peer_collaboration.py::test_canonical_stopped_goal_rejects_peer_request_before_any_write) - repeat until the walker lists a.lock.holder.jsonthat disappears before its read. - Or directly: hold
exclusive_cross_runtime_file_lockover a file in a loop and, from another thread, walk the directory - observe transientFileNotFoundErroron....lock.holder.jsonand*.jsonglob matches.
Expected behavior
Directory walks over runtime state should be deterministic: either the holder sidecar is not named like a state file, or walkers (and the tests that assert on walks) skip/ignore it consistently on every platform.
Actual behavior
On Windows the sidecar is caught mid-lifecycle: glob assertions see persisted holders after simulated crashes, and walkers raise FileNotFoundError on holders deleted between listing and reading.
Sanitized diagnostics
FAILED ... test_canonical_stopped_goal_rejects_peer_request_before_any_write
E FileNotFoundError: [Errno 2] No such file or directory:
'...\.local\manager-context\entries\<hash>\<hash>.lock.lock.holder.json'
FAILED ... test_reserved_managed_start_recovers_after_death_before_the_first_journal_write
E assert not [...\turns\<hash>.json.lock.holder.json] # glob("*.json")Additional context
Possible directions, maintainer preference welcome:
- Rename the sidecar so it does not end in
.json(it is lock metadata, not state), or move holder files into a dedicatedlocks/subdirectory; - Ship a small shared walk helper (or fixture) that skips
.lock.holder.jsonand tolerates per-entryFileNotFoundError, and use it in the two affected suites; - Minimally, patch the two test sites (glob filter + per-entry tolerance) so Windows runs are deterministic.
Happy to implement whichever direction is preferred.
Activity
BigDataDZ commentedon Oct 10, 2026
Root cause found - and it is product-level, not just test flakiness. A third instance surfaced while chasing the
KeyError: binding_idclue from the PR #6075 review, and it pins the mechanism precisely.lock_holder_pathinloopx/file_lock.pyis Windows-only in its sidecar shape:On Windows the holder record is written to a separate file ending in
.json(<locked>.json.lock.holder.json); on POSIX the record is written into the lock file itself, so no.json-suffixed sidecar exists - which is exactly why this class has never been seen on Linux CI.Product crash, third and cleanest instance:
ChatExternalConversations(Lark / native chat) admits each external request underexclusive_file_lock(<requests>/<ref>.json), and the holder sidecar for that lock lands in the same directory._admit_preparedthen callspending(), which is[_read_json(p) for p in sorted(self.root.glob("*.json"))]- it reads the holder sidecar of the request currently being admitted as if it were an external chat request and crashes:Reproduced deterministically on native Windows (zh-CN, cp936, Python 3.12) at tree
07eb7a128viatests/extensions/test_lark_private_manager_returns.py::test_production_pump_returns_to_original_private_source_after_restart(8/8 parametrizations error in fixture setup).commission_evidencein the same module readsrow["binding_id"]overpending()and has the same exposure. This breaks chat admission on Windows wheneverpending()runs while any request file in the directory is locked - i.e., exactly the concurrent-admission case the fence exists to serialize.This upgrades the suggested directions: the cleanest fix is likely
lock_holder_pathreturning a non-.jsonname on Windows (the sidecar is lock metadata, not state), which would fix the test flakes from the original report and this admission crash in one place; product walkers that must stay.json-shaped should still filter the holder name defensively.fix(file-lock): name the Windows holder sidecar without a .json suffix
fix(file-lock): clear the pre-rename holder sidecar and guard record …
fix(file-lock): keep the Windows holder record out of *.json scans
AronSwan commentedon Oct 11, 2026
Post-fix verification on current main (
8d5a6fc17, which includes the5347064ffix): the reproduction shape from your Minimal reproduction step 2 now comes back clean. While holdingexclusive_cross_runtime_file_lockin a loop,glob("*.json")no longer matches the holder sidecar (it is nowstate.json.lock.holder, no.jsonsuffix), and a concurrent directory walker survived a 20-second run — 6,557 read attempts, zeroFileNotFoundError, zeroPermissionError.tests/test_file_lock.pyon this head: 20 passed, 3 skipped (Windows, Python 3.14.2). The legacy*.lock.holder.jsonread-compat path is a thoughtful touch for upgrade windows.One disclosure, since it belongs on the record: we hit this same family on 2026-10-04 — a bare
glob("*.json")intest_reply_delivery_does_not_replace_receiver_dispositioncounted one real turn file plus two lock sidecars as three items. We sandbox-verified a test-side filter, deferred a small PR until after our #5554 landed, and then failed to follow up — our miss. Your report framed the class properly, and direction 1 was the right call.Thanks for the clean report and the fix.
test: make control-plane path assertions host-independent on Windows