Repository navigation
seed-prefetch: a paused group is always continued, even after a crash or SIGKILL - #1631
Merged
Merged
Conversation
… or SIGKILL The Governor SIGSTOPs the prefetch's process group while a job downloads. stop() continued it, but a counter that raised ended the loop with the group still stopped, and a SIGKILLed run (launchd, OOM, power) left it stopped forever. Now the loop's end continues the group; each pause is recorded (pgid plus the leader's start time) before the SIGSTOP and dropped after the SIGCONT; and every run first continues any group a dead run left recorded, only while its leader is the same process. With no record, no pause. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ead error Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Owner
Author
|
Owned by an agent session · session |
…ot settle, no ps under the Governor lock Review (Codex): thaw ran only inside prefetch(), outside the lock, dropped records on a ps or SIGCONT failure, ran ps without a timeout under the Governor's lock, and missed a group whose leader exited. Now run() thaws first under the prefetch lock on every run (records live in its work folder); a group whose leader exited is continued while members live; ps has a timeout and runs once outside the lock; a record stays when ps cannot tell or SIGCONT fails. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…SIGCONT keeps the record Review (Codex): ps lstart has one-second precision, and resume() treated any SIGCONT error as success and dropped the record. Now leader_start reads proc_pidinfo (macOS, sec.usec) or /proc starttime with the boot id (Linux), and the record and pause stay unless SIGCONT succeeded or the group is gone. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
sr2: "SIGCONT before releasing a paused group."
Why
On cosmic-gray (Leo's interactive Mac), several processes niced 5/10 sat stopped (state
T) for days. One was pid 40950, agit fsmonitor--daemonon ~/Projects/cmux, stopped since Oct 20. Its FSEvents stream was never drained, and fseventsd likely grew because of it: a 73 GB footprint, 10-14 GB RSS, 60-190% CPU, and a full 50 GB swap. Others: 52713git ls-files, 89790git pack-objects, 2784glaeda-disk. The seed-prefetch Governor is the one glaeda code path that SIGSTOPs a whole process group, and before this change it could leave the group stopped. The attribution is likely, not proven: without root I couldn't trace which process sent the SIGSTOP.Change (scripts/glaeda-seed-prefetch)
read_interface/read_own/job), the Governor's loop ended with the group still stopped until the prefetch timeout (up to 1500 s) or forever.loop()now always continues the group in afinallyand marks itself stopped.STATE/seed-prefetch-paused/<pgid>.json: pgid and the leader'sps lstart), and removes it after the SIGCONT. Every prefetch run first callsthaw_left_paused, which SIGCONTs any group a dead run left recorded, but only while the leader is the same process (same pid and same start time), so a reused pid is never signalled. If no record can be written, no pause is taken.stop()goes through the sameresume()and doesn't join its own thread.Audit of other SIGSTOP users
scripts/glaeda-cmux-runner-hookListenerGate.stop: it already SIGCONTs every frozen pid in afinallyon every return and exception path. Only a SIGKILL between the STOP and the CONT (milliseconds) could leave it frozen. Not changed here.scripts/benchmark-linker-throughput: it stops itself and its parent continues it (benchmark only). Not changed.Tests
Four new tests in
scripts/test-glaeda-seed-prefetch.py: a raising counter never leaves the group stopped; a pause is recorded and a later run continues a group a killed run left; a reused pid or a bad record is never signalled; no record means no pause. The full suite (53 tests) passed on huge-blue.pr-overlap: no open PR overlaps (I checked the disk-glaeda stack #1622-#1626, which doesn't touch seed-prefetch).Agent session: agent session
🤖 Generated with Claude Code