Repository navigation
fleet-cas prune: the node store is a hard cap (writer too, builds or not); glaeda-disk pressure passes run it - #1624
Merged
Merged
Conversation
Owner
Author
|
Owned by an agent session · session |
config/fleet-reservations.json is glaeda's copy of leo-workspace's fleet-registry reservations (a test compares the two where the registry is checked out); mini-setup installs it. On a host an active entry (or the host-local reservation.json) names, glaeda-disk runs no idle sweep and a pressure pass acts only below the runner floor, back to floor + target_margin_gib. glaeda-fleet-cas-prune leaves a reserved host above its floor alone. An unreadable list fails closed. Entries lapse by themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…not); glaeda-disk pressure passes run it The node store was pruned only while no build ran, which on a busy mini is never, and never on the writer: cmux15's reached 73 GiB (2026-10-26). The writer writes through, so its node store is a cache like a reader's. Files the node touched within 2 h (twice its TOUCH_EVERY) stay as a running build's working set. The local Xcode CAS still goes only while no build runs, now above 20 GiB. glaeda-disk's pressure passes (before every job, under the floor) run `glaeda-fleet-cas-prune --apply --node-only`. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…kv entries, its objects go Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo
force-pushed
the
disk-glaeda-fleet-cas-cap
branch
from
October 26, 2026 21:51
c502020 to
e9b8e60
Compare
teamleaderleo
changed the base branch from
main
to
disk-glaeda-reservation-guard
October 26, 2026 21:51
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
… closed on any malformed entry The list no longer ships in this repository. glaeda-disk and glaeda-fleet-cas-prune read it only from ~/.local/share/glaeda/fleet-reservations.json in the home of the user that owns /Users/Shared/cmux-build-fleet (written there by the fleet's own writer), plus FLEET_ROOT/reservation.json. A missing, unreadable or malformed list (hosts not a list of names, a bad time on any entry, aliases not lists) reads as reserved, whatever the host-local file says. Also from review: the prune registers the glaeda-disk module in sys.modules before running it, so its dataclasses load and the gate is real; the reservation gate runs before store_gc; --runner-caches is held to the same floor-only rule; and a reserved host's pressure starts at the floor itself, not floor + low_margin. Test fixtures use made-up ids, hosts and owners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…e.tmp<pid>.<seq>) Review finding: the hard-cap walk included the daemon's temporaries under node-store/cas, and unlinking one mid-write fails the rename and the build. Skip any name with a dot, as fleet-cas gc.rs does. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-fleet-cas-cap # Conflicts: # scripts/glaeda-fleet-cas-prune
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 35581672 | Triggered | Generic Password | 5922735 | scripts/test-glaeda-disk.py | View secret |
| 35581672 | Triggered | Generic Password | c91dda3 | scripts/test-glaeda-disk.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
…e, reads as reserved The writer (leo-workspace host-baseline ci-minis) stamps expires 13 h after each write and refreshes it every 6 h, so a mini the writer stopped reaching (for one, a host reserved after its last write) falls back to reserved by itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-fleet-cas-cap
…ouched 6 h counts and goes first Review findings: the node cap needs no build check, so a ps failure no longer skips it under --node-only; an interrupted daemon's name.tmp<pid>.<seq> left for TMP_STALE_S (6 h) is abandoned, counted against the cap and removed before any object, while a younger one is still an in-flight write and stays. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
… markers and intervals fail closed Review findings on #1626: - a FLEET_ROOT/reservation.json that is there but unreadable, not glaeda-reservation/v1 or without integer since/until holds the host, as host_busy reads it (absent: no hold); - a fleet entry with from >= until fails the whole list closed; - on a reserved host, after the floor-pressure pass, no stale-marker clearing, paused-work cleanup, test-residue reaping, home-clone gc or runner repack runs, and --dedupe / --dedupe-builds do nothing (the reservation is asked for them too); - filesystems() gets reserved= only when set, so callers and mocks that pass positionals keep working. Each gate has a test that fails when its check is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-fleet-cas-cap
…ous commit lost this edit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-fleet-cas-cap
…ure since holds, as the runner hook reads it Review findings on #1626: lexists() read a permission error as absence; only FileNotFoundError on the read is absence now. The runner hook holds while now < until whatever since says; host_reservation matches it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-fleet-cas-cap
…-cap # Conflicts: # scripts/glaeda-fleet-cas-prune # scripts/test-glaeda-disk.py
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Leo, 2026-10-26: "You run code, you build code, and then you run out of space, and then somebody has to save it. ... Just throw it away. Just delete more stuff, more often, more often. Fix glaeda."
The largest single disk use on the minis is
/Users/Shared/cmux-build-fleet/xcode/node-store, the fleet compilation-cache node store. It had a 20 GiB budget, but the pruner deferred while any build or Runner.Worker ran (on a busy mini that is always) and never pruned the writer's store at all.What changes
put_throughandkv_putacknowledge only once the fleet store took the object or entry), so its node store is a cache like a reader's: a missing file is fetched again. A file with an mtime in the lastNODE_ACTIVE_S(2 h, twice the node'sTOUCH_EVERY) is never deleted, because a running build may still be reading it. The node-store unlinks no longer take the host lock.--local-cas-gib, now 20 (was 30). It goes only while no build runs and the host lock is free, because a build has it open. The writer's local CAS stays as before (every writer build empties it).--pressurepass on a fleet mini runsglaeda-fleet-cas-prune --apply --node-only, bounded at 300 s. The runner hook's pass before each job and its floor eviction are both such passes. The hourly LaunchAgent stays.Before (sizes on 2026-10-26, 13:45 PDT)
After: cmux15's node store at 20 GiB or less, minus the files used in the last 2 h. Posted here after deploy.
Risk
Medium. More node-store misses mean more fetches from the fleet store over the LAN, at about 100 s per full catch-up refill. On the writer, an evicted object comes back from the fleet store on its next read, or is written again by its next build. Nothing a build touched in the last 2 h is deleted. The local CAS rules are unchanged apart from the budget.
Tests:
test_writer_prunes_its_node_store_and_keeps_its_local_cas,test_the_node_cap_holds_while_a_build_runs_and_keeps_its_working_set,test_the_local_cas_defers_while_ps_fails_or_a_build_holds_the_lock,test_node_only_caps_the_node_store_and_leaves_the_rest, andFleetCasNodeCapTest.Agent session: agent session
🤖 Generated with Claude Code