Repository navigation
glaeda-disk: mirrors go at 12 h, unpushed commits and local-only refs bundled to ~/archive first - #1625
Merged
Conversation
Owner
Author
|
Owned by an agent session · session |
config/fleet-reservations.json is glaeda's copy of leo-workspace's fleet-registry reservations (a test compares the two where the registry is checked out); mini-setup installs it. On a host an active entry (or the host-local reservation.json) names, glaeda-disk runs no idle sweep and a pressure pass acts only below the runner floor, back to floor + target_margin_gib. glaeda-fleet-cas-prune leaves a reserved host above its floor alone. An unreadable list fails closed. Entries lapse by themselves. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo
force-pushed
the
disk-glaeda-12h-age
branch
from
October 26, 2026 21:48
01a5016 to
7a40403
Compare
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
… h go above the floor Leo, 2026-10-26: "When have we ever had something useful greater than 12 hours?" DerivedData, CoreSimulator runtimes, package caches, runner work dirs and build dirs now go once idle DEFAULT_AGE_HOURS (12 h). The per-host cmux-next build root, the newest warm compile cache per builder, is exempt by role (only under the floor). The git-mirror window stays 168 h: past it a checkout's unpushed commits are not checked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…es (no lock, no commit only here) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ervation lapses Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo
force-pushed
the
disk-glaeda-12h-age
branch
from
October 26, 2026 21:53
7a40403 to
3ee1a9e
Compare
…y refs bundled to ~/archive first Before a mirror family's item (cmux-job-cache, the hq build cache) goes, archive_mirror finds every repository it takes along (checkouts, bare repos, a linked worktree's repository inside it, submodule stores, embedded submodules) and writes each one's commits no remote holds, plus branches and tags only this host has, into git bundles under ~/archive/glaeda-disk, named with a 30-day expiry and pruned by that name. Each bundle passes `git bundle verify` and lists every ref it must hold before the tree is deleted; a failure keeps the tree, records blocked:bundle-failed in the receipt and run line, and alerts on stderr. CMUX_JOB_MIRROR_HOURS drops from 168 to DEFAULT_AGE_HOURS (12). Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo
force-pushed
the
disk-glaeda-mirror-bundle
branch
from
October 26, 2026 21:56
348cda4 to
510e97e
Compare
… closed on any malformed entry The list no longer ships in this repository. glaeda-disk and glaeda-fleet-cas-prune read it only from ~/.local/share/glaeda/fleet-reservations.json in the home of the user that owns /Users/Shared/cmux-build-fleet (written there by the fleet's own writer), plus FLEET_ROOT/reservation.json. A missing, unreadable or malformed list (hosts not a list of names, a bad time on any entry, aliases not lists) reads as reserved, whatever the host-local file says. Also from review: the prune registers the glaeda-disk module in sys.modules before running it, so its dataclasses load and the gate is real; the reservation gate runs before store_gc; --runner-caches is held to the same floor-only rule; and a reserved host's pressure starts at the floor itself, not floor + low_margin. Test fixtures use made-up ids, hosts and owners. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…eep dirty trees, recheck before delete Addresses the Codex review of #1625: 1. "held" comes only from what the servers confirm (ls-remote heads and tags, and fetch --negotiate-only acks, asked one commit at a time when a batch fails); stale remote-tracking refs are only candidates. 2. Bundle names carry a hash of the repository's real path and ref, the pid and the time, and are linked into place with os.link, so an existing bundle is never replaced. 3. A work tree with tracked edits, untracked non-ignored files or files hidden from status keeps the item (blocked:uncommitted); ignored build output still goes. 4. Checkouts inside ignored directories are bundled like the others. 5. An unreadable submodule store keeps the item (os.walk onerror). 6. Right before the delete, fresh process evidence and a snapshot of every ref, HEAD, reflog and work tree status taken before bundling are checked again; any change keeps the tree. 7. Pruning deletes only bundles manifest.jsonl lists, only at the sha256 it recorded, and only after the process-evidence gate. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…d every worktree's HEAD; explicit policy keys stay Review findings on #1622: - the above-floor veto now also keeps a checkout with FETCH_HEAD.lock, MERGE_HEAD, REBASE_HEAD, a rebase or sequencer dir, CHERRY_PICK_HEAD, REVERT_HEAD or BISECT_LOG (files or dirs, linked worktrees' git dirs too); - a commit that only a linked worktree's HEAD names keeps the checkout; - keep_legacy_ages leaves a key disk-policy.json sets alone even when its value equals the new default (policy_overrides gives the file's keys). Tracked edits and reflog-only commits still go with a runner checkout: actions/checkout's next job runs git clean -ffdx and git reset --hard there, so they never carry from one job to the next. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-12h-age # Conflicts: # scripts/glaeda-disk # scripts/test-glaeda-disk.py
…laeda-mirror-bundle
|
| GitGuardian id | GitGuardian status | Secret | Commit | Filename | |
|---|---|---|---|---|---|
| 35581672 | Triggered | Generic Password | 5922735 | scripts/test-glaeda-disk.py | View secret |
| 35581672 | Triggered | Generic Password | 4271160 | scripts/test-glaeda-disk.py | View secret |
| 35581672 | Triggered | Generic Password | 7adc2af | scripts/test-glaeda-disk.py | View secret |
🛠 Guidelines to remediate hardcoded secrets
- Understand the implications of revoking this secret by investigating where it is used in your code.
- Replace and store your secrets safely. Learn here the best practices.
- Revoke and rotate these secrets.
- If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.
To avoid such incidents in the future consider
- following these best practices for managing and storing secrets including API keys and other credentials
- install secret detection on pre-commit to catch secret before it leaves your machine and ease remediation.
🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.
…e, reads as reserved The writer (leo-workspace host-baseline ci-minis) stamps expires 13 h after each write and refreshes it every 6 h, so a mini the writer stopped reaching (for one, a host reserved after its last write) falls back to reserved by itself. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-12h-age
…laeda-mirror-bundle
… markers and intervals fail closed Review findings on #1626: - a FLEET_ROOT/reservation.json that is there but unreadable, not glaeda-reservation/v1 or without integer since/until holds the host, as host_busy reads it (absent: no hold); - a fleet entry with from >= until fails the whole list closed; - on a reserved host, after the floor-pressure pass, no stale-marker clearing, paused-work cleanup, test-residue reaping, home-clone gc or runner repack runs, and --dedupe / --dedupe-builds do nothing (the reservation is asked for them too); - filesystems() gets reserved= only when set, so callers and mocks that pass positionals keep working. Each gate has a test that fails when its check is removed. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…d keeps what it cannot read Review findings on #1622: GIT_DIR/GIT_WORK_TREE from the caller would point git -C at another repository (as _checkout_git already strips them); an unreadable directory under .git (a hidden worktree's lock) now raises in the walk and keeps the checkout. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-12h-age # Conflicts: # scripts/glaeda-disk
…os and rechecks owners before delete Addresses the second Codex review of #1625: 1. mirror_repos queues every initialized submodule work tree (embedded or in modules/), so a submodule's own submodules are bundled too. 2. and 3. mirror_moved finds the item's repositories again right before the delete: a repository created meanwhile, or a work tree no longer clean (hidden-file flags included), keeps the tree. 4. mirror_moved re-reads owner_retired and the fleet artifact references after bundling. 5. A locked linked worktree keeps the tree, as in git_loss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ous commit lost this edit) Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-12h-age
…ure since holds, as the runner hook reads it Review findings on #1626: lexists() read a permission error as absence; only FileNotFoundError on the read is absence now. The runner hook holds while now < until whatever since says; host_reservation matches it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-12h-age
…d errors, honours locks inside Addresses the third Codex review of #1625: 1. The commits the bundles leave to the servers (their prerequisites from rev-list --boundary, and tips no bundle holds) are asked about again right before the delete; one no server holds any more keeps the tree. 2. The repository search for a mirror item is strict: a directory that cannot be read (EIO, ESTALE) makes it incomplete, and the item stays. 3. A locked linked worktree keeps the item even when its repository is inside it, as in git_loss. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…stores keep, empty repos go Addresses the fourth Codex review of #1625: 1. mirror_moved asks the servers first and rereads the repositories and their snapshot last, so a local commit made while the servers were asked keeps the tree. 2. A symlinked modules/ store, or a symlinked directory inside one, is unreadable, as in submodule_loss. 3. An unborn repository (no refs, no reflog, HEAD naming a branch with no commit) needs no bundle; the snapshot reads HEAD as a file, so it is comparable too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…laeda-mirror-bundle
…and worktree reads fail closed Addresses the fifth Codex review of #1625: 1. mirror_moved now asks the servers, then rereads git, then the owner, artifact references and fresh process evidence last, so a job that starts during the slow steps keeps the tree. 2. The unborn-repository fast path applies only when the repository has no linked worktrees. 3. Linked worktree records are read strictly: an unreadable worktrees/ or record keeps the item. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
# Conflicts: # scripts/glaeda-disk # scripts/test-glaeda-disk.py
… worktrees inside the item Addresses the sixth Codex review of #1625: 1. Reflog directories are walked with files_strict (os.walk with onerror), so an unreadable one makes local_tips, the unborn check and the snapshot fail closed instead of skipping it. 2. A linked worktree inside the item (even one under ignored output such as target/) is walked like any work tree: it must be clean and unlocked. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…laeda-mirror-bundle
…ble linked worktrees as unreadable Addresses the seventh Codex review of #1625: 1. files_strict returns None when a directory below the root is a symlink, which os.walk would not enter, so a symlinked logs/refs/heads cannot hide a reflog-only commit. 2. Linked worktrees are probed with os.stat: only FileNotFoundError means absent; EACCES or EIO keeps the item instead of skipping the outside-worktree check. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ndle # Conflicts: # scripts/glaeda-disk # scripts/test-glaeda-disk.py
…he rename; references rechecked Addresses the eighth Codex review of #1625: 1. mirror_rename_recheck runs as remove()'s after_rename: once the tree is renamed aside, a hash of every git dir's HEAD, packed-refs, refs, logs and worktrees (mirror_git_digest, read relative to the item) must equal the one taken before bundling, or the tree is renamed back and kept (changed:in-use). mirror_moved compares the same digest too. 2. mirror_moved re-runs referenced_main_checkout for git-disposable families. 3. files_strict also refuses a symlinked or special file, which is_file() readers would skip. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ndle check again; kept trees are marked Addresses the ninth Codex review of #1625 (replaces the metadata digest): 1, 2. mirror_rename_recheck runs mirror_repos on the renamed tree (a fresh strict scan of repositories and linked worktrees, every work tree clean, staged and untracked included, and unlocked) and compares each repository's local tips with what was bundled or left to the servers; any new repository or commit, or any error, puts the tree back. 3. local_tips refuses a symlinked or special HEAD or packed-refs. 4. A tree that cannot be renamed back gets .glaeda-disk-keep, which finish_deletion and the interrupted-deletions family (keep_verdict) honour, so nothing deletes it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…s refs; keep marker first Addresses the tenth Codex review of #1625: 1. repo_unborn needs a regular HEAD, no packed-refs and an empty refs/ read strictly. 2. mirror_tips records each repository's local refs by name as well as its tips; the post-rename recheck requires the refs to be unchanged, so a branch or tag named after the last check puts the tree back. 3. The keep marker is a sidecar written before the rename (a failed write leaves the tree in place) and removed only once the recheck passes, so a kept aside tree always carries it. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Addresses the eleventh Codex review of #1625: worktree records name absolute paths, so once the item is renamed aside its linked worktrees cannot be found and checked again. Any repository in a mirror item with linked worktree records (inside or outside the item) now keeps the item; the post-rename recheck finds the same records and keeps it too. Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com> Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #1622 (landed); based on main.
Leo: "When have we ever had something useful greater than 12 hours?"
Coordinator (sr2): "keeping unpushed commits safe is right, but 168 h of whole checkouts is what fills disks. Fix the root instead."
What changes
CMUX_JOB_MIRROR_HOURSgoes from 168 toDEFAULT_AGE_HOURS(12). It covers the families that usemirror_idle_hours:cmux-job-cache(Linux and macOS) andhq-build-fleet-cache.archive_mirrorfinds every repository it takes along. It enumerates them the way_nested_lossdoes:nested_checkouts, bare repos, the repository of a linked worktree inside the item,modules/submodule stores, and embedded submodules. For each repository,bundle_unpushedreuseslocal_tips(refs, HEAD, reflogs, worktree-private tips),network_remotesandremote_hasto find commits no remote holds. Heads and tags the remotes advertise (ls-remote --heads --tags) count as held, so bare SwiftPM-style mirrors don't bundle their whole history. Those commits go into one bundle. Commits that only a reflog or HEAD holds are named by temporaryrefs/glaeda-archive/<sha>refs, which are removed afterwards. Each branch or tag that exists only locally and sits on a pushed commit gets its own small bundle (ref --not ref^@).~/archive/glaeda-disk/<stamp>-<repo>.expires-<YYYY-MM-DD>.bundle. Each one must passgit bundle verifyandlist-headsmust show every expected ref; only then is it renamed into place and the tree deleted. If nothing is unpushed, no bundle is written.applythat has a mirror family prunes expired bundles withprune_mirror_archive. It only touches files in~/archive/glaeda-diskthat match its own name pattern and whose date has passed. Nothing else in~/archiveis touched.blocked:bundle-failed(verdictblocked, so the run line'sblockedcounts it), the run line gets a newbundle_failedlist, and stderr prints anALERT:line.changed:in-use). Uncommitted work is kept asblocked:uncommitted..glaeda-disk-keep-<name>sidecar is written before the rename and removed only after the recheck passes. That way a tree that can't be renamed back is never finished byfinish_deletionor the interrupted-deletions pass. If the sidecar can't be written, nothing is renamed.HEAD,packed-refsor reflog counts as unreadable, so the tree stays.prune_mirror_archiveprunes from the bundle manifest, with sha256 checks.CMUX_JOB_MIRROR_HOURSstays 168 h throughLEGACY_AGES.Tests
New: an unpushed commit, a detached-HEAD commit, a local-only tag and a local-only branch all end up in verified bundles and the tree is gone; a bundle failure keeps the tree and alerts; a mirror with nothing unpushed goes with no bundle; expired bundles are pruned while unexpired, foreign-named and outside-dir files stay. Updated: the cmux-job and retired-hq-worker mirror tests now expect 12 h and bundling. Locally the hook pin check passes, as do HostReservationTest, LinuxLayoutTest, FleetCasNodeCapTest and the age tests. The full GlaedaDiskTest passes except one known local-environment test. CI passes 18/18. Codex Astra SHIPped head 4b65485 with no P0/P1/P2 after several rounds.
Numbers
Before: a reload unit or hq cache entry idle less than 168 h with a commit no remote holds stayed whole, so unit trees sat for a week (107 GiB across eight minis on 2026-10-11). After: such trees go at 12 h, and only their unpushed objects stay, as bundles of KiB to MiB. This is the policy change; the per-host 24 h free-disk minimum gets measured once this deploys.
Size
1,185 changed lines. The coordinator kept it as one PR: the bundle, the recheck and the delete are one safety path, and splitting them would land a 12 h delete without its archive.
Risk
work/<slot>(rsynced from a developer's laptop, which still has them) are not bundled, same as at 168 h today. A reload unit idle 12 to 168 h used to keep its warm clone and now rebuilds cold.blocked:bundle-failedwith an ALERT, so it's visible and not silent.ls-remoteand, at most,fetch --negotiate-onlyper repository per deletion. Bundles are written under a timeout of 600 s.Agent session: agent session
🤖 Generated with Claude Code