Skip to content

glaeda-disk: mirrors go at 12 h, unpushed commits and local-only refs bundled to ~/archive first - #1625

Merged
teamleaderleo merged 40 commits into
mainfrom
disk-glaeda-mirror-bundle
Oct 27, 2026
Merged

teamleaderleo merged 40 commits into
mainfrom
disk-glaeda-mirror-bundle

Conversation

@teamleaderleo

@teamleaderleo teamleaderleo commented Oct 26, 2026 •

Copy link
Copy Markdown
Owner

Follows #1622 (landed); based on main.

Leo: "When have we ever had something useful greater than 12 hours?"
Coordinator (sr2): "keeping unpushed commits safe is right, but 168 h of whole checkouts is what fills disks. Fix the root instead."

What changes

  • CMUX_JOB_MIRROR_HOURS goes from 168 to DEFAULT_AGE_HOURS (12). It covers the families that use mirror_idle_hours: cmux-job-cache (Linux and macOS) and hq-build-fleet-cache.
  • Before such an item is deleted, archive_mirror finds every repository it takes along. It enumerates them the way _nested_loss does: nested_checkouts, bare repos, the repository of a linked worktree inside the item, modules/ submodule stores, and embedded submodules. For each repository, bundle_unpushed reuses local_tips (refs, HEAD, reflogs, worktree-private tips), network_remotes and remote_has to find commits no remote holds. Heads and tags the remotes advertise (ls-remote --heads --tags) count as held, so bare SwiftPM-style mirrors don't bundle their whole history. Those commits go into one bundle. Commits that only a reflog or HEAD holds are named by temporary refs/glaeda-archive/<sha> refs, which are removed afterwards. Each branch or tag that exists only locally and sits on a pushed commit gets its own small bundle (ref --not ref^@).
  • Bundles go to ~/archive/glaeda-disk/<stamp>-<repo>.expires-<YYYY-MM-DD>.bundle. Each one must pass git bundle verify and list-heads must show every expected ref; only then is it renamed into place and the tree deleted. If nothing is unpushed, no bundle is written.
  • Every apply that has a mirror family prunes expired bundles with prune_mirror_archive. It only touches files in ~/archive/glaeda-disk that match its own name pattern and whose date has passed. Nothing else in ~/archive is touched.
  • When bundling fails, the tree is kept. The receipt gets outcome blocked:bundle-failed (verdict blocked, so the run line's blocked counts it), the run line gets a new bundle_failed list, and stderr prints an ALERT: line.
  • Ownership, live-process, lock/lease, job-busy and wt-owner checks are unchanged. Bundling runs after all of them, right before the delete.
  • After the bundle, right before the delete, the item is renamed aside and run through the same check again: a fresh repo and worktree scan with strict reads, a clean check covering staged, tracked and untracked changes, and every local ref compared by name and value with the refs recorded before bundling. Any difference or error renames the tree back and keeps it (changed:in-use). Uncommitted work is kept as blocked:uncommitted.
  • A .glaeda-disk-keep-<name> sidecar is written before the rename and removed only after the recheck passes. That way a tree that can't be renamed back is never finished by finish_deletion or the interrupted-deletions pass. If the sidecar can't be written, nothing is renamed.
  • Any repo with linked-worktree records, inside or outside the item, keeps the item. Git stores those records as absolute paths, so the recheck can't find them after the rename.
  • A symlinked or non-regular HEAD, packed-refs or reflog counts as unreadable, so the tree stays.
  • prune_mirror_archive prunes from the bundle manifest, with sha256 checks.
  • On a reserved host, CMUX_JOB_MIRROR_HOURS stays 168 h through LEGACY_AGES.
  • If a tree can't be covered safely, it is kept as blocked: its repo has a worktree outside the item, a linked worktree whose repo is outside holds commits only its own reflog has, an ignored nested checkout would lose something, the walk is too big, or no network remote answers.

Tests

New: an unpushed commit, a detached-HEAD commit, a local-only tag and a local-only branch all end up in verified bundles and the tree is gone; a bundle failure keeps the tree and alerts; a mirror with nothing unpushed goes with no bundle; expired bundles are pruned while unexpired, foreign-named and outside-dir files stay. Updated: the cmux-job and retired-hq-worker mirror tests now expect 12 h and bundling. Locally the hook pin check passes, as do HostReservationTest, LinuxLayoutTest, FleetCasNodeCapTest and the age tests. The full GlaedaDiskTest passes except one known local-environment test. CI passes 18/18. Codex Astra SHIPped head 4b65485 with no P0/P1/P2 after several rounds.

Numbers

Before: a reload unit or hq cache entry idle less than 168 h with a commit no remote holds stayed whole, so unit trees sat for a week (107 GiB across eight minis on 2026-10-11). After: such trees go at 12 h, and only their unpushed objects stay, as bundles of KiB to MiB. This is the policy change; the per-host 24 h free-disk minimum gets measured once this deploys.

Size

1,185 changed lines. The coordinator kept it as one PR: the bundle, the recheck and the delete are one safety path, and splitting them would land a 12 h delete without its archive.

Risk

  • Medium. Mirror trees now go after 12 h instead of a week. Uncommitted edits in work/<slot> (rsynced from a developer's laptop, which still has them) are not bundled, same as at 168 h today. A reload unit idle 12 to 168 h used to keep its warm clone and now rebuilds cold.
  • Trees that can't be bundled now stay where before they would have gone at 168 h: no network remote reachable, a walk too large (5M entries), or an outside worktree. Each one shows as blocked:bundle-failed with an ALERT, so it's visible and not silent.
  • Bundling runs ls-remote and, at most, fetch --negotiate-only per repository per deletion. Bundles are written under a timeout of 600 s.

Agent session: agent session

🤖 Generated with Claude Code

@teamleaderleo

Copy link
Copy Markdown
Owner Author

Owned by an agent session · session

config/fleet-reservations.json is glaeda's copy of leo-workspace's fleet-registry
reservations (a test compares the two where the registry is checked out); mini-setup
installs it. On a host an active entry (or the host-local reservation.json) names,
glaeda-disk runs no idle sweep and a pressure pass acts only below the runner floor,
back to floor + target_margin_gib. glaeda-fleet-cas-prune leaves a reserved host above
its floor alone. An unreadable list fails closed. Entries lapse by themselves.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo and others added 6 commits October 26, 2026 14:52
… h go above the floor

Leo, 2026-10-26: "When have we ever had something useful greater than 12 hours?"
DerivedData, CoreSimulator runtimes, package caches, runner work dirs and build dirs
now go once idle DEFAULT_AGE_HOURS (12 h). The per-host cmux-next build root, the
newest warm compile cache per builder, is exempt by role (only under the floor).
The git-mirror window stays 168 h: past it a checkout's unpushed commits are not checked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…es (no lock, no commit only here)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ervation lapses

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo and others added 2 commits October 26, 2026 14:53
…y refs bundled to ~/archive first

Before a mirror family's item (cmux-job-cache, the hq build cache) goes, archive_mirror finds every
repository it takes along (checkouts, bare repos, a linked worktree's repository inside it, submodule
stores, embedded submodules) and writes each one's commits no remote holds, plus branches and tags only
this host has, into git bundles under ~/archive/glaeda-disk, named with a 30-day expiry and pruned by
that name. Each bundle passes `git bundle verify` and lists every ref it must hold before the tree is
deleted; a failure keeps the tree, records blocked:bundle-failed in the receipt and run line, and alerts
on stderr. CMUX_JOB_MIRROR_HOURS drops from 168 to DEFAULT_AGE_HOURS (12).

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
@teamleaderleo
teamleaderleo force-pushed the disk-glaeda-mirror-bundle branch from 348cda4 to 510e97e Compare October 26, 2026 21:56
teamleaderleo and others added 5 commits October 26, 2026 15:14
… closed on any malformed entry

The list no longer ships in this repository. glaeda-disk and glaeda-fleet-cas-prune read it only from
~/.local/share/glaeda/fleet-reservations.json in the home of the user that owns /Users/Shared/cmux-build-fleet
(written there by the fleet's own writer), plus FLEET_ROOT/reservation.json. A missing, unreadable or
malformed list (hosts not a list of names, a bad time on any entry, aliases not lists) reads as reserved,
whatever the host-local file says.

Also from review: the prune registers the glaeda-disk module in sys.modules before running it, so its
dataclasses load and the gate is real; the reservation gate runs before store_gc; --runner-caches is held to
the same floor-only rule; and a reserved host's pressure starts at the floor itself, not floor + low_margin.
Test fixtures use made-up ids, hosts and owners.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…eep dirty trees, recheck before delete

Addresses the Codex review of #1625:
1. "held" comes only from what the servers confirm (ls-remote heads and tags, and fetch --negotiate-only
   acks, asked one commit at a time when a batch fails); stale remote-tracking refs are only candidates.
2. Bundle names carry a hash of the repository's real path and ref, the pid and the time, and are linked
   into place with os.link, so an existing bundle is never replaced.
3. A work tree with tracked edits, untracked non-ignored files or files hidden from status keeps the item
   (blocked:uncommitted); ignored build output still goes.
4. Checkouts inside ignored directories are bundled like the others.
5. An unreadable submodule store keeps the item (os.walk onerror).
6. Right before the delete, fresh process evidence and a snapshot of every ref, HEAD, reflog and work tree
   status taken before bundling are checked again; any change keeps the tree.
7. Pruning deletes only bundles manifest.jsonl lists, only at the sha256 it recorded, and only after the
   process-evidence gate.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…d every worktree's HEAD; explicit policy keys stay

Review findings on #1622:
- the above-floor veto now also keeps a checkout with FETCH_HEAD.lock, MERGE_HEAD, REBASE_HEAD, a rebase or
  sequencer dir, CHERRY_PICK_HEAD, REVERT_HEAD or BISECT_LOG (files or dirs, linked worktrees' git dirs too);
- a commit that only a linked worktree's HEAD names keeps the checkout;
- keep_legacy_ages leaves a key disk-policy.json sets alone even when its value equals the new default
  (policy_overrides gives the file's keys).
Tracked edits and reflog-only commits still go with a runner checkout: actions/checkout's next job runs
git clean -ffdx and git reset --hard there, so they never carry from one job to the next.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-12h-age

# Conflicts:
#	scripts/glaeda-disk
#	scripts/test-glaeda-disk.py
@gitguardian

gitguardian Bot commented Oct 26, 2026 •

Copy link
Copy Markdown

⚠️ GitGuardian has uncovered 3 secrets following the scan of your pull request.

Please consider investigating the findings and remediating the incidents. Failure to do so may lead to compromising the associated services or software components.

🔎 Detected hardcoded secrets in your pull request
GitGuardian id GitGuardian status Secret Commit Filename
35581672 Triggered Generic Password 5922735 scripts/test-glaeda-disk.py View secret
35581672 Triggered Generic Password 4271160 scripts/test-glaeda-disk.py View secret
35581672 Triggered Generic Password 7adc2af scripts/test-glaeda-disk.py View secret
🛠 Guidelines to remediate hardcoded secrets
  1. Understand the implications of revoking this secret by investigating where it is used in your code.
  2. Replace and store your secrets safely. Learn here the best practices.
  3. Revoke and rotate these secrets.
  4. If possible, rewrite git history. Rewriting git history is not a trivial act. You might completely break other contributing developers' workflow and you risk accidentally deleting legitimate data.

To avoid such incidents in the future consider


🦉 GitGuardian detects secrets in your source code to help developers and security teams secure the modern development process. You are seeing this because you or someone else with access to this repository has authorized GitGuardian to scan your pull request.

teamleaderleo and others added 10 commits October 26, 2026 15:37
…e, reads as reserved

The writer (leo-workspace host-baseline ci-minis) stamps expires 13 h after each write and refreshes it every
6 h, so a mini the writer stopped reaching (for one, a host reserved after its last write) falls back to
reserved by itself.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
… markers and intervals fail closed

Review findings on #1626:
- a FLEET_ROOT/reservation.json that is there but unreadable, not glaeda-reservation/v1 or without integer
  since/until holds the host, as host_busy reads it (absent: no hold);
- a fleet entry with from >= until fails the whole list closed;
- on a reserved host, after the floor-pressure pass, no stale-marker clearing, paused-work cleanup, test-residue
  reaping, home-clone gc or runner repack runs, and --dedupe / --dedupe-builds do nothing (the reservation is
  asked for them too);
- filesystems() gets reserved= only when set, so callers and mocks that pass positionals keep working.
Each gate has a test that fails when its check is removed.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…d keeps what it cannot read

Review findings on #1622: GIT_DIR/GIT_WORK_TREE from the caller would point git -C at another repository (as
_checkout_git already strips them); an unreadable directory under .git (a hidden worktree's lock) now raises
in the walk and keeps the checkout.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…nto disk-glaeda-12h-age

# Conflicts:
#	scripts/glaeda-disk
…os and rechecks owners before delete

Addresses the second Codex review of #1625:
1. mirror_repos queues every initialized submodule work tree (embedded or in modules/), so a submodule's
   own submodules are bundled too.
2. and 3. mirror_moved finds the item's repositories again right before the delete: a repository created
   meanwhile, or a work tree no longer clean (hidden-file flags included), keeps the tree.
4. mirror_moved re-reads owner_retired and the fleet artifact references after bundling.
5. A locked linked worktree keeps the tree, as in git_loss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ous commit lost this edit)

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
teamleaderleo and others added 11 commits October 26, 2026 15:59
…ure since holds, as the runner hook reads it

Review findings on #1626: lexists() read a permission error as absence; only FileNotFoundError on the read is
absence now. The runner hook holds while now < until whatever since says; host_reservation matches it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…d errors, honours locks inside

Addresses the third Codex review of #1625:
1. The commits the bundles leave to the servers (their prerequisites from rev-list --boundary, and tips no
   bundle holds) are asked about again right before the delete; one no server holds any more keeps the tree.
2. The repository search for a mirror item is strict: a directory that cannot be read (EIO, ESTALE) makes
   it incomplete, and the item stays.
3. A locked linked worktree keeps the item even when its repository is inside it, as in git_loss.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…stores keep, empty repos go

Addresses the fourth Codex review of #1625:
1. mirror_moved asks the servers first and rereads the repositories and their snapshot last, so a local
   commit made while the servers were asked keeps the tree.
2. A symlinked modules/ store, or a symlinked directory inside one, is unreadable, as in submodule_loss.
3. An unborn repository (no refs, no reflog, HEAD naming a branch with no commit) needs no bundle; the
   snapshot reads HEAD as a file, so it is comparable too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…and worktree reads fail closed

Addresses the fifth Codex review of #1625:
1. mirror_moved now asks the servers, then rereads git, then the owner, artifact references and fresh
   process evidence last, so a job that starts during the slow steps keeps the tree.
2. The unborn-repository fast path applies only when the repository has no linked worktrees.
3. Linked worktree records are read strictly: an unreadable worktrees/ or record keeps the item.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
# Conflicts:
#	scripts/glaeda-disk
#	scripts/test-glaeda-disk.py
… worktrees inside the item

Addresses the sixth Codex review of #1625:
1. Reflog directories are walked with files_strict (os.walk with onerror), so an unreadable one makes
   local_tips, the unborn check and the snapshot fail closed instead of skipping it.
2. A linked worktree inside the item (even one under ignored output such as target/) is walked like any
   work tree: it must be clean and unlocked.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ble linked worktrees as unreadable

Addresses the seventh Codex review of #1625:
1. files_strict returns None when a directory below the root is a symlink, which os.walk would not enter,
   so a symlinked logs/refs/heads cannot hide a reflog-only commit.
2. Linked worktrees are probed with os.stat: only FileNotFoundError means absent; EACCES or EIO keeps the
   item instead of skipping the outside-worktree check.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Base automatically changed from disk-glaeda-12h-age to main October 27, 2026 00:23
teamleaderleo and others added 5 commits October 26, 2026 17:38
…ndle

# Conflicts:
#	scripts/glaeda-disk
#	scripts/test-glaeda-disk.py
…he rename; references rechecked

Addresses the eighth Codex review of #1625:
1. mirror_rename_recheck runs as remove()'s after_rename: once the tree is renamed aside, a hash of every
   git dir's HEAD, packed-refs, refs, logs and worktrees (mirror_git_digest, read relative to the item)
   must equal the one taken before bundling, or the tree is renamed back and kept (changed:in-use).
   mirror_moved compares the same digest too.
2. mirror_moved re-runs referenced_main_checkout for git-disposable families.
3. files_strict also refuses a symlinked or special file, which is_file() readers would skip.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…ndle check again; kept trees are marked

Addresses the ninth Codex review of #1625 (replaces the metadata digest):
1, 2. mirror_rename_recheck runs mirror_repos on the renamed tree (a fresh strict scan of repositories and
   linked worktrees, every work tree clean, staged and untracked included, and unlocked) and compares each
   repository's local tips with what was bundled or left to the servers; any new repository or commit, or
   any error, puts the tree back.
3. local_tips refuses a symlinked or special HEAD or packed-refs.
4. A tree that cannot be renamed back gets .glaeda-disk-keep, which finish_deletion and the
   interrupted-deletions family (keep_verdict) honour, so nothing deletes it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
…s refs; keep marker first

Addresses the tenth Codex review of #1625:
1. repo_unborn needs a regular HEAD, no packed-refs and an empty refs/ read strictly.
2. mirror_tips records each repository's local refs by name as well as its tips; the post-rename recheck
   requires the refs to be unchanged, so a branch or tag named after the last check puts the tree back.
3. The keep marker is a sidecar written before the rename (a failed write leaves the tree in place) and
   removed only once the recheck passes, so a kept aside tree always carries it.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
Addresses the eleventh Codex review of #1625: worktree records name absolute paths, so once the item is
renamed aside its linked worktrees cannot be found and checked again. Any repository in a mirror item with
linked worktree records (inside or outside the item) now keeps the item; the post-rename recheck finds the
same records and keeps it too.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Agent-Session: https://cmux.com/deeplink/agent?id=aid-85ca59f74f02d81734c3
@teamleaderleo
teamleaderleo merged commit 9d61dfe into main Oct 27, 2026
18 checks passed
@teamleaderleo
teamleaderleo deleted the disk-glaeda-mirror-bundle branch October 27, 2026 01:36
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant