Skip to content

fix(sandbox): close the sweep's two windows with a tombstone and a guarded open (#446) - #457

Merged
edgehero merged 2 commits into
mainfrom
fix/446-sandbox-sweep-windows
Sep 28, 2026
Merged

edgehero merged 2 commits into
mainfrom
fix/446-sandbox-sweep-windows

Conversation

@edgehero

Copy link
Copy Markdown
Owner

Closes #446.

Two windows could delete a retained run under an operator:

  1. An unpinned sandbox <id> on a run past its window while the sweep was asking its runtime.
  2. A pin landing while a large directory was already being deleted (a 200k-file clone takes about 10 s).

Sweep: a tombstone

After the fresh read matches, the directory is renamed to .reap-<pid>-<now>-<n> in the same root. The name never carries the job id, so no ENAMETOOLONG. The directory is then read once more through the tombstone, and only then deleted:

  • A pin that landed between the fresh read and the rename puts it back, as manifest-changed.
  • A rename failure holds the directory, as rename-failed, in the same sandbox_reaper_skipped family (OQ-007).
  • Every reader of the root skips tombstones: the listing, listSandboxes, the runtime watch, the network sweep's keep set, doctor's count and retainJobDir.
  • Leftover tombstones are cleared at the start of every pass. A tree that will not delete (root-owned files under a non-root worker) stays a tombstone: it is retried each pass, logged once a day, and named by doctor after 10 minutes.

Open: refuse late, pin first, check after launch

  • A run past its deadline, or within SANDBOX_OPEN_GRACE_MS (5 min) of it, is refused unless --pin is given, and the refusal names the command.
  • With --pin, the pin is written right after the run is resolved, before any runtime call. A failed pin now refuses; it used to only warn, and the launch then bound an auto-created empty directory.
  • Once the container is listed, the manifest is read again. A run swept at launch has its container removed and returns swept-at-launch. The check is aborted as soon as the shell returns.
  • The admin panel has no pin action. It stops offering b for such a run, and its refusal points at the CLI's --pin.

Deadline

retainJobDir writes retainUntil from the worker's window. An unpinned run's deadline is the earlier of retainUntil and createdAt plus the reader's current window. So:

  • an opener with a longer window than the worker (the panel is the usual case) still agrees with the worker;
  • lowering retention, or turning it off, still cleans runs retained earlier, as documented.

The one residual is written down in INT-SANDBOX-CONTRACT: a worker window lowered below both the opener's window and the run's retainUntil, with an unpinned open racing that sweep. The post-launch check reports it.

Also

  • A job id with a leading dot (., .., .reap-...) is re-mapped, so it can never name the root's parent or a tombstone.
  • One worker per retention root is stated as the supported configuration. A young tombstone named by another pid is left alone.

Specs

INT-SANDBOX-CONTRACT, REQ-RESURRECTABLE-SANDBOX, OQ-007, OQ-038; docs/sandbox.md.

…arded open (#446)

Two windows could delete a retained run under an operator: an unpinned
open of a run past its window while the sweep was asking its runtime,
and a pin that landed while a large directory was already being
deleted (a 200k-file clone takes seconds).

Sweep. After the fresh read matches, the sweep re-asks the run's own
runtime, renames the directory to a dot-prefixed tombstone
(.reap-<pid>-<now>-<n>, never the job id), reads it once more through
the tombstone (a pin that landed in between puts it back, displacing
only an empty directory), and only then deletes it. A tombstone with a
live pin is never deleted; it is restored only under its own id's
name. Leftover tombstones are cleared at the start of every pass, a
young foreign one only while its pid lives; every reader of the root
skips them.

Open. A run past its deadline, or within SANDBOX_OPEN_GRACE_MS of it,
is refused unless --pin; the pin is written right after resolving the
run, before any runtime call, and a failed pin refuses. Once the
container is listed the manifest and the directory's inode are read
again: a run lost at launch stops the container (podman with
--time=0), one lost later is reported when the shell exits, and only a
never-listed container that exits 125 is read as a refused mount.

Deadline. retainJobDir writes retainUntil from the worker's window;
an unpinned run's deadline is the earlier of retainUntil and createdAt
plus the reader's window, so an opener with a longer window agrees
with the worker and lowering or turning retention off still cleans
runs retained earlier.

Also: a job id with a leading dot or underscore is escaped, runs
retained before the escape stay openable, a directory holding another
job's run is refused; doctor names stuck and pin-held tombstones with
their exact path and never classifies by pid; --list says when a run
needs --pin; one worker per retention root and one runtime endpoint
are the supported configuration; and a new check fails CI on any
merge-conflict marker in a tracked file.

Specs: INT-SANDBOX-CONTRACT, REQ-RESURRECTABLE-SANDBOX, OQ-007, OQ-038.
Signed-off-by: Rob Boerman <robboerman@live.nl>
…ause (#446)

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
edgehero force-pushed the fix/446-sandbox-sweep-windows branch from 329c645 to c26742e Compare September 28, 2026 06:25
@edgehero
edgehero merged commit db5c9b7 into main Sep 28, 2026
7 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

sandbox sweep: close the two windows left after #429 (unpinned open during a slow sweep, pin during a large delete)

1 participant