fix: sweep the session network a dead shell leaves behind (#337)

[edgehero]

Issue #337, item 1. The other three items are the panel's, and land in their own PR; this one stays open until then.

The defect

openSandbox creates pi-sandbox-<jobId>-net before it launches the container and removes it in its own finally. A terminal closed mid-session never reaches that finally, and nothing else removed it either: the boot reaper lists only pi-job- networks, and commit 5dbcddc (#277) withdrew removing one at OPEN time. So the network survived every restart, and every later open of that run refused with egress-network-exists until an operator ran the two commands by hand.

Why this is not a reversal of #277

#277 withdrew automatic removal at open time, where it raced a concurrent open: the second open removed the first's network after the first had created it, disconnecting the proxy from a live shell. That decision stands, and the refusal at open time is unchanged.

This is a different mechanism with a different input. A background sweep on the retention reaper is keyed on the retained directories, and an open in progress always has one, because resolveSandbox refuses a job whose directory is gone.

The rule

A pi-sandbox-<id>-net is swept only when <id> is absent from the retained directories, is not running, has no pi-sandbox- container attached, and has no container of its own in any state but exited or dead. Endpoints are inspected first, what was seen is detached, and the removal is a plain network rm, never -f.

Four properties carry the argument, and each has a comment beside it:

• The keep set is a union of two directory listings. The one the pass began with covers a run this pass expired whose open passed resolveSandbox a moment earlier. A fresh one covers the other end: retainJobDir creates a retained directory at job END, in this same process, so a job can finish and its run be opened while the pass is still running. The fresh half arrives as a closure so the sweeper reads it after its own candidate listing.
• Candidates first, then the evidence that protects them, freshest last. An open creates its network BEFORE its container and AFTER the directory that made it legal, so evidence read before the candidate listing can be older than the thing it has to protect. An earlier revision of this branch read it first, and a review pass drove exactly that on real docker: 486 ms of exposure, the network removed, docker start dead with "network not found".
• The endpoint guard gates the detach, not the removal. For a running endpoint the daemon refuses the removal itself, so the harm left to prevent is stripping the proxy off a shell someone is sitting in. The test asserts no network disconnect is issued rather than asserting the network survived, because the weaker assertion passes on docker's refusal alone.
• The last call before the removal is a docker ps -a for that id. See the measurement below.

The sweeper is injected rather than imported: sandbox.mjs imports readManifest from sandbox-store.mjs, so the other direction is a cycle, and injection keeps sandbox-store.mjs docker-free in its own tests.

A measurement that corrects one this project shipped eight days ago

Under #357, egress.mjs's networkEndpoints docblock generalised a measurement to "listed and holds the network agree". On docker 27.4.0 that holds for a running endpoint and for a stopped one. It does not hold for created:

$ docker create --name c --network N busybox sleep 600
$ docker network inspect --format '{{json .Containers}}' N
{}
$ docker ps --filter name=c --format '{{.Names}}'          # empty
$ docker network rm N
N                                                          # exit 0: the daemon is NOT a backstop here
$ docker start c
Error response from daemon: network N not found

docker run leaves a sandbox in that state for 230 ms with the image local, and for the whole pull when it is not. The docblock now says what was measured, and the sweep asks docker ps -a for that id as the call immediately before the destructive verb. Both other networkEndpoints callers (live-probes.mjs, doctor's canary) carry the residual in a comment and are deliberately unguarded: each touches only a network whose owning pid is DEAD, and a dead process has no launch in flight.

Naming, and one silence that is deliberate and one that is not

Per OQ-007's property that one grep covers boot and every tick, a per-network outcome is reaped_sandbox_network or sandbox_network_not_reaped with a fixed reason token (unreadable, containers-unreadable, sandbox-attached, sandbox-present, rm-failed); only the sweep's own fault keeps sandbox_reaper_skipped. A network skipped because its run is retained or running is passed over in silence, because a line per retained run per pass is noise. A network held back by a container of its own is SAID, because a container stuck in created never ages out (--rm fires on exit, so never on one that never started), nothing in this project removes a pi-sandbox- container, and it is invisible to docker ps, to pi-dispatch sandbox --list and to network inspect, so without a line nothing on the host would name it.

The cost, which narrows the issue's acceptance line deliberately

#337 asks for "a leftover sandbox network with no sandbox attached" to be swept. This sweeps one with no sandbox attached and no retained directory, so a leftover whose run is still resurrectable survives until its window closes, and then one pass longer, because the pass that deletes a directory still counts that run as retained. Until then the next open of that run refuses with the exact commands, which is today's behaviour unchanged. #337's complaint is unboundedness rather than friction, and the stricter reading is what reintroduces #277's race.

Verification

• Full suite in the CI posture: 3935 tests, 0 fail, 1 skipped (the Linux-only systemd parse). dated-fixture-check, test-count-check and temp-dir-check green.
• Mutations confirmed red across the round: key on the survivors instead of the pre-pass listing; drop the fresh listing, or let it replace rather than union; drop the running consult; drop the attached-container check; drop the container look, or move it to the top of the pass, or after the removal; compare container names by prefix instead of whole name; add created to the sweepable states, or invert the allowlist into a denylist; compare states case-sensitively; let a failed listing fall through; unanchor the network shape; add -f; drop the yield; make the ENOENT root skip again; default retained to empty; pass the sweeper factory uncalled.
• Real docker, the case that matters: a container created MID-PASS, right after the sweeper's network ls, leaves its network untouched with {reason: "sandbox-present"} and docker start then succeeds. With the guard removed, the same run removes the network and docker start fails. A running session blocks the detach; a foreign network outside the name shape survives; the proxy stays up throughout.

Review rounds, and what is filed rather than fixed

Three rounds, three reviewers each, against a merged diff, an executing pass and an adversarial pass. Nineteen confirmed findings, and the number worth recording is that two were in the original code and the rest were in the repairs. The branch's round cap then applies: the last round took one simpler-rule fix and what survived is filed.

#363: the check-then-act window is two commands wide rather than one, because removeNetworkOrSay detaches before it removes, and it widens by one command per endpoint. Reaching it still needs both deeper guards to miss. The same issue records that a retained directory which cannot be removed holds its network indefinitely with only the directory named, and that Podman's .State vocabulary is unmeasured.

#362, found in passing and unrelated to this change: --publish is inert while the egress policy is armed, because the sandbox is on an --internal network, and the CLI prints published: anyway.

Specs and docs

AMENDED: REQ-RESURRECTABLE-SANDBOX, INT-SANDBOX-CONTRACT (one clause corrects "such a network stays until an operator removes it"; another corrects a bullet written last week under #357), INT-EGRESS-POLICY-CONTRACT, DES-EGRESS-DENY-ON-A-DEDICATED-NETWORK, DES-SANDBOX-IS-A-FRESH-CONTAINER (two Rejected entries this change tested rather than reversed), OQ-007.

UNCHANGED, checked: REQ-EGRESS-ALLOWLIST, REQ-DEPLOYMENT-BOOTSTRAP, REQ-LOCAL-JOB-VISIBILITY, INT-LIVE-PROBE-CONTRACT (behaviour untouched; the corrected measurement was checked against it and its call site says why it needs no guard), DES-RETENTION-SWEEPS-ON-A-TIMER, DES-CONCURRENCY-3, CONST-ISOLATION-CONTAINER-PER-JOB (its sandbox clause is about a container name, and the boot reaper's filter is untouched), OQ-016.

Docs: docs/sandbox.md, SECURITY.md's namespace disclosure, and both .env.example copies.

Out of scope, re-filed rather than lost

The sandbox launcher is hard-wired to the docker CLI and sandboxVenueRefusal reopens only the local adapter by name. That belongs to #354, and it is re-filed there rather than disappearing when this issue is eventually resolved.

Refs #337