[sandbox audit] Make command supervision survive PID reuse, timeouts and long runs

[Wauplin]

[sandbox audit] — PR 7 of 13 in this repo's stack; merge in order.
Previous: #20 · Next: #22
Review only the commits this PR adds on top of its base; bases collapse to main as the stack lands.

Two commits, deliberately separable: the supervision fixes, then the orphan reaper (the
riskiest piece — drop that commit if you'd rather it landed on its own).

Why

Five supervision problems, all of which fail quietly.

1. Timeout watchers slept past the child's exit. Every timeout= command spawned a thread
that slept to the deadline whether or not the child was still alive. On the parent commit, 20
commands that finish instantly but ask for timeout=3600:

threads: baseline 1, after 20 commands with timeout=3600 -> 21

Those 21 threads stay for an hour. The watcher now waits on a cancellation channel or the
deadline, whichever comes first.

2. And then signalled a raw PID. After sleeping it did kill(pid, 0) and killed the
group — using a PID captured when the command started. A PID stops identifying a process the
moment it exits, and the kernel may reuse the number; we run as root and often as PID 1, so
the recycled PID could be anything. Commands now carry a ProcHandle holding a pidfd,
which refers to the process, not the number. The PID is kept only for the process-group sweep,
which runs after the pidfd confirms the leader is alive. Kernels without pidfd_open fall
back to the old behaviour.

3. A foreground command was invisible to the idle watchdog. It was never registered in
ProcRegistry, so running_count() returned 0, and last_activity_ms was stamped only when
a request arrived. A single run() longer than idle_timeout with no other traffic shut
down its own job mid-command — with the 10-minute default, that is any long build or training
step. On the parent commit, a 12s command against a 3s idle timeout:

FAIL  command did not survive the idle watchdog: {"event":"start","pid":579}
FAIL  the server exited under its own command

Fixed twice over, both cheap: an RAII ActiveOp guard the watchdog counts, and a
last_activity_ms stamp on every streamed output event (output is evidence of life).

4. Teardown reported success it hadn't achieved. kill_uid gave up after 50 passes and
returned nothing, so a sweep that left processes running was indistinguishable from a clean
one — and the caller then deleted the home directory and answered {"deleted": true}. It now
reports convergence, and DELETE /v1/sandboxes[/{id}] answers 500 with the reason. The
sandbox is still removed from the registry either way (the client should stop using it
regardless); what changes is that it's told.

5. Orphans were never reaped. The server is the adoptive parent of re-parented
grandchildren and only ever waited on its own Child handles, so they stayed zombies for the
job's lifetime.

Plus the registry no longer grows without bound: finished entries are capped per scope
(newest kept), running ones never dropped.

One bug worth recording

My first reaper used waitpid(-1, WNOHANG | WNOWAIT) to peek before reaping. WNOWAIT is a
waitid-only flag — waitpid rejects it with EINVAL, which silently turned the whole
reaper into a no-op. It ran, returned no error, and did nothing. It uses waitid now.

That surfaced only because a second bug made it visible: zombies were being counted as live
processes by the uid kill sweep, so an unreaped zombie made the sweep spin until it gave up
and turned a clean delete into a 500. A zombie holds no memory, files or CPU and cannot be
killed, so it is now excluded from the sweep's view.

Validation

• 38 unit tests still pass.
• scripts/process-supervision-regression.sh — root container, all passing:
  • watchers exit with their commands (threads return to baseline);
  • a genuinely overrunning command is still killed and reported timed_out: true;
  • a 12s command survives a 3s idle timeout, the server stays up, and the watchdog still
    fires once things really are idle (the fix must not disable it);
  • a detached grandchild leaves no zombie;
  • exit codes 0, 1, 3 and 7 are still reported correctly with the reaper running — the check
    that matters most, since a reaper that steals a status corrupts every command's result;
  • 300 short background commands leave a capped registry, while a running process is not
    evicted;
  • a clean delete answers 200; a setsid survivor is caught by the uid sweep.
• Verified against the parent commit: 21 leaked threads, and the server exiting under its
  own foreground command.

One honest gap: the zombie check did not reproduce on the parent commit (0 → 0), so that
particular assertion is not proven to catch the original bug — the timing of when the
grandchild's parent exits matters. The mechanism is plain in the code, and the check did catch
my own broken reaper, which is the next best evidence.

Behaviour changes

• DELETE /v1/sandboxes[/{id}] can now answer 500 with {"deleted": true, "error": …}
  when teardown was incomplete. Previously always 200. A client treating any non-2xx as "not
  deleted" would now retry a delete that did remove the sandbox — worth a look on the client
  side, and the reason the body still says deleted: true.
• /processes no longer lists every finished process forever (capped, newest kept). The
  client docstring promising otherwise is updated in the companion client PR.