Skip to content

[sandbox audit] Bound server resources, and fix two file-protocol bugs - #26

Draft
Wauplin wants to merge 1 commit into
security/scope-health-metadatafrom
security/bound-server-resources
Draft

[sandbox audit] Bound server resources, and fix two file-protocol bugs#26
Wauplin wants to merge 1 commit into
security/scope-health-metadatafrom
security/bound-server-resources

Conversation

@Wauplin

@Wauplin Wauplin commented Sep 8, 2026

Copy link
Copy Markdown
Collaborator

[sandbox audit] — PR 11 of 13 in this repo's stack; merge in order.
Previous: #25 · Next: #24
Review only the commits this PR adds on top of its base; bases collapse to main as the stack lands.

Why

A pool host has no cgroups to partition its sandboxes, so anything one sandbox can consume
without limit, it takes from its neighbours. Several of those limits were missing — and two
file-protocol bugs corrupted data rather than merely wasting memory.

The two data bugs

A shrinking file desynchronised the connection. A read advertised Content-Length from
the file's size, then stopped early if the file shrank mid-transfer — and kept the keep-alive
connection open. The next response then began mid-body and was read as the tail of this one.

A ranged write left a stale tail. Ranged writes deliberately don't truncate, so
overwriting a large file with a smaller one through parallel chunks left the old bytes past
the new end. (This is also what clippy's file opened with create, but truncate behavior not defined was pointing at.)

The bounds

  • max_procs / max_mem_mb came straight from the request body, and max_mem_mb * 1024 * 1024 had no checked_mul — a value near 2^54 wrapped in release builds, producing either
    an effectively unlimited address space or a sandbox where nothing can start.
  • Only RLIMIT_NPROC and RLIMIT_AS were set — nothing stopped one sandbox exhausting
    the host's descriptors, filling its disk, or spinning a core.
  • The output channel was unbounded: a client that stopped reading a 50 MiB producer took
    server RSS to ~52 MiB, per connection.
  • Invalid numeric config fell back to a default — for SBX_CAPACITY that meant an
    unlimited host, so a typo silently removed the packing bound.

Behaviour change worth naming

Bounding the output queue means a stalled client now slows the command rather than growing
the server's heap. That's how docker logs behaves and the right trade on a shared host, but
it is a real change: a command producing faster than its consumer reads will block. The buffer
is ~2 MiB, comfortably more than any interactive command produces between reads.

SBX_CAPACITY also now defaults to 64 rather than unlimited. Unlimited was the absence of
a setting, never a considered choice, and the client always sets it explicitly.

Also

count is bounded by the host's remaining capacity rather than an arbitrary 4096 (the env map
is cloned per sandbox); the create env is capped at 64 KiB; directory listings are paginated
(limit/after, reporting truncated/next); and a non-regular file is refused in dedicated
mode too, so a FIFO can't block a connection thread waiting for a writer.

Validation

scripts/resource-bounds-regression.sh, all passing:

refused SBX_CAPACITY=abc / SBX_PORT=notaport / SBX_MAX_CONNECTIONS=-5 / SBX_CAPACITY=0
max_mem_mb 2^54, u64::MAX, 0 and max_procs 0, 100000 -> 400; a sane request -> 200
count: 4000 on a capacity-4 host -> created 3, rest reported rejected
200 KB env -> 400
ulimit -n inside a sandbox -> 4096
ranged overwrite -> file is exactly what was uploaded, no stale tail
listing: limit honoured, truncation reported, cursor advances (not repeating)
FIFO -> refused promptly (400), not a hang
stalled reader: RSS 648 kB before, 648 kB after 64 MiB of unread output
5 MB round-trip + streamed exec still work

Verified against the parent commit: every invalid config accepted, every out-of-range
limit accepted, the 200 KB env accepted, no pagination.

Two of my own test bugs along the way: a 200 KB env on the command line exceeds ARG_MAX, and
I set SBX_PORT twice in one env invocation so the valid value won — the check passed
against a server that had ignored the bad value entirely.

Wire changes

  • PUT /v1/files/write gains truncate_to.
  • GET /v1/files/list gains limit/after and returns truncated/next (additive).
  • A read of a file that shrinks mid-transfer now closes the connection instead of returning
    short and reusing it.

The client-side half of this finding (full output capture even with callbacks, read_bytes()
uploads, fully-materialised parallel downloads) is a separate PR.

A pool host has no cgroups to partition its sandboxes, so anything one
sandbox can consume without limit, it can take from its neighbours. Several
of those limits were missing, and two file-protocol bugs corrupted data
rather than merely wasting memory.

**A shrinking file desynchronised the connection.** A read advertised
`Content-Length` from the file's size, then stopped early if the file got
smaller mid-transfer -- and kept the keep-alive connection open. The next
response then began mid-body and was read as the tail of this one. It now
fails the connection instead: an honest truncated transfer beats a silently
desynchronised one.

**A ranged write left a stale tail.** Ranged writes deliberately do not
truncate, so overwriting a large file with a smaller one through parallel
chunks left the old bytes past the new end. New `truncate_to` parameter lets
the client state the final size, so the file ends up exactly what was
uploaded.

**Caller-supplied limits are now clamped by the server.** `max_procs` and
`max_mem_mb` came straight from the request body, and `max_mem_mb * 1024 *
1024` was computed without `checked_mul` -- so a value near 2^54 wrapped in
release builds and produced either an effectively unlimited address space or
a sandbox where nothing could start. Both are bounded and rejected outside
their range, and the multiplication is saturating.

**`RLIMIT_NOFILE`, `RLIMIT_FSIZE` and `RLIMIT_CPU`** join the two limits
that were already set, so one sandbox cannot exhaust the host's descriptors,
fill its disk, or spin a core indefinitely.

**Output no longer buffers without bound.** The stdout/stderr channel was
unbounded, so a command producing faster than the client reads was buffered
in the server's heap -- a client that stopped reading a 50 MiB producer took
RSS to ~52 MiB, per connection, on a host shared by dozens of sandboxes. A
bounded channel applies backpressure through the pipe to the command
instead. That is a behaviour change worth naming: a stalled client now slows
the command rather than growing the server. It is how `docker logs` behaves,
and the right trade on a shared host.

**Invalid numeric configuration refuses to start.** These fell back to a
default on a parse failure, which for `SBX_CAPACITY` meant an *unlimited*
host -- a typo silently removed the packing bound. `SBX_CAPACITY` also
defaults to 64 rather than unlimited; unlimited was the absence of a
setting, never a considered choice, and the client always sets it.

Also: `count` is bounded by the host's remaining capacity rather than an
arbitrary 4096 (the env map is cloned per sandbox); the create `env` is
capped at 64 KiB; directory listings are paginated with `limit`/`after` and
report `truncated`/`next`; and a non-regular file is refused in dedicated
mode too, so a FIFO cannot block a connection thread waiting for a writer.

Validation:
- 48 unit tests still pass.
- `scripts/resource-bounds-regression.sh`, all passing: four invalid configs
  refuse to start; five out-of-range limit requests are rejected while a sane
  one succeeds; `count: 4000` on a capacity-4 host creates 3 and reports the
  rest rejected; a 200 KB env is refused; `ulimit -n` inside a sandbox
  reports 4096; a ranged overwrite leaves no stale tail; a listing honours
  `limit`, reports truncation, and its cursor advances rather than repeating;
  a FIFO is refused promptly instead of hanging; a stalled reader leaves RSS
  flat (648 kB before and after 64 MiB of unread output); and a 5 MB
  round-trip plus streamed exec still work.
- Verified against the parent commit: every config accepted, every
  out-of-range limit accepted, the 200 KB env accepted, no pagination.

Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
@Wauplin
Wauplin force-pushed the security/scope-health-metadata branch from 93e8010 to 141cec1 Compare September 8, 2026 14:56
@Wauplin
Wauplin force-pushed the security/bound-server-resources branch from 4605065 to 8e62c8f Compare September 8, 2026 14:56
@Wauplin Wauplin changed the title Bound server resources, and fix two file-protocol bugs [sandbox audit] Bound server resources, and fix two file-protocol bugs Sep 9, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant