OpenShell jobs cannot reach their model or the network: the non-dumpable runner vs OpenShell's connect broker #730

Activity

1. edgehero

   added 3 commits that reference this issue on Oct 21, 2026

   test(ci): investigate OpenShell runner networking, E1-E4 copy mode (#730

   ...

   5c5e37a

   test(ci): investigate OpenShell runner networking, E5 bind mode (#730)

   ...

   435dd8d

   test(ci): #730 probes under the job's own policy, the new path's own …

   ...

   6b25b96
2. edgehero commented on Oct 22, 2026

   

   edgehero

   on Oct 22, 2026

   OwnerAuthor

   More actions

   Phase 1 findings (test-only investigation, draft PR #731)

   Measured on the openshell lane (OpenShell 0.1.4, 5a7728c; pi 1.2.0; job image built from the PR), copy and bind mode, in runs 39655548819 and 39674000732. Every experiment is a printed diag block, none asserted. The probe images are the job image with the entrypoint replaced, plus two variants: runner-node chmod 0755 (same bytes, same path), and a 0755 copy at /opt/pi-dispatch/runner-observable added to dev.pi-dispatch.net-binaries.

   Root cause

   The OpenShell sandbox mediates every workload connect with a seccomp user notification. Its broker runs as the workload uid with no capabilities (the Docker driver drops ALL), and it reads the caller's sockaddr out of the caller's memory: first process_vm_readv, then /proc/<tid>/mem (openshell-sandbox/src/linux/task_memory.rs read_exact). It also hashes /proc/<pid>/exe of the caller and of every ancestor (openshell-binary-identity/src/lib.rs). runner-node is root-owned 0711 (issue #545), so exec leaves the supervisor and the runner NONDUMPABLE. The broker therefore cannot read them, and the connect is refused. A DNS lookup is the first connect, so the error is EAI_AGAIN. OpenShell's own startup probe states the assumption outright: "an exec'd workload must be observable by its same-UID ancestor" (task_memory.rs probe_child_access).

   Every openshell job is affected, copy and bind, any provider. No lane case had made a real network call from the runner before; every earlier probe used the image's plain node.

   Measured

   	copy	bind
   E1: runner-node mode / node mode	511 (0711) root:root / 755	same
   E1: dumpable (inferred: a same-uid sibling opening the process's environ)	node readable (dumpable); runner-node denied (nondumpable)	same
   E1: kernel.yama.ptrace_scope, uid	1, 1001:1001	same
   E1: the broker's denial line	openshell logs <sandbox> --source sandbox --level warn, polled during the run, matched nothing. I did not record the raw line count, so I cannot tell whether the query (source, level, a sandbox deleted at teardown) or the absence of the line is the cause. Not established.	not run
   netpath, shipped image: plain node	200 on GET, streamed POST, with and without auth	same
   netpath, shipped image: runner-node (0711)	EAI_AGAIN on every step, before and after loading pi, and through the OpenAI SDK	same
   E2: runner-node 0755, same path	200 on every step, pi's dispatcher and the OpenAI SDK included (chunks 2)	same
   E2: 0755 copy at /opt/pi-dispatch/runner-observable	200 on every step; the generated policy lists the path (true, label too)	same
   E3: tool child (node fetch, git ls-remote https://github.com/...) under the 0711 runner	fetch EACCES, git code 128 (cannot connect)	same
   E3: the same under a dumpable parent (node, or runner-node 0755)	fetch 200, git code 0	same
   E4: child reads the parent's /proc/<ppid>/environ, parent dumpable	READ (canary seen)	same
   E4: /proc/<ppid>/mem, parent dumpable	EACCES (Yama, scope 1)	same
   E4: process_vm_readv, ptrace(PTRACE_ATTACH), parent dumpable	EPERM both (OpenShell's workload seccomp, child_seccomp.rs)	same
   E4: all of the above against the 0711 runner	EACCES / EPERM	same

   The first run's copy-mode E2 and E3 rows read ENOTFOUND for plain node too. That was a defect in my probe (the probe runs had no policy endpoints), fixed in the second run. It was not a result.

   E4's fd/0 row (READ) is not about the runner. My harness gave its parent a regular file on fd 0, and a regular file can be reopened through /proc. The real runner's fds are covered below.

   The process tree in a real openshell job (from source)

   1. OpenShell sandbox supervisor: nondumpable itself, unfiltered threads, hosts the connect broker.
   2. /opt/pi-dispatch/pi-landlock: a static helper, readable, so dumpable. It confines writes, then execs:
   3. /entrypoint.sh (/bin/sh), which execs:
   4. runner-node --disable-sigusr1 supervise.mjs, the job supervisor. It holds the exit key in memory. Its fd 0 is the exec's stdin, which carries every frame (key frame and steer frames). Today it is nondumpable.
   5. /bin/sh -c '…; exec runner-node --disable-sigusr1 run-job.mjs', the runner. It holds the exit key in memory. Its fd 0 is a Node "pipe" (a socketpair) carrying the key frame, closed at once; fd 3 is a socketpair for later frames, not inherited by children. Reopening a socketpair through /proc is ENXIO (measured in OpenShell: steering and live transcripts over stdio, optional bind mode #686). pi runs in this process and makes the provider calls.
   6. Tools (bash, git, node and so on): they exec readable binaries, so they are dumpable.

   runner-node is one binary at one path for both (4) and (5), so making it 0755 makes BOTH the supervisor and the runner dumpable. E3 shows that any nondumpable ancestor (the supervisor included) also blocks every descendant's network. So a fix that keeps the supervisor nondumpable while making only the runner dumpable cannot work as long as the supervisor is an ancestor.

   What would sit in the environ of every process that becomes dumpable

   /proc/<pid>/environ is the process's initial environment block. Changing process.env later does not change it.

   • Provider keys, the forge token (GH_TOKEN/GITHUB_TOKEN), endpoint-bound bundle variables: OpenShell placeholders only (openshell:resolve:env:<NAME>), resolved by OpenShell outside the sandbox. Measured in the existing lane: the placeholder is in the environment, and no process's environ holds a value (environ-values 0).
   • The exit key: never in any environment. It arrives as frame 1 on stdin.
   • Plain bundle variables: not in any environ. They arrive by file (bundle-env.json), and the runner puts them into its own process.env after start (bundle-env.mjs), which /proc/<pid>/environ does not show. They are values the operator gives the agent anyway, and every tool inherits them by design.
   • Steer frames: never in the environment. The runner's frame descriptors are socketpairs (ENXIO on reopen).
   • The rest: PI_* job configuration, PI_DISPATCH_KEYLESS=keyless, HOME, the Playwright settings and the forge host variables. No secret values.

   So a tool reading a dumpable supervisor's or runner's environ learns configuration and placeholders, not secrets.

   Still open: the supervisor's fd 0 is the exec's stdin, whose kind under OpenShell (pipe or socket) I did not measure. If it is a pipe, a tool could reopen it through a dumpable supervisor's /proc/<pid>/fd/0. Every frame is verified under the key, so that could drop or delay messages, never forge one. Measure this before shipping a dumpable supervisor.

   Does the protection depend on the host?

   Yes, for one route. With the supervisor and runner dumpable, the exit key in their memory is protected by:

   • OpenShell's workload seccomp, for process_vm_readv, ptrace, pidfd_getfd and process_vm_writev. This holds on every host.
   • Yama, for /proc/<pid>/mem. Opening and reading that file is not a filtered syscall. Yama ptrace_scope >= 1 stops a child from attaching to its ancestor. At ptrace_scope = 0, which is the kernel default on hosts that do not set it, a same-uid tool could read a dumpable ancestor's memory, and with it the key.

   Nothing inside the sandbox can lower ptrace_scope. It is a host-wide sysctl, writing it needs CAP_SYS_PTRACE in the initial user namespace, and the sandbox runs with no capabilities and no_new_privs. The venue already requires the gateway to run on the worker's own host (openshell-gateway-not-local), so the worker can read the setting that applies to its jobs.

   Other ways around it (E6, from source)

   • Giving the broker CAP_SYS_PTRACE: no configuration in 0.1.4 does this. The Docker driver hard-codes cap_drop: ALL and adds none.
   • Another read path: none. The broker reads only through process_vm_readv and /proc/<tid>/mem. The notification carries pointers, not the sockaddr. SECCOMP_ADDFD injects descriptors and reads nothing. The workload filter denies pidfd_getfd.
   • prctl(PR_SET_DUMPABLE, 1) from the runner after exec: Node has no binding for it, so it would need a native addon. Calling it before exec does nothing, because exec of an unreadable binary resets the process to nondumpable. Not pursued.

   Decision table

   Option	Restores networking	Exit key protection on openshell	Cost	Risk
   (a) A dumpable (0755) runner binary on the openshell venues only, for the supervisor AND the runner, chosen by a worker-set variable. local/podman keep 0711.	Yes (E2, E3, copy and bind)	Seccomp blocks the syscall routes everywhere; Yama blocks /proc/<pid>/mem at scope ≥ 1. Environ exposes no secret. Refuse openshell before spend when the gateway host's ptrace_scope < 1, plus a doctor line.	Small: an image binary, an entrypoint choice, a worker variable, a capability flag, the check, specs	Depends on Yama, and the check makes that explicit. The supervisor's exec-stdin kind still needs measuring (see above).
   (b) A sibling signer: the entrypoint starts a nondumpable key holder beside a dumpable supervisor and runner, so the key holder is no tool's ancestor.	Yes, in principle (no nondumpable ancestor)	The key never sits in a dumpable process. But a tool that could write the runner's memory (needs /proc/<pid>/mem, so the same Yama dependence) could still make it request signatures. It narrows exfiltration, not integrity.	Large: a new process, a signing and verify protocol over a private socket, a new frame path, its own tests and specs	A new trusted component that is easy to get subtly wrong; does not remove the Yama dependence for integrity
   (c) Leave the openshell venues out of 5.0.0 until OpenShell can observe a nondumpable workload (an upstream ask: a broker with CAP_SYS_PTRACE, or reading the sockaddr another way).	No	Unchanged	Small now (a refusal and docs), indefinite later	The venue ships as blessed but unusable unless refused by name

   Recommendation

   (a), with a hard pre-spend refusal on ptrace_scope < 1 (and a doctor line naming the sysctl). On a host with Yama at 1 or higher, a dumpable runner exposes no secret: the measured routes are closed and the environ holds only placeholders and configuration. On a host without Yama the venue refuses by name rather than running unprotected. Before shipping, measure the supervisor's exec-stdin kind. Record an OQ- entry for the upstream ask, so that (a) can be undone once OpenShell can observe a nondumpable workload. Keep (c) as the fallback if the operator will not accept a Yama dependence. (b) does not justify its size, since it keeps the same dependence for the runner's integrity.

   Phase 2 not started.
3. edgehero

   added 4 commits that reference this issue on Oct 22, 2026

   test(ci): #730 the steered exec's stdin kind and whether it reopens t…

   ...

   46ac166

   test(ci): bind parity for the OpenShell runner's networking and key g…

   ...

   480e8d6

   fix(image,worker): suppress core dumps and require Yama exactly 1 for…

   ...

   74c62bc

   test(ci): make the openshell keyguard probe print its errors, not bla…

   ...

   052f0c1