Repository navigation
fix(podman): read a network's members on Podman 4.9 (#452) - #460
Merged
Merged
Conversation
edgehero
marked this pull request as draft
September 27, 2026 20:13
This was referenced Sep 27, 2026
edgehero
force-pushed
the
fix/452-podman49-network-members
branch
from
September 27, 2026 20:51
7b27a43 to
9f9b1e6
Compare
edgehero
marked this pull request as ready for review
September 28, 2026 08:37
edgehero
force-pushed
the
fix/452-podman49-network-members
branch
from
September 28, 2026 08:37
9f9b1e6 to
7c762ec
Compare
On Podman 4.9.3, `network inspect --format {{json .Containers}}` exits
125 for every network ("can't evaluate field Containers"), so every
podman caller of networkEndpoints got "unreadable": a leftover canary
network was reported on every doctor --live and never removed, and the
boot reaper, sandbox sweep and live-probes peer sweep could not read
theirs either.
Measured on 4.9.3 and 5.8.1: `ps -a --filter network=<net> --format
{{.Names}}\t{{.State}}` lists members in every state on both (a whole
name match; an id or id prefix also matches), and `network exists`
tells absent from present. networkEndpoints uses them on podman;
docker's `.Containers` read is unchanged, and `local` through
podman-docker on 4.9 falls back to the same read on exactly that
template error. `names` keeps its meaning (running and paused members);
podman adds `parked` for every other state. A lone created or exited
member makes podman's network rm refuse, so docker's
never-startable-container hazard cannot happen there.
Reaching these sweeps on 4.9 means they can detach the egress proxy,
the #458 trigger: the boot reaper and the sandbox sweep ask
makeNetnsDetachGuard first and leave the network (keeper-not-holding)
while the rootless network keeper does not hold on Podman 4.x. The
sandbox sweep detaches a stopped proxy like the other sweeps and names
what holds a network it cannot remove.
The stale-canary sweep is exported; the conformance script builds two
leftovers the way runEgressCanary does, runs the real sweep with no
bystander, and asserts the proxy still reaches out afterwards; without
the keeper it fails on 4.9.3.
Specs: INT-EGRESS-POLICY-CONTRACT, INT-LIVE-PROBE-CONTRACT,
INT-SANDBOX-CONTRACT, DES-EGRESS-DENY-ON-A-DEDICATED-NETWORK,
DES-PODMAN-NATIVE-ROOTLESS-BACKEND.
Signed-off-by: Rob Boerman <robboerman@live.nl>
edgehero
force-pushed
the
fix/452-podman49-network-members
branch
from
September 28, 2026 09:22
7c762ec to
0f07b6e
Compare
PR #460 gate round 3 found the round-2 guards were per site, and three sites were still unguarded on a rootless Podman 4.x reached as docker (podman-docker, or the real docker CLI on the Podman API socket): a plain doctor's canary teardown and its stale sweep cut the proxy's route out (measured), the live-probe peers could do the same, and a job's and a sandbox's teardown were ungated. So the rule moved into the one helper they all share. - egress.mjs: detachEndpoints is now the only `network disconnect` in worker/src. removeJobNetworkWith and removeNetworkOrSay go through it, and it asks the gate first; a refusal detaches nothing and is returned as `blocked`. - netns-keeper.mjs (new leaf, re-exported by podman-stack.mjs): makeDetachGate asks only for a RUNNING member, reads the runtime once per pass (memoised, an unanswered read included, with the facts readers' 15 s / 1 MiB bound), lets Docker Engine, rootful Podman and 5.x through unread, and otherwise needs the keeper holding now. daemon-facts.mjs (new leaf) holds parseDaemonFacts and parsePodmanInfo, moved unchanged and re-exported, so egress.mjs can use them without an import cycle. - doctor: the canary asks the gate before building anything, on both venues, and the stale sweep says `held` when refused (a new CANARY_LINES shape, row in docs/egress.md). The run's one `docker info` is shared by the job-user section and the gate. The live probes' peers and their sweep use the same gate. - The boot reaper and the sandbox sweep build one gate per pass (a hanging read had cost 30 s per leftover network). The sandbox sweep's runner now hands on execFile's stderr (withStderr), so the podman-docker `.Containers` fallback fires there. - A teardown cannot be held, so nothing egress-armed starts where one would be refused: the podman venue's preflight and sandbox opener refuse without a holding keeper on 4.x, and `local` refuses every rootless daemon (pinned for both routes to a rootless Podman 4.9). - service uninstall names the Valkey volume only when a Valkey unit was removed. Specs: INT-EGRESS-POLICY-CONTRACT AMENDED (a new "every detach goes through one gate" paragraph, and the stated choice for teardowns), INT-SANDBOX-CONTRACT and INT-LIVE-PROBE-CONTRACT AMENDED; DES-PODMAN-NATIVE-ROOTLESS-BACKEND AMENDED (the #458 bullet states the one rule). The #452 revision rows are dated 2026-09-28 and moved to the end of each table. Checked: the worker, receiver and admin suites in CI posture pass. All 20 new mutants go red (three survived first and got tests). On pd-ubuntu (Podman 4.9.3, a throwaway account, deleted afterwards), the three measured breakers leave the proxy reaching out: plain doctor through podman-docker, plain doctor through the API socket, and the local sandbox sweep through podman-docker. Signed-off-by: Rob Boerman <robboerman@live.nl>
…#452) The round-cap re-review of e9aea31 found that every local job or sandbox teardown made a fresh `docker info` for the detach gate. When that read failed, answered ServerErrors or ran out its bound, the gate said runtime-unreadable. On Docker Engine it then asked for a keeper that never exists and refused, and the job's network was left with no line (measured). On a long-running worker that ends in exhausted subnet pools and every egress job failing. - A teardown hands the gate the runtime its job or session was admitted on, and reads only when there is none (a gate's readRuntime may now answer `undefined`, "read it yourself"): - local: the job-user resolver's cached answer (resolveJobUser.peek) - podman: the kept `podman info` (cachedPodmanInfo's peek) - sandbox: the facts its own job-user decision read, carried beside the answer and outside its shape - A refused teardown is said: `job_network_not_removed {network, reason}` on the worker's log, and a warning line from the sandbox CLI and the panel. - The boot reaper's step passes the gate's own bound (15 s, 1 MiB) through to its exec. - The conformance script's failure-path cleanup detaches through removeNetworkOrSay and a gate, like everything in worker/src. - The podman `egress-network-exists` refusal says to check and start the keeper before its manual disconnect loop, which is #458's trigger itself. docker's text is unchanged. Specs: INT-EGRESS-POLICY-CONTRACT's teardown sentence is CORRECTED (a refusal was not only a keeper that died mid-run), and it adds the admitted runtime, the logged refusal, the reaper's gate bound and the keeper-first text. DES-PODMAN-NATIVE-ROOTLESS-BACKEND gets one sentence. Both #452 rows are amended in place. Checked: the worker, receiver and admin suites in CI posture pass. All 13 new mutants go red (one survived first and got a CLI test). Signed-off-by: Rob Boerman <robboerman@live.nl>
… at open everywhere (#452) The focused check of c79cfca found four items. - A `local` job's teardown read the job-user resolver's cache. That cache is per endpoint state, so after a `docker context use` it could hold another daemon's answer. The runtime is now recorded per job id at the job's own job-user preflight, and handed to that job's teardown (teardownRuntime(job)), then forgotten. The record is bounded at 1000 entries for jobs refused before a teardown. resolveJobUser.peek is gone. - A sandbox opened on macOS, Windows or a remote endpoint decided its uid without the daemon's facts, so its teardown read `docker info` again, and a failed read kept the network (the Docker Desktop case). The facts are now read once at open on those paths too, and carried beside the answer. - The conformance cleanup's runner passes the gate's bound through. - DES-PODMAN-NATIVE-ROOTLESS-BACKEND names the kept `podman info` version as a residual, next to the kept cgroup manager: a Podman up- or downgraded under a running worker is judged by its old version until a restart. Specs: INT-EGRESS-POLICY-CONTRACT's teardown sentence names the per-job record and the sandbox read. Both #452 rows are amended in place. Checked: the worker, receiver and admin suites in CI posture pass. All 7 new mutants go red. Signed-off-by: Rob Boerman <robboerman@live.nl>
edgehero
added a commit
that referenced
this pull request
Sep 28, 2026
PR #460 gate round 3 found the round-2 guards were per site, and three sites were still unguarded on a rootless Podman 4.x reached as docker (podman-docker, or the real docker CLI on the Podman API socket): a plain doctor's canary teardown and its stale sweep cut the proxy's route out (measured), the live-probe peers could do the same, and a job's and a sandbox's teardown were ungated. So the rule moved into the one helper they all share. - egress.mjs: detachEndpoints is now the only `network disconnect` in worker/src. removeJobNetworkWith and removeNetworkOrSay go through it, and it asks the gate first; a refusal detaches nothing and is returned as `blocked`. - netns-keeper.mjs (new leaf, re-exported by podman-stack.mjs): makeDetachGate asks only for a RUNNING member, reads the runtime once per pass (memoised, an unanswered read included, with the facts readers' 15 s / 1 MiB bound), lets Docker Engine, rootful Podman and 5.x through unread, and otherwise needs the keeper holding now. daemon-facts.mjs (new leaf) holds parseDaemonFacts and parsePodmanInfo, moved unchanged and re-exported, so egress.mjs can use them without an import cycle. - doctor: the canary asks the gate before building anything, on both venues, and the stale sweep says `held` when refused (a new CANARY_LINES shape, row in docs/egress.md). The run's one `docker info` is shared by the job-user section and the gate. The live probes' peers and their sweep use the same gate. - The boot reaper and the sandbox sweep build one gate per pass (a hanging read had cost 30 s per leftover network). The sandbox sweep's runner now hands on execFile's stderr (withStderr), so the podman-docker `.Containers` fallback fires there. - A teardown cannot be held, so nothing egress-armed starts where one would be refused: the podman venue's preflight and sandbox opener refuse without a holding keeper on 4.x, and `local` refuses every rootless daemon (pinned for both routes to a rootless Podman 4.9). - service uninstall names the Valkey volume only when a Valkey unit was removed. Specs: INT-EGRESS-POLICY-CONTRACT AMENDED (a new "every detach goes through one gate" paragraph, and the stated choice for teardowns), INT-SANDBOX-CONTRACT and INT-LIVE-PROBE-CONTRACT AMENDED; DES-PODMAN-NATIVE-ROOTLESS-BACKEND AMENDED (the #458 bullet states the one rule). The #452 revision rows are dated 2026-09-28 and moved to the end of each table. Checked: the worker, receiver and admin suites in CI posture pass. All 20 new mutants go red (three survived first and got tests). On pd-ubuntu (Podman 4.9.3, a throwaway account, deleted afterwards), the three measured breakers leave the proxy reaching out: plain doctor through podman-docker, plain doctor through the API socket, and the local sandbox sweep through podman-docker. Signed-off-by: Rob Boerman <robboerman@live.nl>
This was referenced Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #452. (Rebased onto the keeper, #463.)
What was wrong
On Podman 4.9.3,
podman network inspect --format '{{json .Containers}}'exits 125 for every existing network (can't evaluate field Containers in type interface {}). Every podman caller ofnetworkEndpointstherefore saw "unreadable":kill -9eddoctor --livecanary was reported "could not be read" on every run and never removed.Measured (4.9.3 and 5.8.1)
network inspect --format {{json .Containers}}ps -a --filter network=N --format {{.Names}}\t{{.State}}ps -aon a missing networknetwork existsThe filter is an exact name match (a sibling
N2was not listed).Change
networkEndpointsreads members withps -aand tells absent from unreadable withnetwork exists. A line that is not a legal name, a tab and a lowercase state word means unreadable. Docker's.Containersread is unchanged.nameskeeps its meaning on both runtimes (running and paused members); podman addsparkedfor every other state.INT-EGRESS-POLICY-CONTRACT: a dead run's probes are removed in any state, and other stopped members are detached.runEgressCanarydoes and runs the real sweep with no bystander, asserting both are gone and the proxy still reaches out. Without the keeper, or with the podman read disabled, the case goes red on 4.9.3.detachEndpoints, which asks one gate (makeDetachGate,worker/src/netns-keeper.mjs) before it detaches anything that is running:keeper-not-holding), and so is one on a runtime that cannot be read (runtime-unreadable).docker infoat teardown never leaves a job network behind on Docker. Every refused teardown is logged.localthrough podman-docker on 4.9 falls back to the same member read, and the sandbox sweep passes the CLI's stderr through so the fallback fires there too.service uninstallnames the Valkey volume only when a Valkey unit was removed.Specs
INT-EGRESS-POLICY-CONTRACT,INT-LIVE-PROBE-CONTRACT,INT-SANDBOX-CONTRACT,DES-PODMAN-NATIVE-ROOTLESS-BACKEND,DES-EGRESS-DENY-ON-A-DEDICATED-NETWORK;docs/podman.mdstep 8,docs/egress.md.