Skip to content

fix(podman): read a network's members on Podman 4.9 (#452) - #460

Merged
edgehero merged 4 commits into
mainfrom
fix/452-podman49-network-members
Sep 28, 2026
Merged

edgehero merged 4 commits into
mainfrom
fix/452-podman49-network-members

Conversation

@edgehero

@edgehero edgehero commented Sep 27, 2026 •

Copy link
Copy Markdown
Owner

Closes #452. (Rebased onto the keeper, #463.)

What was wrong

On Podman 4.9.3, podman network inspect --format '{{json .Containers}}' exits 125 for every existing network (can't evaluate field Containers in type interface {}). Every podman caller of networkEndpoints therefore saw "unreadable":

  • A network left by a kill -9ed doctor --live canary was reported "could not be read" on every run and never removed.
  • The boot reaper, the sandbox network sweep and the live-probes peer sweep use the same read.

Measured (4.9.3 and 5.8.1)

command 4.9.3 5.8.1
network inspect --format {{json .Containers}} exit 125, every network exit 0, running and paused only
ps -a --filter network=N --format {{.Names}}\t{{.State}} every member, every state identical
ps -a on a missing network exit 0, empty same
network exists 0 present, 1 absent same

The filter is an exact name match (a sibling N2 was not listed).

Change

  • On podman, networkEndpoints reads members with ps -a and tells absent from unreadable with network exists. A line that is not a legal name, a tab and a lowercase state word means unreadable. Docker's .Containers read is unchanged.
  • names keeps its meaning on both runtimes (running and paused members); podman adds parked for every other state.
  • Each caller's rule is stated in INT-EGRESS-POLICY-CONTRACT: a dead run's probes are removed in any state, and other stopped members are detached.
  • The stale-canary sweep is exported. The conformance script builds two leftovers the way runEgressCanary does and runs the real sweep with no bystander, asserting both are gone and the proxy still reaches out. Without the keeper, or with the podman read disabled, the case goes red on 4.9.3.
  • Reaching these sweeps on 4.9 means they can detach the egress proxy, the podman 4.9: a job network teardown cuts the egress proxy off, so every later egress job gets 503 #458 trigger. So every network detach in the worker now goes through one helper, detachEndpoints, which asks one gate (makeDetachGate, worker/src/netns-keeper.mjs) before it detaches anything that is running:
    • Docker Engine, rootful Podman and Podman 5.x pass without a keeper read.
    • Rootless Podman 4.x, reached directly, through podman-docker or through its Docker API, needs the netns keeper holding right now; otherwise the network is left (keeper-not-holding), and so is one on a runtime that cannot be read (runtime-unreadable).
    • The runtime is read once per pass, bounded at 15 s / 1 MiB. A job teardown uses the runtime the job was admitted on, recorded per job, and a sandbox the runtime read at open, so a slow docker info at teardown never leaves a job network behind on Docker. Every refused teardown is logged.
    • The callers are job and sandbox teardown, the boot reaper, the sandbox sweep, and doctor's canary, stale-canary sweep and live-probe peers, on both venues. Opening an egress-armed sandbox on Podman 4.x refuses while the keeper does not hold.
  • local through podman-docker on 4.9 falls back to the same member read, and the sandbox sweep passes the CLI's stderr through so the fallback fires there too.
  • The boot reaper's steps are bounded (30 s each, 15 s for the runtime read).
  • service uninstall names the Valkey volume only when a Valkey unit was removed.

Specs

INT-EGRESS-POLICY-CONTRACT, INT-LIVE-PROBE-CONTRACT, INT-SANDBOX-CONTRACT, DES-PODMAN-NATIVE-ROOTLESS-BACKEND, DES-EGRESS-DENY-ON-A-DEDICATED-NETWORK; docs/podman.md step 8, docs/egress.md.

On Podman 4.9.3, `network inspect --format {{json .Containers}}` exits
125 for every network ("can't evaluate field Containers"), so every
podman caller of networkEndpoints got "unreadable": a leftover canary
network was reported on every doctor --live and never removed, and the
boot reaper, sandbox sweep and live-probes peer sweep could not read
theirs either.

Measured on 4.9.3 and 5.8.1: `ps -a --filter network=<net> --format
{{.Names}}\t{{.State}}` lists members in every state on both (a whole
name match; an id or id prefix also matches), and `network exists`
tells absent from present. networkEndpoints uses them on podman;
docker's `.Containers` read is unchanged, and `local` through
podman-docker on 4.9 falls back to the same read on exactly that
template error. `names` keeps its meaning (running and paused members);
podman adds `parked` for every other state. A lone created or exited
member makes podman's network rm refuse, so docker's
never-startable-container hazard cannot happen there.

Reaching these sweeps on 4.9 means they can detach the egress proxy,
the #458 trigger: the boot reaper and the sandbox sweep ask
makeNetnsDetachGuard first and leave the network (keeper-not-holding)
while the rootless network keeper does not hold on Podman 4.x. The
sandbox sweep detaches a stopped proxy like the other sweeps and names
what holds a network it cannot remove.

The stale-canary sweep is exported; the conformance script builds two
leftovers the way runEgressCanary does, runs the real sweep with no
bystander, and asserts the proxy still reaches out afterwards; without
the keeper it fails on 4.9.3.

Specs: INT-EGRESS-POLICY-CONTRACT, INT-LIVE-PROBE-CONTRACT,
INT-SANDBOX-CONTRACT, DES-EGRESS-DENY-ON-A-DEDICATED-NETWORK,
DES-PODMAN-NATIVE-ROOTLESS-BACKEND.

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
edgehero force-pushed the fix/452-podman49-network-members branch from 7c762ec to 0f07b6e Compare September 28, 2026 09:22
PR #460 gate round 3 found the round-2 guards were per site, and three
sites were still unguarded on a rootless Podman 4.x reached as docker
(podman-docker, or the real docker CLI on the Podman API socket): a
plain doctor's canary teardown and its stale sweep cut the proxy's
route out (measured), the live-probe peers could do the same, and a
job's and a sandbox's teardown were ungated. So the rule moved into
the one helper they all share.

- egress.mjs: detachEndpoints is now the only `network disconnect` in
  worker/src. removeJobNetworkWith and removeNetworkOrSay go through
  it, and it asks the gate first; a refusal detaches nothing and is
  returned as `blocked`.
- netns-keeper.mjs (new leaf, re-exported by podman-stack.mjs):
  makeDetachGate asks only for a RUNNING member, reads the runtime
  once per pass (memoised, an unanswered read included, with the facts
  readers' 15 s / 1 MiB bound), lets Docker Engine, rootful Podman and
  5.x through unread, and otherwise needs the keeper holding now.
  daemon-facts.mjs (new leaf) holds parseDaemonFacts and
  parsePodmanInfo, moved unchanged and re-exported, so egress.mjs can
  use them without an import cycle.
- doctor: the canary asks the gate before building anything, on both
  venues, and the stale sweep says `held` when refused (a new
  CANARY_LINES shape, row in docs/egress.md). The run's one
  `docker info` is shared by the job-user section and the gate. The
  live probes' peers and their sweep use the same gate.
- The boot reaper and the sandbox sweep build one gate per pass (a
  hanging read had cost 30 s per leftover network). The sandbox
  sweep's runner now hands on execFile's stderr (withStderr), so the
  podman-docker `.Containers` fallback fires there.
- A teardown cannot be held, so nothing egress-armed starts where one
  would be refused: the podman venue's preflight and sandbox opener
  refuse without a holding keeper on 4.x, and `local` refuses every
  rootless daemon (pinned for both routes to a rootless Podman 4.9).
- service uninstall names the Valkey volume only when a Valkey unit
  was removed.

Specs: INT-EGRESS-POLICY-CONTRACT AMENDED (a new "every detach goes
through one gate" paragraph, and the stated choice for teardowns),
INT-SANDBOX-CONTRACT and INT-LIVE-PROBE-CONTRACT AMENDED;
DES-PODMAN-NATIVE-ROOTLESS-BACKEND AMENDED (the #458 bullet states the
one rule). The #452 revision rows are dated 2026-09-28 and moved to the
end of each table.

Checked: the worker, receiver and admin suites in CI posture pass. All
20 new mutants go red (three survived first and got tests). On pd-ubuntu (Podman 4.9.3, a throwaway
account, deleted afterwards), the three measured breakers leave the
proxy reaching out: plain doctor through podman-docker, plain doctor
through the API socket, and the local sandbox sweep through
podman-docker.

Signed-off-by: Rob Boerman <robboerman@live.nl>
…#452)

The round-cap re-review of e9aea31 found that every local job or
sandbox teardown made a fresh `docker info` for the detach gate. When
that read failed, answered ServerErrors or ran out its bound, the gate
said runtime-unreadable. On Docker Engine it then asked for a keeper
that never exists and refused, and the job's network was left with no
line (measured). On a long-running worker that ends in exhausted
subnet pools and every egress job failing.

- A teardown hands the gate the runtime its job or session was
  admitted on, and reads only when there is none (a gate's
  readRuntime may now answer `undefined`, "read it yourself"):
  - local: the job-user resolver's cached answer (resolveJobUser.peek)
  - podman: the kept `podman info` (cachedPodmanInfo's peek)
  - sandbox: the facts its own job-user decision read, carried beside
    the answer and outside its shape
- A refused teardown is said: `job_network_not_removed {network,
  reason}` on the worker's log, and a warning line from the sandbox CLI
  and the panel.
- The boot reaper's step passes the gate's own bound (15 s, 1 MiB)
  through to its exec.
- The conformance script's failure-path cleanup detaches through
  removeNetworkOrSay and a gate, like everything in worker/src.
- The podman `egress-network-exists` refusal says to check and start
  the keeper before its manual disconnect loop, which is #458's trigger
  itself. docker's text is unchanged.

Specs: INT-EGRESS-POLICY-CONTRACT's teardown sentence is CORRECTED (a
refusal was not only a keeper that died mid-run), and it adds the
admitted runtime, the logged refusal, the reaper's gate bound and the
keeper-first text. DES-PODMAN-NATIVE-ROOTLESS-BACKEND gets one
sentence. Both #452 rows are amended in place.

Checked: the worker, receiver and admin suites in CI posture pass. All
13 new mutants go red (one survived first and got a CLI test).

Signed-off-by: Rob Boerman <robboerman@live.nl>
… at open everywhere (#452)

The focused check of c79cfca found four items.

- A `local` job's teardown read the job-user resolver's cache. That
  cache is per endpoint state, so after a `docker context use` it
  could hold another daemon's answer. The runtime is now recorded per
  job id at the job's own job-user preflight, and handed to that
  job's teardown (teardownRuntime(job)), then forgotten. The record is
  bounded at 1000 entries for jobs refused before a teardown.
  resolveJobUser.peek is gone.
- A sandbox opened on macOS, Windows or a remote endpoint decided its
  uid without the daemon's facts, so its teardown read `docker info`
  again, and a failed read kept the network (the Docker Desktop case).
  The facts are now read once at open on those paths too, and carried
  beside the answer.
- The conformance cleanup's runner passes the gate's bound through.
- DES-PODMAN-NATIVE-ROOTLESS-BACKEND names the kept `podman info`
  version as a residual, next to the kept cgroup manager: a Podman up-
  or downgraded under a running worker is judged by its old version
  until a restart.

Specs: INT-EGRESS-POLICY-CONTRACT's teardown sentence names the
per-job record and the sandbox read. Both #452 rows are amended in
place.

Checked: the worker, receiver and admin suites in CI posture pass. All
7 new mutants go red.

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
edgehero merged commit 62a7a57 into main Sep 28, 2026
9 checks passed
edgehero added a commit that referenced this pull request Sep 28, 2026
PR #460 gate round 3 found the round-2 guards were per site, and three
sites were still unguarded on a rootless Podman 4.x reached as docker
(podman-docker, or the real docker CLI on the Podman API socket): a
plain doctor's canary teardown and its stale sweep cut the proxy's
route out (measured), the live-probe peers could do the same, and a
job's and a sandbox's teardown were ungated. So the rule moved into
the one helper they all share.

- egress.mjs: detachEndpoints is now the only `network disconnect` in
  worker/src. removeJobNetworkWith and removeNetworkOrSay go through
  it, and it asks the gate first; a refusal detaches nothing and is
  returned as `blocked`.
- netns-keeper.mjs (new leaf, re-exported by podman-stack.mjs):
  makeDetachGate asks only for a RUNNING member, reads the runtime
  once per pass (memoised, an unanswered read included, with the facts
  readers' 15 s / 1 MiB bound), lets Docker Engine, rootful Podman and
  5.x through unread, and otherwise needs the keeper holding now.
  daemon-facts.mjs (new leaf) holds parseDaemonFacts and
  parsePodmanInfo, moved unchanged and re-exported, so egress.mjs can
  use them without an import cycle.
- doctor: the canary asks the gate before building anything, on both
  venues, and the stale sweep says `held` when refused (a new
  CANARY_LINES shape, row in docs/egress.md). The run's one
  `docker info` is shared by the job-user section and the gate. The
  live probes' peers and their sweep use the same gate.
- The boot reaper and the sandbox sweep build one gate per pass (a
  hanging read had cost 30 s per leftover network). The sandbox
  sweep's runner now hands on execFile's stderr (withStderr), so the
  podman-docker `.Containers` fallback fires there.
- A teardown cannot be held, so nothing egress-armed starts where one
  would be refused: the podman venue's preflight and sandbox opener
  refuse without a holding keeper on 4.x, and `local` refuses every
  rootless daemon (pinned for both routes to a rootless Podman 4.9).
- service uninstall names the Valkey volume only when a Valkey unit
  was removed.

Specs: INT-EGRESS-POLICY-CONTRACT AMENDED (a new "every detach goes
through one gate" paragraph, and the stated choice for teardowns),
INT-SANDBOX-CONTRACT and INT-LIVE-PROBE-CONTRACT AMENDED;
DES-PODMAN-NATIVE-ROOTLESS-BACKEND AMENDED (the #458 bullet states the
one rule). The #452 revision rows are dated 2026-09-28 and moved to the
end of each table.

Checked: the worker, receiver and admin suites in CI posture pass. All
20 new mutants go red (three survived first and got tests). On pd-ubuntu (Podman 4.9.3, a throwaway
account, deleted afterwards), the three measured breakers leave the
proxy reaching out: plain doctor through podman-docker, plain doctor
through the API socket, and the local sandbox sweep through
podman-docker.

Signed-off-by: Rob Boerman <robboerman@live.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

measure the stale canary network member read on Podman 4.9

1 participant