Repository navigation
fix(podman): keep the rootless network alive across a job teardown on Podman 4.9 (#458) - #463
Merged
Merged
Conversation
edgehero
force-pushed
the
fix/458-podman49-netns-keeper
branch
5 times, most recently
from
September 28, 2026 07:49
8577cb0 to
9329ee2
Compare
… Podman 4.9 (#458) On Podman 4.9.3 (Ubuntu 24.04's), `network disconnect` of the egress proxy from a per-job network tears down the account's shared rootless network namespace and its slirp4netns while the proxy still runs, whenever no other running container is on a bridge network: v4.9.3's cleanup counts the disconnecting container as the caller. The next bridge container starts a new namespace, the proxy loses eth0, and every later egress job gets 503 (measured: 200, then 503 four times). Podman 5's network rewrite counts attachments; 5.8.1 is unaffected. A keeper container holds the helper open: one idle `sleep` on a network of its own (internal, DNS disabled), no port, no mount, no capability, read-only, uid 65534, the proxy's own pinned image, RunInit, restarted after 1 s with no start limit, its network recreated by ExecStartPre after a prune. It ships as a Quadlet pair installed wherever the proxy's units are (service install, up), on every Podman version. It counts only when running in bridge mode on its own network, up for 3 s and not started more than 15 s after the proxy (a keeper that restarted under a running proxy may have let a teardown break it). On Podman below 5 with egress armed, the worker's pre-spend preflight reads it after the proxy and retries the job before anything is reserved when it does not hold (with its own reason and terminal comment, and a boot-time warning); doctor fails on it and does not run its canary or peer networks without it; up restarts the proxy after a keeper restart; doctor also checks the keeper's network is internal with DNS off. service install and up refuse when the user manager's XDG_RUNTIME_DIR or XDG_CONFIG_HOME is not the account's own (the generator and every unit inherit them). The conformance job pins both for its account before the manager starts, starts the keeper from the shipped units, and runs two egress jobs through the real teardown; without the keeper it fails. Specs: DES-PODMAN-STACK-AS-QUADLET-UNITS, DES-PODMAN-NATIVE-ROOTLESS-BACKEND, DES-EGRESS-DENY-ON-A-DEDICATED-NETWORK, INT-EGRESS-POLICY-CONTRACT, INT-LIVE-PROBE-CONTRACT, INT-RUN-HISTORY-FILE-CONTRACT, REQ-DEPLOYMENT-BOOTSTRAP. Signed-off-by: Rob Boerman <robboerman@live.nl>
edgehero
force-pushed
the
fix/458-podman49-netns-keeper
branch
from
September 28, 2026 08:05
9329ee2 to
0c151da
Compare
This was referenced Sep 28, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #458. #460 (#452) must merge after this.
The defect
On Podman 4.9.3, Ubuntu 24.04's packaged version and the one the required conformance job runs,
podman network disconnectof the egress proxy from a per-job network tears down the account's shared rootless network helper while the proxy still runs. It happens whenever no other running container is on a bridge network. Every later egress job then gets 503 until the proxy restarts. Measured with the repo's owncreateJobNetwork/removeJobNetwork: 200, then 503 four times.libpod/networking_linux.gocounts the disconnecting container as the caller.The fix: a keeper
One idle container that keeps the account's rootless network helper alive:
sleep infinityon a network of its own (--internal, DNS disabled);It ships as a Quadlet pair (
deploy/pi-dispatch-netns-keeper.{container,network}, mirrored intoworker/deploy/). It is installed wherever the proxy's units are (service install,up), on every Podman version: harmless on 5.x, and simpler than a version read.Restart=alwaysbrings it back after an out-of-band stop.SuccessExitStatus=143keeps a clean stop from reading as failed.doctor and CI
doctorchecks the keeper when egress is armed: ✗ on Podman 4.x (or an unreported version) when it is not running.doctor --livedoes not run its egress canary, its stale-canary sweep or its peer networks, because their own teardown is the trigger. It says why.docs/podman.mdstep 6 documents the defect, the keeper, and the aardvark-dns repair for an account already damaged (upstream #20396).Specs
DES-PODMAN-STACK-AS-QUADLET-UNITS,DES-PODMAN-NATIVE-ROOTLESS-BACKEND,DES-EGRESS-DENY-ON-A-DEDICATED-NETWORK,INT-EGRESS-POLICY-CONTRACT,INT-LIVE-PROBE-CONTRACT.