Repository navigation
fix(size): offer the apply call exactly when admission would accept t… #468
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| # The native `podman` venue's conformance run (issue #354), on a real rootless Podman. | |
| # | |
| # `.github/scripts/podman-conformance.mjs` builds the REAL `podman` bundle and runs the backend conformance harness | |
| # against it, reading the eight container properties back off live containers and failing on any it could not read. | |
| # It has to run as an unprivileged account with its own rootless Podman, which a runner's default user is not set up | |
| # to be, so this job makes one: `pdjob`, uid 1234 (the uid the Fedora lab measured with, and not the image's own | |
| # 1001, so keep-id is what makes the job's files readable), with subordinate ids, linger and its systemd user | |
| # instance, the job image loaded into ITS store and the egress proxy running under ITS Podman on a named bridge | |
| # network, exactly as docs/podman.md tells an operator to set one up, with the rootless network keeper beside it | |
| # (issue #458) started from its shipped Quadlet unit. | |
| # | |
| # REQUIRED on main since issue #432, the sixth required context and the only one outside pi-upgrade-check.yml | |
| # (CLAUDE.md lists all six). It ran advisory first, because Ubuntu 24.04 ships Podman 4.9, one major behind the 5.8.1 | |
| # the venue was measured on, and was promoted after it had passed on every PR and main push of the podman follow-up | |
| # round (#428 to #437): 38 of 39 runs green, the one red a real defect on the branch that built the venue. A red run | |
| # here is still a finding to read before it is a defect to fix, but it now blocks the merge. The pull_request | |
| # trigger is UNFILTERED for pi-upgrade-check.yml's reason, and must stay so: a required check that never reports | |
| # blocks a merge forever. | |
| # Renaming the job below renames the required context, so a rename must be paired with a branch-protection update. | |
| name: podman conformance | |
| on: | |
| push: | |
| branches: [main] | |
| paths: | |
| - "worker/src/**" | |
| - "image/**" | |
| - "deploy/egress-proxy.conf" | |
| - "deploy/pi-dispatch-netns-keeper.container" | |
| - "deploy/pi-dispatch-netns-keeper.network" | |
| - ".github/scripts/podman-conformance.mjs" | |
| - ".github/workflows/podman-conformance.yml" | |
| pull_request: | |
| workflow_dispatch: | |
| concurrency: | |
| group: ${{ github.workflow }}-${{ github.ref }} | |
| cancel-in-progress: true | |
| permissions: | |
| contents: read | |
| jobs: | |
| conformance: | |
| name: the rootless podman venue holds its declarations | |
| runs-on: ubuntu-24.04 | |
| timeout-minutes: 40 | |
| env: | |
| PDJOB: pdjob | |
| PDJOB_UID: "1234" | |
| steps: | |
| - uses: actions/checkout@v4 | |
| - uses: actions/setup-node@v4 | |
| with: | |
| node-version-file: ".nvmrc" | |
| - run: npm ci --no-audit --no-fund | |
| - name: Build the job image | |
| run: docker build -f image/Dockerfile -t pi-job:ci . | |
| # netavark and aardvark-dns give a rootless named network its DNS, which is how a job reaches the proxy by name; | |
| # catatonit is what `--init` runs as PID 1; uidmap carries newuidmap. Installed explicitly rather than assumed | |
| # from the runner image, so a missing one fails here rather than as a confusing verdict later. | |
| - name: Install rootless Podman's parts | |
| run: | | |
| sudo apt-get update | |
| sudo apt-get install -y podman uidmap netavark aardvark-dns catatonit slirp4netns | |
| podman --version | |
| # Ubuntu 24.04 forbids unprivileged user namespaces through AppArmor by default, and rootless Podman is one. | |
| - name: Allow unprivileged user namespaces | |
| run: sudo sysctl -w kernel.apparmor_restrict_unprivileged_userns=0 | |
| - name: Create the worker account, with subordinate ids, linger and delegated controllers | |
| run: | | |
| sudo useradd --create-home --uid "$PDJOB_UID" --user-group --shell /bin/bash "$PDJOB" | |
| if ! grep -q "^$PDJOB:" /etc/subuid; then sudo usermod --add-subuids 200000-265535 "$PDJOB"; fi | |
| if ! grep -q "^$PDJOB:" /etc/subgid; then sudo usermod --add-subgids 200000-265535 "$PDJOB"; fi | |
| grep "^$PDJOB:" /etc/subuid /etc/subgid | |
| # The venue credits `isolation` only while cpu, memory and pids are delegated; docs/podman.md's drop-in. | |
| sudo mkdir -p /etc/systemd/system/user@.service.d | |
| printf '[Service]\nDelegate=cpu cpuset io memory pids\n' | sudo tee /etc/systemd/system/user@.service.d/delegate.conf | |
| sudo systemctl daemon-reload | |
| # Issue #458 (PR #463 round 2): this runner's image writes XDG_CONFIG_HOME into /etc/environment | |
| # (actions/runner-images, configure-environment.sh), and on Ubuntu /etc/environment is also environment.d's | |
| # 99-environment.conf, which every user manager reads, its generators included. Measured on pd-ubuntu with | |
| # Podman 4.9.3: a user manager whose environment carries ANOTHER home's XDG_CONFIG_HOME makes the Quadlet | |
| # generator look there, and the keeper's unit is "not found" (exit 5), which is exactly what this job met on | |
| # runs 36348645154 and 36383743068. An operator's account resolves it to its own home; this one is pinned to | |
| # its own in its own environment.d (read after /etc/environment, so it wins), before its manager first starts. | |
| # PR #463 round 3 (run 36387381332): the image writes the runner's XDG_RUNTIME_DIR (/run/user/1001) there too, and | |
| # with it every podman command in the account's units failed "XDG_RUNTIME_DIR directory \"/run/user/1001\" is not | |
| # owned by the current user" (reproduced on pd-ubuntu). Both are pinned to the account's own; the runner's PATH | |
| # and tool variables also arrive, and are left: units call podman by absolute path, and none of them is read. | |
| grep -n '^XDG_' /etc/environment || echo "no XDG_ variable in /etc/environment" | |
| sudo -u "$PDJOB" mkdir -p "/home/$PDJOB/.config/environment.d" | |
| printf 'XDG_CONFIG_HOME=/home/%s/.config\nXDG_RUNTIME_DIR=/run/user/%s\n' "$PDJOB" "$PDJOB_UID" | sudo -u "$PDJOB" tee "/home/$PDJOB/.config/environment.d/zz-pi-dispatch-ci.conf" | |
| sudo loginctl enable-linger "$PDJOB" | |
| sudo systemctl start "user@$PDJOB_UID.service" | |
| cat "/sys/fs/cgroup/user.slice/user-$PDJOB_UID.slice/user@$PDJOB_UID.service/cgroup.controllers" | |
| # Every later step runs as the account through this one wrapper: its runtime directory and its user bus, so | |
| # Podman uses the systemd cgroup manager in the account's own delegated tree, as it would under a unit with | |
| # `User=` and linger. The checkout is copied into the account's home, since the runner's home is not readable | |
| # to other users; node_modules comes with it. The `cd` happens AS the account, inside the sudo, because the | |
| # runner's own user cannot enter a home directory created 0750. | |
| - name: Hand the checkout to the worker account | |
| # `env -i`: the runner's own XDG_CONFIG_HOME survived `sudo -H` and podman, as pdjob, then read the runner's | |
| # containers config (`stat /home/runner/.config/containers/storage.conf: permission denied`). Every step that | |
| # needs a variable passes it explicitly through `env NAME=...` after the wrapper. | |
| run: | | |
| sudo cp -a "$GITHUB_WORKSPACE" "/home/$PDJOB/repo" | |
| sudo chown -R "$PDJOB:" "/home/$PDJOB/repo" | |
| nodedir="$(dirname "$(command -v node)")" | |
| sudo tee /usr/local/bin/as-pdjob > /dev/null <<EOF | |
| #!/bin/sh | |
| exec sudo -u $PDJOB -H env -i HOME=/home/$PDJOB USER=$PDJOB LOGNAME=$PDJOB XDG_CONFIG_HOME=/home/$PDJOB/.config XDG_RUNTIME_DIR=/run/user/$PDJOB_UID DBUS_SESSION_BUS_ADDRESS=unix:path=/run/user/$PDJOB_UID/bus PATH=$nodedir:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin sh -c 'cd /home/$PDJOB/repo && exec "\$@"' as-pdjob "\$@" | |
| EOF | |
| sudo chmod 0755 /usr/local/bin/as-pdjob | |
| cat /usr/local/bin/as-pdjob | |
| as-pdjob podman info --format json | |
| - name: Load the job image into the account's own store | |
| run: | | |
| docker save pi-job:ci | as-pdjob podman load | |
| # The name this store gives it, which is what the worker account's PI_JOB_IMAGE has to say. | |
| name="$(as-pdjob podman image inspect --format '{{index .RepoTags 0}}' pi-job:ci)" | |
| echo "loaded as $name" | |
| echo "PI_JOB_IMAGE=$name" >> "$GITHUB_ENV" | |
| # Issue #458: on Podman 4.9, which this runner ships, a job network's teardown disconnects the proxy and, with no | |
| # other bridge container running, tears this account's rootless network helper down under it: every egress job | |
| # after the first then gets 503. The keeper is what prevents that, so it runs here as it runs on a real host, from | |
| # the SHIPPED Quadlet files through the account's own user manager (the generator, the unit and all), rather than | |
| # a hand-typed copy of them. The harness's teardown case goes red without it. BEFORE the proxy, as docs/podman.md | |
| # orders it: a keeper that starts well after the proxy reads to the worker and doctor as one that restarted under it | |
| # (PR #463 round 2), which asks for a proxy restart. | |
| - name: Start the rootless network keeper from its shipped Quadlet unit | |
| run: | | |
| # On any failure below, say why before the step ends: what the generator saw and made, and what it logged. | |
| diagnose() { | |
| echo "::group::keeper diagnostics" | |
| echo "--- /etc/environment"; grep -n 'XDG' /etc/environment || true | |
| echo "--- the account's user manager environment"; as-pdjob systemctl --user show-environment || true | |
| echo "--- the Quadlet generator"; ls -la /usr/lib/systemd/user-generators/ || true | |
| echo "--- the unit files"; as-pdjob ls -la "/home/$PDJOB/.config/containers/systemd" || true | |
| echo "--- what the generator made"; sudo ls -la "/run/user/$PDJOB_UID/systemd/generator" || true | |
| echo "--- quadlet -dryrun over the same files"; as-pdjob sh -c 'QUADLET_UNIT_DIRS="$HOME/.config/containers/systemd" /usr/libexec/podman/quadlet -dryrun -user' || true | |
| echo "--- the units"; as-pdjob systemctl --user status pi-dispatch-netns-keeper-network.service pi-dispatch-netns-keeper.service --no-pager || true | |
| echo "--- the account's journal"; sudo journalctl _UID="$PDJOB_UID" --no-pager -n 80 -o cat | grep -iE 'quadlet|generator|netns-keeper' || true | |
| echo "::endgroup::" | |
| } | |
| trap diagnose ERR | |
| # The generator reads the MANAGER's XDG_CONFIG_HOME, and every unit runs with its XDG_RUNTIME_DIR: another | |
| # account's of either makes the units not found or failing (see the account step). `service install` and `up` | |
| # refuse on the same rule (managerEnvRefusal); here it is a loud failure before anything is started. | |
| managerenv="$(as-pdjob systemctl --user show-environment)" | |
| xdg="$(printf '%s\n' "$managerenv" | sed -n 's/^XDG_CONFIG_HOME=//p')" | |
| run="$(printf '%s\n' "$managerenv" | sed -n 's/^XDG_RUNTIME_DIR=//p')" | |
| bus="$(printf '%s\n' "$managerenv" | sed -n 's/^DBUS_SESSION_BUS_ADDRESS=//p')" | |
| echo "the account's user manager has XDG_CONFIG_HOME=${xdg:-(unset)} XDG_RUNTIME_DIR=${run:-(unset)} DBUS_SESSION_BUS_ADDRESS=${bus:-(unset)}" | |
| if [ -n "$xdg" ] && [ "$xdg" != "/home/$PDJOB/.config" ]; then echo "::error::the user manager's XDG_CONFIG_HOME is not this account's own"; false; fi | |
| if [ -n "$run" ] && [ "$run" != "/run/user/$PDJOB_UID" ]; then echo "::error::the user manager's XDG_RUNTIME_DIR is not this account's own"; false; fi | |
| if [ -n "$bus" ] && [ "$bus" != "unix:path=/run/user/$PDJOB_UID/bus" ]; then echo "::error::the user manager's DBUS_SESSION_BUS_ADDRESS is not this account's own"; false; fi | |
| as-pdjob mkdir -p "/home/$PDJOB/.config/containers/systemd" | |
| as-pdjob cp deploy/pi-dispatch-netns-keeper.network deploy/pi-dispatch-netns-keeper.container "/home/$PDJOB/.config/containers/systemd/" | |
| as-pdjob systemctl --user daemon-reload | |
| as-pdjob systemctl --user start pi-dispatch-netns-keeper.service | |
| # Running, in bridge mode, on its own network: the shape judgeNetnsKeeper counts (a keeper on any other network | |
| # mode runs and holds nothing open, measured). | |
| state="$(as-pdjob podman inspect --format '{{.State.Status}}|{{.HostConfig.NetworkMode}}|{{range $k, $v := .NetworkSettings.Networks}}{{$k}},{{end}}' pi-dispatch-netns-keeper)" | |
| echo "the keeper is $state" | |
| test "$state" = "running|bridge|pi-dispatch-netns-keeper," | |
| - name: Start the egress proxy under the account's Podman, on a named bridge network | |
| run: | | |
| as-pdjob mkdir -p "/home/$PDJOB/deploy" | |
| as-pdjob sh -c "cd /home/$PDJOB/deploy && node /home/$PDJOB/repo/worker/src/cli.mjs init" | |
| image="$(sed -n 's/^ *image: *\(ubuntu\/squid@sha256:[0-9a-f]\{64\}\) *$/\1/p' deploy/docker-compose.yml)" | |
| test -n "$image" | |
| as-pdjob podman network create pi-dispatch-egress-out | |
| as-pdjob podman run -d --name pi-dispatch-egress-proxy --network pi-dispatch-egress-out \ | |
| -v "/home/$PDJOB/repo/deploy/egress-proxy.conf:/etc/squid/squid.conf:ro,z" \ | |
| -v "/home/$PDJOB/deploy/egress-allowlist.conf:/etc/pi-dispatch/allowlist.conf:ro,z" \ | |
| -v "/home/$PDJOB/deploy/model-endpoints.conf:/etc/pi-dispatch/model-endpoints.conf:ro,z" \ | |
| "docker.io/$image" | |
| for i in $(seq 1 60); do | |
| if as-pdjob podman exec pi-dispatch-egress-proxy bash -c 'exec 3<>/dev/tcp/127.0.0.1/3128' 2>/dev/null; then | |
| echo "the proxy is listening" | |
| exit 0 | |
| fi | |
| sleep 2 | |
| done | |
| as-pdjob podman logs pi-dispatch-egress-proxy | |
| exit 1 | |
| # PI_EGRESS=1, said rather than inherited from the unset default: the teardown case (#458) and the live rootless | |
| # network case (#450) SKIP with egress off, and a required check whose cases skip is blind. `as-pdjob` runs with | |
| # `env -i`, so only what this line passes reaches the script. | |
| - name: Run the conformance harness against the real podman bundle | |
| run: as-pdjob env PI_JOB_IMAGE="$PI_JOB_IMAGE" PI_EGRESS=1 node .github/scripts/podman-conformance.mjs |