fix(egress): plain HTTP reaches a listed host on port 80 only, and doctor proves it (#508)

.github/scripts/podman-conformance.mjs

@@ -226,8 +226,9 @@ const withBrokenEnumeration = () => makeReaper({ log: () => {}, bin: `pi-dispatc
 
 /**
  * The egress readings `egressVerdict` takes, from doctor's own canary (`runEgressCanary`, issue #431) under this
- * account's Podman: a job-shaped `--internal` network with the proxy attached, and two probe containers built by the
- * podman builder as a job's are, as the job user, running the runner's route to the network. This script carried a copy
+ * account's Podman: a job-shaped `--internal` network with the proxy attached, and three probe containers built by the
+ * podman builder as a job's are, as the job user: two running the runner's route to the network, and one sending plain
+ * HTTP to a listed host off port 80, which the proxy must refuse (issue #508). This script carried a copy
  * of its own until then, with a plain `fetch`, which proved the network and the allowlist and not the runner's route;
  * the one canary now serves `doctor --live` and this run alike. Every line it prints that is not a pass is repeated
  * here, since the harness only sees the readings.
@@ -387,6 +388,8 @@ async function staleCanarySweep() {
 	const [emptyPid, fullPid] = dead;
 	const emptyNet = egressCanaryNetwork(emptyPid);
 	const fullNet = egressCanaryNetwork(fullPid);
+	// The first two slugs only: two leftover shapes (one running, one stopped) are what the sweep is tested on, and the
+	// third slug (issue #508) is matched by the same pattern, so a third leftover would add nothing.
 	const [runningProbe, stoppedProbe] = CANARY_PROBE_SLUGS.map((slug) => egressCanaryProbe(slug, fullPid));
 	const exists = async (kind, name) => (await podman([kind, "exists", name])).code === 0;
 	try {

.github/scripts/podman-host-check.mjs

@@ -287,6 +287,8 @@ const fromDoctor = (id, pattern, what) => {
 fromDoctor("D-health", /^Egress proxy health: healthy/, "the proxy's health");
 fromDoctor("EG1", /^Egress policy reaches the provider/, "the provider");
 fromDoctor("EG2", /^Egress policy denies an unlisted host/, "an unlisted host");
+// Issue #508: the proxy's port rule for plain (untunnelled) requests, read by doctor's third canary probe.
+fromDoctor("EG8", /^Egress policy refuses plain HTTP/, "plain HTTP to a listed host off port 80");
 fromDoctor("D-egress", /^read back on local: egress holds/, "egress");
 fromDoctor("EG6", /^read back on local: jobToJobIsolation holds/, "jobToJobIsolation");
 fromDoctor("D-localFolders", /^read back on local: localFolders holds/, "localFolders");
@@ -363,7 +365,9 @@ const ROWS = [
 	{ row: "SELinux: an operator's local folder, unlabelled", result: "refused: job-inputs-unreadable", needs: /^(S-\w+:(none|ro)$|R-local$|R-control$)/ },
 	{ row: "SELinux: the global overlay, unlabelled", result: "refused: job-inputs-unreadable", needs: /^(R-overlay|R-control)$/ },
 	{ row: "SELinux: SecurityOptions carries name=selinux", result: "measured", needs: /^SECOPT$/ },
-	{ row: "nftables: egress reaches the provider and denies an unlisted host", result: "measured", needs: /^(EG[1-57]|EG-net|D-egress)$/ },
+	// EG8 (issue #508) rides this row rather than a row of its own: it is the same canary on the same network, and a new
+	// row would need a measured date on this page's SELinux host before it could be printed.
+	{ row: "nftables: egress reaches the provider and denies an unlisted host", result: "measured", needs: /^(EG[1-578]|EG-net|D-egress)$/ },
 	{ row: "nftables: jobToJobIsolation", result: "measured", needs: /^EG6$/ },
 	{ row: "Health checks under systemd", result: "measured", needs: /^(H-|D-health$)/ },
 	{ row: "SELinux: the compose file's config mounts", result: "argv: :ro,z", needs: /^(C-compose|H-proxy)$/ },

deploy/egress-proxy.conf

@@ -6,9 +6,11 @@
 # misordered one silently allows everything, so the file an operator edits contains no ordering at all.
 #
 # What this policy is, and is not:
-#   - Hostname filtering on CONNECT, to port 443 only. The provider, the forge and the registry are
-#     ordinary entries in the allowlist, and nothing is special. The one address-based rule is a DENY:
-#     a listed name that resolves to a fixed loopback or link-local address of this host is refused.
+#   - Hostname filtering, and exactly two shapes pass (issue #508): a CONNECT tunnel to a listed host
+#     on port 443, or a plain (untunnelled) request to a listed host on port 80. Nothing else. The
+#     provider, the forge and the registry are ordinary entries in the allowlist, and nothing is
+#     special. The one address-based rule is a DENY: a listed name that resolves to a fixed loopback
+#     or link-local address of this host is refused.
 #   - TLS is NEVER terminated. This proxy sees the name a client asks for and no byte inside the tunnel,
 #     so it cannot read a credential and cannot account for a token. A proxy that decrypts provider
 #     traffic is OQ-011's mechanism, a materially larger change, and it is not this.
@@ -29,6 +31,15 @@ http_port 3128
 acl allowed dstdomain -n "/etc/pi-dispatch/allowlist.conf"
 
 acl SSL_ports port 443
+# Stock squid's name, narrowed to one port (issue #508). Without it `allow allowed` below let any
+# tool that forwards plain HTTP (npm undici's EnvHttpProxyAgent from 8.7 on without proxyTunnel, a
+# package manager, a curl -x a job brings) reach ANY port of a listed host, in clear text. What every
+# job has never takes this path: pi and the runner tunnel every origin, Node 22's built-in fetch sends a
+# CONNECT even for http://, and the job image's git ignores HTTP_PROXY for an http:// remote. A side
+# effect: a plain forward `GET https://host/`, where this proxy would open the TLS itself, is refused
+# too, since it names port 443 without a CONNECT. Measured on Docker and rootless Podman 4.9.3 and 5.8.1
+# (docs/egress.md): port 8080 went from 200 to 403, port 80 stayed 200.
+acl Safe_ports port 80
 acl CONNECT method CONNECT
 
 # The host's own services, by address (issue #428). On rootless Podman an account's containers.conf
@@ -55,9 +66,14 @@ acl to_host_local dst 127.0.0.0/8 0.0.0.0/32 169.254.0.0/16 10.0.2.2/32 ::1 ::/1
 # debug_options 78,3: a CONNECT to an unlisted name sent a DNS query, where without the rule it sent
 # none), which is a DNS channel out of an egress-armed job. With `allowed` first, only a listed name
 # is ever resolved here, and it is the only kind the allows below could let through anyway.
+# The port rule for plain requests (issue #508) sits right before the allow it narrows: first match
+# wins, so after it it would never apply. It names only port and method ACLs, which resolve nothing,
+# so no unlisted name is resolved by it. It never matches a CONNECT, so the tunnel rules (including a
+# CONNECT allow later placed between the first two rules, see #503) are untouched by it.
 http_access deny allowed to_host_local
 http_access deny CONNECT !SSL_ports
 http_access allow CONNECT allowed
+http_access deny !Safe_ports !CONNECT
 http_access allow allowed
 http_access deny all
 

docs/egress.md

@@ -82,7 +82,7 @@ than everything else combined.
 | | |
 |---|---|
 | One `--internal` network **per job** | `pi-job-<id>-net`, created at job start and removed at job end. Holds exactly two endpoints: the container and the proxy. If the worker dies before it can remove one, the next boot removes it, detaching whatever is still on it first, and says so in the log if it cannot. |
-| One long-lived proxy | `pi-dispatch-egress-proxy`, squid, hostname filtering on `CONNECT` to port 443, and a listed name that resolves to one of this host's fixed loopback or link-local addresses (or slirp4netns's `10.0.2.2`) is refused (issue #428). The allowlist is matched as written (`dstdomain -n`): an IP-literal request is refused unless that literal is itself listed, and the proxy makes no reverse lookup of it. A host mapped to another address (pasta's `--map-host-loopback <address>` or `--map-gw`, slirp4netns's `cidr=`) is not covered here; on the `podman` venue the worker refuses the containers.conf that would do that, and the account's running rootless network while it still does after the key is gone (issue #450). Publishes no port. |
+| One long-lived proxy | `pi-dispatch-egress-proxy`, squid, filtering by hostname. Exactly two shapes pass, both to a listed host: a `CONNECT` tunnel to port 443, or a plain (untunnelled) request to port 80. Nothing else passes (issue #508). "A plain request to port 80" also covers squid's own gateways for other schemes, but only to port 80 (`ftp://host:80/`, `https://host:80/`); every other scheme and port is refused. A plain forward request for an `https://` URL on 443, where the proxy would open the TLS itself, is refused too. A listed name that resolves to one of this host's fixed loopback or link-local addresses (or slirp4netns's `10.0.2.2`) is refused (issue #428). The allowlist is matched as written (`dstdomain -n`): an IP-literal request is refused unless that literal is itself listed, and the proxy makes no reverse lookup of it. A host mapped to another address (pasta's `--map-host-loopback <address>` or `--map-gw`, slirp4netns's `cidr=`) is not covered here; on the `podman` venue the worker refuses the containers.conf that would do that, and the account's running rootless network while it still does after the key is gone (issue #450). Publishes no port. |
 | One upstream network | `pi-dispatch-egress-out`. Only the proxy is on it. |
 
 **Per job, not one shared network**, and that is the part worth understanding. A shared network is a shared
@@ -129,6 +129,7 @@ deployment. `doctor` does it once, when you ask.
 ✓ Egress proxy health: healthy
 ✓ Egress policy reaches the provider (api.anthropic.com answered, so the whole path works and no key was spent)
 ✓ Egress policy denies an unlisted host (the deny direction is the half an allowlist can silently lose)
+✓ Egress policy refuses plain HTTP to a listed host off port 80 (api.anthropic.com:443 without a tunnel; only a CONNECT reaches 443)
 ```
 
 ### On the native `podman` venue
@@ -142,7 +143,8 @@ mount), so a pass is about what a job gets. Its lines carry the section's prefix
 ```
 ✓ podman: Egress policy reaches the provider (api.anthropic.com answered, so the whole path works and no key was spent)
 ✓ podman: Egress policy denies an unlisted host (the deny direction is the half an allowlist can silently lose)
-✓ read back on podman: egress holds (the provider was reached and an unlisted host was not)
+✓ podman: Egress policy refuses plain HTTP to a listed host off port 80 (api.anthropic.com:443 without a tunnel; only a CONNECT reaches 443)
+✓ read back on podman: egress holds (the provider was reached, an unlisted host was not, and plain HTTP off port 80 was refused)
 ```
 
 A plain `pi-dispatch doctor` does not run it there: the first keep-id start of an image copies its layers, which
@@ -158,7 +160,7 @@ them.
 ### Lines about leftovers
 
 You may also see a line about a leftover. Those are the canary's own objects, `pi-dispatch-egress-doctor-<pid>`
-and its two probe containers. The rule is simple enough to rely on: **every one of them is removed at the end
+and its three probe containers. The rule is simple enough to rely on: **every one of them is removed at the end
 of the run that made it, or reported in that same run** -- so what a later run finds is what a run that was
 KILLED left behind, and doctor does not have to guess which. What it knows is that the process in the name is
 no longer alive. The next run sweeps whatever belongs to a process that is no longer alive
@@ -186,7 +188,7 @@ generator.
 
 <!-- /CANARY-LINES -->
 
-The two policy lines each run a throwaway container on a throwaway network, using **your job image's own node and
+The provider and unlisted-host lines each run a throwaway container on a throwaway network, using **your job image's own node and
 its runner's own module** (pi loaded, then the runner's proxy restore, issue #427), so they prove the route your jobs'
 provider calls take once the runner has called that module; that it does, before its first request, is checked in CI. (Before #427 they used a plain `fetch`, which never loads pi and stayed green while every job
 failed.) They cost nothing: `api.anthropic.com` answers `401` to an
@@ -195,6 +197,32 @@ without spending a token. The deny probe asks for `example.com`, a host that res
 that lets everything out is caught; it is only contacted if your proxy lets the request out, which is the
 finding. A probe container that does not run at all is reported as not run, never as a deny.
 
+The third line (issue #508) runs a third container the same way, but not the runner's route: that route tunnels
+every request, so it would only ever send a `CONNECT`. This one sends a plain forward request with raw `node:http`,
+`GET http://api.anthropic.com:443/`, the way a tool that forwards plain HTTP would (npm undici's `EnvHttpProxyAgent`
+without `proxyTunnel`, a package manager, or a `curl -x` a job brings). It asks for port
+443 because that host certainly listens there, so a proxy that lets the request through gets an answer (a `400` for
+clear text on a TLS port) instead of a timeout, and nothing extra has to be listed. It counts as refused only when
+squid itself refuses it: a `403` with an `X-Squid-Error` header starting `ERR_ACCESS_DENIED`. Any other answer means
+the proxy let it through. A proxy that cannot be reached or never answers is no reading. The check is best effort:
+an upstream on port 443 that answered clear text with a `403` and a forged `X-Squid-Error` header would read as
+refused, which needs control of the provider's own answer.
+
+**Right after an upgrade this line can fail.** squid reads its rules only at start, so until the proxy restarts on
+the refreshed `egress-proxy.conf` it still runs the old rules: the line reads `Egress policy lets plain HTTP through
+to api.anthropic.com on port 443, ...` and `--live`'s egress verdict fails. `pi-dispatch up` refreshes the file and
+restarts the proxy; on the `podman` venue `pi-dispatch service install --force` does. A proxy you started by hand
+needs the updated file in its mount and a restart (`docker restart pi-dispatch-egress-proxy`, or `podman restart`).
+
+**With the policy on, a forge must be served over `https://` on port 443.** A job gets the proxy variables in
+uppercase only, and git ignores `HTTP_PROXY` for an `http://` remote, so it never sends one through the proxy: from
+behind the job's `--internal` network an `http://` forge fails on any port, 80 included (measured in the job image,
+git 2.39.5; this was so before issue #508 too). An `https://` remote is a `CONNECT`, which the proxy refuses to any
+port but 443. The clone runs on the host before the container exists, so with an `http://` forge it is a job's
+git push and fetch that fail; on a port other than 80 its API calls fail too (glab and tea do use `HTTP_PROXY`, and
+port 80 is allowed). `pi-dispatch doctor` warns when `GITLAB_URL` or `FORGEJO_URL` is anything but `https://` on 443. The ways
+out are `https://` on 443, or `PI_EGRESS=0`.
+
 An absent proxy is a **hard failure** in doctor, because every job is refused while it is down. Everything
 that needs the network to answer is a **warning**, because a custom provider base URL or a transient blip
 would each make a red there a false alarm. So is any leftover the canary could not clear: a network nobody is
@@ -250,9 +278,11 @@ env-proxy dispatcher itself, from pi's own `undici`, right after pi is loaded, a
 Before, it was a plain `fetch` and stayed green through all of this. On a job image built before the fix, the
 canary says the image cannot find that module, rather than reporting a policy result. That the runner's entrypoint
 calls the module, before its first request, is checked in CI, where the job image's contract job runs the real
-entrypoint against a stub proxy. The runner's restored dispatcher tunnels every origin (`proxyTunnel: true`,
-as pi's own CLI does since undici 8.7 stopped tunnelling `http://`), so an `http://` provider is a CONNECT
-to port 80, which the shipped policy refuses, as before.
+entrypoint against a stub proxy. pi and the runner always tunnel (`proxyTunnel: true`, as pi's own CLI does
+since undici 8.7 stopped tunnelling `http://`). So a provider configured with an `http://` URL is a CONNECT to
+that URL's port, and the shipped policy refuses it unless the port is 443. A tool that forwards plain HTTP
+instead, such as npm undici's `EnvHttpProxyAgent` without `proxyTunnel` or a package manager, gets port 80 of a
+listed host and nothing else (issue #508).
 
 ## What this does not buy you
 
@@ -265,6 +295,9 @@ It also accounts for nothing. A staged package that spawns a `pi` subprocess spe
 host, which is on the allowlist by necessity, and a proxy that does not decrypt cannot count tokens
 (`OQ-011`).
 
+Plain HTTP to port 80 of a listed host is allowed, because some package mirrors and redirects still use it
+(issue #508). Whatever a job sends that way crosses the network in clear text.
+
 And it does not hide **this host** from the job. `--internal` stops the network routing anywhere beyond
 itself, but its gateway is still the host, so a service listening on `0.0.0.0` there answers a job container
 that dials the gateway address. Measured on Docker 27.5.1 and on rootful Podman 5.8.2 alike: a listener on
@@ -282,6 +315,15 @@ All of it was run. The method costs nothing and is worth repeating on your own h
 - **The whole path, end to end**, with the shipped compose profile and a real per-job network: the provider
   answered `401` in 228 ms, an unlisted host was denied in 18 ms, and `api.github.com` answered in 205 ms
   through the `.github.com` rule.
+- **The port rule for plain HTTP** (issue #508), measured on 2026-09-30 with the pinned squid (6.13) on Docker
+  29.1.3, rootless Podman 4.9.3 and rootless Podman 5.8.1 (SELinux enforcing), all three alike. Before the rule, a
+  plain `GET` to a listed host on port 8080 answered `200`; after it, `403`. Port 80 answered `200` before and after.
+  A `CONNECT` to port 8080 was `403` both times. A plain `GET http://api.anthropic.com:443/`, the canary's third
+  probe, got Cloudflare's `400` before and squid's `403` with `X-Squid-Error: ERR_ACCESS_DENIED` after. A plain
+  forward `GET https://api.anthropic.com/` made squid open the TLS itself before (`503`), and is `403` after. An
+  unlisted name was never resolved, before or after. npm undici 8.10.0's `EnvHttpProxyAgent` without
+  `proxyTunnel` went from `200` to `403` on port 8080 and stayed `200` on 80. Node 22's built-in `fetch` with
+  `NODE_USE_ENV_PROXY=1` sends a `CONNECT` even for `http://`, so it was already refused off port 443.
 - **The pre-spend refusal**, in both directions: with the proxy stopped the preflight returns
   `proxyStopped`, with it removed `proxyMissing`, and with `PI_EGRESS` unset it returns admit **without
   spawning docker at all**.

docs/forgejo.md

@@ -32,6 +32,12 @@ FORGEJO_BOT_ID=42          # the harness account's numeric id, from its profile
 FORGEJO_WEBHOOK_SECRET=... # a long random string
 ```
 
+With the egress policy on (the default), Forgejo must be served over `https://` on port 443. git in a job never
+sends an `http://` remote through the proxy, and the proxy refuses a `CONNECT` to any other port (issue #508,
+`docs/egress.md`). With Forgejo's default `http://forgejo.example:3000` a job's git push and fetch fail, and its
+API calls too, since the port is not 80. Plain `http://` works only with `PI_EGRESS=0`. `pi-dispatch doctor` warns when `FORGEJO_URL` is anything
+but `https://` on 443.
+
 There is no source to choose here either. `FORGEJO_AUTH_SOURCE` accepts only `pat` and defaults to it,
 for the same reason GitLab's does: Forgejo has no App or installation token, so the variable exists to
 refuse a wrong assumption of symmetry with `GITHUB_AUTH_SOURCE` rather than to offer an alternative, and it