Proxy boot: stop paying the stage-3c migration on every deploy of a host that has already migrated

[mhenrixon]

Proxy boot: stop paying the stage-3c migration on every deploy of a host that has already migrated

Follow-up to #154 (PR 5, #159 left this row alone on purpose) and to the stage 3 epic #118 (sits between 3c #124, shipped, and 3d, the 5.0 deletion of the bridge).

Problem / Goal

Ensure dash-proxy is the most expensive row of a deploy by round trips, and most of what it does on a host that has already been through the 3c rename is checking, again, that it has been through the 3c rename.

A real 4-host deploy on dash 4.1.0 (3 web hosts, 1 job host, auto-activated load balancer, no build, secrets present, port_holder off):

  Finished all in 122.4 seconds
  Startup (load, config)                  0.6s
  Validate config and secrets             0.9s
  Pull app image                         25.1s  16 ssh    94.1s
  Acquire deploy lock                     0.7s   5 ssh     1.6s
  Ensure dash-proxy                      17.1s  54 ssh    26.0s
  Acquire server lock                     1.0s   3 ssh     1.0s
  Detect stale containers                 1.0s   8 ssh     3.5s
  Boot                                   71.2s  43 ssh   118.9s
  Loadbalancer                            2.3s   3 ssh     1.2s
  Prune                                   3.7s  14 ssh     8.1s
  Acquire server lock                     0.9s   3 ssh     0.8s

54 round trips and 17 wall seconds to conclude, on every host, that the proxy is already running the right config. Reading Dash::Cli::Proxy#boot for that topology, one proxy host on the no-drift path issues:

#	Round trip	Origin	Needed once a host has migrated?
1	docker network create dash (fails "already exists", rescued)	network_hosts, every host + LB	no, after the first boot
2	docker login	proxy boot	yes (image pull)
3	connect_legacy_network_containers	LegacyRename#bridge_network	no
4	copy_legacy_config_volume	LegacyRename#adopt_config_volume	no
5	remove_legacy_container	LegacyRename#replace_legacy_container	no
6	remove_legacy_holder_container	LegacyRename#replace_legacy_container	no
7	container_id capture	Drift#container_exists?	yes, but see below
8	config_digest capture	Drift#current_digest	yes, but see below
9	version capture	boot, minimum-version gate	yes, but see below
10	ensure_proxy_directory (or remove_proxy_secrets_file)	secrets	yes
11	ensure_apps_config_directory	boot	yes
12	start_or_run	boot	yes

The load-balancer host repeats the pattern: connect_legacy_network_containers and copy_legacy_config_volume (2 legacy round trips), then container_id and config_digest as two captures, then LoadbalancerClaim#claim_run_config (a read and an ensure_directory).

Two things are wrong:

1. The legacy bridge costs 4 round trips per proxy host and 2 on the load balancer, forever. Every step is guarded host-side (exists || ! legacy_exists || …, || true), so it is a no-op on a migrated host, and on a host that was installed fresh on 4.x and never had a kamal-proxy at all. But a no-op that takes four SSH round trips is not free: on this deploy that is 14 of the 54. And because the gem never removes the kamal network (documented manual cleanup), connect_legacy_network_containers finds the network on every migrated host and runs its inspect + xargs sweep on every deploy until 3d.
2. Three captures answer one question. container_id, config_digest and version are three docker invocations against the same container; one docker inspect --format returns id, image tag and digest label together.

Done looks like: a migrated host (or a fresh 4.x host) pays zero round trips for the 3c bridge, drift detection is one capture per host, and the Ensure dash-proxy row on the deploy above drops by roughly half, with before/after tables in the PR. Nothing changes for a host that still runs kamal-proxy: it migrates exactly as today.

Context (read these first)

• lib/dash/cli/proxy.rb — boot: network_hosts create-network sweep, then per proxy host registry.login → LegacyRename#run → Drift → version gate → secrets → ensure_apps_config_directory → start_or_run; the load-balancer branch below it repeats the legacy and drift steps inline. network_hosts (line ~661).
• lib/dash/cli/proxy/legacy_rename.rb — the three steps and the ordering contract in the header comment: bridge network, then copy volume, then replace container, all before anything reads the new container/volume/network. "Every step is idempotent and guarded on its destination not already existing."
• lib/dash/cli/proxy/drift.rb — container_exists? (container_id capture), current_digest (config_digest capture), expected_digest is local when proxy_run_config is present.
• lib/dash/commands/proxy.rb — copy_legacy_config_volume (read the comment: the chain must start with a real program because SSHKit prefixes /usr/bin/env; 4.0.0 lost every host's routing table by getting this wrong), remove_legacy_container, remove_legacy_holder_container, version (docker inspect --format '{{.Config.Image}}' | awk), config_digest (CONFIG_DIGEST_FORMAT reads the new label, falls back to the org.kamal.* one), container_id → Base#container_id_for (docker container ls --filter name=^…$ --quiet), ensure_apps_config_directory → make_directory. remove_container / remove_image also carry a legacy half (only dash proxy remove, not per deploy).
• lib/dash/commands/docker.rb — create_network, connect_legacy_network_containers (inspect kamal network, pipe container names into docker network connect dash each || true).
• lib/dash/commands/loadbalancer.rb — copy_legacy_config_volume; lib/dash/cli/proxy/loadbalancer_claim.rb — claim_run_config read + ensure_directory + upload!.
• lib/dash/configuration/proxy.rb — LEGACY_* constants; lib/dash/configuration/proxy/boot.rb — host_directory = <run_directory>/proxy, apps_directory under it. The run directory (.dash/) is gem-owned and already the home of proxy secrets and apps-config, so a migration marker belongs there.
• lib/dash/commands/auditor.rb — record_then from perf(deploy): cut a quarter of a deploy's SSH round trips #159: the pattern for folding independent commands into one round trip inside the Commands layer.
• lib/dash/cli/base.rb — ensure_run_directory memoises per host per process (perf(deploy): cut a quarter of a deploy's SSH round trips #159); DASH.run_directory_ensured_on in lib/dash/commander.rb.
• Tests: test/cli/proxy_test.rb (boot command sequence assertions, round-trip counts from perf(deploy): cut a quarter of a deploy's SSH round trips #159's pattern), test/commands/proxy_test.rb, test/commands/docker_test.rb, test/cli/doctor_test.rb (fix(doctor): recognise a running legacy proxy instead of blaming the ports #132 made the doctor recognise a still-running kamal-proxy; keep it working).
• Rules: CLAUDE.md (stage 3c is DONE, 3d drops the fallbacks at 5.0, so everything added here is bridge code that 3d deletes and must be marked as such), .claude/rules/performance.md (round trips are the metric; before/after from the deploy table, no bench harness), .claude/rules/coding-style.md (shell built in Dash::Commands::*).

Decision

Fold the bridge into one guarded round trip that a migrated host skips, then collapse the drift captures into one inspect. No change to what a still-legacy host does or in what order.

1. One command for the whole bridge, skipped behind a marker. Dash::Commands::Proxy#legacy_rename (name open) returns a single shell string: [ -f <marker> ] || { bridge_network; copy_legacy_config_volume; remove_legacy_container; remove_legacy_holder_container; } && <write marker when migrated>. The four existing builders keep their bodies; they are composed, not rewritten, so the /usr/bin/env and precedence lessons in their comments still hold. The marker is <proxy_boot.host_directory>/.legacy-renamed (inside .dash/proxy/, next to the secrets file and apps-config, created with the same make_directory).
   • The marker is written only when the host is verifiably past the rename: the kamal-proxy container and the kamal-proxy-net holder no longer exist after the chain ran. It is not written on the strength of the chain's exit status, because every step ends in || true and a failed volume copy must not be recorded as done. A fresh 4.x host that never had kamal-proxy qualifies on its first boot.
   • LegacyRename#run becomes one execute. 4 → 1 round trip on the first deploy after this lands, then the guard makes the docker work a no-op; the round trip itself goes away in step 3.
   • Same shape for the load-balancer branch: its two inline legacy executes become one, with its own marker under the loadbalancer's directory (connect_legacy_network_containers + copy_legacy_config_volume).
2. One capture for drift + version. Dash::Commands::Proxy#inspect_state (name open): docker inspect <container> --format '{{.Id}} {{.Config.Image}} <CONFIG_DIGEST_FORMAT body>' with raise_on_non_zero_exit: false; empty output means no container. Drift takes the parsed struct (id, image tag, digest) and boot reads the version from it instead of a third capture; the version builder stays for dash proxy details/doctor. 3 → 1 per proxy host, 2 → 1 on the load balancer. Drift#drifted?, expected_digest and the minimum-version gate keep their semantics and error messages.
3. Fold the guarded bridge into a round trip the host already pays. ensure_apps_config_directory is an idempotent mkdir -p on every proxy host; move it ahead of the drift read (it has no dependency on it) and combine make_directory(apps_directory) with the guarded bridge in one round trip. Ordering contract preserved: bridge before anything reads the new container/volume/network. After this a migrated host pays 0 round trips for the bridge. Do the same on the load-balancer host.
4. Optional, only if the table still justifies it: network_hosts runs docker network create dash on every host including the job host and the LB, rescuing "already exists" on every deploy after the first. Fold it into the same per-host prepare command (docker network inspect dash >/dev/null 2>&1 || docker network create dash) for the proxy/LB hosts, and keep a plain create only for hosts that get no other proxy-boot command. Measure first; it is one round trip per host, not four.

Alternatives considered

• Detect "migrated" from the gem side with a capture, then skip the bridge. Costs the round trip it saves. Rejected; the guard has to live in the same shell as the work.
• A proxy: legacy_migration: false config switch. Puts the burden on the operator to know their fleet's state and is wrong the day a host is restored from an old image. Rejected; host state, not config, decides. (Revisit only if the marker approach proves fragile.)
• Delete the bridge now. That is 3d and 5.0 (Stage 3 (epic): rename server artifacts with a rolling-upgrade bridge #118), and a host that has never deployed with ≥ 4.0.0 still needs it. Rejected.
• Remove the kamal network once bridged so connect_legacy_network_containers finds nothing. Accessories may still be attached to it; Stage 3c: container identity — proxy container, docker network, volumes, image label, in-image paths #124 made its removal manual on purpose. Rejected.

Design decisions the executor must not reopen

• A still-legacy host runs the three bridge steps in the documented order, before any read of the new container/volume/network, exactly as today. No semantics change for the migration itself, and the brief per-host outage it documents stays as is.
• The marker lives in the gem-owned run directory, never in a docker label or volume, and is written only on verified absence of the legacy containers. 3d deletes the marker handling together with LegacyRename; tag every added line with the same "Stage 3c … 3d deletes" comment convention the existing code uses.
• Shell strings are built in Dash::Commands::Proxy / Dash::Commands::Loadbalancer (with combine/chain/any), never inline in Dash::Cli::Proxy.
• No MINIMUM_VERSION change; nothing in ../kamal-proxy. The proxy image, container name, network and volume names are untouched.
• No new SSH or docker command is added anywhere; every step is a fold or a skip. The Ensure dash-proxy row's round-trip count is the acceptance metric.

Implementation steps

One PR (fix/proxy-boot-migrated-hosts off fresh main), or two if step 2 is easier to review alone. Before touching code, run one real deploy with the current main against a multi-host target and paste the table as the baseline; a fresh-host baseline (never had kamal-proxy) is worth having too since it is the case that gains most.

1. lib/dash/commands/proxy.rb — legacy_rename (guarded fold of the four existing builders + marker write on verified absence), inspect_state (one inspect, parsed by a small Dash::Commands::Proxy::State or a Struct), marker path helper reading config.proxy_boot.host_directory. lib/dash/commands/loadbalancer.rb — the two-step equivalent.
2. lib/dash/cli/proxy/legacy_rename.rb — run executes the one command. lib/dash/cli/proxy/drift.rb — takes the parsed state; boot passes it and reads the version from it. lib/dash/cli/proxy.rb — load-balancer branch uses the folded bridge and single inspect; ensure_apps_config_directory moves ahead of the drift read and is combined with the bridge.
3. Tests, RED first: test/commands/proxy_test.rb (exact shell for legacy_rename including the marker guard and the verified-absence write; inspect_state format), test/commands/loadbalancer_test.rb, test/cli/proxy_test.rb (boot sequence on a Printer backend: a proxy host issues N round trips, down from the count on main; the bridge string appears once and before any inspect; the load-balancer branch likewise), test/cli/doctor_test.rb unchanged and green, test/cli/main_test.rb cost-guard sequence unchanged (subcommands are stubbed there, so pin the reduction in test/cli/proxy_test.rb as perf(deploy): cut a quarter of a deploy's SSH round trips #159 did).
4. Real deploys through the integration harness and against a staging target: (a) a host still on kamal-proxy migrates and ends with the marker; (b) the next deploy on that host issues no legacy docker commands (check -v output) and the marker is untouched; (c) a fresh host gets the marker on first boot; (d) a deploy with reboot_on_deploy: false and a drifted proxy still prints the stale warning; (e) an older proxy image still trips the minimum-version error. Paste the before/after tables.

Verification gates

• Unit suite green: bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].grep_v(/integration/).each { |f| require File.expand_path(f) }'
• bundle exec rubocop --parallel — no offenses
• bin/test — full suite (Docker + published proxy image; MINIMUM_VERSION does not move)
• PR description shows the Ensure dash-proxy row before and after on the same topology, plus the per-host round-trip count (web <host> N ssh), and states plainly which reductions are folds (same commands, fewer round trips) and which are skips (commands no longer run on migrated hosts).
• The cost-guard sequence in test/cli/main_test.rb is unchanged.

Out of scope

• Deleting the bridge, the LEGACY_* constants, the org.kamal.* digest fallback or the dual prune filters: that is 3d (Stage 3 (epic): rename server artifacts with a rolling-upgrade bridge #118, 5.0).
• Removing the kamal network or kamal-proxy-config volume from hosts: manual cleanup by design (Stage 3c: container identity — proxy container, docker network, volumes, image label, in-image paths #124).
• Other rows the same report shows (Pull app image 16 ssh, Detect stale containers 2 captures per host, Boot 7–9 per host): separate issues if the numbers justify them.
• Any change to proxy reboot, port-holder handoff, drain, or the health barrier.
• Anything in ../kamal-proxy; no MINIMUM_VERSION change.

Execution

Hand to a fresh implementation session on the sonnet tier. Baseline first (real deploy table on main), then steps 1–4 in order. The Context section names every file; the ordering contract in legacy_rename.rb's header is the invariant to protect.