Repository navigation
feat(proxy): automatic drift-detected proxy reboot on deploy - #24
Merged
Merged
Conversation
Groundwork for detecting proxy drift during deploy: - Proxy::Run.digest / #config_digest: SHA256 over the materialized run invocation (image, run command, docker options), schema-versioned so a future serialization change causes exactly one converging reboot. - Commands::Proxy#run and #start_or_run accept digest: and stamp it as the org.kamal.proxy-config-digest container label, on both the proxy/run and legacy boot_config paths. - New commands: pull (pre-pull the proxy image before stopping the old container), list (docker exec kamal-proxy list), config_digest (read the label back), container_id, and stop with a --time override.
kamal deploy (and setup / proxy boot) now compares the running proxy container's config-digest label against the digest the current configuration produces. On drift it reboots the proxy automatically — one host at a time, so a fleet never loses all proxies at once — before app boot registers the new version. Previously changed proxy run options were silently ignored by start_or_run, and a version bump required a manual, unverified kamal proxy reboot. The safe-reboot worker (Kamal::Cli::Proxy::Reboot) is shared by the auto path and the hardened kamal proxy reboot command: pull image (before stopping — registry outages can no longer strand a host) -> stop with drain headroom -> remove -> run with digest label -> wait for the control socket -> re-register this app's roles and accessories -> verify them against kamal-proxy list, failing loudly with the missing service names. Re-registration heals the exact failure mode where the proxy restores from a torn state file with zero services and 404s everything. Opt out with proxy: reboot_on_deploy: false (root level) — deploy then prints a drift warning instead of silently ignoring the change. Containers started by older kamal versions carry no digest label and converge with one automatic reboot on the first deploy. Unlabelled-drift, opt-out warning, matching-digest no-op, and hardened reboot sequences are covered in CLI tests; an integration test exercises the full deploy -> drift -> auto-reboot -> converge cycle.
Bumps MINIMUM_VERSION to v0.9.2.2 — the fork release carrying the zero-downtime reboot hardening (atomic state writes, restore recovery ladder, list --json, drain-on-stop, restore re-verification, reuse-port, drain, hold). The proxy run command now always passes --recheck-targets-on-restore (proxy/run path): after a reboot, restored targets are re-verified with live health checks instead of trusted blindly, so a target that died while the proxy was down demotes to an honest 503 and rejoins on recovery, rather than serving 502s forever. Both changes alter the config digest, so the first deploy after upgrading converges every host with one automatic serial reboot. Release ordering: ghcr.io/mhenrixon/kamal-proxy:v0.9.2.2 must be published (script/release-dash) before this ships in a gem release.
The MINIMUM_VERSION bump missed setup.sh, whose hardcoded seed tag must equal MINIMUM_VERSION (documented in .claude/rules/upstream-sync.md) — app_with_roles pulls the proxy via the local registry:4443 mirror, so every fixture that boots a proxy failed with manifest unknown. Also drop the stray kamal envify from the drift integration test: the app fixture has no .env.erb and the deploy does not need one.
Since kamal-proxy v0.9.2.2 the server flushes kamal-proxy.state on every shutdown, so any boot after the very first restores state — the log assertion in the proxy integration test asserted the old behavior.
3 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Implements Phase 1b of the zero-downtime proxy reboot plan (companion to the mhenrixon/kamal-proxy hardening series #32-#37).
What
Release ordering
Proxy image first: merge the kamal-proxy series, tag + publish ghcr.io/mhenrixon/kamal-proxy, then bump MINIMUM_VERSION here — the digest includes the version, so the first deploy converges every host.
Refs: basecamp/kamal#1055 (auto-reboot wontfix due to downtime — this removes the downtime), basecamp/kamal#1298, basecamp/kamal-proxy#86, basecamp/kamal#1552.