Boot: wait for readiness server-side and take the container id from docker run

[mhenrixon]

Boot: wait for readiness server-side and take the container id from docker run

Follow-up to #154 (PR 5, #159, which folded the clash check into boot_state and the audits into their actions). Sibling of #160 (proxy boot) and the pull and stale-container issues filed from the same deploy report.

Problem / Goal

On a real 4-host deploy on dash 4.1.0 (3 web behind the proxy, 1 job with a docker healthcheck):

  Boot                                   71.2s  43 ssh   118.9s
    web 94.237.81.47                       29.0s   7 ssh    27.9s (healthy after 19.2s)
    web 94.237.84.183                      30.8s   7 ssh    29.8s (healthy after 22.0s)
    web 212.147.230.64                     34.6s   7 ssh    33.6s (healthy after 23.3s)
    job 94.237.82.224                      31.3s   9 ssh    20.2s (healthy after 11.7s)

Reading Dash::Cli::App::Boot#run for those two role shapes:

Round trip	web (proxy role)	job (healthcheck, no proxy)
boot_state capture (#159)	1	1
audit + ensure_env_directory (#159); upload! of secrets is SFTP, not a command	1	1
docker run --detach …	1	1
container_id_for_version capture, to get the proxy target	1	–
dash-proxy deploy (blocks until healthy)	1	–
status capture per poll attempt, backoff 1 s, 2 s, 3 s …	–	~5
stop old version	1	1
clean_up_assets	1	–
	7	9

Two of those rows are avoidable:

1. The readiness poll for non-proxy roles is a round trip per attempt. Dash::Cli::Healthcheck::Poller.wait_for_healthy runs the loop on the laptop: capture status, sleep, capture again, with the attempt number as the sleep. A container that takes 12 s to become healthy costs five captures; one that takes 60 s costs ten. A proxy role, by contrast, pays one blocking dash-proxy deploy and lets the host do the waiting. The non-proxy path should do the same.
2. docker run --detach already prints the container id, and the very next round trip asks docker for that id again (container_id_for_version) so it can be handed to dash-proxy deploy as the target. Capturing the run's stdout gives the same id for free.

Done looks like: a healthchecked non-proxy role pays one round trip for its readiness wait regardless of how long the container takes (two when a readiness delay applies), the "not ready yet, Ns elapsed" progress lines still print while it waits, a proxy role pays 6 round trips instead of 7, and every readiness decision the poller makes today (healthy, unchecked-but-running, readiness delay, healthcheck drift, exec probe, timeout) is made identically, with the same messages. Before/after per-host rows in the PR.

Context (read these first)

• lib/dash/cli/app/boot.rb — run, start_new_version (execute *app.run(hostname:) then, for running_proxy?, capture_with_info(*app.container_id_for_version(version)) → BootError if empty → app.deploy(target: endpoint); otherwise Poller.wait_for_healthy(role:) { health_status }), health_status (exec probe vs status capture), exec_probe_status, dump_diagnostics (keeps its own container_id_for_version), timing_healthy, stop_old_version (hook → stop → hook → cleanups).
• lib/dash/cli/healthcheck/poller.rb — the whole decision tree: unchecked?, docker_state, acceptable?, ensure_no_healthcheck_drift (raises DriftError when the role declares a healthcheck but the container reports none), announce_missing_gate, the readiness-delay re-check, the retry loop with deploy_timeout and the Container not ready yet, retrying in Ns (Xs elapsed, Ys left) line, Container is healthy!. lib/dash/cli/healthcheck.rb for Error / DriftError.
• lib/dash/commands/app.rb — run (docker run --detach … --name <container_name> …: with --detach, docker prints the 64-character container id on stdout and nothing else), status(version:) (container_id_for_version | xargs docker inspect --format DOCKER_HEALTH_STATUS_FORMAT), health_probe(version:) (docker exec <name> <probe>; the probe is single-quoted by shell), stop, container_id_for_version. lib/dash/commands/base.rb — NO_HEALTHCHECK, DOCKER_HEALTH_STATUS_FORMAT (healthy|unhealthy|starting when the container has a healthcheck, else no-healthcheck:<state>), container_id_for (docker container ls --filter name=^…$ --quiet, which prints the 12-character short id: this is what dash-proxy deploy receives today).
• lib/dash/commands/app/proxy.rb — deploy(target:); lib/dash/configuration/role.rb — readiness_delay, readiness_source, readiness_gated?, healthcheck (exec?), running_proxy?.
• lib/dash/build/progress_parser.rb and lib/dash/cli/build.rb (interaction_handler: near line 180) — the precedent for streaming a long-running command's stdout through an SSHKit interaction handler (on_data(command, stream, data, channel)) while it runs. lib/dash/sshkit_with_ext.rb — CommandEnvMerge passes interaction_handler: straight through to SSHKit::Command.
• lib/dash/configuration/docs/role.yml (readiness_delay, healthcheck, the exec-probe wording near line 124) and docs/ pages that describe the readiness wait; update wording only if the printed lines change.
• Tests: test/cli/app_test.rb (boot sequences, the perf(deploy): cut a quarter of a deploy's SSH round trips #159 single-round-trip tests, recorded_commands), test/cli/healthcheck/poller_test.rb or wherever the poller's cases live (grep -rl wait_for_healthy test/), test/commands/app_test.rb, test/cli/main_test.rb (cost-guard; app:boot is stubbed there, so pin reductions in app_test).
• Rules: .claude/rules/performance.md (round trips are the metric), .claude/rules/coding-style.md (shell in Dash::Commands::App, the CLI orchestrates), .claude/rules/testing.md. perf(deploy): cut a quarter of a deploy's SSH round trips #159's PR description for the reporting format (per-host rows, folds vs skips).

Decision

Move the wait onto the host as one blocking command whose stdout is streamed back for progress, keep every decision in the poller, and read the container id from docker run.

1. Server-side wait, streamed. Dash::Commands::App#wait_for_ready(version:, timeout:) (name open) builds a shell loop that, once a second until timeout seconds have passed, evaluates the same status the laptop reads today (DOCKER_HEALTH_STATUS_FORMAT via docker inspect, or for healthcheck.exec the docker exec probe's exit status) and:
   • exits 0 the moment the status is decidable: healthy, unhealthy, or any no-healthcheck:<state> (the poller must see those to raise drift or accept an unchecked running container), printing that status as the last line;
   • otherwise prints a progress line with the status and elapsed seconds, sleeps 1 s, and loops;
   • on deadline prints the last status and exits non-zero.
     Boot#start_new_version runs it with capture_with_info(..., interaction_handler: <handler>, raise_on_non_zero_exit: false); the handler turns each streamed progress line into the existing Container not ready yet, retrying in 1s (Xs elapsed, Ys left) info line (same wording, retrying in 1s because the host loop is fixed-interval), so the operator sees exactly the cadence they see today. The final line is handed to Poller.wait_for_healthy as the block's result. The poller keeps its tree unchanged: healthy → done; no-healthcheck:running → announce_missing_gate, sleep readiness_delay on the laptop, one plain status capture to confirm (the second round trip for unchecked roles, as today); drift → DriftError; anything else after the deadline → Healthcheck::Error with the status in the message. The poller's own retry loop is what shrinks: it no longer needs to re-call the block for starting, because the host loop already waited. Net: healthchecked roles N → 1 round trip, unchecked roles N → 2.
   • The starting state is the only one the host loop waits through; unhealthy returns immediately so a failing probe is reported at the first observation rather than at the deadline (today it retries until deploy_timeout; keep the message identical, and if the executor finds a reason the retry-on-unhealthy matters, keep that too and say so in the PR).
   • The loop is a Dash::Commands::App builder composed with combine/chain/shell, tested as an exact string; the CLI passes the handler and reads the result.
2. Container id from docker run. start_new_version captures app.run(hostname:) instead of executing it, takes the first 12 characters of the printed id as the proxy target (the same short id container_id_for_version returns today, so dash-proxy deploy receives an identical target), and drops the follow-up capture. An empty capture keeps raising BootError with the current message. dump_diagnostics and stop_new_version keep using container_id_for_version; they run only on failure.
3. Optional, only if the harness table justifies it: clean_up_assets / clean_up_error_pages after the old version stops are separate round trips; they could ride with stop in one record_then-style fold, but that moves them ahead of the post-app-stop hook. Do not fold them unless the PR states that hook-ordering change explicitly; the default is to leave them.

Alternatives considered

• Keep client-side polling and only take the docker run id. Saves one round trip on proxy roles and nothing on the job host, which is the row that scales with boot time. Rejected in interview.
• Server-side wait, silent (no streaming). One round trip, but an operator sees nothing for up to deploy_timeout; the progress lines are the only feedback a slow boot gives. Rejected in interview.
• docker wait / docker events --filter health_status. docker wait returns only on exit; events needs a healthcheck and gives nothing for no-healthcheck roles or exec probes. The inspect loop is the one shape that covers all three readiness sources. Rejected.
• Use dash-proxy deploy for non-proxy roles too. They are not registered with the proxy by definition. Rejected.

Settled in interview:

• Server-side wait with streamed progress lines (not silent, not client-side).

Design decisions the executor must not reopen

• Every readiness decision stays in Dash::Cli::Healthcheck::Poller and its messages stay word-for-word; the host loop only decides when to return, never what it means. Drift detection, announce_missing_gate, the readiness-delay confirm and the timeout error are all exercised by tests before and after.
• No new SSH or docker command: the wait replaces N captures with one; the id read replaces one capture with none.
• The shell loop lives in Dash::Commands::App; the interaction handler is a small class under lib/dash/cli/healthcheck/ (no SSH of its own), mirroring Dash::Build::ProgressParser.
• Health barrier, gatekeeper?/queuer? ordering, hooks, and stop_old_version order are untouched.

Implementation steps

One PR (perf/boot-server-side-readiness off fresh main), or two if the executor prefers to land the docker run id read first. Baseline first: a real multi-host deploy with at least one healthchecked non-proxy role and one proxy role; paste the per-host rows.

1. lib/dash/commands/app.rb — wait_for_ready(version:, timeout:) for the inspect path and the exec-probe path (one builder, two branches, or two builders; exact strings unit-tested in test/commands/app_test.rb including the deadline, the 1 s interval, and that no-healthcheck:* and unhealthy return immediately).
2. lib/dash/cli/healthcheck/ — the streaming handler (progress line → info in today's wording). Unit test it with synthetic chunks split mid-line, as the build parser tests do.
3. lib/dash/cli/app/boot.rb — start_new_version uses the capture for run, derives the 12-char target, and runs the host wait via the poller for non-proxy roles. lib/dash/cli/healthcheck/poller.rb — the block is called once for the wait and once more only for the readiness-delay confirm; retries only remain for the case where the host loop returns a non-decidable status before the deadline (which it should not; assert that in a test).
4. Tests RED first in test/cli/app_test.rb: boot on a proxy role issues 6 round trips and passes the run's short id to dash-proxy deploy; boot on a healthchecked non-proxy role issues 1 wait round trip with the progress lines printed; unchecked role: wait + one confirm after readiness_delay; drift raises DriftError with the current message; deadline raises Healthcheck::Error with the status; exec probe uses the probe loop. Poller tests updated for the new call pattern.
5. Real deploys through the integration harness (roles fixture with a job role) and a staging target; paste before/after per-host rows; confirm dash-proxy deploy accepts the run-derived id on a real proxy; confirm Ctrl-C during the wait leaves no orphaned loop on the host (the SSH session teardown kills it; verify once).

Verification gates

• bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].grep_v(/integration/).each { |f| require File.expand_path(f) }' — green
• bundle exec rubocop --parallel — no offenses
• bin/test — full suite (Docker + published proxy image; MINIMUM_VERSION does not move)
• PR description shows the Boot row and each host row before and after (expected web 7 → 6, job 9 → 5 on the deploy above), and says which reductions are folds and which are skips.
• Cost-guard sequence in test/cli/main_test.rb unchanged.

Out of scope

• The proxy-role wait (dash-proxy deploy is already one blocking round trip) and the health barrier.
• boot_state, the audit folds and ensure_env_directory (perf(deploy): cut a quarter of a deploy's SSH round trips #159); the secrets upload!.
• Hook ordering around stop_old_version unless explicitly stated per step 3 above.
• No direct pushes to main, no manual lib/dash/version.rb bumps, no MINIMUM_VERSION change, nothing in ../kamal-proxy, no frozen-artifact renames.

Execution

Hand to a fresh implementation session on the sonnet tier. Baseline per-host rows first, then steps 1–5; the poller's decision tree is the invariant to protect.