Epic: dash / dash-proxy improvement roadmap (R1–R5) #13

Tracking epic for the dash / dash-proxy improvement roadmap — bridging selected gaps vs nginx/traefik/caddy/envoy for kamal's audience, without competing head-on. Full rationale and evidence links live in ROADMAP.md in each repo.

Strategy: own what basecamp rejected (headers, rate limiting, compression, PROXY protocol), port what they left stuck (on-demand TLS, mTLS, basic auth, scale-to-zero), ship the table-stakes resilience knobs every other proxy has.

Structure: child issues live in whichever repo the code changes — proxy work in mhenrixon/kamal-proxy, gem work here. Features needing both get one issue per repo, cross-linked. Grouped by R1–R5 milestones.

Release ordering (hard constraint): the proxy image ships BEFORE the gem in any release where MINIMUM_VERSION moves — the gem's minimum must name a published ghcr tag. Versioning: gem 2.12.0.N, proxy v0.9.2.N. See .claude/rules/upstream-sync.md.

R1 — Foundations & fixes

All S-sized: fix real bugs and expose already-built capability (rollout!). Max value per line. Ships as gem 2.12.0.1 + proxy v0.9.2.2.

• R1: Loadbalancer deploy propagates full proxy deploy_options #1 — Loadbalancer deploy propagates full proxy deploy_options gem S
• R1: Fix bind_ips validator checking the wrong config key #2 — Fix bind_ips validator checking the wrong config key gem S
• R1: Validate the loadbalancer: config key #3 — Validate the loadbalancer: config key gem S
• R1: Expose healthcheck.port / healthcheck.host in deploy.yml #4 — Expose healthcheck.port / healthcheck.host in deploy.yml gem S
• R1: Expose --tls-staging and --read-target/writer-affinity in deploy.yml #5 — Expose --tls-staging and --read-target/writer-affinity in deploy.yml gem S
• R1: Expose kamal-proxy rollout as kamal app rollout deploy/set/stop #6 — Expose kamal-proxy rollout as kamal app rollout deploy/set/stop gem M
• R1: Add pre/post-proxy-deploy hooks #7 — Add pre/post-proxy-deploy hooks gem S
• R1: Fill accessory-proxy docs stub and fix stale run defaults #8 — Fill accessory-proxy docs stub and fix stale run defaults gem S
• R1: kamal doctor diagnostics command #9 — kamal doctor diagnostics command gem M
• R1: Boot config banner + eager validation + health-wait progress beacons #10 — Boot config banner + eager validation + health-wait progress beacons gem S
• R1: Start CertificateRenewalManager (never started today) dash-proxy#1 — Start CertificateRenewalManager (never started today) proxy S
• R1: Wire the 3 certificate Prometheus metrics dash-proxy#2 — Wire the 3 certificate Prometheus metrics proxy S
• R1: Slowloris/server-timeout defaults on all listeners dash-proxy#3 — Slowloris/server-timeout defaults on all listeners proxy S

R2 — Timeouts & resilience

The #1 pain (per-route timeouts) plus the resilience story no kamal-proxy user gets today without fronting nginx/traefik.

• R2: Per-route timeouts + whole-request deadline dash-proxy#4 — Per-route timeouts + whole-request deadline proxy M
• R2: Plumb path_timeouts into deploy.yml (proxy per-route timeouts) #11 — Plumb path_timeouts into deploy.yml (proxy per-route timeouts) gem S
• R2: Retries / hold-until-healthy dash-proxy#5 — Retries / hold-until-healthy proxy S
• R2: Custom error pages for upstream 502/503 dash-proxy#6 — Custom error pages for upstream 502/503 proxy S
• R2: Upstream connection-pool / idle-timeout tuning knobs dash-proxy#7 — Upstream connection-pool / idle-timeout tuning knobs proxy S

R3 — Security & access

Independent S-sized middlewares — good filler between larger efforts. Several are explicit basecamp rejections (safe fork moat).

• R3: Basic auth per service/path dash-proxy#8 — Basic auth per service/path proxy S
• R3: Plumb basic auth into deploy.yml #12 — Plumb basic auth into deploy.yml gem S
• R3: IP allow/deny (CIDR, per service/path) dash-proxy#9 — IP allow/deny (CIDR, per service/path) proxy S
• R3: Per-IP rate limiting (token bucket, burst, allowlist) dash-proxy#10 — Per-IP rate limiting (token bucket, burst, allowlist) proxy M
• R3: PROXY protocol support dash-proxy#11 — PROXY protocol support proxy S
• R3: mTLS client certs (--tls-client-ca-path) dash-proxy#12 — mTLS client certs (--tls-client-ca-path) proxy M

R4 — TLS & custom domains

The single biggest differentiator (on-demand TLS) — a waiting user base on the stuck upstream PR. Schedule a focused block.

• R4: On-demand TLS with ask endpoint dash-proxy#13 — On-demand TLS with ask endpoint proxy L
• R4: Min-TLS version / cipher config dash-proxy#14 — Min-TLS version / cipher config proxy S

R5 — Traffic shaping & headers

Grab bag ordered by demand: headers, weighted canary, redirects, compression, scale-to-zero, observability.

• R5: Header rules (request/response add/remove/set) dash-proxy#15 — Header rules (request/response add/remove/set) proxy S
• R5: Weighted canary split dash-proxy#16 — Weighted canary split proxy M
• R5: Redirect/rewrite rules dash-proxy#17 — Redirect/rewrite rules proxy M
• R5: Response compression (gzip/zstd/brotli) dash-proxy#18 — Response compression (gzip/zstd/brotli) proxy S
• R5: Scale-to-zero idle services dash-proxy#19 — Scale-to-zero idle services proxy L
• R5: Observability batch (log format, OTel traceparent, metrics excludes) dash-proxy#20 — Observability batch (log format, OTel traceparent, metrics excludes) proxy S

R6 — Deploy safety for non-proxied roles

Gem-only: no proxy code, so MINIMUM_VERSION does not move and this schedules independently of the proxy release train.

Closes the readiness gap that lets dash stop an old non-proxied container ~7s after the new one merely starts. This is not a Kamal bug — 0efb5ccf ("Remove the healthcheck step") traded it for deploy speed and said so in the commit body, then mitigated it with a barrier that only ever gates on the primary role's healthcheck. A non-primary role's own readiness is still never checked. Realized as a live card decline on a NATS listener role: the old container was SIGTERMed before the new one's subscription existed.

Order matters — #36 first, it unblocks most of the rest.

• R6: First-class role-level healthcheck block for non-proxied roles #36 — First-class role-level healthcheck block for non-proxied roles gem M
• R6: Refuse a non-proxied role with no readiness definition #37 — Refuse a non-proxied role with no readiness definition gem S
• R6: Poller cannot distinguish "no healthcheck" from "healthcheck passed" #38 — Poller cannot distinguish "no healthcheck" from "healthcheck passed" gem S
• R6: ${...} in a role option expands on the deploy host, not in the container #39 — ${...} in a role option expands on the deploy host, not in the container gem S
• R6: Pre-stop hook for non-proxied roles #40 — Pre-stop hook for non-proxied roles gem S
• R6: Per-role rolling boot via SSHKit runner options #41 — Per-role rolling boot via SSHKit runner options gem M
• R6: Healthcheck exec probe for images you cannot rebuild #42 — Healthcheck exec probe for images you cannot rebuild gem S
• R6: Dump container and health logs on any failed boot, not just the primary role #43 — Dump container and health logs on any failed boot, not just the primary role gem S
• R6: kamal doctor — config-only readiness check per role #44 — kamal doctor config-only readiness check per role gem S
• R6: Print readiness source per role in the deploy banner #45 — Print readiness source per role in the deploy banner gem S
• boot.limit percentage counts accessory hosts but slices app hosts #46 — boot.limit percentage counts accessory hosts but slices app hosts gem S
• boot.wait sleeps after the final host group #47 — boot.wait sleeps after the final host group gem S

Suggested landing order: #45 + #43 (visibility, zero behaviour change) → #36 + #39 (the capability) → #37 warn-only + #44 → #38 and #37-as-error behind minimum_version → #40, #41, #42 as needed. #46 and #47 are independent and can go any time.

R7 — kamal-proxy flag coverage

The proxy shipped 23 feat commits since v0.9.2.2 and the gem's config surface never followed. Measured against internal/cmd/:

The run side is the worse of the two, and for a non-obvious reason: proxy.run.options looks like a passthrough for proxy flags but is consumed by docker_options_args — it passes options to docker run, not to kamal-proxy run. run_command_options emits exactly debug, metrics-port and recheck-targets-on-restore. Everything else is reachable only through a flag's env-var fallback, if it has one.

Net effect: most of R3 (rate limiting, IP allow lists, mTLS), most of R5 (cache, compression, headers, redirects, canary, scale-to-zero) and all of the ACME/DNS-01 configuration are built, tested and shipping in the image — and cannot be turned on from deploy.yml.

• Coverage guard: every kamal-proxy flag is mapped or explicitly waived #82 — Coverage guard: every kamal-proxy flag mapped or explicitly waived gem M
• First-class proxy.run.acme block with secrets-backed DNS credentials #73 — First-class proxy.run.acme block with secrets-backed DNS credentials gem M
• Expose on-demand TLS, mTLS client CA and ACME cache path in deploy.yml #74 — On-demand TLS, mTLS client CA, ACME cache path gem M
• Expose the response cache in deploy.yml (policy per service, store proxy-wide) #75 — Response cache (policy per service, store proxy-wide) gem M
• Expose response compression in deploy.yml #76 — Response compression gem S
• Expose rate limiting, IP allow lists and client-IP identification in deploy.yml #77 — Rate limiting, IP allow lists, client-IP identification gem M
• Expose header rules, redirects, rewrites and error interception in deploy.yml #78 — Header rules, redirects, rewrites, error interception gem M
• Expose session affinity and scale-to-zero sleep/wake in deploy.yml #79 — Session affinity and scale-to-zero sleep/wake gem L
• Expose target connection-pool tuning, retry windows and request timeouts in deploy.yml #80 — Target pool tuning, retry windows, request timeouts gem M
• Expose the kamal-proxy run flag surface (26 unreachable) plus a run.flags escape hatch #81 — The kamal-proxy run flag surface + a run.flags escape hatch gem L

Landing order: #82 first — it lands the current gap as issue-linked waivers, which makes every issue after it self-verifying (implement one, delete its waiver, the test proves the flag is really wired). Then #73 (unblocks the DNS epic's value to dash users) and #81 (the run surface, which #73 and #75 both build on). The rest are independent.

Not in this epic: ACME DNS-01 provider coverage in the proxy — zoolutions/dash-proxy#77. That epic adds providers to the Go binary; #73 here makes them configurable. Distinct concerns, distinct repos, no ordering dependency between them.

Release plan

Two tags, ordered — the proxy image must be pullable at the tag MINIMUM_VERSION names before the gem ships.

1. kamal-proxy → v1.0.0.0 — 57 commits since v0.9.2.2 (38 non-merge: 23 feat, 3 fix), spanning R2–R5.
2. dash gem → v3.0.0 — MINIMUM_VERSION moves to v1.0.0.0, plus whatever R7 lands.

Known constraint on v1.0.0.0: Gem::Version.new("1.0.0.0") == Gem::Version.new("1.0.0") — trailing zeros are stripped in comparison, so this tag sorts equal to a future upstream v1.0.0 rather than above it. Consequences, both bounded:

• Kamal::Utils.older_version? would accept a running upstream v1.0.0 proxy as satisfying MINIMUM_VERSION: v1.0.0.0. In practice a dash deploy pulls the ghcr fork image, so this is only reachable if someone points proxy.run.repository at basecamp's.
• The next fork counter (v1.0.0.1) sorts above both, so the invariant self-corrects on the first patch release.

The git tags themselves never collide. Recorded here so the next person reading .claude/rules/git-workflow.md — which specifies counters starting at .1 for exactly this reason — knows the deviation was deliberate.

dash v3.0.0 also departs from the dash-v<upstream>.<n> gem grammar, declaring a fork major rather than tracking upstream's base. Follow-on: lib/kamal/version.rb will now conflict on every upstream sync, where the playbook currently says "take upstream's". Update .claude/rules/upstream-sync.md as part of the release so the conflict rule matches reality.

Generated from ROADMAP.md. Each issue carries its evidence (basecamp issue/PR numbers) and code anchors.