v3.0.0 release gate: LB layering discipline, secret redaction, surface freeze, port_holder, docs tiering

Epic-level gate produced by the 2026-07-30 pre-release audit (18 independent reviewers over every merged PR in both repos since dash 2.12.0: gem #14–#91, proxy #23–#83). Companion proxy gate: zoolutions/dash-proxy#84 (must publish v1.0.0.1 before this repo's release step). Parent epic: #13.

Key fact the whole gate rests on: only dash 2.12.0 is on rubygems (tagged at the loadbalancing merge). Every config key added since — all of R1–R7 — is unreleased. Renames, regroupings and cuts are free right now and breaking forever after. v3.0.0 freezes the surface.

Problem / Goal

The audit verdict: implementation quality is high (proxy Go code repeatedly survived adversarial review; validation/docs machinery is genuinely excellent), but 8 blockers and 33 majors stand between origin/dash and an excellent v3.0.0. One systemic root cause produces most of them: the fork's loadbalancer auto-activates for any >1-host primary role, and the older option groups each got an explicit edge/per-app disposition (host/tls/access-control stripped per-app and re-added at the LB; header/redirect/rewrite rules stripped from the LB, with comments) — but the new groups (cache, compress, session_affinity, sleep, target pool tuning, read-routing, run.acme) got neither, so each is broken, doubled, or undecided in exactly the fork's headline multi-host topology.

Done looks like: every blocker fixed, every major fixed or explicitly waived in writing, the surface renamed/cut per the settled decisions below, docs restructured for progressive disclosure, and bin/release-dash 3.0.0 actually able to run — in that order, gem release last, after ghcr.io/mhenrixon/kamal-proxy:v1.0.0.1 is published.

Context (read these first)

• lib/kamal/configuration/proxy.rb — deploy_options; the load_balancing? strip block for host/tls/access-control/basic-auth is the pattern every new group must join. basic_auth_credential (plaintext leak, see C1). lifecycle_options (unstripped, see B1).
• lib/kamal/configuration/loadbalancer.rb — deploy_options calls super, re-adds tls_options, deletes traffic_options keys with a comment. This file is where every disposition becomes explicit.
• lib/kamal/commands/loadbalancer.rb — run/run_args/volume_mounts: the LB boots kamal-proxy with no run flags and no apps-config mount — root cause of the mTLS blocker, the "acme vanishes at the edge" major, and cache's edge story.
• lib/kamal/configuration/proxy/run.rb — MINIMUM_VERSION, run_command, cache_options, config_digest (its comment explains why ACME values stay out — the pattern C2 extends).
• lib/kamal/configuration/proxy/acme.rb + lib/kamal/cli/proxy.rb — the 0600 --env-file secrets pattern to generalize.
• lib/kamal/cli/app/ssl_certificates.rb — uploads role-scoped TLS material to app hosts only; the mTLS reshape lands here.
• lib/kamal/configuration/validator/proxy.rb + lib/kamal/configuration/docs/proxy.yml — the docs file IS the validation schema (unknown keys error), so doc reordering is free and every rename/cut touches both.
• test/proxy_flag_coverage_test.rb — the flag canary + R7_BACKLOG waivers; regenerate the manifest via bin/sync-proxy-flags at the MINIMUM_VERSION bump.
• lib/kamal/cli/proxy/reboot.rb, lib/kamal/commands/proxy.rb — Phase-1 stop→remove→run reboot only; port_holder (E1) replaces this path.
• .claude/rules/upstream-sync.md, .claude/rules/git-workflow.md, bin/release-dash — all still teach the abandoned dash-v<upstream>.<n> grammar; the release script's regex hard-rejects 3.0.0.

Decision

Chosen approach: fix everything in-release rather than cut features — the maintainer explicitly chose the ambitious option at every fork in the road. The one systemic invention (rather than eight point fixes) is B1+B2: a single explicit per-option-group layering contract enforced by a canary test, mirroring the flag-coverage canary that already guards the proxy flag surface.

Alternatives rejected:

• Cut mTLS / defer auto-reboot / ship-and-file the credential leaks — all offered, all declined; nothing knowingly broken or leaky ships under a 3.0 banner.
• Patch CertificateRegistry in place — rejected in favor of full unification (see kamal-proxy#84).
• Per-app-only or two-tier cache — rejected in favor of edge-only with real LB run plumbing (below).
• advanced: re-nesting of the schema — rejected; flat schema + sectioned docs (F1) is the progressive-disclosure mechanism.

Settled in interview (2026-07-30) — the executor must not re-litigate:

• Release train: proxy v1.0.0.1 (cert unification + proxy test(proxy): fail the build when kamal-proxy grows a flag the gem cannot emit #83 rides along) → MINIMUM_VERSION = v1.0.0.1 → gem 3.0.0. Never retag v1.0.0.0.
• Cert fix scope: unify now (kamal-proxy#84), not patch-in-place.
• Auto-reboot: merge port_holder now (gem PR feat(proxy): zero-downtime generation handoff via port-holder #25 / feat/proxy-auto-reboot branch), keep reboot_on_deploy defaulting true; the default may only stay true because reboots become genuinely zero-downtime.
• mTLS: rework in-release — client_ca_path becomes client_ca_pem (secret name, mirroring ssl.certificate_pem), with a working LB upload path.
• Cache: edge-only under load balancing; plumb LB run flags now so run.cache.store reaches the edge. Single-proxy (no LB) topology unchanged.
• Renames/cuts, all four: tls_domains → ssl_domains; path_timeouts → path_response_timeouts; nest the read-routing trio; cut tls.acme_cache_path, run.cache.lease_ttl, run.cache.lease_wait, and scope_cookie_paths.
• Cache-store credential: env-file now (ACME pattern generalized), not ship-and-file.
• Docs: sectioned reorder of docs/proxy.yml (~6 titled sections) + a ~10-key "Proxy basics" page + a per-key layering table.
• Audit recommendations adopted as-is (no re-open): compression default order zstd, br, gzip; content_types REPLACE semantics; boolean-or-hash only (no bare-list compress); cache enabled:-gating hard error stays (uniform pattern); access control edge-only; AC1 warning scope unchanged; healthcheck literal-/ rejection only; --data-dir stays waived; issue Expose the kamal-proxy run flag surface (26 unreachable) plus a run.flags escape hatch #81 (run.flags) stays post-3.0; negative-timeout validations stay; max_idle_conns: 0 gets a warning; sleep without docker_socket keeps raising; docker_socket stays a one-key convenience.

Implementation steps

Ordered; workstreams B–D are parallelizable after B lands its contract. Proxy work (kamal-proxy#84) proceeds concurrently; only step G waits on its published image.

B — Loadbalancer layering contract (the systemic fix; do first)

1. B1 In Proxy#deploy_options + Loadbalancer#deploy_options, give every option group an explicit disposition with comment + test:
   • Edge (strip per-app, re-add at LB): existing host/tls/ssl_domains/access-control/basic-auth; add session_affinity (today both layers pin with the same cookie name and separate HMAC keys — the inner proxy clobbers the edge pin every other request, breaking affinity in the only topology where it matters), add canonical_host + redirects (today per-app placement behind the LB emits http:// Locations to HTTPS clients — redirectURLIfNeeded consults r.TLS only), add cache policy flags (edge-only decision).
   • Per-app only (strip at LB): existing headers/rewrites/intercept_errors; add sleep/lifecycle flags (the LB has no docker socket and its targets are host addresses, not containers — today root-level sleep: + auto-LB fails every deploy at the LB step), add compress (double-running is safe only by accident of the Content-Encoding guard; align with traffic shaping).
   • Both, deliberately (comment + test): target pool tuning / timeouts (each layer has a real connection pool; state it, don't leave it implicit).
   • Read-routing: edge-only, decided as part of D2.
2. B2 Add a layering canary test: any key emitted by Proxy#deploy_options that has no recorded disposition (edge / per-app / both) fails the build — the flag-coverage-canary pattern applied to layering, so the next feature can't skip the decision.
3. B3 LB run plumbing: Kamal::Commands::Loadbalancer#run gains the proxy's run-command surface (shared run_config bookkeeping from PR feat(proxy): support multiple kamal apps on a shared loadbalancer host #52 already enforces cross-app agreement): cache store env-file (C2), ACME env-file + args (fixes "acme silently vanishes at the edge" — the LB terminates TLS, so it must be able to issue), and the apps-config/persistent volume mounts a dedicated LB host is currently missing (audit: dedicated-LB state and ACME certs are not volume-backed today).

C — Secrets & redaction

4. C1 Basic auth: wrap the credential in Kamal::Utils.sensitive (registry-login precedent) so --basic-auth="admin:s3cr3t" never prints at :info verbosity; optionize needs a Sensitive passthrough like argumentize's sensitive: kwarg. Update test/commands/app_test.rb:202 and test/commands/loadbalancer_test.rb:96, asserting the redacted form.
5. C2 Cache store: deliver via the generalized proxy secrets env file (CACHE_STORE=..., 0600, --env-file — the acme.env mechanism made shared), never on the command line; keep the raw URL out of config_digest (hash presence/name, not value, mirroring acme.credential_names). Reaches both the proxy and the LB (B3). Remove the docs caveat that documented the leak.
6. C3 Delete the proxy secrets env file from hosts when the config block is removed (small; mirrors the upload branches).

D — Surface freeze: renames, cuts, validators

7. D1 Renames: tls_domains → ssl_domains; path_timeouts → path_response_timeouts (flag mapping unchanged). No back-compat aliases — nothing is released.
8. D2 Read-routing: nest read_targets/read_target_websockets/writer_affinity_timeout into one block (proposed shape: read_routing: {targets:, websockets:, writer_affinity_timeout:}), declare it edge-only in B1, add the missing loadbalancer_test.rb coverage.
9. D3 mTLS reshape: replace tls.client_ca_path with a secret-name key client_ca_pem; content read from .kamal/secrets, uploaded via StringIO in SslCertificates (and via the LB path from B3); blank secret raises like basic_auth.password_secret. Drop the config-time File.exist? (which today bricks kamal app logs/rollback on machines without the file). With the separate tls: block now empty (acme_cache_path cut, client_ca_pem moved), fold on_demand_url + client_ca_pem into the ssl: hash and delete the tls: block — TLS then has one naming family (ssl*) instead of five homes.
10. D4 Cuts: tls.acme_cache_path (kamal-proxy#84 verifies the proxy's default already persists in the volume), run.cache.lease_ttl/lease_wait, scope_cookie_paths. Remove keys, docs, validator arms, emitted flags; the flag canary waiver list records each cut with its reason.
11. D5 Validator additions (each one line + one test): canonical_host × on_demand_url conflict (proxy rejects the pair; gem currently lets it fail after the SSH round-trip); canonical_host listed in hosts; client_ip.header without trusted_proxies (today it slips through and becomes a client-spoofable X-Forwarded-For rewrite, contradicting the shipped docs); response_timeout rejects negatives (sibling consistency with request_timeout); warn on target.max_idle_conns: 0 ("means the proxy default of 100, not none — leave unset"); reject = inside redirect/rewrite from: (the <from>=<to> wire format cuts at the first = and silently builds a wrong rule).
12. D6 Fix compress: {enabled: false, encodings: [...]} silently keeping compression on — enabled: false must win.
13. D7 ACME polish: document or drop DNS_PROVIDER_ALIASES (one concept, one name); with B3, the "acme + LB" combination becomes supported rather than silently inert.

E — Auto-reboot completion (zero-downtime for real)

14. E1 Merge the port_holder work (gem PR feat(proxy): zero-downtime generation handoff via port-holder #25, sitting on feat/proxy-auto-reboot) into dash: --reuse-port boot, hold/drain generation handoff, kamal-proxy-next → drain → rename. The published proxy image already carries hold/drain/reuse-port (verified at the v1.0.0.0 tag; v1.0.0.1 keeps them).
15. E2 Replace verify_services' plain-text substring match with kamal-proxy list --json (built for exactly this).
16. E3 Give the auto-activated LB the same drift detection/auto-reboot the proxy hosts get (today it has neither, and no stale-config warning).
17. E4 Integration-test the handoff (needs the published image; runs in G). reboot_on_deploy stays true only if this passes.

F — Docs, release mechanics, bookkeeping

18. F1 Reorder docs/proxy.yml into titled sections — 1 Essentials (host/hosts, ssl, app_port, healthcheck, loadbalancer; ~10 keys), 2 Traffic & routing, 3 Security & access, 4 Performance, 5 Fleet (read routing, affinity, sleep), 6 Proxy container (run) — plus a per-key layering table (edge / per-app / both) and the one-line "run.options are docker run options, not kamal-proxy flags" clarification. Mirror the split as a ~10-key "Proxy basics" docs page + full reference.
19. F2 bin/release-dash: accept the three-segment fork grammar (3.0.0, tag dash-v3.0.0); update .claude/rules/upstream-sync.md (the version.rb conflict rule becomes "keep ours — fork declares its own major"; grammar text) and .claude/rules/git-workflow.md to match reality.
20. F3 Release notes must call out, by name: prune retention now per role+destination (more containers retained on multi-role hosts); accessory tag matching now exact (prefix reliance breaks); the R6 readiness warnings; and that the first deploy after upgrading reboots the proxy fleet once, host by host, zero-downtime via port_holder (digest convergence).
21. F4 File the follow-ups that were promised but never filed: kamal doctor reports whether the running proxy was booted with a docker socket (low-pri); kamal proxy cache CLI (stats + purge — the proxy ships a full cache admin subsystem with zero gem surface; docs currently point operators at raw logs); weighted-canary gem exposure is already tracked (kamal#941 / ROADMAP R5) — don't duplicate.

G — Release (strict order)

22. Wait for ghcr.io/mhenrixon/kamal-proxy:v1.0.0.1 (kamal-proxy#84 step 7), verify amd64+arm64.
23. Bump MINIMUM_VERSION to "v1.0.0.1", regenerate the flag manifest (bin/sync-proxy-flags), let the canary prove the mapping.
24. bin/test (full suite incl. integration — Docker + the published image).
25. bin/release-dash 3.0.0 → tag dash-v3.0.0, push the single tag (git push origin tag dash-v3.0.0), gem push.

Verification gates

• bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].grep_v(/integration/).each { |f| require File.expand_path(f) }' — green (ignore the two known Apple-Silicon-only builder failures).
• bundle exec rubocop --parallel — no offenses.
• New tests to exist and pass: layering canary (B2); redacted basic-auth assertion (C1); no credential in any printed command or config_digest input (C2); one LB-topology test per re-homed group (affinity, canonical_host/redirects, cache-at-edge, sleep-stripped, read-routing); every validator addition in D5; port_holder handoff integration test (E4).
• bin/test full suite against v1.0.0.1 before step 25.
• Proxy version assertions interpolate Kamal::Configuration::Proxy::Run::MINIMUM_VERSION — never a literal tag.

Out of scope

• Issue Expose the kamal-proxy run flag surface (26 unreachable) plus a run.flags escape hatch #81 (run.flags escape hatch + remaining waived run flags) — explicitly post-3.0; waivers stay.
• No retag of the published v1.0.0.0 image; no gem-side mention of the file:// cache store until v1.0.0.1 is MINIMUM_VERSION.
• No advanced: re-nesting of the schema; no docs-file split requiring loader changes.
• No edits to kamal.gemspec, bin/release, or other upstream-owned files; no commits to main; no plain v* tags; no git push --tags.
• Per-host allow lists as defence-in-depth behind the LB — possible later as an explicit opt-in, not in this gate.

Execution

Work in slices, one PR per workstream letter (B first), branches off dash, PRs into dash (--repo mhenrixon/kamal --base dash). Hand each slice to a fresh implementation session on the sonnet tier; kamal-proxy#84 runs in ../kamal-proxy in parallel. TDD per .claude/rules/testing.md.