diff --git a/.claude/rules/git-workflow.md b/.claude/rules/git-workflow.md index 4a9b6958..c7eca20a 100644 --- a/.claude/rules/git-workflow.md +++ b/.claude/rules/git-workflow.md @@ -79,7 +79,7 @@ Two builder tests are known-failing on Apple Silicon only (host-arch dependent) | What | Grammar | Example | Push | |---|---|---|---| -| Gem (this repo) | `dash-v.` | `dash-v2.12.0.1` | `git push origin tag dash-v2.12.0.1` | +| Gem (this repo) | `dash-v` (own major from 3.0.0) | `dash-v3.0.0` | `git push origin tag dash-v3.0.0` | | Proxy image (kamal-proxy) | `v.` | `v0.9.2.1` | `git push origin tag v0.9.2.1` | - **NEVER** a bare `v` tag — upstream owns that namespace on both repos @@ -96,7 +96,7 @@ script/release-dash v0.9.2.1 # 2. this repo, on dash — confirm MINIMUM_VERSION matches, then: bin/test -bin/release-dash 2.12.0.1 +bin/release-dash 3.0.0 ``` Full procedure, conflict playbook, and sync runbook: `.claude/rules/upstream-sync.md`. diff --git a/.claude/rules/upstream-sync.md b/.claude/rules/upstream-sync.md index 49bb28ba..ef409c24 100644 --- a/.claude/rules/upstream-sync.md +++ b/.claude/rules/upstream-sync.md @@ -25,7 +25,7 @@ git checkout dash && git merge feat/loadbalancing && && git push | File | Resolution | |---|---| -| `lib/kamal/version.rb` | take upstream's — the fork version is only ever written by `bin/release-dash` at release time | +| `lib/kamal/version.rb` | keep ours — from 3.0.0 the fork declares its own major, so upstream's version never lands here; `bin/release-dash` is the only writer | | `Gemfile.lock` | take either side, run `bundle install`, commit the result | | `lib/kamal/configuration/proxy/run.rb` (`MINIMUM_VERSION`) | upstream bumped their proxy: release the proxy fork first (`v.1`), then set that tag here | | `lib/kamal/configuration/proxy/run.rb` (repository) / `boot.rb` (`repository_name`) | keep `ghcr.io/mhenrixon` | @@ -47,10 +47,10 @@ docker buildx imagetools inspect ghcr.io/mhenrixon/kamal-proxy:v0.9.2.1 # amd6 # 2. this repo, on dash: # ensure Kamal::Configuration::Proxy::Run::MINIMUM_VERSION == that tag bin/test # full suite incl. integration -bin/release-dash 2.12.0.1 # version.rb + Gemfile.lock, tag dash-v2.12.0.1, gem push dash +bin/release-dash 3.0.0 # version.rb + Gemfile.lock, tag dash-v3.0.0, gem push dash ``` -Tag grammar: gem tags `dash-v.`, proxy image tags `v.`. Gem versions are four-segment `.` — `Gem::Version` sorts them above the upstream base and below its next release. Never `-suffix` tags: Gem::Version treats `-` as a prerelease marker that sorts BELOW the base and breaks the proxy minimum-version check. +Tag grammar: gem tags `dash-v`; from 3.0.0 the gem version is plain three-segment semver (`3.0.0`, tag `dash-v3.0.0`) — the fork declares its own major and no longer tracks upstream's number (the legacy four-segment `.` grammar covers only the pre-3.0 line). Proxy image tags stay `v.` — they still track upstream kamal-proxy releases. Never `-suffix` tags: Gem::Version treats `-` as a prerelease marker that sorts BELOW the base and breaks the proxy minimum-version check. ## Upstreaming a feature @@ -74,6 +74,6 @@ Rejected-by-basecamp features (see `ROADMAP.md`'s "safe moat" list) are never up ## Never - commit to `main` or rebase published branches -- `git push --tags` — single-tag pushes only (`git push origin tag dash-v2.12.0.1`) +- `git push --tags` — single-tag pushes only (`git push origin tag dash-v3.0.0`) - release the gem while `MINIMUM_VERSION` names an unpublished proxy tag - edit `kamal.gemspec`, `bin/release`, or other upstream-owned files diff --git a/CLAUDE.md b/CLAUDE.md index 9b0dab2a..b3fd6ee5 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -16,8 +16,8 @@ mhenrixon's maintained fork of [kamal](https://github.com/basecamp/kamal) — de 1. **NO commits on `main`** — it is a fast-forward-only mirror of basecamp/kamal 2. **NO edits to `kamal.gemspec` or `bin/release`** — upstream files kept pristine so syncs never conflict; the fork owns `dash.gemspec` and `bin/release-dash` -3. **NO `v*` git tags** — upstream owns that namespace; fork gem tags are `dash-v` -4. **NO `git push --tags`** — it would push fetched upstream tags to the fork; push single tags (`git push origin tag dash-v2.12.0.1`) +3. **NO `v*` git tags** — upstream owns that namespace; fork gem tags are `dash-v` (own semver major from 3.0.0) +4. **NO `git push --tags`** — it would push fetched upstream tags to the fork; push single tags (`git push origin tag dash-v3.0.0`) 5. **NO suffix proxy versions** like `v0.9.2-dash.1` — Gem::Version parses `-` as a prerelease, which sorts OLDER than the base and hard-fails `kamal proxy boot` 6. **NO gem release before the proxy image exists** — the tag named by `Kamal::Configuration::Proxy::Run::MINIMUM_VERSION` must be pullable from ghcr.io first 7. **NO rebasing published branches** — merge forward; history is shared @@ -36,7 +36,7 @@ mhenrixon's maintained fork of [kamal](https://github.com/basecamp/kamal) — de bin/test # Full suite (integration needs Docker + published proxy image) bundle exec ruby -Itest -e 'Dir["test/**/*_test.rb"].grep_v(/integration/).each { |f| require File.expand_path(f) }' # Unit tests only bundle exec rubocop --parallel # Lint -bin/release-dash 2.12.0.1 # Release the dash gem (proxy image must exist first) +bin/release-dash 3.0.0 # Release the dash gem (proxy image must exist first) git fetch upstream --tags --prune # Start of every sync ``` diff --git a/bin/release-dash b/bin/release-dash index 3d973f63..28f39ea4 100755 --- a/bin/release-dash +++ b/bin/release-dash @@ -2,7 +2,10 @@ # Release the fork gem "dash" to rubygems.org. # -# Versions are four-segment: ., e.g. 2.12.0.1. +# From 3.0.0 the fork declares its own major: versions are ordinary +# three-segment semver (e.g. 3.0.0) and no longer track upstream's number. +# The legacy four-segment . grammar (e.g. 2.12.0.1) stays +# accepted for patch releases of the pre-3.0 line. # Git tags are dash-v - upstream owns the v namespace. # # Ordering rule: the proxy image tag named by @@ -13,8 +16,8 @@ set -euo pipefail VERSION=$1 -if ! echo "$VERSION" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+\.[0-9]+$'; then - echo "Error: version must be four-segment, e.g. 2.12.0.1" +if ! echo "$VERSION" | grep -Eq '^[0-9]+\.[0-9]+\.[0-9]+(\.[0-9]+)?$'; then + echo "Error: version must be three-segment (3.0.0) or the legacy four-segment form (2.12.0.1)" exit 1 fi diff --git a/docs/proxy-basics.md b/docs/proxy-basics.md new file mode 100644 index 00000000..fad32a8b --- /dev/null +++ b/docs/proxy-basics.md @@ -0,0 +1,62 @@ +# Proxy basics + +The ten `proxy:` keys most apps ever need. The full reference — six tiers, +from these essentials down to the proxy container's own runtime — lives in +[`lib/kamal/configuration/docs/proxy.yml`](../lib/kamal/configuration/docs/proxy.yml), +which is also the validation schema: any key that file doesn't show is a key +kamal rejects. + +```yaml +proxy: + # Which hostnames route to this app. One of host / hosts. + host: app.example.com + hosts: + - app.example.com + - www.example.com + + # Automatic HTTPS via Let's Encrypt (needs a host and an open :443), + # or a hash for custom certificates / on-demand TLS / mTLS. + ssl: true + + # Redirect HTTP to HTTPS (default true when ssl is on). + ssl_redirect: true + + # Use Let's Encrypt's staging environment while testing. + ssl_staging: true + + # The port your app container listens on (default 80). + app_port: 3000 + + # What the proxy polls until the new container is ready. + healthcheck: + path: /up + interval: 1 + timeout: 5 + + # Multi-host apps: which host fronts the fleet. Auto-activates on the + # primary role's first host when it has more than one host; set false to + # opt out, or name a dedicated machine. + loadbalancer: lb.example.com + + # Reboot the proxy automatically when its configuration drifts + # (default true; zero-downtime with run.port_holder). + reboot_on_deploy: true +``` + +## Where options apply with a loadbalancer + +Every proxy option lives at exactly one layer — the loadbalancer (TLS, +access control, caching, affinity), the per-host proxies (headers, rewrites, +compression, sleep), or deliberately both (health checks, timeouts, +buffering). You don't place them; kamal does. The per-key table is at the top +of the full reference. + +## When you need more + +| You want | Look at | +|---|---| +| Header rules, redirects, rewrites, canonical host | §2 Traffic & routing | +| Basic auth, IP allow lists, rate limiting, dynamic TLS domains | §3 Security & access | +| Timeouts, connection pools, buffering, compression, response caching | §4 Performance & observability | +| Read/write splitting, session affinity, scale-to-zero | §5 Fleet | +| Ports, ACME/Let's Encrypt DNS credentials, cache store, zero-downtime reboots, escape hatches | §6 Proxy container (`run`) | diff --git a/docs/release-notes/3.0.0.md b/docs/release-notes/3.0.0.md new file mode 100644 index 00000000..d83996df --- /dev/null +++ b/docs/release-notes/3.0.0.md @@ -0,0 +1,76 @@ +# dash 3.0.0 + +The first release where the fork declares its own major. Pairs with +`ghcr.io/mhenrixon/kamal-proxy:v1.0.0.1` — the gem will not boot an older +proxy. Everything in R1–R7 plus the v3.0.0 gate (#93) ships here; only +`dash 2.12.0` was ever on rubygems, so this is the surface's first and final +freeze. + +## Act on these (behavior changes from 2.12.0) + +- **The first deploy after upgrading reboots the proxy fleet once, host by + host.** The drift digest's composition changed (digest schema v1 + + secret-name hashing), so every host converges on the new format with one + reboot. With `proxy: run: port_holder: true` that reboot is zero-downtime — + generations overlap on the published ports via the port-holder handoff. + Without it (and for the loadbalancer container) it is a brief per-host gap. + Adopting port_holder itself takes one final brief-gap reboot; after that, + config and version reboots never drop a request. +- **Prune retention is now per role and destination.** `kamal prune` + previously kept the newest N containers per *host*; it now keeps N per + role+destination, so multi-role hosts retain more containers than before. + If your hosts are disk-tight, lower `retain_containers`. +- **Accessory image tag matching is now exact.** Accessory commands + previously matched running containers by image-name *prefix*; a deployment + relying on prefix matching (e.g. distinguishing `db` from `db-backup` by + prefix alone) must name images exactly. +- **Readiness warnings got teeth (R6).** Non-proxied roles without a + `healthcheck:` block, `boot/wait` without `boot/limit`, rate limiting that + cannot identify clients, `intercept_errors` without `error_pages_path`, and + `target/max_idle_conns: 0` now warn loudly at config time. The healthcheck + warning will become an error in a later release — add a block or opt out + with `healthcheck: false`. + +## Renames and cuts (config update required) + +No aliases — the old spellings fail validation with an unknown-key error: + +| 2.12.0-era key | 3.0.0 | +|---|---| +| `tls_domains` | `ssl_domains` | +| `path_timeouts` | `path_response_timeouts` | +| `read_targets` / `read_target_websockets` / `writer_affinity_timeout` | `read_routing: {targets, websockets, writer_affinity_timeout}` | +| `tls.on_demand_url` | `ssl.on_demand_url` | +| `tls.client_ca_path` (local file path) | `ssl.client_ca_pem` (secret name in `.kamal/secrets`) | +| `tls.acme_cache_path` | cut — the proxy default already persists in the config volume | +| `run.cache.lease_ttl`, `run.cache.lease_wait` | cut — proxy defaults; `run.flags` is the escape hatch | +| `scope_cookie_paths` | cut | + +## Highlights + +- **Loadbalancer layering contract.** Every proxy option now lives at exactly + one layer under load balancing — edge, per-app, or deliberately both — + enforced by a canary test and documented per key at the top of the proxy + reference. Session affinity, canonical-host redirects, response caching and + read routing move to the edge; sleep and compression stay next to the app. +- **Secrets stay secret.** Basic-auth credentials print as `[REDACTED]`; + the cache store URL travels in a 0600 env file (`CACHE_STORE`) instead of + the `docker run` command line; ACME DNS credentials share the same file. + Rotating a secret's *value* doesn't move the drift digest — run + `kamal proxy reboot` after a rotation. +- **Zero-downtime proxy reboots** (`run.port_holder: true`) via the + generation handoff; reboots are verified against `kamal-proxy list --json`. +- **The loadbalancer grew up:** it boots with the full proxy run surface + (ACME issuance, cache store, apps-config mount, state volume at the right + path — dedicated-host routes now survive reboots), gets TLS material + uploaded to its host, and has the same drift detection/auto-reboot as the + proxy hosts. +- **Progressive-disclosure docs**: a ten-key [Proxy basics](../proxy-basics.md) + page, and a reference reorganized into six tiers from essentials to the + proxy container's runtime. + +## Versioning + +From 3.0.0 the gem uses its own three-segment semver (`dash-v3.0.0` tags) and +no longer tracks upstream kamal's version. Proxy image tags still track +upstream kamal-proxy (`v.`, e.g. `v1.0.0.1`). diff --git a/lib/kamal/configuration/docs/proxy.yml b/lib/kamal/configuration/docs/proxy.yml index 2c0c68e6..61f043d2 100644 --- a/lib/kamal/configuration/docs/proxy.yml +++ b/lib/kamal/configuration/docs/proxy.yml @@ -12,6 +12,36 @@ # proxy: + + # Where each option lives when a loadbalancer fronts the fleet + # + # Every deploy option has exactly one home (see + # Kamal::Configuration::Proxy::DEPLOY_OPTION_DISPOSITIONS - the layering + # contract that enforces this): + # + # edge - applied only by the loadbalancer, stripped from the per-host + # proxies: host/hosts, ssl (certificates, on-demand, mTLS), + # ssl_redirect, ssl_staging, ssl_domains, basic_auth, + # allow_ips, client_ip, rate_limit, session_affinity, + # canonical_host, redirects, cache, read_routing + # + # per-app - applied only by the per-host proxies, next to the app: + # headers, rewrites, intercept_errors, sleep, compress + # + # both - each layer runs its own copy, deliberately: healthcheck, + # response/request timeouts, path timeouts, target pool + # tuning, buffering, path_prefix, strip_path_prefix, + # forward_headers, logging, exclude_metrics_paths + # + # Without a loadbalancer the single proxy is every layer at once and the + # whole surface applies to it. `loadbalancer`, `reboot_on_deploy` and `run` + # configure the containers themselves rather than a deployment, so they + # have no layer. + + # ========================================================================== + # 1. Essentials — enough for most apps + # ========================================================================== + # Hosts # # The hosts that will be used to serve the app. The proxy will only route requests @@ -22,10 +52,11 @@ proxy: # # Specify one of `host` or `hosts`. host: foo.example.com + hosts: - foo.example.com - bar.example.com - + # Loadbalancer # # Specify a host to run the loadbalancer on. The loadbalancer will distribute requests @@ -160,6 +191,157 @@ proxy: # Defaults to `false`: ssl_staging: true + # Healthcheck + # + # When deploying, the proxy will by default hit `/up` once every second until we hit + # the deploy timeout, with a 5-second timeout for each request. + # + # Once the app is up, the proxy will stop hitting the healthcheck endpoint. + # + # By default, the healthcheck is sent to the app port. Set `port` to check a + # different port on the container, and `host` to set the Host header sent with + # healthcheck requests. + healthcheck: + interval: 3 + path: /health + timeout: 3 + port: 3001 + host: health.example.com + + + # ========================================================================== + # 2. Traffic & routing + # ========================================================================== + + # Path-based routing + # + # For applications that split their traffic to different services based on the request path, + # you can use path-based routing to mount services under different path prefixes. + # Usage sample: path_prefix: '/api' + # + # You can also specify multiple paths in two ways. + # + # When using path_prefix you can supply multiple routes separated by commas. + path_prefix: "/api,/oauth_callback" + + # You can also specify paths as a list of paths, the configuration will be + # rolled together into a comma separated string. + path_prefixes: + - "/api" + - "/oauth_callback" + + # By default, the path prefix will be stripped from the request before it is forwarded upstream. + # + # So in the example above, a request to /api/users/123 will be forwarded to web-1 as /users/123. + # + # To instead forward the request with the original path (including the prefix), + # specify --strip-path-prefix=false + strip_path_prefix: false + + # Forward headers + # + # Whether to forward the `X-Forwarded-For` and `X-Forwarded-Proto` headers. + # + # If you are behind a trusted proxy, you can set this to `true` to forward the headers. + # + # By default, kamal-proxy will not forward the headers if the `ssl` option is set to `true`, and + # will forward them if it is set to `false`. + forward_headers: true + + # Header rules + # + # Rewrite headers on their way to the app and on their way back out, without + # the app knowing. `set` replaces whatever was there, `add` appends and keeps + # it, `remove` strips it. + # + # Names are canonicalised, and values have their leading and trailing + # whitespace trimmed. A value may contain colons — a CSP naming a scheme or a + # port survives intact. + # + # Two things kamal rejects rather than letting them fail late: a value with a + # newline or carriage return in it (that is response splitting, and shell + # escaping would silently turn it into a literal backslash-n), and a *request* + # rule naming `Host` — Go carries the host outside the header map, so the rule + # would do nothing at all. + # + # Response rules apply to what the app returned. Error pages, redirects, and + # auth or rate-limit rejections come from the proxy itself and are unaffected. + headers: + request: + set: + X-Forwarded-Host: app.example.com + add: + X-Request-Source: kamal + remove: + - X-Internal-Token + response: + set: + Strict-Transport-Security: max-age=31536000 + add: + X-Served-By: kamal-proxy + remove: + - Server + + # Redirects and rewrites + # + # A redirect answers the client with a `Location`; a rewrite changes the path + # the app receives while the client's URL stays as it was — which is what an + # SPA serving its own routes out of `/index.html` needs. + # + # `from` is a regular expression **anchored to the whole path**, so `/old` does + # not fire on `/not-old-either`. `to` is a path on this host, and for a + # redirect may also be a full http(s) URL. Captures are available as `$1`, + # `$2`, … Rules are tried in order and the first match wins. + # + # ### Watch the path prefix + # + # Both match the path **the client asked for, before `path_prefix` stripping**. + # An app mounted at `path_prefix: /api` with `strip_path_prefix` on sees + # `/users`, but a rule here still has to be written against `/api/users`. This + # is the part that is easy to get wrong. + # + # `status` applies to redirects only and must be 301, 302, 303, 307 or 308. + # It defaults to 301, so say 302 explicitly for anything you may want back. + redirects: + - from: /old + to: /new + - from: /gone/(.*) + to: https://elsewhere.example.com/$1 + status: 302 + + rewrites: + - from: /api/(.*) + to: /v2/$1 + + # Canonical host + # + # Redirect every request to this host, to force apex or www one way. + canonical_host: www.example.com + + # Intercept error statuses + # + # Replace these statuses coming from the app with the proxy's own error pages, + # discarding whatever body the app sent. 4xx and 5xx codes only. + # + # This pairs with the root-level `error_pages_path`. It works without it, but + # not usefully: with no pages to render, the proxy falls back to a bare + # plaintext status line, so the app's own error page is thrown away and + # replaced by the words "Bad Gateway". Kamal warns when you do that. + intercept_errors: + - 502 + - 503 + # + # ### Note + # Everything in this section applies on the per-host proxy, including when a + # loadbalancer is configured — unlike TLS and access control, which move to the + # loadbalancer. Applying them at both layers would append an `add` header twice + # and run a rewrite over its own output. + + + # ========================================================================== + # 3. Security & access + # ========================================================================== + # Basic auth # # Require HTTP Basic credentials on every request to this service. Requests @@ -222,26 +404,76 @@ proxy: interval: 300 batch_size: 1 - # Forward headers + # Who the client is # - # Whether to forward the `X-Forwarded-For` and `X-Forwarded-Proto` headers. + # Rate limiting and IP allow lists are both only as correct as the address they + # key on, so configure this first if anything sits in front of kamal-proxy. # - # If you are behind a trusted proxy, you can set this to `true` to forward the headers. + # With no `trusted_proxies`, the client is always the address that opened the + # connection — nothing a client sends can influence it, which is what makes the + # allow list meaningful. # - # By default, kamal-proxy will not forward the headers if the `ssl` option is set to `true`, and - # will forward them if it is set to `false`. - forward_headers: true + # Once you declare `trusted_proxies`, and only when the connecting address is + # one of them, kamal-proxy reads the forwarded chain instead: it walks the + # chain from the nearest hop backwards past every proxy you declared, and the + # first address none of your proxies wrote is the client. **List every hop**, + # not only the one that connects to kamal-proxy — a chain it cannot resolve + # denies the request rather than falling back to the connecting address. + # + # `header` names the header carrying the original client IP (`CF-Connecting-IP` + # behind Cloudflare, `True-Client-IP` behind some others); kamal-proxy reads it + # instead of `X-Forwarded-For`. It is only honoured when `trusted_proxies` is + # set, because otherwise it is just something the client wrote — kamal rejects + # that combination rather than appearing to honour it. + # + # Addresses and ranges are plain IPv4/IPv6 or CIDR. Write IPv4 as IPv4, not as + # an IPv4-mapped IPv6 range, and leave IPv6 zones off — neither matches + # anything. If clients reach you over IPv6, list IPv6 ranges too, or they are + # denied. + client_ip: + header: CF-Connecting-IP + trusted_proxies: + - 173.245.48.0/20 + - 2400:cb00::/32 - # Paths to leave out of the Prometheus metrics + # Rate limiting # - # Request paths that should not be counted, typically health and readiness - # endpoints that would otherwise dominate the histograms. + # A per-client token bucket. Requests over the limit get a 429. IPv6 clients + # are counted per /64, since one client can pick any address inside its own. # - # This is a per-service deploy setting even though it reads like metrics - # configuration — where the metrics are served and who may read them are - # proxy-wide and live under `run/metrics_port` and `run/metrics_allow_ips`. - exclude_metrics_paths: - - /up + # `requests` is requests per second and may be fractional — 0.5 is one request + # every two seconds. `burst` is how many requests a client may make back to + # back before the limit applies (default: the rate, rounded up). `exempt` lists + # addresses and ranges the limit skips, for monitoring and health probes. + rate_limit: + requests: 100 + burst: 20 + exempt: + - 10.0.0.0/8 + + # IP allow list + # + # Serve this service only to these addresses and ranges; everything else gets a + # 403. Combine with `client_ip` above when you are behind a CDN, or the list is + # matched against the CDN's addresses rather than your visitors'. + allow_ips: + - 10.0.0.0/8 + - 192.168.0.0/16 + # + # ### Notes + # - The health check path is served without an address check and without a rate + # limit, so it stays reachable during a deploy. That means it cannot be `/` — + # kamal rejects `healthcheck: path: /` while either feature is on, because it + # would leave the whole service open. The default `/up` is fine. + # - When a loadbalancer is configured, all of this moves to the loadbalancer: + # the per-host proxies see the loadbalancer as their peer, so an allow list + # there would refuse every request and one rate limiter would count the whole + # fleet as a single client. + + + # ========================================================================== + # 4. Performance & observability + # ========================================================================== # Two different deadlines # @@ -257,6 +489,7 @@ proxy: # `response_timeout` defaults to 30 seconds; `request_timeout` defaults to 0, # meaning no limit. response_timeout: 10 + request_timeout: 30 # Per-path timeouts @@ -267,6 +500,7 @@ proxy: path_response_timeouts: "/api/reports": "5m" "/stream": 0 + path_request_timeouts: "/uploads": "10m" "/stream": 0 @@ -316,75 +550,12 @@ proxy: try_duration: 30 try_interval: 1 - # Path-based routing - # - # For applications that split their traffic to different services based on the request path, - # you can use path-based routing to mount services under different path prefixes. - # Usage sample: path_prefix: '/api' - # - # You can also specify multiple paths in two ways. + # Buffering # - # When using path_prefix you can supply multiple routes separated by commas. - path_prefix: "/api,/oauth_callback" - # You can also specify paths as a list of paths, the configuration will be - # rolled together into a comma separated string. - path_prefixes: - - "/api" - - "/oauth_callback" - # By default, the path prefix will be stripped from the request before it is forwarded upstream. + # Whether to buffer request and response bodies in the proxy. # - # So in the example above, a request to /api/users/123 will be forwarded to web-1 as /users/123. - # - # To instead forward the request with the original path (including the prefix), - # specify --strip-path-prefix=false - strip_path_prefix: false - - # Read-only targets - # - # kamal-proxy can split traffic between the deployed (writer) targets and a - # set of read-only targets, e.g. app instances backed by database replicas. - # Read requests are routed to the read targets; write requests always go to - # the writers. - # - # Targets are host:port addresses reachable from the proxy. - # - # `websockets` routes WebSocket traffic to the read targets too (default - # `false`). `writer_affinity_timeout` is how long, in seconds, a client's - # reads stick to the writer after it makes a write, so clients always read - # their own writes (default 1 second). - # - # When a loadbalancer fronts the fleet, read routing is decided there — at - # the only layer that sees the whole fleet. - read_routing: - targets: - - 192.168.0.2:3000 - - 192.168.0.3:3000 - websockets: true - writer_affinity_timeout: 10 - - # Healthcheck - # - # When deploying, the proxy will by default hit `/up` once every second until we hit - # the deploy timeout, with a 5-second timeout for each request. - # - # Once the app is up, the proxy will stop hitting the healthcheck endpoint. - # - # By default, the healthcheck is sent to the app port. Set `port` to check a - # different port on the container, and `host` to set the Host header sent with - # healthcheck requests. - healthcheck: - interval: 3 - path: /health - timeout: 3 - port: 3001 - host: health.example.com - - # Buffering - # - # Whether to buffer request and response bodies in the proxy. - # - # By default, buffering is enabled with a max request body size of 1GB and no limit - # for response size. + # By default, buffering is enabled with a max request body size of 1GB and no limit + # for response size. # # You can also set the memory limit for buffering, which defaults to 1MB; anything # larger than that is written to disk. @@ -395,217 +566,6 @@ proxy: max_response_body: 0 memory: 2_000_000 - # Session affinity - # - # Keep each client on the target that first served it, for apps holding session - # state in the instance. Off by default, and rightly so — every request is - # otherwise free to go to whichever target is best placed to serve it. - # - # The client carries an opaque HttpOnly cookie naming its target. When that - # target leaves the pool the next request falls through to another one and is - # re-pinned, so a deploy does not strand anybody. Reads served by a - # `read_targets` replica are never pinned. - # - # `cookie` renames the pin cookie; kamal-proxy picks a sensible default. - session_affinity: - enabled: true - cookie: _kamal_affinity - - # Scale to zero - # - # Stop this service's containers after `after` seconds with no traffic, and - # start them again on the next request, which is held until they are healthy. - # Health checks and the proxy's own TLS probes are not traffic and never wake a - # sleeping service. - # - # `wake_timeout` is how long a request waits for the containers to come back - # before giving up with a 503. `containers` names the containers to stop and - # start, replacing what the proxy infers from the target address — needed when - # a target names a network alias rather than a container. - # - # ### This needs the container runtime socket - # - # Stopping and starting containers means talking to the runtime, so the proxy - # must have been booted with `run/docker_socket` set. That is a boot-time - # prerequisite for a deploy-time setting, so kamal checks it while reading this - # file rather than letting the first request hang. - # - # Setting `run/docker_socket` also mounts that socket into the proxy container - # — the flag alone only says where to look. **Reaching the container runtime - # socket is root-equivalent on the host**, which is why it is a separate, - # explicit setting and not something enabling sleep does for you. - # - # Not compatible with `tls/on_demand_url`: a sleeping target cannot answer the - # ask endpoint, and waking one would let any hostname on the internet start a - # container. - sleep: - after: 300 - wake_timeout: 30 - containers: - - app-web - - # Header rules - # - # Rewrite headers on their way to the app and on their way back out, without - # the app knowing. `set` replaces whatever was there, `add` appends and keeps - # it, `remove` strips it. - # - # Names are canonicalised, and values have their leading and trailing - # whitespace trimmed. A value may contain colons — a CSP naming a scheme or a - # port survives intact. - # - # Two things kamal rejects rather than letting them fail late: a value with a - # newline or carriage return in it (that is response splitting, and shell - # escaping would silently turn it into a literal backslash-n), and a *request* - # rule naming `Host` — Go carries the host outside the header map, so the rule - # would do nothing at all. - # - # Response rules apply to what the app returned. Error pages, redirects, and - # auth or rate-limit rejections come from the proxy itself and are unaffected. - headers: - request: - set: - X-Forwarded-Host: app.example.com - add: - X-Request-Source: kamal - remove: - - X-Internal-Token - response: - set: - Strict-Transport-Security: max-age=31536000 - add: - X-Served-By: kamal-proxy - remove: - - Server - - # Redirects and rewrites - # - # A redirect answers the client with a `Location`; a rewrite changes the path - # the app receives while the client's URL stays as it was — which is what an - # SPA serving its own routes out of `/index.html` needs. - # - # `from` is a regular expression **anchored to the whole path**, so `/old` does - # not fire on `/not-old-either`. `to` is a path on this host, and for a - # redirect may also be a full http(s) URL. Captures are available as `$1`, - # `$2`, … Rules are tried in order and the first match wins. - # - # ### Watch the path prefix - # - # Both match the path **the client asked for, before `path_prefix` stripping**. - # An app mounted at `path_prefix: /api` with `strip_path_prefix` on sees - # `/users`, but a rule here still has to be written against `/api/users`. This - # is the part that is easy to get wrong. - # - # `status` applies to redirects only and must be 301, 302, 303, 307 or 308. - # It defaults to 301, so say 302 explicitly for anything you may want back. - redirects: - - from: /old - to: /new - - from: /gone/(.*) - to: https://elsewhere.example.com/$1 - status: 302 - rewrites: - - from: /api/(.*) - to: /v2/$1 - - # Canonical host - # - # Redirect every request to this host, to force apex or www one way. - canonical_host: www.example.com - - # Scope cookie paths to the path prefix - # - # When the app is mounted under `path_prefix`, rewrite the `Path` of cookies it - # sets so they are scoped to the prefix rather than to the whole host. - # - # Defaults to `false`: - scope_cookie_paths: true - - # Intercept error statuses - # - # Replace these statuses coming from the app with the proxy's own error pages, - # discarding whatever body the app sent. 4xx and 5xx codes only. - # - # This pairs with the root-level `error_pages_path`. It works without it, but - # not usefully: with no pages to render, the proxy falls back to a bare - # plaintext status line, so the app's own error page is thrown away and - # replaced by the words "Bad Gateway". Kamal warns when you do that. - intercept_errors: - - 502 - - 503 - # - # ### Note - # Everything in this section applies on the per-host proxy, including when a - # loadbalancer is configured — unlike TLS and access control, which move to the - # loadbalancer. Applying them at both layers would append an `add` header twice - # and run a rewrite over its own output. - - # Who the client is - # - # Rate limiting and IP allow lists are both only as correct as the address they - # key on, so configure this first if anything sits in front of kamal-proxy. - # - # With no `trusted_proxies`, the client is always the address that opened the - # connection — nothing a client sends can influence it, which is what makes the - # allow list meaningful. - # - # Once you declare `trusted_proxies`, and only when the connecting address is - # one of them, kamal-proxy reads the forwarded chain instead: it walks the - # chain from the nearest hop backwards past every proxy you declared, and the - # first address none of your proxies wrote is the client. **List every hop**, - # not only the one that connects to kamal-proxy — a chain it cannot resolve - # denies the request rather than falling back to the connecting address. - # - # `header` names the header carrying the original client IP (`CF-Connecting-IP` - # behind Cloudflare, `True-Client-IP` behind some others); kamal-proxy reads it - # instead of `X-Forwarded-For`. It is only honoured when `trusted_proxies` is - # set, because otherwise it is just something the client wrote — kamal rejects - # that combination rather than appearing to honour it. - # - # Addresses and ranges are plain IPv4/IPv6 or CIDR. Write IPv4 as IPv4, not as - # an IPv4-mapped IPv6 range, and leave IPv6 zones off — neither matches - # anything. If clients reach you over IPv6, list IPv6 ranges too, or they are - # denied. - client_ip: - header: CF-Connecting-IP - trusted_proxies: - - 173.245.48.0/20 - - 2400:cb00::/32 - - # Rate limiting - # - # A per-client token bucket. Requests over the limit get a 429. IPv6 clients - # are counted per /64, since one client can pick any address inside its own. - # - # `requests` is requests per second and may be fractional — 0.5 is one request - # every two seconds. `burst` is how many requests a client may make back to - # back before the limit applies (default: the rate, rounded up). `exempt` lists - # addresses and ranges the limit skips, for monitoring and health probes. - rate_limit: - requests: 100 - burst: 20 - exempt: - - 10.0.0.0/8 - - # IP allow list - # - # Serve this service only to these addresses and ranges; everything else gets a - # 403. Combine with `client_ip` above when you are behind a CDN, or the list is - # matched against the CDN's addresses rather than your visitors'. - allow_ips: - - 10.0.0.0/8 - - 192.168.0.0/16 - # - # ### Notes - # - The health check path is served without an address check and without a rate - # limit, so it stays reachable during a deploy. That means it cannot be `/` — - # kamal rejects `healthcheck: path: /` while either feature is on, because it - # would leave the whole service open. The default `/up` is fine. - # - When a loadbalancer is configured, all of this moves to the loadbalancer: - # the per-host proxies see the loadbalancer as their peer, so an allow list - # there would refuse every request and one rate limiter would count the whole - # fleet as a single client. - # Response compression # # Serve gzip, brotli or zstd without the app knowing about it. Responses the @@ -693,6 +653,17 @@ proxy: - locale allow_set_cookie: false + # Paths to leave out of the Prometheus metrics + # + # Request paths that should not be counted, typically health and readiness + # endpoints that would otherwise dominate the histograms. + # + # This is a per-service deploy setting even though it reads like metrics + # configuration — where the metrics are served and who may read them are + # proxy-wide and live under `run/metrics_port` and `run/metrics_allow_ips`. + exclude_metrics_paths: + - /up + # Logging # # Configure request logging for the proxy. @@ -706,6 +677,88 @@ proxy: - X-Request-ID - X-Request-Start + + # ========================================================================== + # 5. Fleet — multi-host behaviour + # ========================================================================== + + # Read-only targets + # + # kamal-proxy can split traffic between the deployed (writer) targets and a + # set of read-only targets, e.g. app instances backed by database replicas. + # Read requests are routed to the read targets; write requests always go to + # the writers. + # + # Targets are host:port addresses reachable from the proxy. + # + # `websockets` routes WebSocket traffic to the read targets too (default + # `false`). `writer_affinity_timeout` is how long, in seconds, a client's + # reads stick to the writer after it makes a write, so clients always read + # their own writes (default 1 second). + # + # When a loadbalancer fronts the fleet, read routing is decided there — at + # the only layer that sees the whole fleet. + read_routing: + targets: + - 192.168.0.2:3000 + - 192.168.0.3:3000 + websockets: true + writer_affinity_timeout: 10 + + # Session affinity + # + # Keep each client on the target that first served it, for apps holding session + # state in the instance. Off by default, and rightly so — every request is + # otherwise free to go to whichever target is best placed to serve it. + # + # The client carries an opaque HttpOnly cookie naming its target. When that + # target leaves the pool the next request falls through to another one and is + # re-pinned, so a deploy does not strand anybody. Reads served by a + # `read_targets` replica are never pinned. + # + # `cookie` renames the pin cookie; kamal-proxy picks a sensible default. + session_affinity: + enabled: true + cookie: _kamal_affinity + + # Scale to zero + # + # Stop this service's containers after `after` seconds with no traffic, and + # start them again on the next request, which is held until they are healthy. + # Health checks and the proxy's own TLS probes are not traffic and never wake a + # sleeping service. + # + # `wake_timeout` is how long a request waits for the containers to come back + # before giving up with a 503. `containers` names the containers to stop and + # start, replacing what the proxy infers from the target address — needed when + # a target names a network alias rather than a container. + # + # ### This needs the container runtime socket + # + # Stopping and starting containers means talking to the runtime, so the proxy + # must have been booted with `run/docker_socket` set. That is a boot-time + # prerequisite for a deploy-time setting, so kamal checks it while reading this + # file rather than letting the first request hang. + # + # Setting `run/docker_socket` also mounts that socket into the proxy container + # — the flag alone only says where to look. **Reaching the container runtime + # socket is root-equivalent on the host**, which is why it is a separate, + # explicit setting and not something enabling sleep does for you. + # + # Not compatible with `tls/on_demand_url`: a sleeping target cannot answer the + # ask endpoint, and waking one would let any hostname on the internet start a + # container. + sleep: + after: 300 + wake_timeout: 30 + containers: + - app-web + + + # ========================================================================== + # 6. Proxy container (run) + # ========================================================================== + # Run configuration # # These options are used when booting the proxy container. @@ -928,6 +981,9 @@ proxy: store: redis://cache.example.com:6379/0 store_timeout: 2 memory_size: 134_217_728 + # `options` are `docker run` options for the proxy container - resource + # limits, labels, extra mounts - NOT kamal-proxy flags. A kamal-proxy flag + # the gem has no key for goes in `flags` above instead. options: # Additional options to pass to `docker run` label: - custom.label=kamal-proxy