From b0d6cfaafb332372fcf6c99833f6a2b1bc5e666b Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Wed, 8 Jul 2026 18:28:29 +0200 Subject: [PATCH 1/4] Add Stage 4 load-balancer runbook and weight-change procedure Gcloud equivalents of every console action for provisioning the global external ALB (EXTERNAL_MANAGED): dual IPv4+IPv6 frontends (the current host serves AAAA via ghs.googlehosted.com, so IPv6 clients exist), DNS-authorized Certificate Manager cert for both hostnames, serverless NEGs for App Engine and Cloud Run in us-central1, backend services with timeout 600 and full request logging, and a URL map pinned to app_engine=100/cloud_run=0 so every future ramp step is a weight edit. Committed before execution per the plan's manual-infra rule; exit gates and full teardown included. The recurring weight-change procedure (export -> diff -> edit -> import -> verify -> commit snapshot) lands in the living operations doc. Co-Authored-By: Claude Opus 4.8 (1M context) --- changelog.d/pr4-stage4-lb-runbook.added.md | 1 + docs/migration/cloud-run-operations.md | 22 ++ .../pr4-stage4-load-balancer-runbook.md | 221 ++++++++++++++++++ 3 files changed, 244 insertions(+) create mode 100644 changelog.d/pr4-stage4-lb-runbook.added.md create mode 100644 docs/migration/history/pr4-stage4-load-balancer-runbook.md diff --git a/changelog.d/pr4-stage4-lb-runbook.added.md b/changelog.d/pr4-stage4-lb-runbook.added.md new file mode 100644 index 000000000..653f185f5 --- /dev/null +++ b/changelog.d/pr4-stage4-lb-runbook.added.md @@ -0,0 +1 @@ +Add the Stage 4 load-balancer provisioning runbook and the URL-map weight-change procedure for the Cloud Run host cutover. Documentation only; no infrastructure is created by this change. diff --git a/docs/migration/cloud-run-operations.md b/docs/migration/cloud-run-operations.md index 43d3a5d82..aed16fa13 100644 --- a/docs/migration/cloud-run-operations.md +++ b/docs/migration/cloud-run-operations.md @@ -137,6 +137,28 @@ curl -s -o /dev/null -w '%{http_code}\n' "$SERVICE_URL/readiness-check" curl -sI "$SERVICE_URL/readiness-check" | grep -i x-policyengine-backend ``` +## URL map weight changes (the ramp lever, Stages 5 and 9–11) + +Once the load balancer exists (Stage 4), every traffic shift between App Engine and +Cloud Run is a weight edit on the `lb-api` URL map — and **only** via this procedure. +Console edits leave no audit trail, and `url-maps import` **replaces the entire map**, +so an un-diffed import can silently drop routing config: + +```bash +gcloud compute url-maps export lb-api --project policyengine-api \ + --destination docs/migration/urlmap/$(date -u +%Y%m%dT%H%M%SZ)-lb-api.yaml +# 1. diff against the last committed snapshot — investigate ANY unexpected delta +# 2. edit ONLY the two weight values (must sum to 100) +# 3. import, then re-export and verify the readback matches the edit +gcloud compute url-maps import lb-api --project policyengine-api --source .yaml --quiet +# 4. verify in-band before trusting it: ~20 header-sampled curls against the LB +# counting X-PolicyEngine-Backend values +# 5. commit the new snapshot to docs/migration/urlmap/ +``` + +Rollback during a ramp = the same procedure with `app_engine=100, cloud_run=0` +(measured propagation is minutes; rehearsed in Stage 9). + ## History index - `history/migration-pr1-baseline-runbook.md` — PR 1 baseline capture diff --git a/docs/migration/history/pr4-stage4-load-balancer-runbook.md b/docs/migration/history/pr4-stage4-load-balancer-runbook.md new file mode 100644 index 000000000..2e9a78968 --- /dev/null +++ b/docs/migration/history/pr4-stage4-load-balancer-runbook.md @@ -0,0 +1,221 @@ +# PR 4 Stage 4 Runbook: Provision the Load Balancer + +> Per-stage record: gcloud equivalents of every console action, committed **before** +> manual execution (the plan's infra rule). Current operational behavior: +> [`../cloud-run-operations.md`](../cloud-run-operations.md). Source of truth for stage +> gates: the Stage 4 section of `api-v1-pr4-cloud-run-host-cutover-execution-plan.md` +> in the planning folder. + +Builds the global external Application Load Balancer (`EXTERNAL_MANAGED`) that will +eventually front `api.policyengine.org`, with two serverless NEG backends (App Engine, +Cloud Run) and a weighted URL map pinned to **app_engine=100 / cloud_run=0**. This stage +creates infrastructure only — **no DNS change to `api.policyengine.org`** (the only DNS +edits are the two certificate-authorization CNAMEs; the `api-lb` validation hostname is +Stage 5, the real cutover is Stage 8). + +Facts baked in from pre-checks (2026-07-08): + +- `api.policyengine.org` currently CNAMEs to `ghs.googlehosted.com`, which serves **both + A and AAAA** — IPv6 clients exist today, so the LB gets dual (IPv4 + IPv6) frontends. +- The App Engine app's `locationId` is `us-central` → serverless NEGs go in compute + region **`us-central1`** (same region as the Cloud Run service). +- Executor needs `roles/compute.loadBalancerAdmin` + `roles/certificatemanager.editor` + (or owner) on `policyengine-api`; the CI deploy service account has neither and never + will — this stage is manual by design. + +## 0. Shell variables (paste first) + +```bash +export PROJECT=policyengine-api REGION=us-central1 +export IP4=policyengine-api-lb-ip IP6=policyengine-api-lb-ipv6 +export NEG_GAE=neg-app-engine NEG_CR=neg-cloud-run +export BS_GAE=bs-app-engine BS_CR=bs-cloud-run +export MAP=lb-api REDIRECT_MAP=lb-api-redirect +export PROXY_HTTPS=proxy-api-https PROXY_HTTP=proxy-api-http +export CERT=cert-api-lb CERT_MAP=map-api +``` + +## 1. Static IPs + +```bash +gcloud compute addresses create "$IP4" --project "$PROJECT" --global --ip-version=IPV4 +gcloud compute addresses create "$IP6" --project "$PROJECT" --global --ip-version=IPV6 +gcloud compute addresses list --project "$PROJECT" --global \ + --format 'table(name, address)' # record BOTH in this file when executed +``` + +Executed values: `IP4 = ______` / `IP6 = ______` (fill in at execution). + +## 2. Certificate via DNS authorization (both hostnames) + +```bash +for D in api api-lb; do + gcloud certificate-manager dns-authorizations create "auth-${D}" \ + --project "$PROJECT" --domain "${D}.policyengine.org" +done +gcloud certificate-manager dns-authorizations list --project "$PROJECT" \ + --format 'table(name, dnsResourceRecord.name, dnsResourceRecord.data)' +``` + +Add the two CNAME records printed above **at Squarespace** (they look like +`_acme-challenge.api.policyengine.org → xxxx.authorize.certificatemanager.goog`). +These are additive records — they do not affect serving DNS. Then: + +```bash +gcloud certificate-manager certificates create "$CERT" --project "$PROJECT" \ + --domains="api.policyengine.org,api-lb.policyengine.org" \ + --dns-authorizations="auth-api,auth-api-lb" +gcloud certificate-manager maps create "$CERT_MAP" --project "$PROJECT" +for D in api api-lb; do + gcloud certificate-manager maps entries create "entry-${D}" --project "$PROJECT" \ + --map "$CERT_MAP" --hostname "${D}.policyengine.org" --certificates "$CERT" +done +# GATE (may take minutes after the CNAMEs propagate): +gcloud certificate-manager certificates describe "$CERT" --project "$PROJECT" \ + --format 'value(managed.state, managed.authorizationAttemptInfo)' +# expect state ACTIVE and both domains AUTHORIZED before proceeding to step 5. +``` + +## 3. Serverless NEGs + +```bash +gcloud compute network-endpoint-groups create "$NEG_GAE" --project "$PROJECT" \ + --region "$REGION" --network-endpoint-type=serverless --app-engine-service=default +gcloud compute network-endpoint-groups create "$NEG_CR" --project "$PROJECT" \ + --region "$REGION" --network-endpoint-type=serverless --cloud-run-service=policyengine-api +``` + +Pinning `--app-engine-service=default` is deliberate: the NEG routes straight to the +service, **bypassing `gcp/dispatch.yaml`** — the dispatch file stays as the rollback +path for the direct App Engine domain mapping, not part of the LB path. + +## 4. Backend services (timeout 600, full request logging) + +```bash +for PAIR in "$BS_GAE:$NEG_GAE" "$BS_CR:$NEG_CR"; do + BS="${PAIR%%:*}"; NEG="${PAIR##*:}" + gcloud compute backend-services create "$BS" --project "$PROJECT" --global \ + --load-balancing-scheme=EXTERNAL_MANAGED --protocol=HTTPS \ + --timeout=600 --enable-logging --logging-sample-rate=1.0 + gcloud compute backend-services add-backend "$BS" --project "$PROJECT" --global \ + --network-endpoint-group="$NEG" --network-endpoint-group-region="$REGION" +done +``` + +`--timeout=600`: the default 30s would kill long `/calculate` requests; Cloud Run's own +request cap is 300s and App Engine's is longer, so 600 makes the LB never the binding +constraint. **Validated empirically in Stage 5's long-request proof** — if the timeout +turns out not to be honored for serverless NEGs, that gate catches it before any DNS +change. Reminder: serverless NEGs have **no health checks** — alerting (Stage 6) is the +failover mechanism. + +## 5. URL map at 100/0, proxies, forwarding rules + +```bash +gcloud compute url-maps create "$MAP" --project "$PROJECT" \ + --default-service "$BS_GAE" +gcloud compute url-maps export "$MAP" --project "$PROJECT" \ + --destination /tmp/lb-api.yaml +``` + +Edit `/tmp/lb-api.yaml`: replace the top-level `defaultService: ...` line with the +weighted action (weights are the ramp lever for Stages 9–11): + +```yaml +defaultRouteAction: + weightedBackendServices: + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine + weight: 100 + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run + weight: 0 +``` + +```bash +gcloud compute url-maps import "$MAP" --project "$PROJECT" --source /tmp/lb-api.yaml --quiet +``` + +Then the HTTPS side (certificate map attaches to the proxy — no `--ssl-certificates`): + +```bash +gcloud compute target-https-proxies create "$PROXY_HTTPS" --project "$PROJECT" \ + --url-map "$MAP" --certificate-map "$CERT_MAP" +for PAIR in "fr-api-https:$IP4" "fr-api-https-v6:$IP6"; do + FR="${PAIR%%:*}"; ADDR="${PAIR##*:}" + gcloud compute forwarding-rules create "$FR" --project "$PROJECT" --global \ + --load-balancing-scheme=EXTERNAL_MANAGED --address="$ADDR" \ + --target-https-proxy="$PROXY_HTTPS" --ports=443 +done +``` + +HTTP→HTTPS 301 (separate redirect map; `url-maps create` has no redirect flags, so +import it): + +```bash +cat > /tmp/lb-api-redirect.yaml <<'YAML' +name: lb-api-redirect +defaultUrlRedirect: + httpsRedirect: true + redirectResponseCode: MOVED_PERMANENTLY_DEFAULT + stripQuery: false +YAML +gcloud compute url-maps import "$REDIRECT_MAP" --project "$PROJECT" \ + --source /tmp/lb-api-redirect.yaml --quiet +gcloud compute target-http-proxies create "$PROXY_HTTP" --project "$PROJECT" \ + --url-map "$REDIRECT_MAP" +for PAIR in "fr-api-http:$IP4" "fr-api-http-v6:$IP6"; do + FR="${PAIR%%:*}"; ADDR="${PAIR##*:}" + gcloud compute forwarding-rules create "$FR" --project "$PROJECT" --global \ + --load-balancing-scheme=EXTERNAL_MANAGED --address="$ADDR" \ + --target-http-proxy="$PROXY_HTTP" --ports=80 +done +``` + +## 6. Snapshot the URL map (audit trail for every future weight change) + +```bash +mkdir -p docs/migration/urlmap +gcloud compute url-maps export "$MAP" --project "$PROJECT" \ + --destination "docs/migration/urlmap/$(date -u +%Y%m%dT%H%M%SZ)-lb-api.yaml" +git add docs/migration/urlmap/ && git commit -m "Snapshot LB URL map (Stage 4 initial, 100/0)" +``` + +All future weight changes follow the procedure in +[`../cloud-run-operations.md`](../cloud-run-operations.md) ("URL map weight changes"): +export → diff → edit → import → commit. Console clicks leave no audit trail, and +`url-maps import` **replaces the whole map** — always diff first. + +## Exit gates (from the plan; evaluate before Stage 5) + +```bash +LB_IP=$(gcloud compute addresses describe "$IP4" --project "$PROJECT" --global --format 'value(address)') +curl -sv --resolve "api.policyengine.org:443:${LB_IP}" \ + https://api.policyengine.org/liveness-check -o /dev/null 2>&1 \ + | grep -E "HTTP/|x-policyengine-backend" +# expect: 200 + X-PolicyEngine-Backend: app_engine (traffic reaches GAE through the LB) +curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://${LB_IP}/liveness-check" \ + -H 'Host: api.policyengine.org' +# expect: 301 → https://api.policyengine.org/liveness-check +``` + +- Cert `state: ACTIVE`, both hostnames `AUTHORIZED` (step 2). +- LB request logs visible in Logging with `resource.type="http_load_balancer"` and + `backend_service_name` populated (allow a few minutes after first traffic). +- IPv6 spot check: repeat the `--resolve` curl against the IPv6 address. + +## Teardown (full rollback of this stage; reverse order) + +```bash +gcloud compute forwarding-rules delete fr-api-https fr-api-https-v6 fr-api-http fr-api-http-v6 --global --project "$PROJECT" --quiet +gcloud compute target-https-proxies delete "$PROXY_HTTPS" --project "$PROJECT" --quiet +gcloud compute target-http-proxies delete "$PROXY_HTTP" --project "$PROJECT" --quiet +gcloud compute url-maps delete "$MAP" "$REDIRECT_MAP" --project "$PROJECT" --quiet +gcloud compute backend-services delete "$BS_GAE" "$BS_CR" --global --project "$PROJECT" --quiet +gcloud compute network-endpoint-groups delete "$NEG_GAE" "$NEG_CR" --region "$REGION" --project "$PROJECT" --quiet +gcloud compute addresses delete "$IP4" "$IP6" --global --project "$PROJECT" --quiet +# certificate-manager resources can stay (no cost, reusable) or be deleted last. +``` + +No serving-path risk at any point in this stage: nothing resolves to the LB until the +Stage 5 `api-lb` record, and `api.policyengine.org` continues to point at App Engine +until Stage 8. Standing cost while provisioned: ~$0.025/h per forwarding rule +(4 rules ≈ $73/month) plus egress — starts as soon as step 5 completes. From aa1eabc8107c927c8ef88f3383575c1d9219d570 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Wed, 8 Jul 2026 19:13:42 +0200 Subject: [PATCH 2/4] Record reserved LB addresses from Stage 4 step 1 execution Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/migration/history/pr4-stage4-load-balancer-runbook.md | 3 ++- 1 file changed, 2 insertions(+), 1 deletion(-) diff --git a/docs/migration/history/pr4-stage4-load-balancer-runbook.md b/docs/migration/history/pr4-stage4-load-balancer-runbook.md index 2e9a78968..7a07e139b 100644 --- a/docs/migration/history/pr4-stage4-load-balancer-runbook.md +++ b/docs/migration/history/pr4-stage4-load-balancer-runbook.md @@ -44,7 +44,8 @@ gcloud compute addresses list --project "$PROJECT" --global \ --format 'table(name, address)' # record BOTH in this file when executed ``` -Executed values: `IP4 = ______` / `IP6 = ______` (fill in at execution). +Executed values (2026-07-08): `IP4 = 34.117.200.13` / `IP6 = 2600:1901:0:347b::` +(both `RESERVED`, unattached until step 5's forwarding rules). ## 2. Certificate via DNS authorization (both hostnames) From a752c21419c4cdf25b63e5e02ca5f1bec54bc3a1 Mon Sep 17 00:00:00 2001 From: Anthony Volk Date: Wed, 8 Jul 2026 20:06:02 +0200 Subject: [PATCH 3/4] Amend Stage 4 step 4 with executed reality: serverless-NEG constraints timeoutSec and portName are rejected on backend services with serverless NEG backends; the timeout is platform-fixed at 60 minutes (docs-verified), making the planned --timeout=600 both impossible and unnecessary. Old gcloud sets portName unconditionally at create, so the services were created via the Compute REST API with backends inline. Co-Authored-By: Claude Opus 4.8 (1M context) --- .../pr4-stage4-load-balancer-runbook.md | 42 ++++++++++++++----- 1 file changed, 31 insertions(+), 11 deletions(-) diff --git a/docs/migration/history/pr4-stage4-load-balancer-runbook.md b/docs/migration/history/pr4-stage4-load-balancer-runbook.md index 7a07e139b..043b767d6 100644 --- a/docs/migration/history/pr4-stage4-load-balancer-runbook.md +++ b/docs/migration/history/pr4-stage4-load-balancer-runbook.md @@ -90,24 +90,44 @@ Pinning `--app-engine-service=default` is deliberate: the NEG routes straight to service, **bypassing `gcp/dispatch.yaml`** — the dispatch file stays as the rollback path for the direct App Engine domain mapping, not part of the LB path. -## 4. Backend services (timeout 600, full request logging) +## 4. Backend services (full request logging; timeout is platform-fixed) + +Two platform constraints discovered at execution (2026-07-08), which changed this step +from the original plan: + +- **`timeoutSec` cannot be set** on a backend service with serverless NEG backends + (API error: "Timeout sec is not supported…"). Per the serverless NEG docs, the + backend-service timeout **does not apply — it is fixed at 60 minutes** for serverless + backends, so the plan's `--timeout=600` intent is exceeded by default; Cloud Run's own + 300s request cap is the real binding constraint. Stage 5's long-request proof + validates this empirically. (A `timeoutSec: 30` may appear in describes — it is a + display default and is ignored.) +- **`portName` is likewise rejected**, and older gcloud SDKs (≤ ~461) set it + unconditionally at create time, making `add-backend` fail. Recent gcloud may work; + the reliable path (used here) creates the service via the Compute REST API with the + backend inline and neither field present: ```bash +TOKEN=$(gcloud auth print-access-token) for PAIR in "$BS_GAE:$NEG_GAE" "$BS_CR:$NEG_CR"; do BS="${PAIR%%:*}"; NEG="${PAIR##*:}" - gcloud compute backend-services create "$BS" --project "$PROJECT" --global \ - --load-balancing-scheme=EXTERNAL_MANAGED --protocol=HTTPS \ - --timeout=600 --enable-logging --logging-sample-rate=1.0 - gcloud compute backend-services add-backend "$BS" --project "$PROJECT" --global \ - --network-endpoint-group="$NEG" --network-endpoint-group-region="$REGION" + curl -s -X POST "https://compute.googleapis.com/compute/v1/projects/$PROJECT/global/backendServices" \ + -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d @- < Date: Wed, 8 Jul 2026 20:25:02 +0200 Subject: [PATCH 4/4] Snapshot LB URL map (Stage 4 initial, app_engine=100 / cloud_run=0) Co-Authored-By: Claude Opus 4.8 (1M context) --- docs/migration/urlmap/20260708T182458Z-lb-api.yaml | 12 ++++++++++++ 1 file changed, 12 insertions(+) create mode 100644 docs/migration/urlmap/20260708T182458Z-lb-api.yaml diff --git a/docs/migration/urlmap/20260708T182458Z-lb-api.yaml b/docs/migration/urlmap/20260708T182458Z-lb-api.yaml new file mode 100644 index 000000000..a208c66c9 --- /dev/null +++ b/docs/migration/urlmap/20260708T182458Z-lb-api.yaml @@ -0,0 +1,12 @@ +creationTimestamp: '2026-07-08T11:10:19.533-07:00' +defaultRouteAction: + weightedBackendServices: + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine + weight: 100 + - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run + weight: 0 +fingerprint: HmNPfwPFsXk= +id: 1171555047161670757 +kind: compute#urlMap +name: lb-api +selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/urlMaps/lb-api