PolicyEngine · anth-volk · Jul 8, 2026 · Jul 8, 2026 · Jul 8, 2026 · Jul 8, 2026
diff --git a/changelog.d/pr4-stage4-lb-runbook.added.md b/changelog.d/pr4-stage4-lb-runbook.added.md
@@ -0,0 +1 @@
+Add the Stage 4 load-balancer provisioning runbook and the URL-map weight-change procedure for the Cloud Run host cutover. Documentation only; no infrastructure is created by this change.
diff --git a/docs/migration/cloud-run-operations.md b/docs/migration/cloud-run-operations.md
@@ -137,6 +137,28 @@ curl -s -o /dev/null -w '%{http_code}\n' "$SERVICE_URL/readiness-check"
 curl -sI "$SERVICE_URL/readiness-check" | grep -i x-policyengine-backend
 ```
 
+## URL map weight changes (the ramp lever, Stages 5 and 9–11)
+
+Once the load balancer exists (Stage 4), every traffic shift between App Engine and
+Cloud Run is a weight edit on the `lb-api` URL map — and **only** via this procedure.
+Console edits leave no audit trail, and `url-maps import` **replaces the entire map**,
+so an un-diffed import can silently drop routing config:
+
+```bash
+gcloud compute url-maps export lb-api --project policyengine-api \
+  --destination docs/migration/urlmap/$(date -u +%Y%m%dT%H%M%SZ)-lb-api.yaml
+# 1. diff against the last committed snapshot — investigate ANY unexpected delta
+# 2. edit ONLY the two weight values (must sum to 100)
+# 3. import, then re-export and verify the readback matches the edit
+gcloud compute url-maps import lb-api --project policyengine-api --source <edited>.yaml --quiet
+# 4. verify in-band before trusting it: ~20 header-sampled curls against the LB
+#    counting X-PolicyEngine-Backend values
+# 5. commit the new snapshot to docs/migration/urlmap/
+```
+
+Rollback during a ramp = the same procedure with `app_engine=100, cloud_run=0`
+(measured propagation is minutes; rehearsed in Stage 9).
+
 ## History index
 
 - `history/migration-pr1-baseline-runbook.md` — PR 1 baseline capture

diff --git a/docs/migration/history/pr4-stage4-load-balancer-runbook.md b/docs/migration/history/pr4-stage4-load-balancer-runbook.md
@@ -0,0 +1,242 @@
+# PR 4 Stage 4 Runbook: Provision the Load Balancer
+
+> Per-stage record: gcloud equivalents of every console action, committed **before**
+> manual execution (the plan's infra rule). Current operational behavior:
+> [`../cloud-run-operations.md`](../cloud-run-operations.md). Source of truth for stage
+> gates: the Stage 4 section of `api-v1-pr4-cloud-run-host-cutover-execution-plan.md`
+> in the planning folder.
+
+Builds the global external Application Load Balancer (`EXTERNAL_MANAGED`) that will
+eventually front `api.policyengine.org`, with two serverless NEG backends (App Engine,
+Cloud Run) and a weighted URL map pinned to **app_engine=100 / cloud_run=0**. This stage
+creates infrastructure only — **no DNS change to `api.policyengine.org`** (the only DNS
+edits are the two certificate-authorization CNAMEs; the `api-lb` validation hostname is
+Stage 5, the real cutover is Stage 8).
+
+Facts baked in from pre-checks (2026-07-08):
+
+- `api.policyengine.org` currently CNAMEs to `ghs.googlehosted.com`, which serves **both
+  A and AAAA** — IPv6 clients exist today, so the LB gets dual (IPv4 + IPv6) frontends.
+- The App Engine app's `locationId` is `us-central` → serverless NEGs go in compute
+  region **`us-central1`** (same region as the Cloud Run service).
+- Executor needs `roles/compute.loadBalancerAdmin` + `roles/certificatemanager.editor`
+  (or owner) on `policyengine-api`; the CI deploy service account has neither and never
+  will — this stage is manual by design.
+
+## 0. Shell variables (paste first)
+
+```bash
+export PROJECT=policyengine-api REGION=us-central1
+export IP4=policyengine-api-lb-ip IP6=policyengine-api-lb-ipv6
+export NEG_GAE=neg-app-engine NEG_CR=neg-cloud-run
+export BS_GAE=bs-app-engine BS_CR=bs-cloud-run
+export MAP=lb-api REDIRECT_MAP=lb-api-redirect
+export PROXY_HTTPS=proxy-api-https PROXY_HTTP=proxy-api-http
+export CERT=cert-api-lb CERT_MAP=map-api
+```
+
+## 1. Static IPs
+
+```bash
+gcloud compute addresses create "$IP4" --project "$PROJECT" --global --ip-version=IPV4
+gcloud compute addresses create "$IP6" --project "$PROJECT" --global --ip-version=IPV6
+gcloud compute addresses list --project "$PROJECT" --global \
+  --format 'table(name, address)'   # record BOTH in this file when executed
+```
+
+Executed values (2026-07-08): `IP4 = 34.117.200.13` / `IP6 = 2600:1901:0:347b::`
+(both `RESERVED`, unattached until step 5's forwarding rules).
+
+## 2. Certificate via DNS authorization (both hostnames)
+
+```bash
+for D in api api-lb; do
+  gcloud certificate-manager dns-authorizations create "auth-${D}" \
+    --project "$PROJECT" --domain "${D}.policyengine.org"
+done
+gcloud certificate-manager dns-authorizations list --project "$PROJECT" \
+  --format 'table(name, dnsResourceRecord.name, dnsResourceRecord.data)'
+```
+
+Add the two CNAME records printed above **at Squarespace** (they look like
+`_acme-challenge.api.policyengine.org → xxxx.authorize.certificatemanager.goog`).
+These are additive records — they do not affect serving DNS. Then:
+
+```bash
+gcloud certificate-manager certificates create "$CERT" --project "$PROJECT" \
+  --domains="api.policyengine.org,api-lb.policyengine.org" \
+  --dns-authorizations="auth-api,auth-api-lb"
+gcloud certificate-manager maps create "$CERT_MAP" --project "$PROJECT"
+for D in api api-lb; do
+  gcloud certificate-manager maps entries create "entry-${D}" --project "$PROJECT" \
+    --map "$CERT_MAP" --hostname "${D}.policyengine.org" --certificates "$CERT"
+done
+# GATE (may take minutes after the CNAMEs propagate):
+gcloud certificate-manager certificates describe "$CERT" --project "$PROJECT" \
+  --format 'value(managed.state, managed.authorizationAttemptInfo)'
+# expect state ACTIVE and both domains AUTHORIZED before proceeding to step 5.
+```
+
+## 3. Serverless NEGs
+
+```bash
+gcloud compute network-endpoint-groups create "$NEG_GAE" --project "$PROJECT" \
+  --region "$REGION" --network-endpoint-type=serverless --app-engine-service=default
+gcloud compute network-endpoint-groups create "$NEG_CR" --project "$PROJECT" \
+  --region "$REGION" --network-endpoint-type=serverless --cloud-run-service=policyengine-api
+```
+
+Pinning `--app-engine-service=default` is deliberate: the NEG routes straight to the
+service, **bypassing `gcp/dispatch.yaml`** — the dispatch file stays as the rollback
+path for the direct App Engine domain mapping, not part of the LB path.
+
+## 4. Backend services (full request logging; timeout is platform-fixed)
+
+Two platform constraints discovered at execution (2026-07-08), which changed this step
+from the original plan:
+
+- **`timeoutSec` cannot be set** on a backend service with serverless NEG backends
+  (API error: "Timeout sec is not supported…"). Per the serverless NEG docs, the
+  backend-service timeout **does not apply — it is fixed at 60 minutes** for serverless
+  backends, so the plan's `--timeout=600` intent is exceeded by default; Cloud Run's own
+  300s request cap is the real binding constraint. Stage 5's long-request proof
+  validates this empirically. (A `timeoutSec: 30` may appear in describes — it is a
+  display default and is ignored.)
+- **`portName` is likewise rejected**, and older gcloud SDKs (≤ ~461) set it
+  unconditionally at create time, making `add-backend` fail. Recent gcloud may work;
+  the reliable path (used here) creates the service via the Compute REST API with the
+  backend inline and neither field present:
+
+```bash
+TOKEN=$(gcloud auth print-access-token)
+for PAIR in "$BS_GAE:$NEG_GAE" "$BS_CR:$NEG_CR"; do
+  BS="${PAIR%%:*}"; NEG="${PAIR##*:}"
+  curl -s -X POST "https://compute.googleapis.com/compute/v1/projects/$PROJECT/global/backendServices" \
+    -H "Authorization: Bearer $TOKEN" -H "Content-Type: application/json" -d @- <<JSON
+{
+  "name": "$BS",
+  "loadBalancingScheme": "EXTERNAL_MANAGED",
+  "protocol": "HTTPS",
+  "logConfig": {"enable": true, "sampleRate": 1.0},
+  "backends": [{"group": "https://www.googleapis.com/compute/v1/projects/$PROJECT/regions/us-central1/networkEndpointGroups/$NEG"}]
+}
+JSON
+done
+# verify both services exist with their NEG attached:
+gcloud compute backend-services describe "$BS_GAE" --global --project "$PROJECT" \
+  --format 'value(backends[0].group)'
+```
+
+Reminder: serverless NEGs have **no health checks** — alerting (Stage 6) is the
+failover mechanism.
+
+## 5. URL map at 100/0, proxies, forwarding rules
+
+```bash
+gcloud compute url-maps create "$MAP" --project "$PROJECT" \
+  --default-service "$BS_GAE"
+gcloud compute url-maps export "$MAP" --project "$PROJECT" \
+  --destination /tmp/lb-api.yaml
+```
+
+Edit `/tmp/lb-api.yaml`: replace the top-level `defaultService: ...` line with the
+weighted action (weights are the ramp lever for Stages 9–11):
+
+```yaml
+defaultRouteAction:
+  weightedBackendServices:
+  - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine
+    weight: 100
+  - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run
+    weight: 0
+```
+
+```bash
+gcloud compute url-maps import "$MAP" --project "$PROJECT" --source /tmp/lb-api.yaml --quiet
+```
+
+Then the HTTPS side (certificate map attaches to the proxy — no `--ssl-certificates`):
+
+```bash
+gcloud compute target-https-proxies create "$PROXY_HTTPS" --project "$PROJECT" \
+  --url-map "$MAP" --certificate-map "$CERT_MAP"
+for PAIR in "fr-api-https:$IP4" "fr-api-https-v6:$IP6"; do
+  FR="${PAIR%%:*}"; ADDR="${PAIR##*:}"
+  gcloud compute forwarding-rules create "$FR" --project "$PROJECT" --global \
+    --load-balancing-scheme=EXTERNAL_MANAGED --address="$ADDR" \
+    --target-https-proxy="$PROXY_HTTPS" --ports=443
+done
+```
+
+HTTP→HTTPS 301 (separate redirect map; `url-maps create` has no redirect flags, so
+import it):
+
+```bash
+cat > /tmp/lb-api-redirect.yaml <<'YAML'
+name: lb-api-redirect
+defaultUrlRedirect:
+  httpsRedirect: true
+  redirectResponseCode: MOVED_PERMANENTLY_DEFAULT
+  stripQuery: false
+YAML
+gcloud compute url-maps import "$REDIRECT_MAP" --project "$PROJECT" \
+  --source /tmp/lb-api-redirect.yaml --quiet
+gcloud compute target-http-proxies create "$PROXY_HTTP" --project "$PROJECT" \
+  --url-map "$REDIRECT_MAP"
+for PAIR in "fr-api-http:$IP4" "fr-api-http-v6:$IP6"; do
+  FR="${PAIR%%:*}"; ADDR="${PAIR##*:}"
+  gcloud compute forwarding-rules create "$FR" --project "$PROJECT" --global \
+    --load-balancing-scheme=EXTERNAL_MANAGED --address="$ADDR" \
+    --target-http-proxy="$PROXY_HTTP" --ports=80
+done
+```
+
+## 6. Snapshot the URL map (audit trail for every future weight change)
+
+```bash
+mkdir -p docs/migration/urlmap
+gcloud compute url-maps export "$MAP" --project "$PROJECT" \
+  --destination "docs/migration/urlmap/$(date -u +%Y%m%dT%H%M%SZ)-lb-api.yaml"
+git add docs/migration/urlmap/ && git commit -m "Snapshot LB URL map (Stage 4 initial, 100/0)"
+```
+
+All future weight changes follow the procedure in
+[`../cloud-run-operations.md`](../cloud-run-operations.md) ("URL map weight changes"):
+export → diff → edit → import → commit. Console clicks leave no audit trail, and
+`url-maps import` **replaces the whole map** — always diff first.
+
+## Exit gates (from the plan; evaluate before Stage 5)
+
+```bash
+LB_IP=$(gcloud compute addresses describe "$IP4" --project "$PROJECT" --global --format 'value(address)')
+curl -sv --resolve "api.policyengine.org:443:${LB_IP}" \
+  https://api.policyengine.org/liveness-check -o /dev/null 2>&1 \
+  | grep -E "HTTP/|x-policyengine-backend"
+# expect: 200 + X-PolicyEngine-Backend: app_engine  (traffic reaches GAE through the LB)
+curl -s -o /dev/null -w '%{http_code} %{redirect_url}\n' "http://${LB_IP}/liveness-check" \
+  -H 'Host: api.policyengine.org'
+# expect: 301 → https://api.policyengine.org/liveness-check
+```
+
+- Cert `state: ACTIVE`, both hostnames `AUTHORIZED` (step 2).
+- LB request logs visible in Logging with `resource.type="http_load_balancer"` and
+  `backend_service_name` populated (allow a few minutes after first traffic).
+- IPv6 spot check: repeat the `--resolve` curl against the IPv6 address.
+
+## Teardown (full rollback of this stage; reverse order)
+
+```bash
+gcloud compute forwarding-rules delete fr-api-https fr-api-https-v6 fr-api-http fr-api-http-v6 --global --project "$PROJECT" --quiet
+gcloud compute target-https-proxies delete "$PROXY_HTTPS" --project "$PROJECT" --quiet
+gcloud compute target-http-proxies delete "$PROXY_HTTP" --project "$PROJECT" --quiet
+gcloud compute url-maps delete "$MAP" "$REDIRECT_MAP" --project "$PROJECT" --quiet
+gcloud compute backend-services delete "$BS_GAE" "$BS_CR" --global --project "$PROJECT" --quiet
+gcloud compute network-endpoint-groups delete "$NEG_GAE" "$NEG_CR" --region "$REGION" --project "$PROJECT" --quiet
+gcloud compute addresses delete "$IP4" "$IP6" --global --project "$PROJECT" --quiet
+# certificate-manager resources can stay (no cost, reusable) or be deleted last.
+```
+
+No serving-path risk at any point in this stage: nothing resolves to the LB until the
+Stage 5 `api-lb` record, and `api.policyengine.org` continues to point at App Engine
+until Stage 8. Standing cost while provisioned: ~$0.025/h per forwarding rule
+(4 rules ≈ $73/month) plus egress — starts as soon as step 5 completes.
diff --git a/docs/migration/urlmap/20260708T182458Z-lb-api.yaml b/docs/migration/urlmap/20260708T182458Z-lb-api.yaml
@@ -0,0 +1,12 @@
+creationTimestamp: '2026-07-08T11:10:19.533-07:00'
+defaultRouteAction:
+  weightedBackendServices:
+  - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-app-engine
+    weight: 100
+  - backendService: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/backendServices/bs-cloud-run
+    weight: 0
+fingerprint: HmNPfwPFsXk=
+id: 1171555047161670757
+kind: compute#urlMap
+name: lb-api
+selfLink: https://www.googleapis.com/compute/v1/projects/policyengine-api/global/urlMaps/lb-api
Original file line number	Diff line number	Diff line change
		@@ -0,0 +1 @@
		Add the Stage 4 load-balancer provisioning runbook and the URL-map weight-change procedure for the Cloud Run host cutover. Documentation only; no infrastructure is created by this change.