Migration PR 4, Stages 2+3: capacity qualification record and production Cloud Run configuration#3741
Merged
Merged
Conversation
Committed locally on the Stage 2 branch; pushed with the Stage 3 PR. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Full lean-protocol campaign: cold-start distribution, traffic profile, warm parity (post-gzip-fix, per-probe PASS), worker/concurrency cells, and soaks. Decision: WEB_CONCURRENCY=2, --concurrency 4, service-level min-instances 1, max-instances 4, memory unchanged at 16Gi. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
--concurrency 4 and WEB_CONCURRENCY=2 per the capacity campaign: two workers beat one decisively (GIL-bound engine), and admission control at 4 is what keeps the two-worker instance memory-safe in 16Gi (the concurrency-8 soak OOM-killed a worker at 92% memory; concurrency 4 soaked clean at 69%). Pinned explicitly on every deploy because gcloud inherits unspecified template fields — a mid-campaign CI deploy shipped candidates with an inherited test concurrency — and because --concurrency default resolves to the platform default (640), not the historical 80. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Production capacity is an explicit workflow decision, not a script default: max-instances 4 sized from the Stage 2 campaign (peak real traffic ~11 RPS, per-instance ceiling ~1-2 concurrent uncached calculates). Staging pins min 0 / max 1 explicitly. Revision-level min-instances stays 0 everywhere; warm capacity is set service-level, once, outside CI (revision-level would warm one 16Gi instance per accumulated revision tag). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## master #3741 +/- ##
==========================================
+ Coverage 79.82% 79.89% +0.07%
==========================================
Files 70 70
Lines 4336 4332 -4
Branches 808 807 -1
==========================================
Hits 3461 3461
+ Misses 655 652 -3
+ Partials 220 219 -1 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Worker recycling (--max-requests) is removed rather than retuned: review analysis showed the 500-request budget recycles faster than the ~170s re-import at peak (the counter advances on cheap requests too), jitter 100 cannot decorrelate two workers, a mid-life re-import runs without cpu-boost and CPU-throttled at idle, and the qualifying soak never executed a recycle. The ops doc now records why it needs its own qualification. Also: pin assertions get boundary anchors; the ops doc's source-of-truth section acknowledges push.yml job pins; stale cold-start figures replaced with measured values; the changelog credits staging's shape change; the min-instances runbook gains WHEN + verification; the duplicated rationale comments collapse to pointers; load-generator cleanup (single exception path, Counter, always/never cache-bust, dead fields removed). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
anth-volk
force-pushed
the
feat/migration-pr4-stage2-runtime-timing
branch
from
22e389f to
041a882
Compare
…d percentile - The scaling/runtime-shape invariants now scan the whole workflow file: any mention of the min-instances variable must be an explicit zero pin (workflow-level env flows into every job), and the concurrency/worker variables may not appear in the workflow at all — their only home is cloud_run_env.sh, which the dry-run test validates. - The load generator's cache-bust nonce gains a per-run token so re-running a cell within Redis's 300s TTL cannot replay identical body hashes and measure the cache. - The warm-up calculate is best-effort: a timeout no longer loses the cell after the readiness wait. - _percentile is shared from capture_migration_baseline (one definition of p95 across the evidence set) and fixed to round half up — banker's rounding biased small even samples low (p50 of two values picked the minimum). - achieved_rps divides by the measured window rather than the configured duration (in-flight requests at the deadline run to completion). Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Two stages of the Cloud Run host-cutover plan in one reviewable arc: the Stage 2 measurement campaign's tooling and evidence (commits 1–2), the Stage 3 production configuration that encodes its results (commits 3–6), and a self-review fixes commit (7).
Commits
scripts/measure_cloud_run_runtime.py) — closed-loop asyncio/httpx clients with cache-busting via a top-level nonce (/us/calculateis Redis-cached on the full body hash; unbusted load tests measure Redis at ~260ms instead of the engine at ~44s+). Reusable for the Stage 10 ramp watchpoints.docs/migration/history/) — cold-start distribution (n=18 mined + 282.8s live user-facing wake), traffic profile (peak 10.98 RPS, mostly light/cached), warm parity (per-probe PASS after Lower ASGI gzip compresslevel from 9 to 4 #3735 — Cloud Run beats App Engine on every probe), the worker/concurrency cells, and both soaks (c8: OOM SIGKILL at 92% memory; c4: clean at 69%).--concurrency 4+WEB_CONCURRENCY=2explicit on every deploy (both services — staging mirrors production's shape). Explicit pinning is non-negotiable after the campaign experienced template pollution (a mid-campaign CI deploy inherited a test concurrency) and the--concurrency default=platform-640 trap.push.yml: productionmax-instances 4; stagingmin 0 / max 1. Revision-level min-instances stays 0 everywhere — warm capacity is service-level only (revision-level would keep a warm 16Gi instance per accumulatedstage3-*tag).--min-instances 1runbook with WHEN-to-run and verification steps.--max-requests), originally included, was removed after adversarial review verified it was unsafe as configured (the request counter advances on cheap requests, so at peak a worker would recycle every ~90–110s against a ~170s+ re-import that runs CPU-throttled without boost; jitter 100 cannot decorrelate two workers; and the qualifying soak never executed a recycle). It returns only with its own qualification — rationale recorded in the ops doc and the Stage 2 record's follow-ups. Also: docs-drift fixes (source-of-truth layering, measured cold-start figures), changelog honesty about staging, comment dedup, load-generator cleanup.Verification
After merge
gcloud run services update policyengine-api --min-instances 1(service-level; runbook with verification in the ops doc).containerConcurrency: 4/maxScale: 4; after 30+ min idle,/readiness-checkTTFB < 2s; billing shows exactly one warm instance.Refs: Stage 2 section of the cutover execution plan (planning folder); #3734/#3735 (gzip parity fix found by Phase A).
🤖 Generated with Claude Code