Skip to content

refactor(cloudbuild): replace GCE container declarations with startup scripts#3722

Open
djwhitt wants to merge 4 commits into
mainfrom
davidwhittington/o11y-2142-replace-gce-container-declaration-with-a-startup-script
Open

refactor(cloudbuild): replace GCE container declarations with startup scripts#3722
djwhitt wants to merge 4 commits into
mainfrom
davidwhittington/o11y-2142-replace-gce-container-declaration-with-a-startup-script

Conversation

@djwhitt

@djwhitt djwhitt commented Jul 23, 2026

Copy link
Copy Markdown
Contributor

Summary

  • add an idempotent Container-Optimized OS startup script that authenticates to GCR and starts Logflare with the existing privileged, host-networked, always-restarting container behavior
  • preserve Konlet's host firewall rules and rotated json-file logging so the existing COS logging agent continues exporting container logs
  • create plain GCE instance templates with startup, shutdown, image, and per-cluster environment metadata instead of create-with-container
  • migrate the manual instance-template helper and reject stale existing templates that still use the deprecated container declaration

Validation

  • shellcheck cloudbuild/gce-startup.sh cloudbuild/create-instance-template.sh scripts/create-instance-templates.sh test/cloudbuild/gce_scripts_test.sh
  • bash -n for all changed shell scripts
  • test/cloudbuild/gce_scripts_test.sh
  • parsed both changed Cloud Build configs with PyYAML and executed their environment-generation steps with representative substitutions

Staging validation

Exercised the changed staging Cloud Build config in two stages, using the existing staging build/deploy tooling. No serving traffic was affected at any point.

Isolated instance test: built a fresh instance template from this branch and booted a single, disposable instance in its own instance group (not the real serving group), then tore the disposable resources down afterward. Confirmed:

  • the generated instance template carries startup-script/shutdown-script/image/env metadata and no gce-container-declaration
  • the startup script completed successfully on first boot: GCR auth succeeded, the image pulled, and the container started
  • the running container matches today's behavior: privileged, host networking, always-restart, and the same rotated json-file logging configuration
  • the host firewall rules the startup script installs are present and idempotent on rerun
  • the instance joined the real staging Erlang cluster and reported the expected cluster peers, cross-checked two independent ways: the app's own health endpoint, and a separate live RPC call into the running node
  • container logs continue to land in Cloud Logging under the same log stream, with the expected container and application labels intact

Canary on the real serving group: rolled the same tested template out as a single-instance canary on the real staging serving group, then rolled it back. Confirmed:

  • the canary instance received real load-balancer-routed traffic and joined the live staging cluster alongside the existing fleet, with no impact to the rest of the group
  • the shutdown script drains the instance gracefully during the rolling replacement back to the previous template

@djwhitt
djwhitt marked this pull request as ready for review July 23, 2026 16:17
@djwhitt
djwhitt requested a review from Ziinc July 23, 2026 20:05
Comment thread test/cloudbuild/gce_scripts_test.sh Outdated
Comment thread test/cloudbuild/gce_scripts_test.sh Outdated
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants