ci: verify deployed flows and gate releases on passing CI

[tinsever]

Description

CI currently builds the bundled image without starting it, and a release can publish images without confirming that CI passed for its source commit. This adds deployment regression coverage and requires successful CI on the exact main commit before release images are published.

• Exercise browser sign-in, project/task creation, persistence, realtime cache updates and workspace isolation against the bundled image; test realtime both without Redis and across two API instances.
• Upgrade the latest stable release to the candidate with existing accounts, memberships, tasks, comments and private images, and build the standalone API/web images.
• Run the existing S3 upload and Helm security checks, lint workflows and shell scripts, validate PR titles, and check documentation links before merge.
• Reuse CI for nightly verification, give Docker images separate cache scopes, bound job runtimes, and retain runtime failure diagnostics.

Type of Change

• Test addition or update
• Documentation update
• Other: CI reliability and release gating

How Has This Been Tested?

• Unit tests: repository suite, 62 security/release-guard tests, and 89 review-bot tests.
• Integration tests: 472 PostgreSQL tests and eight real MinIO upload tests.
• Runtime testing: production frontend browser flow, both realtime modes, and upgrade from the published v2.27.0 API to the candidate source build against the same PostgreSQL database and MinIO service.
• Other: repository build/typecheck/Biome, translations, OpenAPI, site checks for 45 pages, actionlint/ShellCheck, Helm security checks, Compose validation, and commitlint acceptance/rejection checks.

Local checks used Node 26.8.1; CI remains pinned to Node 24.19.0. The complete bundled-image and standalone-image builds will run in this PR's CI; local runtime verification used source builds to avoid a large additional Docker build cache.

Checklist

• Changes follow the project conventions and have been self-reviewed.
• CI behavior and local test commands are documented.
• New regression coverage exercises the intended behavior.
• New and existing unit tests pass locally.

[chatgpt-codex-connector]

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review	Status	Commit	Review trigger
📝 Code Review	✅ Completed 2026-09-24T22:06:13.319112Z	52329d5	Manual request

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

[qodo-free-for-open-source-projects]

PR Summary by Qodo

Verify deployed flows and gate releases on exact-commit CI

🧪 Tests ✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

AI Description

• Run browser, realtime, storage, upgrade, image, workflow, and documentation checks in CI.
• Require successful CI for the exact main commit before publishing release images.
• Reuse bounded CI nightly while preserving scoped caches and runtime failure diagnostics.

Diagram

graph TD
  Triggers["PR, main, nightly"] --> CI["Reusable CI"] --> Checks["Build and checks"]
  CI --> Runtime["Deployed runtime"] --> Services["Test services"]
  Release["Release dispatch"] --> Guard{"Exact SHA passed?"} -->|Yes| Publish["Publish images"]
  CI -. "run status" .-> Guard

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Run reusable CI inside each release

• ➕ Guarantees the release workflow tests its own source SHA directly.
• ➕ Avoids polling the GitHub Actions API and selecting among reruns.
• ➖ Repeats the full, expensive runtime and image verification suite.
• ➖ Significantly increases release latency and runner consumption.

2. Rely on repository rules and protected environments

• ➕ Centralizes release authorization in GitHub configuration.
• ➕ Can require human approval alongside status checks.
• ➖ Policy lives partly outside version-controlled workflows.
• ➖ May not prove an eligible CI run exists for the exact manually dispatched SHA.
• ➖ Skipped release commits and manual runs still require explicit handling.

Recommendation: Keep the exact-SHA polling guard because it reuses comprehensive main CI without duplicating an expensive deployment suite. Repository rules and protected environments remain useful complementary controls, but they do not replace the version-controlled exact-commit check.

Files changed (17) +1025 / -60

Refactor (1) +1 / -53

nightly.ymlReuse the complete CI workflow nightly +1/-53

Reuse the complete CI workflow nightly

• Replaces duplicated lint, test, integration, and build steps with a reusable call to the main CI workflow. Nightly image publication remains gated on successful verification.

.github/workflows/nightly.yml

Tests (4) +469 / -0

browser.mjsExercise production browser workflows and isolation +161/-0

Exercise production browser workflows and isolation

• Adds a Playwright regression covering sign-in, project and task creation, realtime cache invalidation, persistence, and cross-workspace authorization. Failures capture a screenshot and browser trace.

scripts/ci/browser.mjs

realtime.mjsVerify local and Redis-backed realtime delivery +78/-0

Verify local and Redis-backed realtime delivery

• Adds authenticated WebSocket checks proving task updates reach multiple clients on one instance and clients connected to separate Redis-backed API instances.

scripts/ci/realtime.mjs

require-ci.test.mjsTest release CI run selection rules +57/-0

Test release CI run selection rules

• Covers exact SHA, branch, and event filtering plus pending, failed, cancelled, skipped, rerun, and newer-run behavior.

scripts/ci/require-ci.test.mjs

upgrade.mjsVerify stable-to-candidate data compatibility +173/-0

Verify stable-to-candidate data compatibility

• Seeds the stable release with credentials, membership, project, task, comment, and private image data. After replacement, it verifies preserved values, asset privacy and bytes, authentication, and subsequent writes.

scripts/ci/upgrade.mjs

Documentation (1) +67 / -0

README.mdDocument CI coverage and local runtime checks +67/-0

Document CI coverage and local runtime checks

• Documents the expanded CI suite, release authorization behavior, retained diagnostics, disposable service topology, reserved ports, and commands for running deployment checks locally.

scripts/ci/README.md

Other (11) +488 / -7

build-images.ymlBound image builds and isolate matrix caches +3/-2

Bound image builds and isolate matrix caches

• Adds a 45-minute timeout and scopes GitHub Actions caches by image name, preventing bundled, API, and web builds from overwriting one another.

.github/workflows/build-images.yml

ci.ymlExpand CI into deployed-system verification +140/-4

Expand CI into deployed-system verification

• Makes CI reusable, bounds every job, and adds site, workflow, shell, storage, standalone-image, and release-guard checks. The bundled image now runs browser, realtime, Redis fan-out, and stable-upgrade scenarios with retained failure artifacts and guaranteed cleanup.

.github/workflows/ci.yml

helm-chart.ymlTrigger Helm checks for security-script changes +1/-0

Trigger Helm checks for security-script changes

• Runs the Helm workflow when the chart secret-preservation checker changes, ensuring modifications to that guard receive chart validation.

.github/workflows/helm-chart.yml

helm-validate.ymlExecute Helm security regression checks +5/-0

Execute Helm security regression checks

• Adds a five-minute timeout and runs the existing secret-preservation and security checker against the default rendered chart.

.github/workflows/helm-validate.yml

pr-title.ymlValidate pull request titles securely +35/-0

Validate pull request titles securely

• Adds a bounded commitlint workflow for pull request titles. It checks out the base SHA so untrusted pull request code cannot alter the validation policy.

.github/workflows/pr-title.yml

release.ymlGate release images on exact-source CI +26/-1

Gate release images on exact-source CI

• Adds a release guard that waits for successful main CI on the exact release source SHA before image publication. It also bounds planning, release, and promotion job runtimes.

.github/workflows/release.yml

compose.ymlDefine disposable deployed-test services +90/-0

Define disposable deployed-test services

• Introduces a loopback-only Compose stack containing PostgreSQL, MinIO, optional Redis, two application instances, and an isolated upgrade instance. Profiles support single-instance, distributed realtime, and migration scenarios.

scripts/ci/compose.yml

http.mjsProvide safe authenticated CI HTTP helpers +100/-0

Provide safe authenticated CI HTTP helpers

• Adds loopback-origin enforcement, health polling, cookie-aware API requests, account authentication, and reusable workspace, project, and task fixture creation.

scripts/ci/http.mjs

init-databases.sqlCreate an isolated upgrade database +1/-0

Create an isolated upgrade database

• Creates a separate PostgreSQL database so upgrade fixtures remain isolated from browser and realtime scenarios.

scripts/ci/init-databases.sql

require-ci.mjsAuthorize releases using exact-commit CI status +60/-0

Authorize releases using exact-commit CI status

• Queries GitHub Actions for the latest eligible main CI run matching the release SHA. It waits for pending or missing runs and rejects every completed non-success conclusion.

scripts/ci/require-ci.mjs

upgrade.shOrchestrate digest-pinned release upgrades +27/-0

Orchestrate digest-pinned release upgrades

• Resolves and validates the latest stable release, pins its image digest, seeds fixtures, and recreates the service with the candidate image against unchanged PostgreSQL and MinIO data.

scripts/ci/upgrade.sh

[qodo-free-for-open-source-projects]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📎 Requirement gaps (0) 🎨 UX issues (0) 🔗 Cross-repo conflicts (0) 📜 Skill insights (0)

1. Observer tab errors go undetected ✓ Resolved 🐞 Bug ☼ Reliability

Description

browser.mjs attaches its pageerror listener only to the primary page, while the separately
created observer page never contributes errors to failures. Any uncaught error that leaves the
observer sufficiently functional to satisfy the status locator allows the regression check to pass
despite a browser runtime failure.

Code

scripts/ci/browser.mjs[83]

+  const observer = await context.newPage();

Evidence

The primary page alone registers a pageerror handler, the observer is created without one, and the
final assertion checks only the shared array populated by the primary handler.

scripts/ci/browser.mjs[31-33]
scripts/ci/browser.mjs[83-105]
scripts/ci/browser.mjs[143-147]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The browser regression records runtime errors from the primary page but ignores errors emitted by the observer page, allowing some observer failures to escape the check.
## Fix Focus Areas
- scripts/ci/browser.mjs[31-33]
- scripts/ci/browser.mjs[83-105]
## Recommended Fix
Attach a `pageerror` listener to the observer immediately after creating it and append those errors to the shared `failures` collection, preserving the existing final assertion.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Upgrade checks can fail before storage starts ✓ Resolved 🐞 Bug ☼ Reliability

Description

The minio service has no healthcheck, so Compose considers it started as soon as its container is
running rather than when its S3 endpoint accepts requests. upgrade.mjs immediately creates the
fixture bucket after the API becomes healthy, and that request can reach the newly started storage
service before it is ready.

Code

scripts/ci/compose.yml[R25-26]

+  minio:
+    image: quay.io/minio/minio@sha256:14cea493d9a34af32f524e538b8346cf79f3321eff8e708c1e2960462bd8936e

Evidence

The added Compose service defines MinIO and exposes its port but has no healthcheck, whereas
PostgreSQL, Redis, and the app services explicitly define readiness checks. The upgrade seed path
performs CreateBucketCommand immediately after only waiting for the API health endpoint, so it
does not independently establish that MinIO is available.

scripts/ci/compose.yml[3-33]
scripts/ci/compose.yml[52-58]
scripts/ci/upgrade.mjs[21-39]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The runtime CI Compose stack does not wait for MinIO to become ready before upgrade seeding creates an S3 bucket. This can make the newly added upgrade coverage fail intermittently during storage startup.
## Fix Focus Areas
- scripts/ci/compose.yml[25-58]
- scripts/ci/upgrade.mjs[21-39]
## Recommended Fix
Add a MinIO readiness healthcheck to the `minio` service and make services that use it depend on `minio` being healthy, or explicitly retry/wait for the MinIO ready endpoint before `CreateBucketCommand` runs. Keep the wait bounded and preserve the current disposable loopback-only setup.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

Tip of the day

💡 Did you know, you can enable the Remediation agent and Qodo fixes findings in a dedicated fix PR

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

[tinsever]

@codex review

[tinsever]

@codex review

[chatgpt-codex-connector]

Codex Review: Didn't find any major issues. You're on a roll.

Reviewed commit: 52329d52e5

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".