Skip to content
This repository was archived by the owner on Aug 12, 2026. It is now read-only.

Add OTR testnet deployment workflow - #189

Closed
neekolas wants to merge 1 commit into
testnet-agent-invite-urlfrom
deploy-otr-testnet
Closed

Add OTR testnet deployment workflow#189
neekolas wants to merge 1 commit into
testnet-agent-invite-urlfrom
deploy-otr-testnet

Conversation

@neekolas

@neekolas neekolas commented Apr 15, 2026

Copy link
Copy Markdown
Collaborator

Summary

Activates OTR testnet deployment support after the app understands the testnet environment.

  • Adds otr-testnet to AWS deployment triggers and manual deployment options.
  • Adds a deploy_otr_testnet Terraform deployment job using workspace convos-otr-testnet.
  • Adds deploy-time and scheduled S3 lifecycle verification for testnet using vars.TESTNET_API_URL.
  • Updates release docs to promote through otr-dev -> otr-testnet -> otr-prod.

Verification

  • bun check
  • YAML parsed with Ruby Psych.load_file for both modified workflow files.
  • bun test tests/renew-batch.test.ts tests/agent-join.test.ts tests/config-environment.test.ts

Note

Add OTR testnet deployment workflow and lifecycle verification

  • Adds otr-testnet branch trigger, workflow_dispatch option, and repository_dispatch type deploy-otr-testnet to deploy-aws.yml, deploying to the convos-otr-testnet Terraform workspace.
  • Adds a test_lifecycle_testnet job that waits for service health and asserts the lifecycle renewal endpoint returns HTTP 200 with .success == true.
  • Adds a verify-testnet job to s3-lifecycle-verify.yml that checks lifecycle canary status against TESTNET_API_URL and publishes a workflow summary.
  • Documents the new testnet release stage in RELEASE.md: fast-forward merge otr-dev into otr-testnet, verify, then promote to otr-prod.

Macroscope summarized fb28045.

Summary by CodeRabbit

  • New Features

    • Added a Testnet environment with automated deployment, lifecycle testing, and verification.
  • Chores

    • Deployment workflows updated to support Testnet and to gate production releases after Testnet verification.
  • Documentation

    • Release process updated to require changes pass through Testnet before production.
  • Bug Fixes

    • Improved error logging for authentication middleware to produce clearer messages.

@coderabbitai

coderabbitai Bot commented Apr 15, 2026

Copy link
Copy Markdown
Contributor

Walkthrough

Added a testnet deployment stage and lifecycle verification: workflows now support otr-testnet deployment and automated lifecycle checks; release docs require changes to flow through otr-testnet before production.

Changes

Cohort / File(s) Summary
Deployment Workflow
​.github/workflows/deploy-aws.yml
Added otr-testnet to triggers (push, workflow_dispatch, repository_dispatch). Added deploy_otr_testnet job (Terraform workspace convos-otr-testnet) and test_lifecycle_testnet job (polls healthcheck, POSTs lifecycle test and asserts success).
Verification Workflow
​.github/workflows/s3-lifecycle-verify.yml
Added verify-testnet job mirroring existing lifecycle verification: POSTs to ${{ vars.TESTNET_API_URL }}/api/v2/assets/test/lifecycle-status, asserts HTTP 200 and .status == "healthy", emits GitHub step summary for testnet.
Release Documentation
RELEASE.md
Inserted Testnet section and updated release flow to require merging otr-devotr-testnet before merging to otr-prod; added rule prohibiting pushing commits to otr-prod that are not on otr-testnet.

Sequence Diagram

sequenceDiagram
    participant Push as Git Push / CI
    participant Registry as Container Registry
    participant Terraform as Terraform Deployer
    participant Testnet as Testnet Environment
    participant Healthcheck as Healthcheck Poller
    participant LifecycleAPI as Lifecycle API

    Push->>Registry: build & push image (digest)
    Registry->>Terraform: trigger deploy_otr_testnet (uses image digest)
    Terraform->>Testnet: deploy via workspace "convos-otr-testnet"
    Testnet-->>Healthcheck: service becomes reachable
    Healthcheck->>Testnet: poll ${vars.TESTNET_API_URL}/healthcheck (<=~100s)
    Testnet-->>Healthcheck: 200 OK
    Healthcheck->>LifecycleAPI: POST /api/v2/assets/test/lifecycle (with token)
    LifecycleAPI-->>Healthcheck: 200 + { success: true }
    Healthcheck-->>Push: test_lifecycle_testnet completes (pass)
Loading

Estimated code review effort

🎯 3 (Moderate) | ⏱️ ~22 minutes

Possibly related PRs

Poem

🐰 I hopped the branches, soft and fleet,
From dev to testnet — a tidy feat.
I watched the healthchecks hum and sing,
Then danced as deployments took to wing. 🎉

🚥 Pre-merge checks | ✅ 3
✅ Passed checks (3 passed)
Check name Status Explanation
Title check ✅ Passed The title accurately and clearly describes the primary change: adding OTR testnet deployment workflow. It directly corresponds to the main modifications across deploy-aws.yml and s3-lifecycle-verify.yml.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Description Check ✅ Passed Check skipped - CodeRabbit’s high-level summary is enabled.

✏️ Tip: You can configure your own custom pre-merge checks in the settings.

✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch deploy-otr-testnet

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands and usage tips.

neekolas commented Apr 15, 2026

Copy link
Copy Markdown
Collaborator Author

Warning

This pull request is not mergeable via GitHub because a downstack PR is open. Once all requirements are satisfied, merge this PR as a stack on Graphite.
Learn more

This stack of pull requests is managed by Graphite. Learn more about stacking.

@claude

claude Bot commented Apr 15, 2026

Copy link
Copy Markdown

Claude finished @neekolas's task —— View job


PR Review Complete ✓

Review checklist:

  • Read and analyze deploy-aws.yml changes
  • Read and analyze s3-lifecycle-verify.yml changes
  • Read and analyze RELEASE.md changes
  • Read and analyze src/middleware/auth.ts changes
  • Provide comprehensive feedback on code quality, bugs, performance, security, and test coverage

Review Feedback

Overall Assessment

This PR successfully adds testnet deployment infrastructure following established patterns. The changes are well-structured and consistent with existing dev/prod deployments. I've identified a few areas for improvement below.

Code Quality & Best Practices

✅ Strengths

  • Consistent patterns: The testnet deployment job (deploy_otr_testnet) correctly mirrors the structure of deploy_otr_dev and deploy_otr_prod
  • Proper conditionals: The if condition in .github/workflows/deploy-aws.yml:147 correctly handles all trigger scenarios (workflow_dispatch, push, repository_dispatch)
  • Documentation: RELEASE.md clearly documents the new three-stage deployment flow (dev → testnet → prod)

⚠️ Code Duplication Issue

File: .github/workflows/s3-lifecycle-verify.yml

The new verify-testnet job (lines 81-150) duplicates ~70 lines of shell script logic from verify-dev and verify-prod. This creates maintenance burden - any bug fix or enhancement needs to be applied in three places.

Recommendation: Extract the shared verification logic into a reusable workflow that accepts environment name and API URL as parameters. This would:

  • Reduce the file from ~220 lines to ~100 lines
  • Ensure consistency across all three environments
  • Make future updates to the verification logic trivial (single source of truth)

CodeRabbit already flagged this, and I strongly second that recommendation.

Minor: Repository dispatch type inconsistency

File: .github/workflows/deploy-aws.yml:23

repository_dispatch:
  types: [deploy-otr-testnet, deploy-otr-prod]

The deploy-otr-dev type is missing from the repository_dispatch types array, while testnet and prod are included. This creates an inconsistency:

  • Dev can only be deployed via push or workflow_dispatch
  • Testnet and prod can also be deployed via repository_dispatch

Question: Is this intentional? If not, consider adding deploy-otr-dev for consistency.

Potential Bugs

⚠️ Missing repository_dispatch check in deploy_otr_testnet condition

File: .github/workflows/deploy-aws.yml:147

The condition logic has a subtle inconsistency. Compare:

deploy_otr_testnet (line 147):

if: (inputs.ref == 'otr-testnet') || (inputs.environment == 'otr-testnet') || (github.ref == 'refs/heads/otr-testnet' && !inputs.ref && !inputs.environment) || (github.event_name == 'repository_dispatch' && github.event.client_payload.ref == 'otr-testnet')

deploy_otr_prod (line 125):

if: (inputs.ref == 'otr-prod') || (inputs.environment == 'otr-prod') || (github.ref == 'refs/heads/otr-prod' && !inputs.ref && !inputs.environment) || (github.event_name == 'repository_dispatch' && github.event.client_payload.ref == 'otr-prod')

Both include the repository_dispatch check with client_payload.ref, which is correct given that line 23 includes these types.

deploy_otr_dev (line 103):

if: (inputs.ref == 'otr-dev') || (inputs.environment == 'otr-dev') || (github.ref == 'refs/heads/otr-dev' && !inputs.ref && !inputs.environment)

This lacks the repository_dispatch check. This is consistent with deploy-otr-dev not being in the types array, but creates an asymmetry. ✅ Not a bug if intentional, but worth documenting why.

Security Concerns

✅ No security issues identified

  • Secrets are properly referenced (${{ secrets.LIFECYCLE_TEST_TOKEN }}, ${{ secrets.TERRAFORM_TOKEN }})
  • Authorization headers are correctly used for API calls
  • No hardcoded credentials or sensitive data
  • The testnet environment uses the same security model as dev/prod

🔒 Auth.ts changes (unrelated to this PR's primary goal)

The changes to src/middleware/auth.ts:160 appear to be formatting-only (adding isNSEOnly to logging). This is fine, but it's worth noting that this file wasn't mentioned in the PR summary. Consider whether this change should be in a separate commit or if it was an accidental inclusion.

Performance Considerations

⏱️ Health check timing

Files: .github/workflows/deploy-aws.yml (lines 206-218, 242-254, 170-182)

All three test_lifecycle_* jobs use a 100-second timeout (10 attempts × 10 seconds) for health checks. This is reasonable, but:

Potential optimization: The testnet job duplicates the exact same logic. If you refactor the verification workflow (as suggested above), consider also extracting the health check logic.

Current approach: ✅ Acceptable - 100s is a reasonable timeout for deployment stabilization

⚡ No performance regressions

  • The new testnet jobs run in parallel with dev/prod (when applicable)
  • No blocking dependencies between environments
  • Lifecycle tests are properly gated behind their respective deployments

Test Coverage

⚠️ Missing explicit test validation

The PR description mentions:

  • bun check
  • YAML parsed with Ruby Psych.load_file for both modified workflow files.
  • bun test tests/renew-batch.test.ts tests/agent-join.test.ts tests/config-environment.test.ts

However, there's no automated test in the CI that verifies:

  1. The YAML syntax is valid (GitHub will validate this, but a pre-merge check would be helpful)
  2. The deployment workflows work end-to-end in testnet

Recommendation: Consider adding a workflow validation step (e.g., using actionlint or similar) to catch YAML errors before merge.

✅ Runtime verification is present

The test_lifecycle_testnet job (lines 201-235) properly validates:

  • Deployment health via /healthcheck endpoint
  • Lifecycle functionality via /api/v2/assets/test/lifecycle endpoint
  • This provides good runtime coverage

Documentation

✅ Release process well-documented

File: RELEASE.md

  • Clear three-stage flow (dev → testnet → prod)
  • Explicit fast-forward merge instructions
  • Rules section prevents common mistakes (e.g., pushing to prod without going through testnet)

📝 Suggested improvement

Consider adding a section to RELEASE.md about:

  • How to verify testnet is healthy before promoting to prod
  • Expected timeline between stages (e.g., "let testnet run for 24 hours before prod")
  • Rollback procedures if testnet reveals issues

Summary

Verdict: ✅ Approve with minor suggestions

This PR successfully implements testnet deployment infrastructure. The changes are solid and follow established patterns. The main improvement opportunity is refactoring the duplicated verification logic in s3-lifecycle-verify.yml into a reusable workflow.

Priority Actions:

  1. High: Consider refactoring s3-lifecycle-verify.yml to use reusable workflows (reduces tech debt)
  2. Medium: Clarify if deploy-otr-dev should be added to repository_dispatch types
  3. Low: Add workflow validation to CI (prevents YAML syntax errors)
  4. Low: Document testnet verification timeline in RELEASE.md

No blocking issues identified - the code is production-ready as-is.


This was referenced Apr 15, 2026
@neekolas
neekolas force-pushed the deploy-otr-testnet branch from 9d542ea to 797d80e Compare April 15, 2026 22:38
@neekolas
neekolas force-pushed the testnet-agent-invite-url branch from 3eab43d to 09c00e0 Compare April 15, 2026 22:38
@neekolas
neekolas force-pushed the deploy-otr-testnet branch from 797d80e to 9d954ee Compare April 15, 2026 23:00
@neekolas
neekolas force-pushed the testnet-agent-invite-url branch from 09c00e0 to 5550f93 Compare April 15, 2026 23:00

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
.github/workflows/s3-lifecycle-verify.yml (1)

81-150: Correct implementation, but consider extracting to a reusable workflow.

The verify-testnet job correctly mirrors the existing verify-dev and verify-prod jobs. The endpoint, authorization, and validation logic are all consistent.

However, all three jobs share ~70 lines of identical shell script logic. Consider extracting the verification logic into a reusable workflow or a composite action that accepts the API URL and environment name as inputs. This would reduce maintenance burden when updating the verification logic.

💡 Example reusable workflow structure

Create .github/workflows/s3-lifecycle-verify-env.yml:

name: S3 Lifecycle Verify (Reusable)
on:
  workflow_call:
    inputs:
      environment:
        required: true
        type: string
      api_url:
        required: true
        type: string
    secrets:
      LIFECYCLE_TEST_TOKEN:
        required: true

jobs:
  verify:
    name: Verify Lifecycle (${{ inputs.environment }})
    runs-on: ubuntu-latest
    steps:
      - name: Verify S3 lifecycle canaries
        run: |
          # ... shared verification logic using ${{ inputs.api_url }}

Then call it from the main workflow:

jobs:
  verify-dev:
    uses: ./.github/workflows/s3-lifecycle-verify-env.yml
    with:
      environment: Dev
      api_url: ${{ vars.DEV_API_URL }}
    secrets:
      LIFECYCLE_TEST_TOKEN: ${{ secrets.LIFECYCLE_TEST_TOKEN }}
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In @.github/workflows/s3-lifecycle-verify.yml around lines 81 - 150, The
verify-testnet job duplicates ~70 lines of shared shell logic also present in
verify-dev and verify-prod; extract that logic into a reusable workflow (e.g.,
.github/workflows/s3-lifecycle-verify-env.yml) or a composite action that
exposes inputs environment and api_url and expects the LIFECYCLE_TEST_TOKEN
secret, move the "Verify S3 lifecycle canaries" script into the reusable
workflow (job name verify), and replace the
verify-dev/verify-testnet/verify-prod jobs with uses:
./.github/workflows/s3-lifecycle-verify-env.yml + with: environment/api_url and
the secrets mapping so all three jobs call the single shared implementation.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In @.github/workflows/s3-lifecycle-verify.yml:
- Around line 81-150: The verify-testnet job duplicates ~70 lines of shared
shell logic also present in verify-dev and verify-prod; extract that logic into
a reusable workflow (e.g., .github/workflows/s3-lifecycle-verify-env.yml) or a
composite action that exposes inputs environment and api_url and expects the
LIFECYCLE_TEST_TOKEN secret, move the "Verify S3 lifecycle canaries" script into
the reusable workflow (job name verify), and replace the
verify-dev/verify-testnet/verify-prod jobs with uses:
./.github/workflows/s3-lifecycle-verify-env.yml + with: environment/api_url and
the secrets mapping so all three jobs call the single shared implementation.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: e2cca702-a46f-4140-963c-aca1bcc86655

📥 Commits

Reviewing files that changed from the base of the PR and between 5550f93 and 9d954ee.

📒 Files selected for processing (3)
  • .github/workflows/deploy-aws.yml
  • .github/workflows/s3-lifecycle-verify.yml
  • RELEASE.md

@neekolas
neekolas marked this pull request as ready for review April 15, 2026 23:14
@neekolas
neekolas requested a review from a team as a code owner April 15, 2026 23:14
@macroscopeapp

macroscopeapp Bot commented Apr 15, 2026

Copy link
Copy Markdown

Approvability

Verdict: Needs human review

This PR adds testnet deployment infrastructure and includes a minor change to auth middleware logging. While the changes are straightforward additions, the author does not own any of the modified files (all owned by @xmtplabs/engineering), and one change touches the sensitive auth middleware directory. The designated code owners should review these changes.

You can customize Macroscope's approvability policy. Learn more.

@neekolas
neekolas force-pushed the deploy-otr-testnet branch from 9d954ee to fb28045 Compare April 16, 2026 21:43

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🧹 Nitpick comments (1)
src/middleware/auth.ts (1)

46-47: Consider applying the same error normalization to the other catch blocks.

The normalization here is a nice improvement (avoids dumping non-Error throwables as {} in structured logs). For consistency, the catch blocks in authMiddleware (Line 97) and authMiddlewareAllowNSE (Line 166) still log the raw { error } object and would benefit from the same treatment.

🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.

In `@src/middleware/auth.ts` around lines 46 - 47, The catch blocks in
authMiddleware and authMiddlewareAllowNSE should normalize the caught value
before logging like the AppCheck block does; replace logging of the raw { error
} with a normalized message computed as const message = error instanceof Error ?
error.message : String(error) and call req.log.error({ error: message }, "...")
(use the same log text as the existing catch) so non-Error throwables are logged
consistently; update the catch handlers inside the authMiddleware and
authMiddlewareAllowNSE functions to use that normalization.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.

Nitpick comments:
In `@src/middleware/auth.ts`:
- Around line 46-47: The catch blocks in authMiddleware and
authMiddlewareAllowNSE should normalize the caught value before logging like the
AppCheck block does; replace logging of the raw { error } with a normalized
message computed as const message = error instanceof Error ? error.message :
String(error) and call req.log.error({ error: message }, "...") (use the same
log text as the existing catch) so non-Error throwables are logged consistently;
update the catch handlers inside the authMiddleware and authMiddlewareAllowNSE
functions to use that normalization.

ℹ️ Review info
⚙️ Run configuration

Configuration used: Organization UI

Review profile: CHILL

Plan: Pro

Run ID: ebc1f0a8-75c4-4158-8a0a-affef5915d44

📥 Commits

Reviewing files that changed from the base of the PR and between 9d954ee and fb28045.

📒 Files selected for processing (4)
  • .github/workflows/deploy-aws.yml
  • .github/workflows/s3-lifecycle-verify.yml
  • RELEASE.md
  • src/middleware/auth.ts
✅ Files skipped from review due to trivial changes (1)
  • RELEASE.md
🚧 Files skipped from review as they are similar to previous changes (2)
  • .github/workflows/s3-lifecycle-verify.yml
  • .github/workflows/deploy-aws.yml

@fbac

fbac commented Jun 26, 2026

Copy link
Copy Markdown
Collaborator

@neekolas with the TF env removed, we can close this one?

@neekolas neekolas closed this Jun 26, 2026
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

3 participants