Skip to content

test: enforce api test boundary lint rules#20713

Merged
e7h4n merged 1 commit into
mainfrom
fix/api-test-boundary-lint
Jul 9, 2026
Merged

test: enforce api test boundary lint rules#20713
e7h4n merged 1 commit into
mainfrom
fix/api-test-boundary-lint

Conversation

@e7h4n

@e7h4n e7h4n commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

Summary

  • Delete service-level API tests and tighten API lint so src/signals/services/**/*.test.ts, sibling *.service imports from API tests, and direct signals/external/db imports from API tests are rejected.
  • Migrate all vitest API tests off HTTP fixture routes. 12 test-only contract modules deleted (route + contract + helper each): test-relationship-state, test-usage-state, test-generation-state, test-memory-state, test-chat-thread-state, test-chat-messages-state, test-workflow-trigger-state, test-email-state, test-webhooks-state, test-billing-redeem-state, test-onboarding-status-state, test-user-export-state.
  • Migrated tests now construct state through real production APIs (connector OAuth callbacks, chat sends, run creation, storage prepare/commit, inbound email/Stripe/Clerk/GitHub webhooks, runner claim/checkpoint/complete callbacks, cron endpoints), mock external providers with MSW, isolate via random org/user/thread/message IDs (all fixture-teardown endpoints removed), and assert through product read surfaces.
  • State that production APIs cannot construct deterministically lives in a new narrow non-HTTP fixture layer apps/api/src/test-fixtures/ (14 small doc-commented modules, e.g. identical-created_at chat messages for cursor tie-breaks, platform vm0_api_keys rows, usage_pricing config, aged email-outbox rows, expired credit grants, legacy pre-event-sourcing threads). No HTTP route, no contract.
  • Replace the firewall-auth refresh-timeout test seam (setFirewallAuthRefreshTimeoutMsForTests) with an optional FIREWALL_AUTH_REFRESH_TIMEOUT_MS env read, and make the email-outbox drain's Resend pacing overridable via EMAIL_OUTBOX_DRAIN_DELAY_MS (tests disable the pacing instead of widening timeouts).
  • Delete the orphaned services/connector-catalog-artifacts/ directory (its only importer was a deleted service-level test).

Global-sweep cron isolation

Removing fixture teardown surfaced a class of cross-file interference: several crons (summarize-memory, drain-email-outbox, refresh-storage-presigned-urls) sweep shared tables globally, so one file's leftover fixtures leak into another file's sweep. Fixes, all through product surfaces:

  • Tests that enable the MemoryViewer switch (cron-summarize-memory.test.ts, run-lifecycle.bdd.test.ts) now opt their fixture users back out via the same product feature-switch API in onTestFinished, keeping later sweeps scoped.
  • The summarize-memory idempotency test asserts fixture-scoped activity (via GET /api/zero/memory/activity) instead of the global cron response.
  • The bounded presigned-URL refresh test asserts the per-tick budget on the global counter (refreshed <= 3) and then polls repeated cron ticks until its fixture rows converge — the same way production converges — instead of asserting a single tick's global counts.

Deployed E2E infrastructure is explicitly out of scope

An earlier revision of this PR removed all test routes from the global route table and deleted the Slack/Telegram/OAuth mock routes. That broke deploy-api ("Generate E2E test tokens" calls /api/cli/auth/test-token on the preview) and would break cli-e2e-01-serial / cli-e2e-02-playwright / runner suites plus the test-oauth connectors in packages/connectors, whose provider endpoints are /api/test/oauth-provider/* on the deployed preview. Those consumers run against a remote deployment where in-process MSW is impossible.

This PR therefore splits the surface:

  • Removed: every fixture route whose only consumers were vitest tests (list above).
  • Kept, documented, and quarantined: 13 deployed-E2E infrastructure modules (cli-auth-test, test-oauth-provider-* ×5, test-slack-mock, test-slack-state, test-slack-dispatch-probe, test-telegram-mock, test-telegram-state, test-telegram-dispatch-probe, test-zero-agent-state), re-mounted through a single documented registry apps/api/src/signals/e2e-routes.ts (E2E_ROUTES). All remain env-gated by isTestEndpointAllowed (development / preview-with-bypass only; 404 in production). The E2E_SLACK_MOCK_ENABLED / E2E_TELEGRAM_MOCK_ENABLED preview redirection in the Slack/Telegram clients is restored for the same reason. The lint guard still prevents src/signals/route.ts from importing ./routes/test-* or ./routes/cli-auth-test directly — e2e-routes.ts is the only sanctioned aggregation point.

Current test-only API surface

  • Global route table: only E2E_ROUTES (13 env-gated deployed-E2E modules); zero vitest-fixture routes.
  • Remaining test-only contract modules: 21 (down from 33 on main) — 13 deployed-E2E infra + 8 vitest-harness modules not yet migrated (test-automations-state, test-banking-state, test-computer-use-state, test-cron-cleanup-sandboxes-state, test-cron-sync-skills-state, test-system-storage-presigned-url-cache-state, test-usage-insight-state, test-workflow-skill-storage-presigned-url-cache-state), tracked as follow-up.
  • Some vitest tests still seed Slack/Telegram fixtures through the retained e2e-infra state routes (helpers/zero-integrations-slack.ts, helpers/zero-telegram.ts, helpers/api-bdd-firewall.ts / api-bdd-auth-device.ts provisioning); migrating those to connector-OAuth + MSW is follow-up work.

Coverage notes

  • Rebased over test(api): enforce external behavior test boundary #20735/feat: inject zero memory profile context #20726: the deterministic alias-race trigger and the runtime-injection memory seeds that landed on main inside test-relationship-state were ported into src/test-fixtures/relationship-memory.ts (non-HTTP), so the alias-race, injection-preview, and injection-window tests keep their coverage while the fixture route stays deleted.
  • Relationship Gmail mocks now authenticate a per-fixture access token and 401/404 foreign requests: the drain cron works a shared global job queue, so another worker can claim a fixture's job — with token-checked mocks that attempt fails and the queue retries until the owning worker processes it, instead of fabricating foreign content into the fixture's org.
  • A small number of assertions moved from internal-row reads to behavioral equivalents (e.g. promo-redemption rows proven via resume/rotate/fresh-session responses; usage charges proven via billing-status balance drops and GET /api/zero/usage/record; webhook dedup proven via no-duplicate-run). Tests that only exercised fixture internals or production-unreachable defensive branches were deleted with in-place justification.

Verification

  • pnpm -F api lint
  • pnpm -F @vm0/api-contracts check-types
  • NODE_OPTIONS=--max-old-space-size=4096 pnpm -F api check-types
  • pnpm knip (clean)
  • pnpm format, git diff --check
  • Full pnpm -F api exec vitest run locally against PostgreSQL: 152 files / 2010 tests, green in consecutive runs

Local-dev caveat

Random-ID isolation on a persistent local database trades teardown for accumulation: global-sweep crons (summarize-memory, aggregate-insights, presigned-URL refresh) slowly gain work from leftover fixtures across runs. CI is unaffected (fresh database per run). The affected tests now scope their fixtures (MemoryViewer opt-out) or pre-drain/converge through repeated cron ticks; if a long-lived local database still gets slow, scripts/prepare.sh / a DB reseed clears the backlog.

@codecov

codecov Bot commented Jul 8, 2026

Copy link
Copy Markdown

Codecov Report

✅ All modified and coverable lines are covered by tests.
✅ All tests successful. No failed tests found.

📢 Thoughts on this report? Let us know!

@github-actions

github-actions Bot commented Jul 8, 2026

Copy link
Copy Markdown
Contributor

API preview: https://pr-20713-api.vm6.ai

@e7h4n e7h4n force-pushed the fix/api-test-boundary-lint branch 6 times, most recently from de0cd1f to 10b2340 Compare July 8, 2026 14:00
@e7h4n e7h4n changed the title test: enforce API test boundary lint rules test: enforce api test boundary lint rules Jul 9, 2026
@e7h4n e7h4n force-pushed the fix/api-test-boundary-lint branch from 6464415 to c9f41eb Compare July 9, 2026 01:51
Enforce the API test boundary model: no service-level API tests, no direct
DB access from tests or helpers, and no vitest usage of /api/test/* fixture
routes. Tests construct state through production APIs with MSW-mocked
providers, isolate by random IDs, and assert through product read surfaces.
Deletes 13 test-only contract modules; production-unreachable state moves to
a narrow non-HTTP fixture layer under src/test-fixtures/.

Keep the deployed-E2E infrastructure (cli-auth-test provisioning, test-oauth
provider routes, Slack/Telegram mock/state/probe routes and client
redirection) behind a single documented E2E_ROUTES registry, still env-gated
to development/preview.

Scope global-sweep crons in tests: MemoryViewer fixtures opt back out after
each test, drain pacing and the firewall refresh timeout become optional env
reads, and bounded-sweep assertions poll to convergence instead of asserting
one tick's global counters.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@e7h4n e7h4n force-pushed the fix/api-test-boundary-lint branch from c9f41eb to 0399e10 Compare July 9, 2026 02:00
@e7h4n e7h4n added this pull request to the merge queue Jul 9, 2026
Merged via the queue into main with commit 2cb7a68 Jul 9, 2026
92 checks passed
@github-project-automation github-project-automation Bot moved this from Backlog to Done in VM0 Kanban Jul 9, 2026
@github-actions github-actions Bot deleted the fix/api-test-boundary-lint branch July 9, 2026 04:21
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

Status: Done

Development

Successfully merging this pull request may close these issues.

1 participant