test(model): check Triage's rules across event orders with the real workflows

AGENTS.md

@@ -61,6 +61,7 @@ Check these for a stock route or primitive before touching the hub, sidecars or
 - Tests beside the code are fast and offline; anything that runs real services lives in `e2e/`.
 - The one exception is `tests/chain/`: it starts the real hub and local sidecars but reaches no outside service (a fake GitHub and a stub decision model), and CI runs it as its own job, `bun run test:chain`.
 - `tests/policy-corpus/` holds one policy file per case with its expected outcome, run through every reader of the policy; a new policy key needs a fixture there.
+- `tests/model/` checks R1, R3 and R5 of [docs/INVARIANTS.md](docs/INVARIANTS.md) across event orders, running the real GitHub listener and manager in one process against the hub's own routes and a fake GitHub under a deterministic policy; what it stands in for is listed there. A change to the listener, the manager, the workflows or the contracts names the rules it touches and keeps it green.
 - Never read `process.env` in tests.
 - A failed chain shard is read, never rerun.
 - A change to how a workflow judges or reviews states its package's eval result before and after.

docs/DEV.md

@@ -91,6 +91,8 @@ Hub tests connect to `postgres://postgres:postgres@localhost:5432/interchange` a
 
 The chain starts its hub with its waits short (`TIMINGS` and `MAIL_HOLD_MS` in `tests/chain/stack.ts`, set through the `TRIAGE_*_MS` variables in [SELF_HOST.md](SELF_HOST.md#triage-tuning) and `HUB_MAIL_HOLD_TTL_MS`). A scenario polls every 100 ms for the state it names and never sleeps for a fixed time, which `bun run lint` enforces under `tests/chain/`. The real waits left are stock's one-minute ping timeout in the scenarios that freeze a listener's sidecar, the one-minute mail hold `unstarted` waits out, and the manager's 30-second poll in `woken`. A failed shard is handled as the [Tests rules](../AGENTS.md#tests) say.
 
+`tests/model/` checks the rules in [INVARIANTS.md](INVARIANTS.md) across event orders. It runs the real GitHub listener and manager definitions in the test's own process through Interchange's `runLocal`, against the hub's own intake and manager routes on a database of its own and the chain's fake GitHub, with steps authorized against the grants a deployed run holds. Stock's deploy and stop-run routes are stood in for, mail between runs waits in a queue the test delivers, and webhooks enter at the hub's intake; [INVARIANTS.md](INVARIANTS.md#how-the-model-check-works) lists every stand-in and what it leaves out. It is a Postgres test, so `bun run test:postgres` runs it.
+
 In CI the chain runs in five shards, each with its own list of files in `.github/workflows/ci.yml`; a new chain file must be added to one. A pull request that changes none of `apps/hub`, `apps/sidecar`, `apps/web/scripts`, `apps/web/src/lib`, `tooling/`, `packages/`, `tests/chain/`, `vendor/`, the lockfile, the root `package.json` or the CI workflow and setup actions skips the chain, so a change limited to the portal's pages and components, docs, e2e or research does not run it; every push to `main` runs it.
 
 Scripts are listed in [IMPLEMENTATION.md](IMPLEMENTATION.md#scripts).

docs/IMPLEMENTATION.md

@@ -111,7 +111,7 @@ A WXT browser extension that checks the open GitHub pull request locally; see
 | `bun run dev` | Database migrations, then hub and portal with prefixed logs (`tooling/dev.ts`). |
 | `bun run db:migrate` | Apply every database migration the hub needs to `DATABASE_URL`; run before the hub starts. |
 | `bun run hub` | Hub only; does not migrate. |
-| `bun run check` | Typecheck tooling, portal, extension, hub, packages, `e2e/` and `tests/chain/`, and bundle the hub. |
+| `bun run check` | Typecheck tooling, portal, extension, hub, packages, `e2e/`, `tests/chain/` and `tests/model/`, and bundle the hub. |
 | `bun run test` | Unit and integration tests; hub tests need Postgres. |
 | `bun run test:unit`, `test:postgres`, `test:dom` | The three parts of `bun run test`: tests that need no Postgres, the `*.postgres.test.ts` tests, and the portal's DOM tests. CI runs unit and Postgres tests in three shards each (`--shard=1/3`). |
 | `bun run test:chain` | The live triage chain from webhook to pull request on one machine, against a fake GitHub and a stub decision model; needs Postgres and reaches the npm registry for native modules. |

docs/INVARIANTS.md

@@ -0,0 +1,59 @@
+# Invariants
+
+The rules Triage keeps on every pull request, whatever order its events arrive in. Breaking one is a defect. A pull
+request that changes how Triage keeps a rule names the rule.
+
+## Rules
+
+- **R1** One final verdict per pull request head, for the current head. Checked by `tests/model/rules.postgres.test.ts`: once the pull request settles it has exactly one current Triage comment, which is a verdict naming the head, and no waiting label.
+- **R2** Triage approves or merges only on Pass with a complete code review that meets the policy's findings condition, all for the current head. No automated check yet.
+- **R3** Every host write happens once under its effect id. Checked by `tests/model/rules.postgres.test.ts` over every write the pull request received, whichever pass made it: no Triage comment is created while one is current, unless that one was superseded, and no label is added while it is on.
+- **R4** Unreadable input never passes or approves. No automated check yet.
+- **R5** Nothing is left running after the final post. Checked by `tests/model/rules.postgres.test.ts`: once the pull request settles, none of its managers is live, no live manager was deployed for no single pull request, and no mail for it was lost to a run that had ended. Settling delivers every queued mail, so no mail is left waiting by construction.
+- **R6** An older run never overwrites a newer head's comment, label, status or approval. No automated check yet.
+- **R7** A pull request waiting on CI or a reviewer always gets its verdict. No automated check yet; R1's check covers a head waiting on CI.
+- **R8** GitHub and Bitbucket Data Center behave the same except where documented. No automated check yet.
+- **R9** Every reader of the policy agrees on what it says. No automated check yet.
+- **R10** The portal shows only what the hub says. No automated check yet.
+
+A label may come off and go back on while a pass runs; R1 judges the labels a pull request ends with, and R3 only an
+add while the label is on.
+
+## How the model check works
+
+`tests/model/` runs the real `triage-listener-github` and `triage-manager-github` definitions in the test's process
+through Interchange's `runLocal` and the stock step invoker. They run against the hub's own intake and manager routes on
+a database of their own and the chain's fake GitHub. Steps authorize through Interchange's credentials-backed authorize
+against the grants a deployed run holds: those its definition's walk yields, its credential bindings, and the grants
+recorded for it at deploy.
+
+Events played: open, push, CI pending, success, failure or cancelled on the head, CI finishing on an older head, an
+edit, conversion to draft and back to ready, a mail delivered twice, and which queued mail is delivered next. Every
+sequence of none to two of them after the open is played, and 300 seeded random walks of ten. A pull request settles
+once it is ready for review, CI on its head has finished and every mail has been delivered. A failure names its
+sequence, and a walk's seed, so it replays; a run that ends other than completed, or a sequence that throws, fails too.
+
+Stand-ins, and what they leave out:
+
+- Stock's deploy route records the run, its definition's grant requirements as creator grants, and its run tokens. It
+  checks the caller holds the deploy and the asset's read and refuses a body that names no published release, but it
+  does not resolve the requirements against the creator's own grants. It ties each manager it deploys to the pull
+  request named by the mail the listener was running, which R5 reads.
+- Stock's stop-run route checks the caller's `manage` grant and ends the run at once, so the window while a cancelled
+  run still runs is not played.
+- Mail waits in the test's queue. Stock's `mailAccept` check and its hold on mail to a just-deployed run are bypassed;
+  mail to a run that has ended is dropped or refused, as stock does.
+- Webhooks enter at the hub's intake as the coalescer flushes them. The bridge and coalescer do not run; the bridge's
+  rule that a draft's webhooks are dropped is played by the test.
+- The intake's hooks are fixed: the listener is always routable, no pull request is paused, the App's login is fixed,
+  and code review and backlog reads are not served.
+- Runs use the real clock, so CI's start grace, the manager's poll and its lifetime are not played.
+- The policy is deterministic only, with no judge, no code review and no model.
+
+## Not covered by the model check
+
+- Interleaving inside a pass: a push or a second manager while a pass runs, and contention for the claim.
+- The cancellation window, `mailAccept` and the listener's `onTrigger` admission, and the real coalescer.
+- A fake clock for CI and lifetime timers.
+- Judge and code review stand-ins, and so R2.
+- Every sequence of three events and more, which takes longer than CI allows on one shard; the seeded walks reach some.

package.json

@@ -9,6 +9,7 @@
     "e2e",
     "tests/chain",
     "tests/policy-corpus",
+    "tests/model",
     "vendor/interchange/packages/*",
     "vendor/interchange/apps/*",
     "vendor/interchange/tests/admin-ui-e2e"
@@ -33,7 +34,7 @@
     "test:unit": "bun --conditions=intx-src test --path-ignore-patterns='**/*.dom.test.tsx' --path-ignore-patterns='**/*.postgres.test.ts' ./apps/ ./packages/ ./tooling/ ./tests/policy-corpus/",
     "test:postgres": "bun --conditions=intx-src test .postgres.test.ts",
     "test:dom": "bun --conditions=intx-src test --isolate --preload ./apps/web/src/lib/dom-preload.ts .dom.test",
-    "check": "bun tooling/require-bun.ts && bun tooling/exact-deps-check.ts && bun check --no-script -p tooling && bun check --no-script -p apps/web && bun check --no-script -p apps/extension && bun check --no-script -p apps/extension/tsconfig.background.json && bun check --no-script -p apps/extension/e2e && bun check --no-script -p apps/hub && bun check --no-script -p packages/triage-workflows && bun check --no-script -p packages/triage-manager && bun check --no-script -p packages/triage-judge && bun check --no-script -p packages/pr-code-review && bun check --no-script -p packages/pr-code-review-anthropic && bun check --no-script -p packages/provisioner-test && bun check --no-script -p packages/provisioner-template && bun check --no-script -p packages/provisioner-docker && bun check --no-script -p packages/provisioner-e2b && bun check --no-script -p packages/provisioner-railway && bun check --no-script -p packages/policy-check-action && bun check --no-script -p e2e && bun check --no-script -p tests/chain && bun check --no-script -p tests/policy-corpus && bun build --conditions=intx-src --external @ast-grep/napi apps/hub/src/server.ts --target bun --outfile /dev/null",
+    "check": "bun tooling/require-bun.ts && bun tooling/exact-deps-check.ts && bun check --no-script -p tooling && bun check --no-script -p apps/web && bun check --no-script -p apps/extension && bun check --no-script -p apps/extension/tsconfig.background.json && bun check --no-script -p apps/extension/e2e && bun check --no-script -p apps/hub && bun check --no-script -p packages/triage-workflows && bun check --no-script -p packages/triage-manager && bun check --no-script -p packages/triage-judge && bun check --no-script -p packages/pr-code-review && bun check --no-script -p packages/pr-code-review-anthropic && bun check --no-script -p packages/provisioner-test && bun check --no-script -p packages/provisioner-template && bun check --no-script -p packages/provisioner-docker && bun check --no-script -p packages/provisioner-e2b && bun check --no-script -p packages/provisioner-railway && bun check --no-script -p packages/policy-check-action && bun check --no-script -p e2e && bun check --no-script -p tests/chain && bun check --no-script -p tests/policy-corpus && bun check --no-script -p tests/model && bun build --conditions=intx-src --external @ast-grep/napi apps/hub/src/server.ts --target bun --outfile /dev/null",
     "test:chain": "bun --conditions=intx-src test --parallel=2 tests/chain",
     "lint": "oxlint",
     "eval:review": "bun --conditions=intx-src packages/pr-code-review-anthropic/evals/review.eval.ts",

tests/chain/github.ts

@@ -10,7 +10,7 @@ const TOKEN = "installation-token";
 const READ_TOKEN = "read-token";
 
 export type File = { filename: string; status: string; additions: number; deletions: number; patch: string };
-export type CheckRun = { name: string; status: "queued" | "in_progress" | "completed"; conclusion: "success" | "failure" | null };
+export type CheckRun = { name: string; status: "queued" | "in_progress" | "completed"; conclusion: "success" | "failure" | "cancelled" | null };
 export type Pull = {
   number: number;
   title: string;
@@ -21,6 +21,7 @@ export type Pull = {
   commits: string[];
   labels: string[];
   comments: { id: number; user: typeof BOT; body: string }[];
+  draft?: boolean;
 };
 export type Write = { method: string; path: string; body: unknown };
 
@@ -57,7 +58,7 @@ export function fakeGithub() {
 
   function pullJson(repo: Repo, pull: Pull) {
     return {
-      number: pull.number, title: pull.title, body: pull.body, body_html: `<p>${pull.body}</p>`, state: "open", draft: false, merged: false, mergeable: true, mergeable_state: "clean",
+      number: pull.number, title: pull.title, body: pull.body, body_html: `<p>${pull.body}</p>`, state: "open", draft: pull.draft ?? false, merged: false, mergeable: true, mergeable_state: "clean",
       user: { login: pull.author, type: "User" }, author_association: "CONTRIBUTOR", head: { sha: pull.head, ref: `pr-${pull.number}`, repo: { id: 1, full_name: repo.name } }, base: { ref: "main", sha: "0".repeat(40), repo: { id: 1, full_name: repo.name } },
       additions: pull.files.reduce((sum, file) => sum + file.additions, 0), deletions: pull.files.reduce((sum, file) => sum + file.deletions, 0),
       changed_files: pull.files.length, labels: pull.labels.map((name) => ({ name })), requested_reviewers: [], requested_teams: [], assignees: [],
@@ -97,6 +98,7 @@ export function fakeGithub() {
       if (commit[2] === "status") return Response.json({ state: "success", statuses: [], total_count: 0 });
       return Response.json({ total_count: 0, check_suites: [] });
     }
+    if (rest.startsWith("/compare/")) return Response.json({ status: "ahead", ahead_by: 1, behind_by: 0, commits: [] });
     if (rest.startsWith("/git/trees/")) return Response.json({ sha: "tree", tree: [], truncated: false });
     if (rest.startsWith("/collaborators/")) return Response.json({ permission: "read", role_name: "read" });
     if (rest === "/labels" || rest.startsWith("/labels?")) return Response.json([]);
@@ -201,6 +203,9 @@ export function fakeGithub() {
     setChecks(repo: string, head: string, runs: CheckRun[]): void {
       repos.get(repo)!.checks.set(head, runs);
     },
+    checksOf(repo: string, head: string): CheckRun[] {
+      return repos.get(repo)!.checks.get(head) ?? [];
+    },
     writesTo(repo: string, number: number): Write[] {
       return writes.filter((write) => write.path.startsWith(`/repos/${repo}/issues/${number}/`) || write.path.startsWith(`/repos/${repo}/pulls/${number}/`));
     },