Skip to content

Commit 480f108

Browse files
committed
test(model): check Triage's rules across event orders with the real workflows
1 parent e310e35 commit 480f108

12 files changed

Lines changed: 298 additions & 67 deletions

File tree

‎AGENTS.md‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -60,6 +60,7 @@ Check these for a stock route or primitive before touching the hub, sidecars or
6060
- Only load-bearing tests, end to end where possible.
6161
- Tests beside the code are fast and offline; anything that runs real services lives in `e2e/`.
6262
- The one exception is `tests/chain/`: it starts the real hub and local sidecars but reaches no outside service (a fake GitHub and a stub decision model), and CI runs it as its own job, `bun run test:chain`.
63+
- `tests/model/` runs the real listener and manager in one process against the hub's own routes and a fake GitHub, and checks the rules in [docs/INVARIANTS.md](docs/INVARIANTS.md) across event orders. A change to the listener, the manager, the workflows or the contracts names the rules it touches and keeps it green.
6364
- Never read `process.env` in tests.
6465
- A failed chain shard is read, never rerun.
6566
- A change to how a workflow judges or reviews states its package's eval result before and after.

‎bun.lock‎

Lines changed: 1 addition & 0 deletions
Some generated files are not rendered by default. Learn more about customizing how changed files appear on GitHub.

‎docs/DEV.md‎

Lines changed: 2 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -91,6 +91,8 @@ Hub tests connect to `postgres://postgres:postgres@localhost:5432/interchange` a
9191

9292
The chain starts its hub with its waits short (`TIMINGS` and `MAIL_HOLD_MS` in `tests/chain/stack.ts`, set through the `TRIAGE_*_MS` variables in [SELF_HOST.md](SELF_HOST.md#triage-tuning) and `HUB_MAIL_HOLD_TTL_MS`). A scenario polls every 100 ms for the state it names and never sleeps for a fixed time, which `bun run lint` enforces under `tests/chain/`. The real waits left are stock's one-minute ping timeout in the scenarios that freeze a listener's sidecar, the one-minute mail hold `unstarted` waits out, and the manager's 30-second poll in `woken`. A failed shard is handled as the [Tests rules](../AGENTS.md#tests) say.
9393

94+
`tests/model/` checks the rules in [INVARIANTS.md](INVARIANTS.md) across event orders. It runs the real listener and manager definitions in the test's own process through Interchange's `runLocal`, against the hub's own intake and manager routes on a database of its own and the chain's fake GitHub; stock's deploy and stop-run routes are stood in for, and mail between runs waits in a queue the test delivers. It is a Postgres test, so `bun run test:postgres` runs it, in about a minute and a half.
95+
9496
In CI the chain runs in five shards, each with its own list of files in `.github/workflows/ci.yml`; a new chain file must be added to one. A pull request that changes none of `apps/hub`, `apps/sidecar`, `apps/web/scripts`, `apps/web/src/lib`, `tooling/`, `packages/`, `tests/chain/`, `vendor/`, the lockfile, the root `package.json` or the CI workflow and setup actions skips the chain, so a change limited to the portal's pages and components, docs, e2e or research does not run it; every push to `main` runs it.
9597

9698
Scripts are listed in [IMPLEMENTATION.md](IMPLEMENTATION.md#scripts).

‎docs/IMPLEMENTATION.md‎

Lines changed: 1 addition & 1 deletion
Original file line numberDiff line numberDiff line change
@@ -111,7 +111,7 @@ A WXT browser extension that checks the open GitHub pull request locally; see
111111
| `bun run dev` | Database migrations, then hub and portal with prefixed logs (`tooling/dev.ts`). |
112112
| `bun run db:migrate` | Apply every database migration the hub needs to `DATABASE_URL`; run before the hub starts. |
113113
| `bun run hub` | Hub only; does not migrate. |
114-
| `bun run check` | Typecheck tooling, portal, extension, hub, packages, `e2e/` and `tests/chain/`, and bundle the hub. |
114+
| `bun run check` | Typecheck tooling, portal, extension, hub, packages, `e2e/`, `tests/chain/` and `tests/model/`, and bundle the hub. |
115115
| `bun run test` | Unit and integration tests; hub tests need Postgres. |
116116
| `bun run test:unit`, `test:postgres`, `test:dom` | The three parts of `bun run test`: tests that need no Postgres, the `*.postgres.test.ts` tests, and the portal's DOM tests. CI runs unit and Postgres tests in three shards each (`--shard=1/3`). |
117117
| `bun run test:chain` | The live triage chain from webhook to pull request on one machine, against a fake GitHub and a stub decision model; needs Postgres and reaches the npm registry for native modules. |

‎docs/INVARIANTS.md‎

Lines changed: 21 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,21 @@
1+
# Invariants
2+
3+
The rules Triage keeps on every pull request, whatever order its events arrive in. Each rule here has a check:
4+
`tests/model/rules.postgres.test.ts` plays sequences of events on a pull request through the real listener and manager
5+
workflows, lets it settle, and checks the rule on what GitHub ends up holding and what the hub stored. A pull request has
6+
settled once it is ready for review, CI on its head has finished and every mail has been delivered.
7+
8+
Events played: open, push, CI pending, success, failure or cancelled on the head, CI finishing on an older head, an
9+
edit, conversion to draft and back to ready, a mail delivered twice, and which queued mail is delivered next. Every
10+
sequence of up to two of them after the open is played, and 300 seeded random walks of ten. A failure names the
11+
sequence, and a walk's seed, so it replays.
12+
13+
| Rule | What holds once the pull request has settled | Enforced by |
14+
| --- | --- | --- |
15+
| R1 | The pull request has exactly one current Triage verdict comment, and it names the head. | `tests/model/rules.postgres.test.ts` |
16+
| R3 | No effect is written twice: no comment appears on the pull request twice. | `tests/model/rules.postgres.test.ts` |
17+
| R5 | No manager run is live and no mail is left undelivered. | `tests/model/rules.postgres.test.ts` |
18+
19+
Labels are judged by where they end. A pass that triages a head again may take a verdict label off and put it back, so
20+
a label can flicker while a pass runs; the label set GitHub ends with is what counts. GitHub keeps labels as a set and the
21+
hub keeps one verdict row per manager run and pull request, so neither can hold a duplicate.

‎tests/chain/github.ts‎

Lines changed: 7 additions & 2 deletions
Original file line numberDiff line numberDiff line change
@@ -10,7 +10,7 @@ const TOKEN = "installation-token";
1010
const READ_TOKEN = "read-token";
1111

1212
export type File = { filename: string; status: string; additions: number; deletions: number; patch: string };
13-
export type CheckRun = { name: string; status: "queued" | "in_progress" | "completed"; conclusion: "success" | "failure" | null };
13+
export type CheckRun = { name: string; status: "queued" | "in_progress" | "completed"; conclusion: "success" | "failure" | "cancelled" | null };
1414
export type Pull = {
1515
number: number;
1616
title: string;
@@ -21,6 +21,7 @@ export type Pull = {
2121
commits: string[];
2222
labels: string[];
2323
comments: { id: number; user: typeof BOT; body: string }[];
24+
draft?: boolean;
2425
};
2526
export type Write = { method: string; path: string; body: unknown };
2627

@@ -57,7 +58,7 @@ export function fakeGithub() {
5758

5859
function pullJson(repo: Repo, pull: Pull) {
5960
return {
60-
number: pull.number, title: pull.title, body: pull.body, body_html: `<p>${pull.body}</p>`, state: "open", draft: false, merged: false, mergeable: true, mergeable_state: "clean",
61+
number: pull.number, title: pull.title, body: pull.body, body_html: `<p>${pull.body}</p>`, state: "open", draft: pull.draft ?? false, merged: false, mergeable: true, mergeable_state: "clean",
6162
user: { login: pull.author, type: "User" }, author_association: "CONTRIBUTOR", head: { sha: pull.head, ref: `pr-${pull.number}`, repo: { id: 1, full_name: repo.name } }, base: { ref: "main", sha: "0".repeat(40), repo: { id: 1, full_name: repo.name } },
6263
additions: pull.files.reduce((sum, file) => sum + file.additions, 0), deletions: pull.files.reduce((sum, file) => sum + file.deletions, 0),
6364
changed_files: pull.files.length, labels: pull.labels.map((name) => ({ name })), requested_reviewers: [], requested_teams: [], assignees: [],
@@ -97,6 +98,7 @@ export function fakeGithub() {
9798
if (commit[2] === "status") return Response.json({ state: "success", statuses: [], total_count: 0 });
9899
return Response.json({ total_count: 0, check_suites: [] });
99100
}
101+
if (rest.startsWith("/compare/")) return Response.json({ status: "ahead", ahead_by: 1, behind_by: 0, commits: [] });
100102
if (rest.startsWith("/git/trees/")) return Response.json({ sha: "tree", tree: [], truncated: false });
101103
if (rest.startsWith("/collaborators/")) return Response.json({ permission: "read", role_name: "read" });
102104
if (rest === "/labels" || rest.startsWith("/labels?")) return Response.json([]);
@@ -201,6 +203,9 @@ export function fakeGithub() {
201203
setChecks(repo: string, head: string, runs: CheckRun[]): void {
202204
repos.get(repo)!.checks.set(head, runs);
203205
},
206+
checksOf(repo: string, head: string): CheckRun[] {
207+
return repos.get(repo)!.checks.get(head) ?? [];
208+
},
204209
writesTo(repo: string, number: number): Write[] {
205210
return writes.filter((write) => write.path.startsWith(`/repos/${repo}/issues/${number}/`) || write.path.startsWith(`/repos/${repo}/pulls/${number}/`));
206211
},

‎tests/model/events.ts‎

Lines changed: 126 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,126 @@
1+
// The events a pull request's sequence is made of, each played on the stack as GitHub and the hub would see it, and the
2+
// seeded random walks the rules are checked over past the sequences played in full.
3+
import { pullOf, sha, type CheckRun } from "../chain/github.ts";
4+
import type { modelStack } from "./stack.ts";
5+
6+
type Stack = Awaited<ReturnType<typeof modelStack>>;
7+
8+
export const LETTERS = ["push", "ci-pending", "ci-success", "ci-failure", "ci-cancelled", "ci-stale", "edit", "draft", "ready", "duplicate", "run-first", "run-second"] as const;
9+
export type Letter = (typeof LETTERS)[number];
10+
11+
/** One pull request a sequence plays on: its number, every head it had, newest last, and whether it is a draft. */
12+
export type Played = { number: number; heads: string[]; draft: boolean };
13+
14+
const BUILD = "build";
15+
16+
function current(pull: Played): string {
17+
return pull.heads.at(-1)!;
18+
}
19+
20+
/**
21+
* The bridge takes in no webhook of a draft but its conversion to one (apps/hub/src/github/bridge.ts), so a draft's
22+
* events reach the hub as nothing.
23+
*/
24+
async function webhook(stack: Stack, pull: Played, event: Parameters<Stack["webhook"]>[1], head = current(pull)): Promise<void> {
25+
if (pull.draft && event !== "drafted") return;
26+
await stack.webhook(pull.number, event, head);
27+
}
28+
29+
/** CI on the head settles; GitHub sends a webhook only for a completed check, never for one still running. */
30+
async function checks(stack: Stack, pull: Played, run: CheckRun, head = current(pull)): Promise<void> {
31+
stack.github.setChecks(stack.repo, head, [run]);
32+
if (run.status === "completed") await webhook(stack, pull, "checks", head);
33+
}
34+
35+
/** Plays `letter`; false when it changes nothing in this state, so the sequence equals a shorter one. */
36+
export async function play(stack: Stack, pull: Played, letter: Letter): Promise<boolean> {
37+
switch (letter) {
38+
case "push": {
39+
const head = sha(`${pull.number}-${pull.heads.length}`);
40+
pull.heads.push(head);
41+
stack.github.pull(stack.repo, pull.number).head = head;
42+
stack.github.setChecks(stack.repo, head, []);
43+
await webhook(stack, pull, "updated");
44+
return true;
45+
}
46+
case "ci-pending":
47+
await checks(stack, pull, { name: BUILD, status: "in_progress", conclusion: null });
48+
return true;
49+
case "ci-success":
50+
await checks(stack, pull, { name: BUILD, status: "completed", conclusion: "success" });
51+
return true;
52+
case "ci-failure":
53+
await checks(stack, pull, { name: BUILD, status: "completed", conclusion: "failure" });
54+
return true;
55+
case "ci-cancelled":
56+
await checks(stack, pull, { name: BUILD, status: "completed", conclusion: "cancelled" });
57+
return true;
58+
case "ci-stale": {
59+
const old = pull.heads.at(-2);
60+
if (old === undefined) return false;
61+
await checks(stack, pull, { name: BUILD, status: "completed", conclusion: "success" }, old);
62+
return true;
63+
}
64+
case "edit":
65+
await webhook(stack, pull, "updated");
66+
return true;
67+
case "draft":
68+
if (pull.draft) return false;
69+
pull.draft = true;
70+
stack.github.pull(stack.repo, pull.number).draft = true;
71+
await webhook(stack, pull, "drafted");
72+
return true;
73+
case "ready":
74+
if (!pull.draft) return false;
75+
pull.draft = false;
76+
stack.github.pull(stack.repo, pull.number).draft = false;
77+
await webhook(stack, pull, "ready");
78+
return true;
79+
case "duplicate": {
80+
const [first] = stack.queue;
81+
if (!first) return false;
82+
stack.queue.push(first);
83+
return true;
84+
}
85+
case "run-first":
86+
if (stack.queue.length === 0) return false;
87+
await stack.run(0);
88+
return true;
89+
case "run-second":
90+
if (stack.queue.length < 2) return false;
91+
await stack.run(1);
92+
return true;
93+
}
94+
}
95+
96+
/** Opens pull request `number` at its first head. */
97+
export async function open(stack: Stack, number: number): Promise<Played> {
98+
const pull = { number, heads: [sha(`${number}-0`)], draft: false };
99+
await stack.open(pullOf(number, current(pull)));
100+
return pull;
101+
}
102+
103+
/**
104+
* Lets the pull request settle as it does once people stop acting on it: a draft is marked ready, CI on its head
105+
* finishes, and every mail is run.
106+
*/
107+
export async function settle(stack: Stack, pull: Played): Promise<void> {
108+
if (pull.draft) await play(stack, pull, "ready");
109+
await stack.drain();
110+
const [run] = stack.github.checksOf(stack.repo, current(pull));
111+
if (run?.status !== "completed") await play(stack, pull, "ci-success");
112+
await stack.drain();
113+
}
114+
115+
/** A random walk of `length` letters from `seed`, the same on every machine. */
116+
export function walk(seed: number, length: number): Letter[] {
117+
let state = seed >>> 0;
118+
function next(): number {
119+
state = (state + 0x6d2b79f5) >>> 0;
120+
let t = state;
121+
t = Math.imul(t ^ (t >>> 15), t | 1);
122+
t ^= t + Math.imul(t ^ (t >>> 7), t | 61);
123+
return ((t ^ (t >>> 14)) >>> 0) / 4_294_967_296;
124+
}
125+
return Array.from({ length }, () => LETTERS[Math.floor(next() * LETTERS.length)]!);
126+
}

‎tests/model/one-pull.postgres.test.ts‎

Lines changed: 0 additions & 29 deletions
This file was deleted.

‎tests/model/package.json‎

Lines changed: 1 addition & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -5,6 +5,7 @@
55
"license": "SEE LICENSE IN LICENSE.md",
66
"type": "module",
77
"dependencies": {
8+
"@corbits/github-reads": "workspace:*",
89
"@corbits/triage-contracts": "workspace:*",
910
"@corbits/triage-manager": "workspace:*",
1011
"@intx/agent": "workspace:*",

‎tests/model/rules.postgres.test.ts‎

Lines changed: 66 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -0,0 +1,66 @@
1+
// Requires PostgreSQL at localhost:5432 (docs/DEV.md). Checks the rules in docs/INVARIANTS.md after every sequence of
2+
// events on a pull request, each on a pull request of its own, so no reset is needed between them.
3+
import { afterAll, expect, mock, setDefaultTimeout, test } from "bun:test";
4+
import { emptyPack } from "@corbits/triage-contracts";
5+
import { OWNER } from "../chain/github.ts";
6+
import { LETTERS, open, play, settle, walk, type Letter } from "./events.ts";
7+
import { broken } from "./rules.ts";
8+
import { modelStack } from "./stack.ts";
9+
10+
/** Every sequence up to this many events after the open is played, less those with an event that changes nothing. */
11+
const LENGTH = 2;
12+
const WALKS = 300;
13+
const WALK_LENGTH = 10;
14+
15+
const MANAGER_RELEASE = { workflow: { assetId: "ast_triage-manager-github", commitSha: "c0ffee", packageName: "@corbits/triage-manager", entry: "./triage-manager-github.mjs" }, offerings: { ids: ["off_model"], defaultId: "off_model" } };
16+
mock.module(new URL("../../packages/triage-manager/src/release.json", import.meta.url).pathname, () => ({ default: { github: { manager: MANAGER_RELEASE } } }));
17+
18+
const REPO = `${OWNER}/model`;
19+
// Under `ask`, the pr-triage role holds only posting: its labels and its comment.
20+
const stack = await modelStack(REPO, { pack: { ...emptyPack(REPO), checks: { "max-files-changed": { enabled: true }, "ci-green": { enabled: true } } }, automatic: ["labels", "comment"] });
21+
setDefaultTimeout(5 * 60_000);
22+
afterAll(() => stack.stop());
23+
24+
let pulls = 0;
25+
26+
type Checked = { played: Letter[]; breaks: string[] };
27+
28+
/** Plays `letters` after an open on a pull request of its own, each that changes nothing left out, and settles it. */
29+
async function check(letters: readonly Letter[]): Promise<Checked> {
30+
const missed = stack.github.misses.length;
31+
const pull = await open(stack, ++pulls);
32+
const played: Letter[] = [];
33+
for (const letter of letters) if (await play(stack, pull, letter)) played.push(letter);
34+
await settle(stack, pull);
35+
return { played, breaks: await broken(stack, pull, missed) };
36+
}
37+
38+
function replayOf(letters: readonly Letter[]): string {
39+
return ["open", ...letters].join(" ");
40+
}
41+
42+
test(`every sequence up to ${LENGTH} events after an open keeps the rules`, async () => {
43+
const found: string[] = [];
44+
let level: Letter[][] = [[]];
45+
for (let size = 1; size <= LENGTH; size++) {
46+
const next: Letter[][] = [];
47+
for (const letters of level.flatMap((prefix) => LETTERS.map((letter) => [...prefix, letter]))) {
48+
const checked = await check(letters);
49+
// A sequence with an event that changed nothing is a shorter one already played, and so is every sequence after it.
50+
if (checked.played.length < letters.length) continue;
51+
next.push(letters);
52+
found.push(...checked.breaks.map((rule) => `${replayOf(letters)}: ${rule}`));
53+
}
54+
level = next;
55+
}
56+
expect(found).toEqual([]);
57+
});
58+
59+
test(`${WALKS} seeded walks of ${WALK_LENGTH} events keep the rules`, async () => {
60+
const found: string[] = [];
61+
for (let seed = 0; seed < WALKS; seed++) {
62+
const { played, breaks } = await check(walk(seed, WALK_LENGTH));
63+
found.push(...breaks.map((rule) => `seed ${seed}, ${replayOf(played)}: ${rule}`));
64+
}
65+
expect(found).toEqual([]);
66+
});

0 commit comments

Comments
 (0)