Repository navigation
test(model): check Triage's rules across event orders with the real workflows - #746
Conversation
480f108 to
805c033
Compare
3db42ea to
e38817f
Compare
|
Triage: Pass No changes needed from you. A maintainer will review this pull request. Notes:
Approve held: code review did not read the whole change on this head. |
There was a problem hiding this comment.
Adds a model test that runs Triage's real GitHub listener and manager workflows through Interchange's runLocal, plays many orders of pull request events, and checks rules R1, R3 and R5 once each pull request settles. The most important defect is that deployRun keeps only the first run-token row, so every hub consumer of a run gets the same token, which overstates the authorization coverage.
This review did not read the whole change; approve and merge are held until a review does.
There was a problem hiding this comment.
Adds a model test that runs Triage's real GitHub listener and manager workflows through many orders of pull request events, then checks rules R1, R3 and R5 once each pull request settles. The main defect is that the unattributed-manager check can record managers as reported inside checks whose results are then discarded, so the R5 failure for those managers never surfaces.
This review did not read the whole change; approve and merge are held until a review does.
There was a problem hiding this comment.
Adds a model-check test that runs Triage's real GitHub listener and manager workflows through Interchange's runLocal against stand-ins for deploy, stop, mail delivery and GitHub. It plays every sequence of up to two events after open, plus 300 seeded random walks, then checks rules R1, R3 and R5 once each pull request settles. No findings were raised in this review.
This review did not read the whole change; approve and merge are held until a review does.
Triage's real listener and manager workflows now run in one test process. The test plays many orders of pull request events through them and, once each pull request settles, checks three of Triage's rules: R1, R3 and R5 in
docs/INVARIANTS.md. This branch builds on #748, which moves the chain helpers.What the test runs:
tests/model/runs the real GitHub listener and manager definitions through Interchange'srunLocaland the stock step invoker. They run against the hub's own intake and manager routes on a database of their own, with the chain's fake GitHub.The rules checked once a pull request has settled (ready, CI finished, every mail delivered):
Stand-ins and what they leave out (also in
docs/INVARIANTS.mdanddocs/DEV.md):managegrant and ends the run at once, so the cancellation window is not played.mailAcceptcheck and its mail hold on a just-deployed run are bypassed. Mail to a run that has ended is refused at send and dropped at delivery, and the hub'sdeliverthrowsRunUnavailableErrorfor a run that is not live.Not covered yet: interleaving inside a pass (a push or a second manager while a pass runs, and contention for the claim), the cancellation window,
mailAcceptand the real coalescer, a fake clock, judge and review stand-ins (so R2), and every sequence of three events.Docs:
docs/INVARIANTS.mdlists all ten rules from the issue, each with its check or "No automated check yet". AGENTS.md says what the model check covers and that a change to the listener, manager, workflows or contracts names the rules it touches. DEV.md and IMPLEMENTATION.md describe the test.Verification:
bun run checkandbun run lintpass.tests/modelpassed on the final code: 121 sequences checked plus 300 walks, with no rule broken, in 58 s./manager/deployedand the stop route not ending the run. It reported "managers still live: run_…" even though the hub was never told about those managers. Skipping/manager/deployedalone breaks no rule.deployedgrant made the listener's run fail.Part of CL-11739