Skip to content

fix(triage): read a large pull request's whole diff for the judge - #691

Closed
TheGreatAxios wants to merge 1 commit into
mainfrom
cl-11613-split-a-large-diff-across-several-judge-calls
Closed

TheGreatAxios wants to merge 1 commit into
mainfrom
cl-11613-split-a-large-diff-across-several-judge-calls

Conversation

@TheGreatAxios

Copy link
Copy Markdown
Contributor

The judge already splits a large diff by file across several calls, up to 64 per pull request, and combines focused and docs in code (any slice that is a separate change fails focused, every serving slice passes it, anything else is Unsure; any file part needing docs fails docs). The trim that left #688 Unsure sat earlier, in the facts step: it stopped reading patches after 270,000 chars, three calls' worth, so later files reached the judge without lines and focused listed them as unread.

The facts step now reads patches up to what the 64 calls can hold (64 × 90,000 = 5,760,000 chars). Past that budget, files are listed as unread and the rule is Unsure naming them, as before. A split needing more than 64 calls leaves the judged rules Unsure with "pull request too large for the decision model (N requests)", as before. Nothing is trimmed silently. Small pull requests are unchanged. Bitbucket Data Center lists no patches, so it is unaffected.

  • facts.ts: MAX_PATCH_CHARS is MAX_JUDGE_REQUESTS * MAX_SYSTEM_ONE_STATE_BYTES. quality.ts exports MAX_JUDGE_REQUESTS.
  • docs/DECISION_MODEL.md: the limit, the unread-files text and the cost of the largest pull request. ARCHITECTURE.md and the user manual do not mention this budget.

Verification

  • feat: make the policy file the one source of what Triage may do #688 replayed offline through the real listFiles and prepareQualityEvaluation, using its files from the GitHub API. Before: 58 of 141 files had patches, 4 calls, and 78 files unread by focused. After: every file with lines has its patch (135; the other 6 are removed PNGs), 9 calls, and 0 files unread by focused.
  • The multi-call test in pr-triage.test.ts, which runs the facts director against a fake GitHub, now uses more than 270,000 chars of patch. It checks that every file reaches the judge across at least 4 calls. On main it fails with 10 files missing; on this branch it passes.
  • bun --conditions=intx-src test ./packages/triage-workflows/: 285 pass, 0 fail. bun run lint: exit 0, no new warnings.
  • bun run test:chain: locally it printed nothing for 300 s and was stopped, so CI runs it. Its fixture has a one-line patch and does not reach the budget.
  • Judge eval: not run, because it calls a paid model. Its result is the same before and after by construction. The change only matters above 270,000 chars of patch, and the largest eval pull request has 83,508 (refactor(web): rebuild the repository panel in the mockup's sentence layout #157). The largest mix is about 105,000 (feat(triage): let a policy request changes on a pull request #190 with refactor(hub): sign the GitHub App setup state instead of storing it #186). Every eval case was measured through the GitHub compare API.

Waiting on you

  • The patch budget per pull request. A pull request up to the 64-call bound is now read whole. At the bound that is about 640 GitHub file-page reads and up to 5.8 MB of patch carried through the workflow steps. My recommendation: keep it tied to the 64-call bound so there is one limit. The alternative is a lower shared bound, such as 16 calls / 1.44 MB, which still covers feat: make the policy file the one source of what Triage may do #688 (9 calls).
  • The verdict question. It is still one call that holds as much of the diff as fits after the rule results. It does not decide focused or docs. Splitting it would be a separate issue.

Closes CL-11613

The facts step stopped reading patches after 270,000 chars, three judge
requests' worth, so on a larger pull request the files past it reached
the judge without lines and focused and docs stayed unconfirmed. The
judge already splits a diff by file across up to 64 requests and combines
the answers in code; the patch budget now matches what those 64 requests
hold, and past it the unread files are listed as before.
@linear-code

linear-code Bot commented Oct 27, 2026

Copy link
Copy Markdown

CL-11613

@TheGreatAxios

Copy link
Copy Markdown
Contributor Author

Replaced by #692, which makes the change in the judge's budget in quality.ts on top of #690 instead of in facts.ts.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant