Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
44 changes: 24 additions & 20 deletions docs/ARCHITECTURE.md
Original file line number Diff line number Diff line change
Expand Up @@ -764,26 +764,30 @@ label off that head, so it never stands on a head not yet triaged.
pages as it takes, until it has read 2,000,000 characters of the pull
request (`MAX_CODE_RULES_PATCH_CHARS`), each pull request in a batch
counting its own, since one sidecar holds every deployment's step outputs;
lockfiles and other generated files are listed without their patch, which
no rule reads. The file that passes it partway and every later one are
listed without patches. The judge's view of the diff has its own, smaller
budget (`judgedFiles` in `logic/quality.ts`). A match in any checked file
fails the rule. Without one, a file that adds code but has no patch,
because the host sent none (GitHub omits it for a very large or binary
file) or the pull request passed that ceiling, or whose content was not
read, leaves the rule unconfirmed, naming the files and why; a file GitHub
counts no added lines in, such as a rename or a mode change, has nothing to
check. A rule ast-grep refuses leaves the rule unconfirmed. `vendor/`,
`dist/`, `build/` and `generated/` at the repository root and
`node_modules/` anywhere are skipped and listed as skipped in the evidence,
and a `?? 0` inside arithmetic is a counter's real default. With such a
rule on, the facts step reads each file whose patch adds lines at the head
through the host's `read_file` (`whole`), ten at a time and at most 300 per
pull request (more leaves the rule unconfirmed without reading any), runs
the rules there and outputs only the matches. Bitbucket Data Center lists
no patch, so there the rule is unconfirmed whenever an in-scope TypeScript
or JavaScript file is added, edited, moved or copied. The extension leaves
code-pattern rules out and swaps the native module for a stub.
lockfiles and other generated files are listed without their patch, which no
rule reads. The file that passes it partway and every later one are listed
without patches. The judge reads the same patches (`judgedFiles` in
`logic/quality.ts`) across up to 64 first-round requests, plus the verdict
request and at most 30 focused re-sends; a diff that needs more than 64 in
the first round sends the first 64, and the files only later requests hold
are unread, so the judged rules that ask about them are Unsure, naming them.
A match in any checked file fails the rule. Without one, a
file that adds code but has no patch, because the host sent none (GitHub
omits it for a very large or binary file) or the pull request passed that
ceiling, or whose content was not read, leaves the rule unconfirmed, naming
the files and why; a file GitHub counts no added lines in, such as a rename
or a mode change, has nothing to check. A rule ast-grep refuses leaves the
rule unconfirmed. `vendor/`, `dist/`, `build/` and `generated/` at the
repository root and `node_modules/` anywhere are skipped and listed as
skipped in the evidence, and a `?? 0` inside arithmetic is a counter's real
default. With such a rule on, the facts step reads each file whose patch
adds lines at the head through the host's `read_file` (`whole`), ten at a
time and at most 300 per pull request (more leaves the rule unconfirmed
without reading any), runs the rules there and outputs only the matches.
Bitbucket Data Center lists no patch, so there the rule is unconfirmed
whenever an in-scope TypeScript or JavaScript file is added, edited, moved
or copied. The extension leaves code-pattern rules out and swaps the native
module for a stub.
3. **route** (manager): deploys the judge when a question needs asking, and
otherwise runs evaluate in the manager.
4. **judge** (judge): one System One call per request of questions. Each first-round request and the verdict request also asks
Expand Down
19 changes: 9 additions & 10 deletions docs/DECISION_MODEL.md
Original file line number Diff line number Diff line change
Expand Up @@ -51,7 +51,7 @@ Questions point into the state by path, such as `` `pullRequest.files[3].hunks[0
- A doc file is `*.md` or `*.mdx` outside `vendor/`, `node_modules/`, `dist/` and `build/`, legal notices excluded.
- A test file is under a `test/`, `tests/`, `Tests/` or `__tests__/` directory (so Rust's `tests/`, Maven's `src/test/` and Swift's `Tests/` count) or a C# `*.Tests/` project such as `Acme.UnitTests/`, a Ruby file under `spec/`, or named by its language's convention: `*.test.*` or `*.spec.*` in JS or TS, `test_*.py` or `*_test.py`, `*_test.go`, `test_*.rb`, `*_test.rb` or `*_spec.rb`, `*_test.exs`, `*_test.dart`, `*Test` or `*Tests` in Java, Kotlin and C#, and `*Test`, `*Spec` or `*Suite` in Scala, the last two outside `src/main/`. It is not asked about.
- Otherwise each file gets one Choice, "How does the change in `pullRequest.files[N]` leave `repository.docs`?", with options in this order: `unaffected` (the docs stay correct and complete), `now_wrong` (a statement the docs make, such as a name, table, route, file, setting or behaviour, is contradicted because the change renames, removes or changes it) and `now_incomplete` (the change adds setup, configuration, permissions or commands the docs must describe). P(needs docs) is 1 − P(`unaffected`), on the Noul band: a file at 0.7 or above needs docs and fails the rule, one at 0.3 or below does not, and anything between leaves the rule unconfirmed. At most 3 files are named.
- **Duplicates open pull request**: the facts step shortlists up to five open pull requests opened earlier on the same base branch that share a changed path (lockfiles left out) or two title words of four or more letters with this one, most shared paths first. With none, it passes without a question. Otherwise one request of its own holds this pull request's metadata (no diff) and the shortlist as `repository.openPullRequests` (number, title and up to 50 changed paths each), and asks one Score per shortlisted pull request, question id `duplicates-open-pull-request-NNN`: "How does this pull request relate to `repository.openPullRequests[N]`, another open pull request, by both titles, this pull request's body and both lists of changed paths?", with levels `same change`, `overlaps` and `unrelated`, each described in the question. The pull request with the highest P(`same change`) decides it on the unsure band: 0.7 or above fails it and names that pull request, 0.3 or below passes it, and anything between is unconfirmed. Its default `onFail` is `ask`, so a failure makes the verdict Unsure. The shortlist sits under `repository`, which the instructed question covers, since other pull requests' titles are untrusted too. When its request would take the pull request past 64 requests, it is not asked and the rule is unconfirmed. The shortlist is measured without a model (`bun run eval:duplicates`, 8 cases: 8 of 8 put the true duplicate on it and keep lockfile-only and common-word overlaps off). The judge's answer is unmeasured: no eval case asks it, so how often a failure names a real duplicate is unknown.
- **Duplicates open pull request**: the facts step shortlists up to five open pull requests opened earlier on the same base branch that share a changed path (lockfiles left out) or two title words of four or more letters with this one, most shared paths first. With none, it passes without a question. Otherwise one request of its own holds this pull request's metadata (no diff) and the shortlist as `repository.openPullRequests` (number, title and up to 50 changed paths each), and asks one Score per shortlisted pull request, question id `duplicates-open-pull-request-NNN`: "How does this pull request relate to `repository.openPullRequests[N]`, another open pull request, by both titles, this pull request's body and both lists of changed paths?", with levels `same change`, `overlaps` and `unrelated`, each described in the question. The pull request with the highest P(`same change`) decides it on the unsure band: 0.7 or above fails it and names that pull request, 0.3 or below passes it, and anything between is unconfirmed. Its default `onFail` is `ask`, so a failure makes the verdict Unsure. The shortlist sits under `repository`, which the instructed question covers, since other pull requests' titles are untrusted too. Its request always has a slot: when the split needs more, the files give one up. The shortlist is measured without a model (`bun run eval:duplicates`, 8 cases: 8 of 8 put the true duplicate on it and keep lockfile-only and common-word overlaps off). The judge's answer is unmeasured: no eval case asks it, so how often a failure names a real duplicate is unknown.
- **Custom model rules** (`packages/triage-contracts/src/check-pack.ts`):
- `is-true` with scope `pull-request` (the default): one Noul, `<name>: <claim>`.
- `is-true` with scope `file` or `hunk`, optionally limited by `when` globs: one Noul per file or hunk in scope, "Does the change in … meet this rule: <claim>?". At most 3 items are named on failure.
Expand Down Expand Up @@ -95,23 +95,22 @@ Two different numbers come out of the answers:
| State per request | 90,000 bytes (32k tokens is about 105 KB of diff) | `MAX_SYSTEM_ONE_STATE_BYTES` |
| Questions per request | 256 (no documented cap; 300 were answered) | `MAX_SYSTEM_ONE_QUESTIONS` |
| Room for files per request | at least 8 KB | `MIN_FILE_ROOM` |
| Requests per pull request | 64 (about 5.8 MB of state, $0.17 at most) | `MAX_JUDGE_REQUESTS` |
| First-round requests per pull request | 64 (about 5.8 MB of state) | `MAX_JUDGE_REQUESTS` |
| Repository docs | 24,000 bytes, default branch | `REPO_DOCS_MAX_BYTES` |
| Contribution rules | 8,000 bytes, default branch | `REPO_RULES_MAX_BYTES` |
| Patch the judge reads per file | 65,536 chars | `MAX_PATCH_INPUT_CHARS` |
| Patch the judge reads per pull request | 270,000 chars | `MAX_PATCH_CHARS` |
| Patch the rules that run in code read per pull request | 2,000,000 chars | `MAX_CODE_RULES_PATCH_CHARS` |
| Patch the facts step reads per pull request, for the rules that run in code and the judge | 2,000,000 chars | `MAX_CODE_RULES_PATCH_CHARS` |
| Patch text in one facts step's output | 2,000,000 chars: one pull request per step, as the hub records backlog and catch-up mails per pull request and each manager reads its own | `MAX_CODE_RULES_PATCH_CHARS` × 1 |
| GitHub tool page | 9,000 chars | `PR_FILES_PAGE_CHARS` |

- **Docs.** Docs are read from the default branch in this order: root `README`, then `docs/`, then other root markdown, then nested markdown, each in path order. Reading stops at 24,000 bytes; a doc that crosses the cap is cut and later docs are not seen. The same docs are in every request's state.
- **Contribution rules.** Read under their own 8,000-byte budget, in this order: `AGENTS.md`, `CLAUDE.md`, `CONTRIBUTING` and `PULL_REQUEST_TEMPLATE` at the root, then `CONTRIBUTING` and `PULL_REQUEST_TEMPLATE` in `.github/` (including `PULL_REQUEST_TEMPLATE/`), then the same two in `docs/`. An `AGENTS.md` or `CLAUDE.md` inside a package stays an ordinary doc. Only requests that ask a custom rule carry them and give up room for them; requests that ask only built-in rules are unchanged.
- **Request splitting.** Files fill each request's state in order up to the budget left after metadata and docs. A file too large for one request keeps its leading hunks there for every rule, and focused and docs read its remaining hunks in the next requests, docs asking about each part (any part needing docs fails it); a hunk too large for any request is cut at line ends across several. Each request asks only about the files it holds; whole-PR questions go in the first, and focused asks in each. If the title, body, commits, path list and docs leave less than 8 KB of the budget for files, nothing is asked and the model rules stay unconfirmed ("pull request too large for the decision model (N KB)", the size of that metadata). The same holds when the split needs more than 64 requests ("… (N requests)").
- **Oversized patches.** A patch larger than one tool page is read across pages and joined, and every patch is read whole for the rules that run in code until 2,000,000 chars of the pull request, each pull request in a backlog batch counting its own; lockfiles and other generated files are listed without their patch, so they do not count. The file that passes it partway and every later one are listed without patches. The judge reads 65,536 chars of a file, which is then marked cut. Once the judge's 270,000-char budget is spent, later files are sent to it without patches.
- **Unread files.** A file with lines changed but no patch to read (GitHub left it out, the judge's budget was spent, or not even its first hunk fits a request) is unread, and a rule that would ask about it stays unconfirmed. Treatment differs by rule:
- **Request splitting.** Files fill each request's state in order up to the budget left after metadata and docs. A file too large for one request keeps its leading hunks there for every rule, and focused and docs read its remaining hunks in the next requests, docs asking about each part (any part needing docs fails it); a hunk too large for any request is cut at line ends across several. Each request asks only about the files it holds; whole-PR questions go in the first, and focused asks in each. If the title, body, commits, path list and docs leave less than 8 KB of the budget for files, nothing is asked and the model rules stay unconfirmed ("pull request too large for the decision model (N KB)", the size of that metadata). When the split needs more than 64 requests (63 when duplicates has a request), the first 64 (or 63) are sent and the files only later requests hold are unread.
- **Oversized patches.** A patch larger than one tool page is read across pages and joined, and every patch is read whole for the rules that run in code until 2,000,000 chars of the pull request, each pull request in a backlog batch counting its own; lockfiles and other generated files are listed without their patch, so they do not count. The file that passes it partway and every later one are listed without patches. The judge reads every patch the facts step read, up to 65,536 chars of a file, which is then marked cut, across as many first-round requests as it takes up to 64.
- **Unread files.** A file with lines changed but no patch to read (GitHub left it out, the facts step passed its 2,000,000-char ceiling, not even its first hunk fits a request, or only requests past the 64th, or the 63rd when duplicates has a request, hold it) is unread, and a rule that would ask about it stays unconfirmed. Treatment differs by rule:
- Docs reads a file too large for one request across requests as focused does. A file whose patch was cut while reading is asked about from the hunks that were read and is unread, so it can fail docs but never pass it. A changed doc file passes docs by rule regardless.
- Scoped custom rules ask about a cut file (its patch cut while reading, or too large for one request) from the hunks that were read and treat it as unread.
- Focused reads every line of every patch it has. Its only unread files are those whose patch it does not have whole: GitHub left it out, the judge's 270,000-char budget was spent, or the file passed 65,536 chars. Any unread file leaves focused unconfirmed, listing them, unless another slice already fails it.
- Focused reads every line of every patch it has. Its only unread files are those whose patch it does not have whole: GitHub left it out, the facts step passed its 2,000,000-char ceiling, the file passed 65,536 chars, or part of it was only in requests past the 64th (the 63rd when duplicates has a request). Any unread file leaves focused unconfirmed, listing them, unless another slice already fails it.
- Binary files and pure renames have no lines and are never unread.

## Reused answers
Expand All @@ -129,9 +128,9 @@ Estimates at TypeSafe's direct price, about 3.3 bytes per token (32k tokens ≈

- A small pull request: one request. With 24 KB of docs and a few KB of diff that is about 9k tokens, about $0.0004.
- One full request: at most 64k tokens, about $0.0027.
- The largest pull request Triage reads: the 270,000-char patch budget fills about four to five requests when the docs take their full 24 KB, at most about $0.015. Hunk-scoped custom rules add one question per hunk, uncapped, so wide diffs can need more requests.
- The largest pull request Triage judges: 64 first-round requests, the verdict request and, when focused fails, at most 30 re-sent requests (below), so at most 95 requests, about $0.26 at $0.0027 per full request. Files go into requests whole, so a diff fills more requests than its size alone suggests; one that needs more than 64 in the first round has its later files unread, so the rules that ask about them are Unsure, naming them. corbits-triage#688, 141 files and about 620,000 chars of patch, takes 9 requests, at most about $0.025. Hunk-scoped custom rules add one question per hunk, uncapped, so wide diffs can need more requests, and files past the 64th request (the 63rd when duplicates has a request) are unread.
- Focused asks one question in each request that holds files. It adds requests only for slices no other rule needs: the rest of a file too large for one request, or files no other rule asks about. On the judge eval the largest pull requests (#46, #119, #157, #158) take 2 calls each. With 24 KB of docs and small patches, a pull request of 200 files takes 2 requests, 500 files 4, 1,000 files 8 and 3,000 files 24, under 66 KB of state each.
- A failed focused rule sends each failing slice's request again with its file questions: one extra request per failing slice, the same state re-sent whole. On the judge eval's mixes that is one or two extra requests of 34–83 KB each, about $0.0004–0.0011 apiece.
- A failed focused rule sends each failing slice's request again with its file questions: one extra request per failing slice, the same state re-sent whole, until 30 files are named. On the judge eval's mixes that is one or two extra requests of 34–83 KB each, about $0.0004–0.0011 apiece.

## Measured accuracy

Expand Down
Loading
Loading