Skip to content

Commit 6f4cf17

Browse files
docs(skill): make the setup skill analyze a repository and suggest its policy (#687)
1 parent 1b7eaf8 commit 6f4cf17

5 files changed

Lines changed: 396 additions & 86 deletions

File tree

‎docs/Corbits Triage User Manual/Corbits Triage User Manual.md‎

Lines changed: 9 additions & 9 deletions
Original file line numberDiff line numberDiff line change
@@ -171,7 +171,7 @@ rules:
171171
172172
A file written before rules carried reasons keeps working as it is.
173173
174-
A coding agent can write the file for you with the skill in [`skills/corbits-triage-setup`](../../skills/corbits-triage-setup/SKILL.md). It looks at the repository, asks only what it can't tell, suggests the starter rules and any that fit, writes `.corbits-triage.yml` and checks it the same way Triage reads it. To use it, copy the `corbits-triage-setup` folder into your coding agent's skills folder, such as `.claude/skills/` in the repository for Claude Code or `.agents/skills/` for agents that read that path, then ask the agent to set up Corbits Triage.
174+
A coding agent can write the file for you with the skill in [`skills/corbits-triage-setup`](../../skills/corbits-triage-setup/SKILL.md). It reads your last 50 merged pull requests on GitHub and suggests rules from what the repository already does, each with the number behind it, such as size limits at the 90th percentile or a linked-issue rule when most pull requests name an issue. You pick which to keep; it suggests and never decides a stricter policy for you. On Bitbucket Data Center, or with no merged pull requests, the size limits stay at the starter's 30 files and 3,000 lines. It then writes or edits `.corbits-triage.yml` and checks it the same way Triage reads it. When nobody is there to answer, as in an agent run, it writes its suggestions and lists each one under **Waiting on you** in the pull request. To use it, copy the `corbits-triage-setup` folder into your coding agent's skills folder, such as `.claude/skills/` in the repository for Claude Code or `.agents/skills/` for agents that read that path, then ask the agent to set up Corbits Triage.
175175
176176
A file can also start from another repository's policy with `extends: owner/name` (on Bitbucket, `PROJECT/slug`), and then set only what differs for this repository. This is useful when many repositories should follow the same rules: keep them in one repository's `.corbits-triage.yml` and extend it everywhere else. That repository must be connected to the same workspace, and its own file must not use `extends`. A change merged into it applies to every repository extending it from their next verdict.
177177

@@ -200,21 +200,21 @@ On GitHub, a mistake in the file can be caught before it reaches the default bra
200200
```yaml
201201
name: Corbits Triage policy
202202
on: pull_request
203-
permissions:
204-
checks: read
205-
contents: read
206-
pull-requests: read
207203
jobs:
208-
check:
204+
corbits-triage-policy:
209205
runs-on: ubuntu-latest
206+
permissions:
207+
checks: read
208+
contents: read
209+
pull-requests: read
210210
steps:
211211
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
212212
with:
213213
persist-credentials: false
214-
- uses: corbitsdev/corbits-triage/packages/policy-check-action@0aa2ba2ebeadc02530e15c222306c7baa9b4330c
214+
- uses: corbitsdev/corbits-triage/packages/policy-check-action@844696731de693ac70876ddf37cca016b065f25f
215215
```
216216

217-
Each pull request then gets a **Corbits Triage policy / check** check. It reads the file exactly as Triage does and fails if Triage could not read it. Each problem is marked on its line in the pull request's **Files changed**, or on the file as a whole when it has no line.
217+
Each pull request then gets a **Corbits Triage policy / corbits-triage-policy** check, named after the workflow and the job. To run it from a workflow you already have, add the `corbits-triage-policy` job to it; the check is then named after that workflow. A workflow written with the earlier job id, `check`, keeps reporting **Corbits Triage policy / check** until you edit it; when you replace that job with this one, change a branch protection rule or ruleset that requires the old name to the new one. It reads the file exactly as Triage does and fails if Triage could not read it. Each problem is marked on its line in the pull request's **Files changed**, or on the file as a whole when it has no line.
218218

219219
It also runs the rules that need no model, from the policy on the branch the pull request merges into, such as size, linked issue, merge conflicts and forbidden paths, against the pull request. Each failed rule shows as an error when its failure means Fail, and as a warning otherwise, and the run's summary lists every rule with its result and how many passed. Rules a model answers, such as **focused** or your own written rules, show as skipped, and so does **CI must pass before merge**, since the check is itself still running. A policy that uses `extends` is not checked, and one Triage cannot read shows a warning instead. The check fails when a rule whose failure means Fail fails. It only reads the pull request, never posts to it, and works on pull requests from forks. It needs no Triage account and no secrets, and sets up its own Bun for the job. Both actions are pinned to a full commit SHA, so the job only runs code you chose; to update the check, replace its SHA with a newer commit from `main`.
220220

@@ -258,7 +258,7 @@ Each rule's `onFail` says what happens when it fails:
258258

259259
- `fail`, the default for every rule but the size rules, regression risk, the judged duplicate rule, sensitive paths and risk: the verdict is **Fail** and the comment asks the author to fix it.
260260
- `ask`, the default for the judged duplicate rule, sensitive paths and risk: the verdict is **Unsure** and the comment lists the rule under **Asking a maintainer about:**, so the pull request waits for you in **Needs you** instead of on its author.
261-
- `label`, the default for the size rules and regression risk: the verdict does not change. When the rule fails, or the decision model can't tell, Triage adds its label to the pull request and a line under **Notes:** in its comment. Triage's comment and the reading pane leave a `label` rule out of their rule list; it shows only as a note and a label (on Bitbucket Data Center, only the note). The browser extension, the **Corbits Triage policy / check** check and the local check still list it. The label follows the rule: once the rule passes, such as a pull request cut back under its size limit, Triage takes the label off. The level labels of sensitive paths and risk (`sensitive: medium`, `sensitive: high`, `risk: medium` and `risk: high`) follow the level: when it changes or drops to low, Triage takes the old one off. A label the policy names for a rule follows that rule, even if a person added it; a label the policy does not name for a rule is never taken off. A label a `labels` step in the policy also names is left alone. The label is `size: large` for the size rules, `docs: check` for the docs rule, `sensitive: high` and `risk: high` for sensitive paths and risk, and the rule's id followed by `: check` for any other, such as `regression-risk: check`; `label:` names a different one, but not one starting `Triage:` or `rule: `, a verdict label, or a level label of sensitive paths or risk other than the rule's own, which Triage takes back itself; a `labels` step may not name a level label either. `onFail: label` suits rules that are hygiene rather than a gate, such as size limits an atomic change has to cross. A top-level `onFail` sets it for every rule in the file that does not set its own. The Policy section shows a rule that does not fail the pull request with what it does instead, such as "(on fail, labels size: large)".
261+
- `label`, the default for the size rules and regression risk: the verdict does not change. When the rule fails, or the decision model can't tell, Triage adds its label to the pull request and a line under **Notes:** in its comment. Triage's comment and the reading pane leave a `label` rule out of their rule list; it shows only as a note and a label (on Bitbucket Data Center, only the note). The browser extension, the **Corbits Triage policy / corbits-triage-policy** check and the local check still list it. The label follows the rule: once the rule passes, such as a pull request cut back under its size limit, Triage takes the label off. The level labels of sensitive paths and risk (`sensitive: medium`, `sensitive: high`, `risk: medium` and `risk: high`) follow the level: when it changes or drops to low, Triage takes the old one off. A label the policy names for a rule follows that rule, even if a person added it; a label the policy does not name for a rule is never taken off. A label a `labels` step in the policy also names is left alone. The label is `size: large` for the size rules, `docs: check` for the docs rule, `sensitive: high` and `risk: high` for sensitive paths and risk, and the rule's id followed by `: check` for any other, such as `regression-risk: check`; `label:` names a different one, but not one starting `Triage:` or `rule: `, a verdict label, or a level label of sensitive paths or risk other than the rule's own, which Triage takes back itself; a `labels` step may not name a level label either. `onFail: label` suits rules that are hygiene rather than a gate, such as size limits an atomic change has to cross. A top-level `onFail` sets it for every rule in the file that does not set its own. The Policy section shows a rule that does not fail the pull request with what it does instead, such as "(on fail, labels size: large)".
262262

263263
To make a size limit block, give it `onFail: fail` or `onFail: ask`:
264264

‎packages/policy-check-action/README.md‎

Lines changed: 7 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -14,21 +14,21 @@ It needs no Triage account and no secrets, and works on pull requests from forks
1414

1515
```yaml
1616
on: pull_request
17-
permissions:
18-
checks: read
19-
contents: read
20-
pull-requests: read
2117
jobs:
22-
check:
18+
corbits-triage-policy:
2319
runs-on: ubuntu-latest
20+
permissions:
21+
checks: read
22+
contents: read
23+
pull-requests: read
2424
steps:
2525
- uses: actions/checkout@11d5960a326750d5838078e36cf38b85af677262 # v4.4.0
2626
with:
2727
persist-credentials: false
28-
- uses: corbitsdev/corbits-triage/packages/policy-check-action@0aa2ba2ebeadc02530e15c222306c7baa9b4330c
28+
- uses: corbitsdev/corbits-triage/packages/policy-check-action@844696731de693ac70876ddf37cca016b065f25f
2929
```
3030
31-
Pin both actions to a full commit SHA; for this Action, a commit from `main`. `github-token` defaults to the job's token. Locally, from here: `bun --conditions=intx-src src/main.ts <repository root> <owner/name>`;
31+
Pin both actions to a full commit SHA; for this Action, a commit from `main`. The check is named `<workflow name> / corbits-triage-policy`; a workflow written with the earlier job id, `check`, keeps reporting `Corbits Triage policy / check` until it is edited, and when that job is replaced, a branch protection rule or ruleset that requires the old name must be changed to the new one. `github-token` defaults to the job's token. Locally, from here: `bun --conditions=intx-src src/main.ts <repository root> <owner/name>`;
3232
with `GITHUB_EVENT_NAME=pull_request`, `GITHUB_EVENT_PATH`, `GITHUB_API_URL`, `GITHUB_TOKEN` and `GITHUB_STEP_SUMMARY` set it also runs the rules.
3333

3434
## Check locally

‎packages/triage-contracts/src/setup-skill.test.ts‎

Lines changed: 15 additions & 8 deletions
Original file line numberDiff line numberDiff line change
@@ -8,19 +8,26 @@ import { checkPolicyFile } from "./policy-file.ts";
88
const REPO = "acme/widgets";
99
const REFERENCE = readFileSync(join(import.meta.dir, "../../../skills/corbits-triage-setup/reference.md"), "utf8");
1010

11-
/** The first yaml block under a `## heading` of the setup skill's reference. */
12-
function yamlUnder(heading: string): string {
13-
const section = REFERENCE.split(`\n## ${heading}\n`)[1];
14-
const block = section?.match(/```yaml\n([\s\S]*?)```/)?.[1];
15-
if (block === undefined) throw new Error(`No yaml block under ${heading}.`);
16-
return block;
11+
/** Every yaml block under a `## heading` of the setup skill's reference. */
12+
function yamlUnder(heading: string): string[] {
13+
const section = REFERENCE.split(`\n## ${heading}\n`)[1]?.split("\n## ")[0];
14+
if (section === undefined) throw new Error(`No ${heading} section.`);
15+
const blocks = Array.from(section.matchAll(/```yaml\n([\s\S]*?)```/g), (match) => match[1] as string);
16+
if (blocks.length === 0) throw new Error(`No yaml block under ${heading}.`);
17+
return blocks;
1718
}
1819

1920
test("the setup skill's starter file is the starter policy, and every other example is valid with it", () => {
20-
const starter = yamlUnder("Starter file");
21+
const [starter] = yamlUnder("Starter file") as [string];
2122
const exported = checkPolicyFile(exportPolicyFile(starterPack(REPO)).text, REPO);
2223
expect(checkPolicyFile(starter, REPO)).toEqual(exported);
24+
const [groups] = yamlUnder("Rules") as [string];
25+
const withGroups = starter.replace(/ {2}- check: sensitive-paths\n(?: {4}.*\n)+/, groups);
26+
expect(withGroups).not.toBe(starter);
27+
expect(checkPolicyFile(withGroups, REPO)).toMatchObject({ ok: true });
2328
for (const heading of ["Extras, written out", "Actions", "Labels", "Blocks and decide"]) {
24-
expect(checkPolicyFile(`${starter}${yamlUnder(heading)}`, REPO)).toMatchObject({ ok: true });
29+
for (const block of yamlUnder(heading)) {
30+
expect(checkPolicyFile(`${starter}${block.replaceAll("<login>", "octocat")}`, REPO)).toMatchObject({ ok: true });
31+
}
2532
}
2633
});

0 commit comments

Comments
 (0)