feat(skills): add aicr-cross-review skill for multi-agent PR review#1915
feat(skills): add aicr-cross-review skill for multi-agent PR review#1915yuanchen8911 wants to merge 1 commit into
Conversation
8da1b78 to
3b76441
Compare
|
Note Reviews pausedIt looks like this branch is under active development. To avoid overwhelming you with review comments due to an influx of new commits, CodeRabbit has automatically paused this review. You can configure this behavior by changing the Use the following commands to manage reviews:
Use the checkboxes below for quick actions:
📝 WalkthroughWalkthroughAdds the Estimated code review effort: 4 (Complex) | ~60 minutes 🚥 Pre-merge checks | ✅ 4✅ Passed checks (4 passed)
✨ Finishing Touches🧪 Generate unit tests (beta)
Comment |
0058899 to
0ab5828
Compare
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/aicr-cross-review/scripts/codex-watch.sh:
- Around line 57-67: Move the STATEKEY and OUR_JOB_ID environment-variable
assignments from the node side to the consuming python3 side of both pipelines:
the reaper probe around probe and the job-status probe around job_phase. Ensure
the Python filters read the actual state key and job ID from os.environ,
preserving the existing CODEX_DEAD, CODEX_ALIVE, and job-phase behavior.
- Line 40: The polling calls in the watchdog, including the workspace-root
lookup around node "$comp" and the reaper invocation near the corresponding
polling logic, lack bounded execution. Wrap each node "$comp" and node "$REAPER"
call with the repository’s supported timeout mechanism, preserving their
existing output parsing, error suppression, and polling behavior while ensuring
a hung IPC call cannot block the watchdog indefinitely.
In @.agents/skills/aicr-cross-review/scripts/coverage-check.sh:
- Around line 22-23: Make PRREF unique for each invocation of the coverage-check
script instead of deriving it solely from PRNUM. Update the PRREF assignment
near WORK and the corresponding ref cleanup/use around the later ref-handling
logic, incorporating the per-run WORK identifier or another invocation-unique
value while preserving the existing PR association and lifecycle.
In @.agents/skills/aicr-cross-review/scripts/workflow.mjs:
- Around line 60-83: Update EVAL_SCHEMA to conditionally require evidence
whenever an evaluation verdict is AGREE or DISAGREE, while allowing it for
OPEN_QUESTION to remain optional. In tally(), filter or otherwise gate
AGREE/DISAGREE counts through evidenceFor() before applying the confirmation and
disagreement thresholds, including the primary two-of-three path.
- Around line 316-341: Update addCandidate so findings sharing the same path and
line are merged only when their substantive content matches; preserve separate
candidates for divergent summaries, evidence, or impact. Ensure each distinct
finding retains its own data and source, while identical findings continue to
share a candidate.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 7a28a6c8-ef05-48e9-806f-0a9c05fa6af9
📒 Files selected for processing (5)
.agents/skills/aicr-cross-review/SKILL.md.agents/skills/aicr-cross-review/scripts/codex-watch.sh.agents/skills/aicr-cross-review/scripts/coverage-check.sh.agents/skills/aicr-cross-review/scripts/workflow.mjsdocs/contributor/skills.md
There was a problem hiding this comment.
Actionable comments posted: 4
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/aicr-cross-review/scripts/codex-watch.sh:
- Around line 72-80: Update the job_phase probe’s JSON parsing in the heredoc to
fail closed: when node status execution fails, produces empty output, or JSON
parsing fails, emit RUNNING (or the established distinct error token) instead of
DONE. Preserve DONE only when valid status data is parsed and no matching job
remains, and align this behavior with the reaper probe’s safe failure default.
In @.agents/skills/aicr-cross-review/scripts/coverage-check.sh:
- Around line 81-82: Update the coverage-threshold comment in the script to
match the current .settings.yaml value of 80%, while preserving the instruction
to rely on the configured quality.coverage_threshold rather than hardcoding it
in logic.
In @.agents/skills/aicr-cross-review/scripts/workflow.mjs:
- Around line 89-98: Update REFUTE_SCHEMA validation and the verify-phase
handling around the verdict application logic so CONFIRMED and REFUTED responses
require non-empty evidence, while UNVERIFIABLE may retain its existing evidence
behavior. Reject or route verdicts missing required evidence before accepting
them, preserving the existing reason requirement and verdict-specific handling.
In @.agents/skills/aicr-cross-review/SKILL.md:
- Around line 167-211: Add a dedicated Residual Risk section to the Phase 4
consensus report template in SKILL.md, positioned alongside the other findings
sections. Define a placeholder that renders the aggregated residualRisk entries
returned by workflow.mjs, ensuring this collected data has an explicit
destination in the generated markdown.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 74048a37-be66-44d4-8458-392c2154c70b
📒 Files selected for processing (5)
.agents/skills/aicr-cross-review/SKILL.md.agents/skills/aicr-cross-review/scripts/codex-watch.sh.agents/skills/aicr-cross-review/scripts/coverage-check.sh.agents/skills/aicr-cross-review/scripts/workflow.mjsdocs/contributor/skills.md
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/aicr-cross-review/SKILL.md:
- Around line 111-121: Update the Go coverage instructions around
coverage-check.sh to require explicit AskUserQuestion confirmation before
executing PR-supplied Go code, unless the user has already explicitly authorized
it. If confirmation is not granted, skip coverage and report that it was skipped
due to missing authorization; retain the existing background execution and
HEAD_SHA validation for authorized runs.
- Line 90: Update the workflow instructions around the pinned diff and Phase 2
verification to create a detached worktree at HEAD_SHA, pass that immutable
checkout as the file-read workspace, and clean it up after review. Replace the
current repoPath/git show preference with this required workflow so all reads
correspond to the pinned commit.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: f82047a1-a6ff-4cc1-bec4-5407ec89cac8
📒 Files selected for processing (4)
.agents/skills/aicr-cross-review/SKILL.md.agents/skills/aicr-cross-review/scripts/codex-watch.sh.agents/skills/aicr-cross-review/scripts/coverage-check.sh.agents/skills/aicr-cross-review/scripts/workflow.mjs
f526f2f to
22c062a
Compare
There was a problem hiding this comment.
Actionable comments posted: 5
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/aicr-cross-review/scripts/codex-watch.sh:
- Around line 33-35: Validate MAX as a non-negative integer and INTERVAL as a
positive integer immediately after parsing the arguments, exiting with the usage
message for invalid values. In the polling loop, calculate the remaining budget
from elapsed time and sleep for the smaller of INTERVAL and that remaining
duration, avoiding sleep once MAX has been reached.
- Around line 61-70: Update the watchdog’s reaper probe and companion-status
pipeline assignments so each runs within an if-not-success branch, preventing
set -euo pipefail from terminating the script. Treat reaper or JSON parsing
failures as no probe result so the loop continues, and treat companion-status
failures as RUNNING. Preserve the existing successful status parsing and
fallback behavior.
In @.agents/skills/aicr-cross-review/scripts/workflow.mjs:
- Around line 310-341: Update the filesChecked membership test in the
participant findings loop around addCandidate so suffix matches require a
path-separator boundary, preventing basename-only matches across unrelated
files. Preserve matching for exact paths and valid relative-path suffixes, while
ensuring genuinely unchecked files still receive the scrutiny flag.
In @.agents/skills/aicr-cross-review/SKILL.md:
- Around line 164-170: Update the “Merge” bullet to document the deduplication
key used by workflow.mjs addCandidate: path, line, and the normalized summary
via normSummary(summary), while preserving the existing multi-source tracking
behavior.
- Around line 97-107: Update the diff-generation block in the cross-review setup
to use the base branch captured as baseRefName in step 1 instead of hardcoding
origin/main. Resolve or construct the corresponding remote base reference for
the git diff while preserving the HEAD_SHA pinning and temporary ref cleanup
behavior.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 7b2164ee-d330-4739-b0cf-2a54fa1b3607
📒 Files selected for processing (5)
.agents/skills/aicr-cross-review/SKILL.md.agents/skills/aicr-cross-review/scripts/codex-watch.sh.agents/skills/aicr-cross-review/scripts/coverage-check.sh.agents/skills/aicr-cross-review/scripts/workflow.mjsdocs/contributor/skills.md
aac74dd to
1c0be23
Compare
There was a problem hiding this comment.
Actionable comments posted: 2
♻️ Duplicate comments (1)
.agents/skills/aicr-cross-review/SKILL.md (1)
211-251: 🗄️ Data Integrity & Integration | 🟡 Minor | ⚡ Quick winPhase 4 template still has no section for
residualRisk, and now also drops per-candidateflags.
workflow.mjscollects and returnsresidualRiskat the top level (still no home in this template — previously flagged, unresolved), and separately attaches scrutinyflagsto every emitted candidate (e.g. "reported this file without listing it in filesChecked — scrutinize", workflow.mjs lines 358-361, 560) — but the template has no place to surface either, so both are effectively produced and then discarded before reaching the human-readable report.📝 Proposed fix: add sections for both fields
### Open Questions <unverifiable findings + reviewers' open questions> +### Residual Risk + +<risk that remains even after fixes — e.g. known limitations reviewers flagged as acceptable but worth tracking> + +### Scrutiny Flags + +<findings flagged during merge/verification for extra scrutiny, e.g. files cited but absent from a reporter's filesChecked> + ### Test Coverage (Go PRs only)🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In @.agents/skills/aicr-cross-review/SKILL.md around lines 211 - 251, Update the Cross-Review Summary template to add dedicated sections for top-level residualRisk and per-candidate flags. Ensure the output schema preserves and renders both fields in the human-readable report, alongside the existing findings and candidate-related content.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In @.agents/skills/aicr-cross-review/SKILL.md:
- Around line 112-132: Use the captured baseRefName throughout baseline
selection: in .agents/skills/aicr-cross-review/SKILL.md lines 112-132, fetch and
diff against origin/<baseRefName> instead of hardcoded main and pass
<baseRefName> to the Phase 1.5 coverage-check.sh invocation; in
.agents/skills/aicr-cross-review/scripts/coverage-check.sh lines 49-61, accept
an optional base-ref argument defaulting to main and use it for the base fetch
and worktree instead of origin/main.
- Around line 182-194: Update the Codex references in the “Review” and
“Operational notes” sections to match workflow.mjs: describe Codex as running
through the general-purpose agent with the mandatory CODEX_DISPATCH protocol,
and remove the stale codex:codex-rescue and optional-dispatch claims. Preserve
the existing workflow, recovery, and reviewer-count guidance.
---
Duplicate comments:
In @.agents/skills/aicr-cross-review/SKILL.md:
- Around line 211-251: Update the Cross-Review Summary template to add dedicated
sections for top-level residualRisk and per-candidate flags. Ensure the output
schema preserves and renders both fields in the human-readable report, alongside
the existing findings and candidate-related content.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: ASSERTIVE
Plan: Enterprise
Run ID: 4f077e4f-783e-4968-be39-cf91a0441fb1
📒 Files selected for processing (5)
.agents/skills/aicr-cross-review/SKILL.md.agents/skills/aicr-cross-review/scripts/codex-watch.sh.agents/skills/aicr-cross-review/scripts/coverage-check.sh.agents/skills/aicr-cross-review/scripts/workflow.mjsdocs/contributor/skills.md
d3e2699 to
ec1aa4a
Compare
Remaining "never posts" claim fixed (025fd8f)Confirmed — I fixed the frontmatter last round but missed the same claim in the intro paragraph, which still read "The skill never executes the reviewed commit's code and never posts" with no qualifier. It now reads "never posts to the PR unless you explicitly ask (Phase 5)". I audited the whole surface rather than just the reported line, matching across line wraps since that is why the first pass missed it. Four claims exist and all are now correct:
One unqualified "never" is intentional and stays: "Never post Dismissed Findings or Positive Observations." That is absolute even when posting is requested — dismissed findings and praise are local report content, not PR content. The PR description carried the same softer wording and was updated to match. |
025fd8f to
5a1ff55
Compare
Path quoting completed and a posting injection fixed (5a1ff55)Both confirmed. Unquoted paths in SKILL.md. Correct, and the PR-body claim was an overclaim on my part: I quoted the workflow's generated prompts last round and described it as covering generated shell commands, but SKILL.md's own setup, overlay-validation and cleanup blocks still used bare Shell evaluation on explicit posting — this one was worth catching. Reproduced against a realistic finding body: Both the backticks and Phase 5 now instructs writing the filtered summary to a file with the Write tool and posting with I also swept for other unquoted path placeholders in shell blocks; the two remaining occurrences were already quoted. Agreed on rejecting the cleanup traps, extra validators, schema tightening, repoNotes propagation, and lint configuration. Still no live end-to-end Workflow run. |
5a1ff55 to
f3502b1
Compare
Symlink trust bypass and undefined variable fixed (f3502b1)Both confirmed; the first was reproduced. Local-overlay trust bypass. The tracked-status check applies to the link, not its target, so an untracked symlink resolving to a PR-tracked file passed as trusted. Reproduced in a scratch repo where a fork force-adds the ignored overlay: The guard I added last round worked correctly on the real file and was defeated by the alias pointing at it. Your point about checkout behaviour is what makes it reachable: git overwrites ignored files during checkout without complaint, so a reviewer who checks out a fork branch to look at it silently acquires the PR's overlay. Fixed by skipping symlinks before the tracked check, as you suggested — no target resolution. Re-ran both cases:
Undefined Also ran Agreed on the rejected cleanup, restoration, extra head-race, prompt-duplication and lint suggestions. Still no live end-to-end Workflow run — agreed that is the final readiness check. |
f3502b1 to
a812087
Compare
Six findings addressed (a812087)Four verified decisively; two accepted on mechanism with a caveat noted below. Fork remote layout (4). Confirmed the sharpest way available: the fork carries exactly one PR ref — its own PR #1 — while the canonical repo has 1915's head. So Late findings bypass completeness (5). Reproduced. With both lanes returning the same Late candidates are absent from Allowlist position dependency (6). Confirmed by offset: in the CodeRabbit prompt Explore is the wrong agent (3). Agreed, and its own registry description settles it: Explore "reads excerpts rather than whole files, so it locates code; it doesn't review or audit it" — which is precisely what the integration lane does. Switched to general-purpose. That made the Skill tool (1). Added Overlay guard is not a trust boundary (2). Accepted, and the honest fix is to stop claiming otherwise rather than to add machinery. Subagents load the checkout's CLAUDE.md hierarchy before any guard runs, so the symlink/tracked check reduces accidental exposure only. SKILL.md now says so and points to the same operational remedy as the self-review guard: for an untrusted or fork PR, start the session from a trusted checkout. Verification: all seven scenarios pass, Still no live end-to-end Workflow run — and finding 1's verification specifically depends on one. |
a812087 to
1a7d965
Compare
Single-vote dismissal removed, contract text corrected (1a7d965)The blocker was real and reproduced. With CodeRabbit unavailable, the reporter softening to OPEN_QUESTION, and one evidenced DISAGREE from Codex: The finding was Removed the Contract text. All four corrected, prose only:
Eight scenarios pass, Thanks for chasing down the |
1a7d965 to
eddbc04
Compare
Dead branch removed and tally comment corrected (eddbc04)Both applied. No behavioral change — nine scenarios produce identical results before and after. Unreachable consumer backfill. Verified rather than accepted on inspection: enumerating every combination of consumer path and line, 9 key collisions are possible and the backfill condition can fire in 0 of them. Since the consumer coordinates are part of the dedupe key, a key match implies they are already equal. Branch deleted and the merge comment corrected — it claimed to preserve "consumer coordinates" that never needed merging. tally() comment. It was staler than reported: besides "zero support = dismissed", it still described the CodeRabbit arbitration branch removed several revisions ago. Rewritten to state the actual rule — two evidenced AGREEs confirm, two evidenced DISAGREEs dismiss, anything else is contested — and to note the symmetry explicitly, since the asymmetry between those two thresholds was the blocker fixed in the previous commit. The PR description now records that dead code in this skill is removed on proof: the consumer backfill via key-collision enumeration, the arbitration branch via the 128-state vote enumeration. Agreed on the rejected CodeRabbit suggestions — cleanup traps, a second head-race check, candidate batching/recovery, and expanded failure reporting all sit outside the simplicity and fail-closed boundary. With no static blocker remaining, the live Workflow run is the only outstanding gate. |
eddbc04 to
8f29ebd
Compare
Three findings, the dead-code sweep, and template hygiene (8f29ebd)All verified before fixing. 1. Pinned reads treated PR data as shell source. Reproduced the glob half directly — The canonical recipe now reads: There is one recipe, not three: 2. Integration findings could omit consumer coordinates. Confirmed — I implemented this as an intake rejection rather than 3. The 20-minute timebox was unenforceable. Confirmed against this environment: Bash defaults to 120000 ms and is capped at 600000. The lane now runs with an explicit 600000 ms timeout and the prompt states why, so it will not be "simplified" back to a larger number. Your point that this erases the third lane rather than corrupting consensus is right, which is why the fix is a smaller honest deadline rather than backgrounding machinery. Dead-code sweep. All four removed after confirming each: Template hygiene. Correct — I had deleted the unchecked options rather than leaving them. The body now carries the canonical template verbatim, with Documentation update and Build/CI/tooling added under Type of Change and Docs/examples under Components. Checklist entries that do not apply are left unchecked with a short N/A reason rather than removed. Ten scenarios pass, including the new integration-coordinates case. Correction (3c4a9be): the injection half was not fixed by this commit. Double quotes stop globbing but still expand Fixed at 7aaecb7 by using single quotes, which are the only literal form in sh/bash/zsh, with the reason stated inline in the recipe so it is not changed back. |
8f29ebd to
7aaecb7
Compare
Four findings and the cleanup sweep (7aaecb7)1. Double quotes did not stop execution — my previous fix was wrong, and I said otherwise. Reproduced: Last round I quoted the placeholders, verified the zsh glob case, and then claimed in both the commit message and the PR body that this prevented smuggling 2. Malformed integration output produced false consensus. Confirmed: the finding was dropped, nothing was emitted, and 3. PR URLs were not normalized. Correct — 4. Fallback pointed at a command that posts. Both places now stop with the reason. The contradiction was self-inflicted: the same file explains two hundred lines later that Cleanup, all folded in: three stale 20-minute descriptions corrected, the duplicate Ten scenarios pass. Two of this round's four findings were defects in the previous round's fixes, which is the fourth time that has happened — the live Workflow run remains the outstanding gate. |
7aaecb7 to
183be53
Compare
Three findings fixed (183be53)1. Uninspectable filename produced clean consensus. Confirmed — Took your fix rather than adding escaping: a path that cannot be read through the single-quoted recipe is now the lane's failure outcome — 2. Numeric PR input failed in the fork layout. Confirmed. I fixed the fetch to use the canonical URL two rounds ago but left the normalizing 3. Self-review guard consumed Folding 2 and 3 into one edit was right; they were the same ordering problem seen from two angles. Eleven scenarios pass, all six Bash blocks parse, JS parses. Two of these three were again defects in earlier fixes of mine — the unscoped Correction (3c4a9be): the It was defined in one Bash invocation and referenced in three others, and each Bash call is a fresh shell — the same file says so on line 136. Reproduced as The same round also removed no-argument mode: |
183be53 to
3c4a9be
Compare
Two findings fixed (3c4a9be)Both reproduced before fixing. 1. Defined in Phase 0, referenced in Batch B and Phases 3/5 — three separate Bash invocations. The same file states on line 136 that shell variables do not persist between calls, which I wrote, and then last round I introduced a variable that depends on exactly that. All five call sites now name 2. No-argument mode could not find fork PRs. Reproduced from this PR's own worktree: The branch lives on the fork while the PR lives on Six Bash blocks parse, JS parses, eleven scenarios pass. Both findings were introduced by the previous round's fix for the fork layout: I scoped the calls correctly and then chose a mechanism that could not survive the tool boundary, while leaving a no-argument path that the new scoping made unreachable. The live Workflow run remains the outstanding gate. |
All seven review threads verified addressed at 3c4a9be@ArangoGutierrez — each of your seven threads now has a verification reply anchored at the current head, with the evidence inline. Summary so you can close them out:
Four of the seven were resolved by deleting the thing rather than fixing it — the coverage lane and the Codex watchdog together accounted for most of what you flagged. Three of your comments also led somewhere past what the thread said, and those are noted in the individual replies: I have not resolved the threads — they are yours to close if you agree with the dispositions. Live end-to-end run of the skill is in progress against a separate PR; that was the last outstanding readiness item. |
Ready for re-review — 3c4a9be@ArangoGutierrez @mchmarny @njhensley @lockwobr Since the last round of review this became a substantially smaller change: two files instead of four, and the skill no longer executes anything from the reviewed commit. What changed structurally
Where to look if you are re-reviewing selectively
All seven of @ArangoGutierrez's threads have verification replies anchored at this head; four were resolved by deleting the subject rather than fixing it. I have deliberately not resolved them. Behaviour is covered by eleven stubbed-harness scenarios described under Testing. The skill is also being exercised end-to-end against a live PR for the first time; if that surfaces anything I will push a fix and say so here rather than let it pass silently. |
3c4a9be to
1be6fc4
Compare
Head is now e65e917 — live-run result and one fixCurrent head is What changed. The skill has now been run end-to-end against a live PR (#1921), and it did not complete. Result was That is the designed behavior and the part I most wanted exercised: two of six agents failed and nothing degraded into a quorum. It preserved the raw material (2 candidate findings, 6 open questions) explicitly labeled unverified. But it is a partial result and I do not want to oversell it. The run never reached The one defect it surfaced, fixed in e65e917. Verification note. The six threads that today's push marked outdated have per-thread replies re-confirming each check at the new head. All seven remain open for @ArangoGutierrez to close. Pin updated: rebased |
e65e917 to
cfda806
Compare
Adds `aicr-cross-review` under `.agents/skills/` — a three-reviewer consensus workflow (Claude Code, Codex, CodeRabbit) with integration impact analysis and adversarial verification of every confirmed finding. Also registers the skill in the Available Skills catalog. Two files: SKILL.md (phases 0-5 orchestration) and scripts/workflow.mjs (the Workflow-tool consensus engine). The skill never executes the reviewed commit's code. There is no build, test, or coverage step, and every reviewer prompt forbids running tests, builds, package managers, repository scripts, or checked-in binaries from the PR. The only things executed are trusted tools the user already installed: git, gh, the CodeRabbit CLI, and the Codex companion. Coverage is left to CI, whose Merge Gate already enforces the project-wide threshold from .settings.yaml on fork PRs included. No lane may execute the reviewed commit's code. The execution rules appear in all six prompts (both round-1 review lanes, CodeRabbit, integration, the cross-review round, and the adversarial verifier), and CODEX_DISPATCH requires every dispatched Codex task to carry them verbatim, since Codex runs in its own shell. The rule permits the trusted commands a prompt explicitly prescribes — notably CodeRabbit's detached worktree — so it no longer contradicts the lane it governs. CI status is read with `gh pr checks` after re-confirming headRefOid still equals the pinned HEAD_SHA; `gh pr checks` reports the PR's current head, and the head can move during a review. `--required` is deliberately not used: before the aggregate gate job exists it exits 1 with "no required checks reported", making a normal in-progress run look like a failure. The head check is an explicit `if`, not `A && B || C`: chaining would report "head moved" whenever `gh pr checks` merely exited nonzero for pending (8) or failing checks, even with a matching SHA. Availability contract — report incomplete and stop: - Claude, Codex and integration analysis are required in round 1. - Claude and Codex must each return exactly one evaluation per candidate in the cross-review round. A missing evaluation, a duplicate, or an unknown candidate id returns incomplete, so a candidate nobody evaluated cannot ride its initial source votes to confirmed. - An AGREE or DISAGREE returned with blank evidence is fatal, not merely discarded: with no stored evaluation the reviewer falls back to its round-1 source vote, so a dropped verdict would still decide the tally. - CodeRabbit runs once and is best-effort. It is not a pre-flight requirement, and it uses a general-purpose agent rather than the CodeRabbit plugin agent type. When it does not run its fixed vote slot records NONE, which raises the bar (Claude and Codex must then agree) rather than lowering it. No dynamic participants, no quorum arithmetic, no retries. Consensus integrity: - Confirmed = 2 of 3 reviewer slots AGREE with evidence; integration analysis is never a reviewer slot. A finding whose evidence is blank or whitespace-only is rejected at intake, so it never registers its reporter as a source — schema `required` guarantees the field exists, not that it says anything. - One consumer per finding: the dedupe key includes the consumer coordinates, so a changed declaration breaking two callers stays two candidates, each verified separately. - Every confirmed finding goes to a fresh adversarial refuter. UNVERIFIABLE, a dead verifier, or a verdict without a citation lands in `unresolved`, which is emitted and blocks `consensusReached`. Pinning: - The diff comes from `refs/pull/N/head` and the PR's actual `baseRefName` into session-scoped refs, asserted against the `headRefOid` captured at setup. - Every reviewer prompt reads via `git show <headSha>:<path>` and searches via `git grep <pattern> <headSha>` — never the mutable working copy, which may be on another branch during concurrent sessions. - `repoNotes` comes from the base-pinned CLAUDE.md; the untracked local overlay is read only after confirming git does not track it, since .gitignore does not stop `git add -f`. Filesystem paths interpolated into generated shell commands are quoted — in the workflow prompts and in the SKILL.md setup, overlay-validation and cleanup blocks — so a checkout or temp path containing spaces does not split into multiple arguments and fail a required lane. Explicitly requested posting uses `gh pr comment --body-file <report-file>` against a file written by the Write tool, never `--body "..."`. Finding text quotes PR content, so backticks or $(...) in a finding would otherwise be evaluated by the shell before gh runs. All lanes run as general-purpose agents. The integration lane no longer uses Explore, which reads excerpts to locate code and is explicitly not a review agent; with Explore gone every lane inherits the session model, so the model argument and its handoff prose are removed. The skill denies the Skill tool so a reviewer lane cannot auto-invoke a skill that posts. Pinned-read recipes put their placeholders in SINGLE quotes and send the pattern through `git grep -e`. Double quotes stop globbing but still expand $(...) and backticks, so a PR-controlled filename such as docs/$(id).md would execute; single quotes are the only literal form in sh/bash/zsh. An integration finding missing consumerPath or consumerLine stops the run. Integration is a required lane, so dropping its malformed output produced a result that looked clean and reported consensusReached: true while that lane had contributed nothing. A path that cannot be read through the single-quoted recipe makes the lane return unavailable rather than an open question. Open questions do not block consensus, so a filename containing a quote could otherwise yield a clean-looking run in which nothing was actually inspected. Every gh call and the PR-ref fetch name NVIDIA/aicr literally rather than carrying a shell variable: each Bash call is a fresh shell, and the local repository is the contributor's fork in GitHub's standard layout, which has neither the PR nor refs/pull/*. A PR number or URL is required. There is no no-argument mode, because a fork PR cannot be identified from the local branch name alone. The reference is normalized in Phase 0, so the numeric .number is available to the self-review guard that runs immediately after it and to every later temp path and scoped ref. Pre-flight and environment failures stop with a reason instead of suggesting /code-review, which posts its result to the PR. The CodeRabbit lane runs with an explicit 600000 ms timeout. The Bash tool defaults to two minutes and is capped at ten, so the previous ~20-minute timebox was not enforceable and would have silently erased the lane. There is no single-vote dismissal. Dismissal needs two evidenced DISAGREEs; a lone dissent leaves the finding contested, so one slot of three cannot bury an issue that only it ever judged. Findings raised during the cross-review round stay contested: they were never presented to anyone, so their reporters' source votes alone must not confirm them. PR refs are fetched from the canonical repository by URL rather than from `origin`, which is the contributor's fork in GitHub's standard fork layout and carries no refs/pull/*. The local-overlay check is documented as accident-reduction rather than a trust boundary — reviewer subagents load the checkout's CLAUDE.md hierarchy before any guard here runs, so an untrusted PR needs a session started from a trusted checkout. The local-overlay trust check skips symlinks before testing tracked status. The check applies to the link, not its target, so an untracked symlink resolving to a PR-tracked file would otherwise be treated as a trusted overlay. Safety: - The Claude lane reviews directly rather than delegating to the `code-review` command, whose step 8 instructs its agent to `gh pr comment` the result back to the PR. This skill never posts by default. - If the PR modifies this skill, the review stops and asks for a trusted checkout. Signed-off-by: Yuan Chen <yuanchen97@gmail.com>
cfda806 to
078f6d9
Compare
Summary
Adds
aicr-cross-reviewas a shared agent skill under.agents/skills/— a three-reviewer consensus workflow (Claude Code, Codex, CodeRabbit) with integration impact analysis and adversarial verification of every confirmed finding. Also adds the skill to the Available Skills catalog indocs/contributor/skills.md.Two files:
SKILL.md(phases 0–5 orchestration, 395 lines) andscripts/workflow.mjs(the Workflow-tool consensus engine, 731 lines).Motivation / Context
Contributes the cross-review workflow used in this repo to the shared skill set so all contributors can invoke it via
/aicr-cross-reviewor$aicr-cross-review. Namedaicr-cross-review(notcross-review) to avoid shadowing a contributor's global cross-review skill.Fixes: N/A
Related: #1911
Type of Change
Component(s) Affected
cmd/aicr,pkg/cli)cmd/aicrd,pkg/server)pkg/recipe)pkg/bundler,pkg/component/*)pkg/collector,pkg/snapshotter)pkg/validator)pkg/errors,pkg/k8s)docs/,examples/).agents/skills/(agent skill definitions)Implementation Notes
The skill never executes the reviewed commit's code. There is no build, test, or coverage step. The execution rules appear in all six prompts — both round-1 review lanes, CodeRabbit, integration, the cross-review round, and the adversarial verifier — because these are shell-capable agents and running tests is normal review behaviour, so the prohibition has to be stated rather than assumed.
CODEX_DISPATCHadditionally requires every dispatched Codex task to carry the rules verbatim, since Codex runs in its own shell and inherits nothing from the wrapper prompt. The rule permits the trusted commands a prompt explicitly prescribes — notably CodeRabbit's detached worktree — so it does not contradict the lane it governs. Evidence comes from the saved diff plus pinnedgit show/git grepreads.Coverage is deliberately left to CI. The Merge Gate already enforces the project-wide threshold from
.settings.yamland fails the run when coverage is below it, on fork PRs included. An earlier revision of this PR ran coverage locally in throwaway worktrees; that was the skill's only code-execution surface and the source of a disproportionate share of its complexity (fork gating, timeouts, baseline selection, floor attribution). Phase 3 reports CI status instead. It re-confirmsheadRefOidstill equals the pinnedHEAD_SHAbefore reporting, becausegh pr checksreads the PR's current head and the head can move during a 20-minute review; if it moved, the CI line is omitted rather than attributed to the wrong commit.--requiredis deliberately not used — before the aggregate gate job exists it exits 1 with "no required checks reported", so a normal in-progress run looks like a failure. Plaingh pr checksexits 8 while running and 0 when green, and handlesskipped/neutralcorrectly, unlike a raw check-runs query (on a green commit: 13 false failures across 30 of 37 checks). The head check is an explicitifrather thanA && B || C, because chaining would report "head moved" whenevergh pr checkssimply exited nonzero for pending or failing checks with the SHA still matching. It does not parse CI's coverage comment: that comment is posted only for same-repo PRs, carries no head SHA, and is baselined against the last successful main run, so it is absent on fork PRs and can be silently stale elsewhere.Every
ghcall and the PR-ref fetch nameNVIDIA/aicrliterally rather than carrying a shell variable — each Bash call is a fresh shell, and the local repository is the contributor's fork in GitHub's standard layout, which has neither the PR norrefs/pull/*. A PR number or URL is required; there is no no-argument mode, because a fork PR cannot be identified from the local branch name alone. The reference is normalized in Phase 0, so.numberis available to the self-review guard that runs immediately after — previously the guard consumed<n>a phase before it existed. Pre-flight and environment failures stop with a reason rather than suggesting/code-review, which posts its result to the PR.Availability — report incomplete and stop:
general-purposeagents, and the skill setsdisallowed-tools: Skillso a reviewer lane cannot auto-invoke a skill that posts. The integration lane no longer usesExplore, which reads excerpts to locate code and is explicitly not a review agent; withExploregone every lane inherits the session model and themodelargument is removed.incomplete— without this, a candidate nobody actually evaluated could ride its initial source votes toconfirmed.NONE, which raises the bar (Claude and Codex must then agree) rather than lowering it. It is correspondingly not a pre-flight requirement, and it runs on a general-purpose agent rather than the CodeRabbit plugin agent type, so an unavailable CLI degrades instead of blocking. No dynamic participants, no quorum arithmetic, no retries, no arbitration.Consensus integrity:
requiredguarantees the field exists, not that it says anything.consumerPathandconsumerLine; one lacking either stops the run. Integration is a required lane, so dropping its malformed output produced a result that looked clean and reportedconsensusReached: truewhile that lane had contributed nothing. JSON Schema cannot make them conditionally required, and the verifier only traces a consumer when the coordinates exist, so without them the finding's central claim would reach consensus unverified.git grep -e. Double quotes stop globbing but still expand$(...)and backticks, so a PR-controlled filename likedocs/$(id).mdwould execute — single quotes are the only literal form in sh/bash/zsh. The recipe says so inline, because the earlier double-quoted version looked correct. A path that cannot be read this way (one containing a single quote) makes the lane returnunavailablerather than an open question - open questions do not block consensus, so the alternative was a clean-looking run in which nothing was inspected.unresolvedarray that blocksconsensusReached.Commit pinning:
origin: in GitHub's standard fork layoutoriginis the contributor's fork and carries norefs/pull/*. The diff is generated fromrefs/pull/N/headand the PR's actualbaseRefNameinto session-scoped refs, asserted against theheadRefOidcaptured at setup.git show <headSha>:<path>and searches viagit grep <pattern> <headSha>— never the mutable working copy, which may be on a different branch during concurrent sessions.repoNotesis distilled from the base-pinnedCLAUDE.md. The local-overlay check is accident-reduction, not a trust boundary — reviewer subagents load the checkout'sCLAUDE.mdhierarchy before any guard runs, so an untrusted PR needs a session started from a trusted checkout. The overlay is read only after skipping symlinks and confirming git does not track the path:.gitignoredoes not stopgit add -f, and the tracked-status check applies to a link rather than its target, so an untracked symlink resolving to a PR-tracked file would otherwise pass as trusted.Safety:
code-reviewcommand: that command's step 8 instructs its agent togh pr commentthe result back to the PR and carriesBash(gh pr comment:*)in allowed-tools. This skill never posts unless you explicitly ask (Phase 5).gh pr comment --body-fileagainst a file written with the Write tool, never--body "...". Finding text quotes PR content, so backticks or$(...)in a finding would otherwise be evaluated by the shell beforeghruns.SKILL.mdlives inside the reviewed repo.workflow.mjshas been the most frequent defect class in this PR's review, so the doc states the contract the code actually enforces rather than an earlier design.Testing
No Go code affected — skill-only change. Skipping
make qualifyper project policy for skill/doc-only PRs (CLAUDE.local.md); no test, e2e, or Go lint surface can regress from it.Verified:
workflow.mjsparses and was executed end-to-end against stubbedagent/parallelharnesses covering: CodeRabbit unavailable → review completes with its slot recordedNONE; integration lane unavailable →incomplete; a cross-review lane returning 0 of 1 evaluations →incomplete; an unknown candidate id →incomplete; a blank-evidence AGREE in the cross-review round →incomplete; blank-evidence rejection at intake; two consumers separated into distinct candidates verified independently; UNVERIFIABLE →unresolved; REFUTED →dismissed;consensusReached: falsewhileunresolvedis non-empty. Candidate accounting balanced in every run — no finding lost between the emitted arrays.addCandidatewas shown unreachable by enumerating every key collision (0 of 9 could fire), and the CodeRabbit arbitration branch by enumerating all 128 reachable vote states.ReferenceError; a null-dereference introduced when the participant list became a fixed slot list; and a candidate being confirmed from its round-1 source votes after both cross-review lanes returned blank-evidence verdicts that were logged as "not counted".CODEX_DISPATCHcarries the repeat-verbatim instruction. An earlier revision of this PR described that rule as applied when an edit had silently failed to write, so it is now verified by reading the file back.codex/1.0.2:status <job> --wait --timeout-ms --poll-interval-msexists, and job state is at.job.status, not a top-levelstatusfield.gh apiplaceholder substitution confirmed to cover{owner}/{repo}/{branch}only; the inline review-POST recipe that relied on{number}was removed rather than repaired.on-push-comment.yaml(fork PRs skipped; baseline is the last successful main run) and.github/actions/go-coverage/action.yml(threshold enforced, forks included) before deciding to drop the local coverage lane. The Phase 3 command was validated against this PR's own head commit in both the in-progress and green states.Not verified: no live end-to-end run against a real PR. The Codex
--waitinvocation and the CodeRabbit--base-commitblock are still untested against the live tools.Risk Assessment
Rollout notes: N/A
Checklist
make testwith-race) — N/A, no Go code changedmake lint) — N/A, no Go or YAML source changedgit commit -S) — GPG signing info