feat(eval): add rework rate tracking script for agent PRs (#5516)

[Benkapner]

Summary

Adds scripts/rework-rate.sh, a script that calculates how often agent-merged PRs need human cleanup afterward.

How it works

1. Fetches all PRs merged by the fullsend bot in a configurable window (default: 30 days)
2. For each, checks the git log for human commits within N days (default: 7) that touch the same files
3. Reports total agent PRs, reworked count, rework rate percentage, and details of reworked PRs

Usage

./scripts/rework-rate.sh                           # defaults: fullsend-ai/fullsend, 30 days, 7 day follow-up
./scripts/rework-rate.sh myorg/myrepo 60 14        # custom repo, 60 day window, 14 day follow-up

Why this matters

Rework rate is a concrete trust metric. If 5% of agent PRs need human cleanup, you can trust the agent more. If 40% do, you should require human review on everything. The trustworthiness-evidence problem doc identifies this as a gap, and the roadmap references it under Testing (#295).

Related Issue

Closes #5516

Checklist

• PR title follows Conventional Commits
• Commits are signed off (DCO)

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[qodo-code-review]

PR Summary by Qodo

Add script to measure human rework after agent-merged PRs

✨ Enhancement 🕐 10-20 Minutes

AI Description

• Add a CLI script to compute rework rate for agent-authored merged PRs.
• Detect follow-up human commits touching the same files within a configurable window.
• Print aggregate rate plus per-PR follow-up details for auditing.

Diagram

graph TD
  U([Developer]) --> S["scripts/rework-rate.sh"] -->|"API calls"| GH["GitHub API (via gh)"] -. "JSON results" .-> S
  S -->|"parse & compare"| LT["Local tools (jq/comm/date)"] --> S
  S -->|"print report"| R["Console output"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. GraphQL batch query (reduce API calls)

• ➕ Can fetch PRs + files + related commit metadata more efficiently
• ➕ Less likely to hit REST rate limits; fewer per-PR round trips
• ➕ Easier to add pagination for >100 PRs/files/commits
• ➖ More complex implementation than a Bash + gh REST approach
• ➖ Harder for non-experts to modify quickly

2. Scheduled GitHub Action that publishes the metric

• ➕ Produces a continuously updated metric without local setup
• ➕ Can persist results (artifact, issue comment, dashboard) for trend tracking
• ➕ Centralizes auth/rate-limit handling in CI
• ➖ More infra/permissions work; requires action maintenance
• ➖ Harder to run ad-hoc against arbitrary repos/windows

Recommendation: The current Bash + gh approach is a good, low-friction baseline for an exploratory trust metric. If this becomes a regularly consumed KPI (or repos exceed the 100-item API page defaults), consider a GraphQL-based implementation or add explicit pagination and rate-limit handling to keep results complete and resilient.

Files changed (1) +107 / -0

Enhancement (1) +107 / -0

rework-rate.shAdd agent PR rework-rate reporting script using gh + jq +107/-0

Add agent PR rework-rate reporting script using gh + jq

• Introduces a Bash script that searches for merged PRs authored by the agent bot within a lookback window, then checks for subsequent non-bot commits that touch overlapping files within a follow-up window. Outputs total agent PRs, reworked count, computed percentage, and a per-PR list of detected follow-ups.

scripts/rework-rate.sh

[github-actions]

Site preview

Preview: https://69414218-site.fullsend-ai.workers.dev

Commit: 7c60325ea6c9108c7fd1e7bff5a89c1347fd6e46

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. Bot identity mismatch ✓ Resolved 🐞 Bug ≡ Correctness

Description

The script searches for agent PRs with author:app/fullsend-ai-coder, but the repo’s documented
canonical agent identity is the bot login fullsend-ai-coder[bot]; if GitHub search doesn’t map the
app qualifier to that login, the report can incorrectly return zero agent PRs.
This is inconsistent with other repo scripts that key off the ...[bot] login and can make the
metric unusable until the qualifier is aligned/validated.

Code

scripts/rework-rate.sh[R26-28]

+# Fetch merged PRs by bot authors
+BOT_PRS=$(gh api "search/issues?q=repo:${REPO}+is:pr+is:merged+author:app/fullsend-ai-coder+merged:>=${SINCE}&per_page=100&sort=created&order=desc" \
+  --jq '.items[] | {number: .number, title: .title, closed_at: .closed_at}')

Relevance

⭐⭐⭐ High

Team recently fixed bot-login string mismatches; prefers canonical fullsend-ai-coder[bot] identity
in scripts.

PR-#5429

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Repo documentation and other scripts consistently reference the agent as fullsend-ai-coder[bot],
while this new script searches for author:app/fullsend-ai-coder, creating a mismatch risk that can
lead to zero results.

scripts/rework-rate.sh[26-28]
AGENTS.md[144-158]
scripts/check-e2e-authorization.sh[34-36]
PR-#5429

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`scripts/rework-rate.sh` uses the search qualifier `author:app/fullsend-ai-coder`, but the repository’s canonical bot identity references are `fullsend-ai-coder[bot]`. If the search qualifier does not match how PR authors are represented, the script can miss agent PRs entirely.

### Issue Context
The repo explicitly documents which bot login strings to use when referencing agent identities, and other scripts use those logins.

### Fix Focus Areas
- scripts/rework-rate.sh[26-28]

### Suggested fix
- Update the search query to match the canonical login used elsewhere (e.g., `author:fullsend-ai-coder[bot]`).
- If you want to support both representations, query for both (e.g., two searches merged together, or an `OR` query if supported) and dedupe by PR number.
- Add a short comment noting that the qualifier must match PR `user.login`/search representation for this repo.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Missing pagination 🐞 Bug ≡ Correctness

Description

The script fetches PR files and follow-up commits with per_page=100 but does not paginate, so
larger PRs or busy follow-up windows can be truncated and produce false negatives (missed overlaps ⇒
undercounted rework rate).
This also affects the agent PR discovery query (also capped at 100) and can undercount both the
denominator and numerator.

Code

scripts/rework-rate.sh[R45-64]

+  # Get files changed in this PR
+  PR_FILES=$(gh api "repos/${REPO}/pulls/${PR_NUM}/files?per_page=100" \
+    --jq '.[].filename' 2>/dev/null || echo "")
+
+  if [ -z "$PR_FILES" ]; then
+    continue
+  fi
+
+  # Check for human commits touching the same files after merge
+  FOLLOWUP_UNTIL=$(date -d "${MERGED_AT} +${FOLLOWUP_DAYS} days" +%Y-%m-%dT23:59:59Z 2>/dev/null \
+    || date -j -f "%Y-%m-%dT%H:%M:%SZ" "${MERGED_AT}" -v+${FOLLOWUP_DAYS}d +%Y-%m-%dT23:59:59Z 2>/dev/null \
+    || echo "")
+
+  if [ -z "$FOLLOWUP_UNTIL" ]; then
+    continue
+  fi
+
+  # Get commits after merge by non-bot authors
+  FOLLOWUP_COMMITS=$(gh api "repos/${REPO}/commits?since=${MERGED_AT}&until=${FOLLOWUP_UNTIL}&per_page=100" \
+    --jq '[.[] | select(.author.type != "Bot" and .author.login != "fullsend-ai-coder[bot]" and .author.login != "fullsend-ai-fullsend[bot]") | {sha: .sha, author: .author.login, message: .commit.message}]' 2>/dev/null || echo "[]")

Relevance

⭐⭐⭐ High

Repo scripts already use gh api --paginate for list endpoints; truncation bugs are treated as
correctness issues.

PR-#2106

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new script hard-codes per_page=100 on multiple list endpoints without --paginate, which can
truncate results; the repo already demonstrates pagination for similar “list events” API usage.

scripts/rework-rate.sh[26-28]
scripts/rework-rate.sh[45-47]
scripts/rework-rate.sh[62-64]
scripts/check-e2e-authorization.sh[118-121]
PR-#2617

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
Several GitHub list endpoints are queried with `per_page=100` but without pagination. This can truncate PR file lists and commit lists, causing missed overlaps and incorrect rework-rate calculations.

### Issue Context
The repo already uses `--paginate` elsewhere when enumerating potentially long API results.

### Fix Focus Areas
- scripts/rework-rate.sh[26-28]
- scripts/rework-rate.sh[45-47]
- scripts/rework-rate.sh[62-64]

### Suggested fix
- Use pagination for all list endpoints:
 - For PR files: `gh api --paginate "repos/${REPO}/pulls/${PR_NUM}/files?per_page=100" --jq '.[].filename'`.
 - For commits: `gh api --paginate "repos/${REPO}/commits?since=...&until=...&per_page=100" --jq '...'`.
 - For search: use `gh api --paginate ... --jq '.items[] | ...'` (jq runs per page) or switch to a gh command that paginates PR search reliably.
- Consider deduping results if combining pages/responses into a single stream.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Silent API failure skip ✓ Resolved 🐞 Bug ☼ Reliability

Description

On gh api failures, the script suppresses stderr and substitutes empty results (|| echo "" / `||
echo "[]") and then continue`s, so rate limits/auth issues silently reduce detected rework.
Because these failures are indistinguishable from “no files/commits”, the final metric can be
misleading without any warning output.

Code

scripts/rework-rate.sh[R45-68]

+  # Get files changed in this PR
+  PR_FILES=$(gh api "repos/${REPO}/pulls/${PR_NUM}/files?per_page=100" \
+    --jq '.[].filename' 2>/dev/null || echo "")
+
+  if [ -z "$PR_FILES" ]; then
+    continue
+  fi
+
+  # Check for human commits touching the same files after merge
+  FOLLOWUP_UNTIL=$(date -d "${MERGED_AT} +${FOLLOWUP_DAYS} days" +%Y-%m-%dT23:59:59Z 2>/dev/null \
+    || date -j -f "%Y-%m-%dT%H:%M:%SZ" "${MERGED_AT}" -v+${FOLLOWUP_DAYS}d +%Y-%m-%dT23:59:59Z 2>/dev/null \
+    || echo "")
+
+  if [ -z "$FOLLOWUP_UNTIL" ]; then
+    continue
+  fi
+
+  # Get commits after merge by non-bot authors
+  FOLLOWUP_COMMITS=$(gh api "repos/${REPO}/commits?since=${MERGED_AT}&until=${FOLLOWUP_UNTIL}&per_page=100" \
+    --jq '[.[] | select(.author.type != "Bot" and .author.login != "fullsend-ai-coder[bot]" and .author.login != "fullsend-ai-fullsend[bot]") | {sha: .sha, author: .author.login, message: .commit.message}]' 2>/dev/null || echo "[]")
+
+  if [ "$FOLLOWUP_COMMITS" = "[]" ] || [ -z "$FOLLOWUP_COMMITS" ]; then
+    continue
+  fi

Relevance

⭐⭐ Medium

They avoid hidden GitHub API failures, but precedent is mixed on whether to warn/exit versus
continuing.

PR-#2106

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The script explicitly discards API errors and then continues when results are empty, which makes
transient API failures indistinguishable from legitimate empty responses and can silently undercount
rework.

scripts/rework-rate.sh[45-51]
scripts/rework-rate.sh[62-68]
scripts/rework-rate.sh[76-80]
PR-#2398

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The script intentionally suppresses `gh api` errors and treats failures as empty data, then skips processing. This can silently bias the rework-rate metric (especially under rate limiting) with no indication that the report is incomplete.

### Issue Context
Past repo work has treated “failed to fetch files” as a case that should not silently skip decisions.

### Fix Focus Areas
- scripts/rework-rate.sh[45-51]
- scripts/rework-rate.sh[62-68]
- scripts/rework-rate.sh[76-80]

### Suggested fix
- Don’t conflate API failure with “no data”:
 - Capture `gh api` exit status and emit a warning to stderr identifying the PR/sha.
 - Track an `ERRORS`/`UNKNOWN` counter and print it in the summary.
 - Decide a policy: either (a) fail the script (non-zero) when required API calls fail, or (b) mark the PR as `unknown` and exclude from denominator, or (c) “fail open” (treat as reworked) to avoid underreporting.
- Avoid `2>/dev/null` for these calls unless you still surface a structured warning message.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[rh-hemartin]

Hello! You need to fix the body, your process didn't expand \n characters into newlines.

Also I'm running the script locally and it looks too static. You should include something that indicates that the script is actively working. I asked my local agent to introduce a progress system and it looks something like this:

Rework Rate Report
Repository: fullsend-ai/fullsend
Window: last 7 days (since 2026-07-16T00:00:00Z)
Follow-up window: 1 days after merge

  Checking PR 7/39 (#5468)...

[rh-hemartin]

I think something is better than nothing, but I'm not sure about the metric being really a rework one. Do you have any reference to decide file changes? Maybe doing this per line would be better? Also I think we could include the coder into the rework metric, currently we are using it more and more for fixes, so it makes sense that if it passes multiple times on the same file (or line or whatever) then it is considered rework, even if it was itself. What do you think?

[Benkapner]

thanks for the review and for running it locally. pushed fixes addressing all points:

• body rendering: fixed (was a tooling issue with escaped newlines)
• progress indicator: added Checking PR N/M (#XXXX)... output per PR
• bot identity: now uses fullsend-ai-coder[bot] with app/ fallback
• pagination: added --paginate to the PR files API call
• silent failures: API errors now surface as WARNING: lines and are tracked in a Skipped (API errors) count in the output

on the metric question you raised (file-level vs line-level, and whether to include the coder agent's own rework): those are good points. i think this first version is a starting point to get a baseline, and we can refine the metric definition based on what the data shows. happy to iterate on the detection granularity in a follow-up.

[rh-hemartin]

You need to install pre-commit locally, the CI is failing because of that:

9
shellcheck...............................................................Failed
- hook id: shellcheck
- exit code: 1

In scripts/rework-rate.sh line 18:
SINCE=$(date -d "-${DAYS} days" +%Y-%m-%dT00:00:00Z 2>/dev/null || date -v-${DAYS}d +%Y-%m-%dT00:00:00Z)
                                                                           ^-----^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
SINCE=$(date -d "-${DAYS} days" +%Y-%m-%dT00:00:00Z 2>/dev/null || date -v-"${DAYS}"d +%Y-%m-%dT00:00:00Z)


In scripts/rework-rate.sh line 61:
  if [ $? -ne 0 ] || [ -z "$PR_FILES" ]; then
       ^-- SC2181 (style): Check exit code directly with e.g. 'if ! mycmd;', not indirectly with $?.


In scripts/rework-rate.sh line 69:
    || date -j -f "%Y-%m-%dT%H:%M:%SZ" "${MERGED_AT}" -v+${FOLLOWUP_DAYS}d +%Y-%m-%dT23:59:59Z 2>/dev/null \
                                                         ^--------------^ SC2086 (info): Double quote to prevent globbing and word splitting.

Did you mean: 
    || date -j -f "%Y-%m-%dT%H:%M:%SZ" "${MERGED_AT}" -v+"${FOLLOWUP_DAYS}"d +%Y-%m-%dT23:59:59Z 2>/dev/null \


In scripts/rework-rate.sh line 80:
  if [ $? -ne 0 ]; then
       ^-- SC2181 (style): Check exit code directly with e.g. 'if ! mycmd;', not indirectly with $?.


In scripts/rework-rate.sh line 98:
    if [ $? -ne 0 ]; then
         ^-- SC2181 (style): Check exit code directly with e.g. 'if ! mycmd;', not indirectly with $?.

For more information:
  https://www.shellcheck.net/wiki/SC2086 -- Double quote to prevent globbing ...
  https://www.shellcheck.net/wiki/SC2181 -- Check exit code directly with e.g...

[Benkapner]

thanks @rh-hemartin , fixed the shellcheck findings (SC2086 double-quoting, SC2181 direct exit code checks). should pass now i hope

[rh-hemartin]

Example run:

$ bash scripts/rework-rate.sh fullsend-ai/fullsend 3 1
Rework Rate Report
Repository: fullsend-ai/fullsend
Window: last 3 days (since 2026-07-20T00:00:00Z)
Follow-up window: 1 days after merge

Found 28 agent PRs to check.

  Checking PR 1/28 (#5506)...
  Checking PR 2/28 (#5501)...
  Checking PR 3/28 (#5500)...
  Checking PR 4/28 (#5499)...
  Checking PR 5/28 (#5497)...
  Checking PR 6/28 (#5474)...
  Checking PR 7/28 (#5468)...
  Checking PR 8/28 (#5459)...
  Checking PR 9/28 (#5447)...
  Checking PR 10/28 (#5444)...
  Checking PR 11/28 (#5435)...
  Checking PR 12/28 (#5429)...
  Checking PR 13/28 (#5409)...
  Checking PR 14/28 (#5406)...
  Checking PR 15/28 (#5397)...
  Checking PR 16/28 (#5391)...
  Checking PR 17/28 (#5348)...
  Checking PR 18/28 (#5342)...
  Checking PR 19/28 (#5341)...
  Checking PR 20/28 (#5309)...
  Checking PR 21/28 (#5276)...
  Checking PR 22/28 (#5275)...
  Checking PR 23/28 (#5273)...
  Checking PR 24/28 (#5269)...
  Checking PR 25/28 (#5267)...
  Checking PR 26/28 (#5237)...
  Checking PR 27/28 (#5131)...
  Checking PR 28/28 (#4049)...

Results
-------
Agent PRs merged (last 3 days): 28
Reworked by humans: 15
Rework rate: 53.6%

Reworked PRs:

  #5447 - feat(#5431): add WASM net/http host bridge for Cloudflare Worker
    Follow-up: e9a2678 by @ifireball (same files: cmd/mint/main.go,cmd/mint/main_test.go,cmd/mint-wasm/go.mod,)
  #5444 - refactor(#5438)!: per-scenario World in context + in-process repo lease pool
    Follow-up: e380ed5 by @ifireball (same files: pkg/behaviourtest/steps/dispatch.go,)
  #5435 - docs(#5434): add bot identities section to AGENTS.md
    Follow-up: 0dcbe1a by @ifireball (same files: AGENTS.md,)
  #5409 - fix(#5408): deduplicate skills by basename during base composition
    Follow-up: 7132712 by @ggallen (same files: docs/guides/user/bring-your-own-agent.md,)
  #5406 - fix(#5405): use dotted OTel attribute keys for cache tokens
    Follow-up: 5f4d5e0 by @ggallen (same files: internal/cli/run.go,)
  #5348 - feat(#5343): gate GCP/filesystem paths behind build tags for WASM
    Follow-up: e9a2678 by @ifireball (same files: Makefile,)
  #5342 - fix(#4983): accept dots in minted GitHub token validation
    Follow-up: 99d840d by @ascerra (same files: internal/cli/minttoken.go,internal/cli/minttoken_test.go,)
  #5341 - fix(#5337): add missing @vue/server-renderer to lockfile
    Follow-up: 99d840d by @ascerra (same files: website/package-lock.json,)
  #5309 - test(#5206): add fork dispatch feature file, cleanup, and docs
    Follow-up: 129d28b by @ggallen (same files: internal/forge/fake.go,internal/forge/fake_test.go,internal/forge/forge.go,)
  #5275 - feat(#5271): include private repos in repos.yaml glob expansion
    Follow-up: 98e92fa by @ggallen (same files: internal/repos/init.go,)
  #5273 - docs(adr): update repos-management plan to reflect all PRs merged
    Follow-up: 669c9ab by @ggallen (same files: docs/plans/repos-management.md,)
  #5267 - docs(#5266): add review autonomy evidence tracking document
    Follow-up: 4e23848 by @waynesun09 (same files: docs/problems/review-autonomy-evidence.md,)
  #5237 - fix(#5193): pin Claude Code version in sandbox Containerfile
    Follow-up: 89ae507 by @waynesun09 (same files: renovate.json,)
  #5131 - fix(#2569): add 422 fallback for review inline comment failures
    Follow-up: 129d28b by @ggallen (same files: internal/forge/fake.go,internal/forge/fake_test.go,)
  #4049 - feat(#4026): support disabling agents via config.yaml enabled field
    Follow-up: 5f4d5e0 by @ggallen (same files: internal/cli/run.go,internal/cli/run_test.go,)
    ```

[rh-hemartin]

My concerns were fixed.

[codecov]

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

[Benkapner]

thanks for testing it, appreciate the example run output. ready for your approval when you're good.

[rh-hemartin]

LGTM, but I want here feedback from @ascerra and @maruiz93 who are more into the eval space.