Skip to content

feat(retro): add flapping detection to retro analysis skill (#5512) #5513

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

Sign up for GitHub

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Open
Benkapner wants to merge 2 commits into
base: main
Choose a base branch
from
+50 −0
Open

feat(retro): add flapping detection to retro analysis skill (#5512) #5513

Changes from all commits
Commits
Show all changes
2 commits
Select commit Hold shift + click to select a range
46c6304
feat(retro): add flapping detection to retro analysis skill (#5512)
Benkapner Jul 23, 2026
8113425
fix(retro): use commit SHAs for run-to-run diff, add pagination
Benkapner Jul 23, 2026
File filter

Filter by extension

Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
50 changes: 50 additions & 0 deletions internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md
View file
Open in desktop
Original file line number Diff line number Diff line change
Expand Up @@ -96,6 +96,56 @@ After subagents return their findings, use your main context to:
3. Form hypotheses about root causes
4. Decide what changes to propose and where

## Flapping detection

Check whether the workflow exhibits fix-break oscillation. Flapping wastes agent cycles and often indicates a deeper problem (conflicting instructions, flaky tests, or an approach the agent cannot converge on).

### Signals to check

Dispatch a subagent to gather the data:

> "List all code and fix agent runs for PR #N. For each run, get the list of changed files and the diff summary. Also get the CI test results (pass/fail) for each run."

Then check for these patterns:

1. **File oscillation:** the same file was changed in two or more consecutive runs, and the changes reverse each other (lines added in run N were removed in run N+1, or vice versa).
2. **Test result flipping:** a test that passed after run N fails after run N+1, then passes again after run N+2.
3. **Cycle count:** more than 2 review-fix cycles on the same PR without convergence (the review keeps requesting changes on the same findings).

```bash
# Get the last two code/fix run SHAs for this PR
COMMITS=$(gh api "repos/$REPO_FULL_NAME/pulls/$PR_NUMBER/commits" \
--paginate --jq '.[].sha')
CURRENT_SHA=$(echo "$COMMITS" | tail -1)
PREVIOUS_SHA=$(echo "$COMMITS" | tail -2 | head -1)

# Compare files changed in each run
gh api "repos/$REPO_FULL_NAME/compare/${PREVIOUS_SHA}...${CURRENT_SHA}" \
--jq '.files[].filename' | sort > /tmp/current_files.txt

gh api "repos/$REPO_FULL_NAME/pulls/$PR_NUMBER/files" \
--paginate --jq '.[].filename' | sort > /tmp/all_pr_files.txt

# Files changed in both the latest push and the overall PR (potential oscillation)
comm -12 /tmp/current_files.txt /tmp/all_pr_files.txt
```
Comment on lines +115 to +131

@qodo-code-review qodo-code-review Bot Jul 23, 2026

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Run diff comparison wrong 🐞 Bug ≡ Correctness

The flapping-detection snippet compares the full PR file list against git diff HEAD~1 HEAD, which
is not a comparison between consecutive code/fix runs and only yields filename overlap (not reversed
changes). This can cause the retro agent to miss real oscillations or generate false flapping
candidates that aren’t tied to specific run-to-run transitions.
Agent Prompt 
### Issue description
The current flapping-detection bash example doesn’t compare *consecutive code/fix runs* (it compares PR-wide files to the last commit), and it doesn’t provide a way to verify that changes were actually reversed.

### Issue Context
The section describes detecting oscillation between *runs* (run N vs run N+1), but the snippet uses `HEAD~1..HEAD` and PR-wide file listing, which is not run-scoped.

### Fix Focus Areas
- internal/scaffold/fullsend-repo/skills/retro-analysis/SKILL.md[115-125]

### Suggested direction
- Change the snippet to:
  - Use run-specific commit SHAs (e.g., from each run’s produced commit / PR head at the time) and compare consecutive SHAs via `gh api repos/$REPO_FULL_NAME/compare/<shaN>...<shaNplus1>` (or `git diff <shaN>..<shaNplus1>` if available locally).
  - Add an explicit step that checks *reversal* for candidate files (e.g., show `git diff <shaN>..<shaNplus1> -- <file>` and `git diff <shaNplus1>..<shaNplus2> -- <file>` to demonstrate add/remove reversal).

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@Benkapner Benkapner Jul 23, 2026

Copy link
Copy Markdown
Contributor Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

fixed. now compares consecutive commit SHAs instead of using HEAD~1.


### When flapping is detected

Include a proposal with these specifics:

- **title:** Start with "Flapping detected:" followed by what oscillated
- **what_happened:** List each cycle with the run IDs, which files changed, and how the changes reversed
- **what_could_go_better:** Identify what might be causing the loop (conflicting review criteria, flaky test, ambiguous instructions)
- **proposed_change:** Suggest a concrete intervention (clarify the conflicting instruction, fix the flaky test, add a convergence guard)
- **validation_criteria:** "The next code/fix cycle on similar PRs should converge within 2 iterations"

### When NOT to flag

- A single rework cycle (review requested changes, fix addressed them, review approved) is normal, not flapping.
- Different files changing across runs is normal iteration, not oscillation.
- Only flag when you see the same changes being applied and reversed repeatedly.

## Before proposing: check for existing issues

**This step is mandatory.** Before including any proposal in your output, verify that no open issue already covers the same improvement. The retro agent is the primary source of systemic proposals — without this check, repeated runs produce duplicate issues that waste human triage time.
Expand Down
Loading