fix(workflow): feature-research silent-success + strict-mode hardening#28
Merged
chrisleekr merged 2 commits intomainfrom Apr 11, 2026
Merged
Conversation
Three compounding defects caused run 24232817769 to roll up as `success` while producing zero safe-outputs: 1. Copilot CLI v1.0.22 regression (confirmed in gh-aw v0.68.1 release notes + PR #25689 - "0-byte output due to incompatibilities with v1.0.22"). Fix: upgrade gh-aw to v0.68.1, inherit framework-level pin of Copilot CLI to v1.0.21 via install_copilot_cli.sh. 2. gh-aw has no built-in fail-on-empty for safe-outputs. Fix: new post-steps assertion reads /tmp/gh-aw/agent_output.json and exits non-zero when create_issue AND noop counts are both zero. 3. Agent improvised `bun run check` on a Bun-less runner. Fix: new "Read-only scope" section in prompt body forbidding build/test commands. Spec correction: original attribution to copilot-cli#2479/#2481 is wrong; those bugs were closed as fixed (2026-04-09 / 2026-04-03, fixed in CLI v1.0.19) before the 2026-04-10 failing run. Corrected in spec.md Background / FR-002 / Assumptions. Details and walkthrough evidence: specs/20260411-143905-fix-feature-researcher/ Walkthrough T025-T028 still pending - PR opens as draft. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
📝 WalkthroughWalkthroughThis PR adds enforcement and fixes to the Feature Opportunity Researcher GitHub Agentic Workflow to prevent silent failures when the agent produces no safe-output records. It upgrades GitHub Actions framework dependencies (gh-aw v0.68.1, actions/github-script v9), injects a post-steps assertion that fails the workflow when agent output is empty, adds a "Read-only scope" prompt constraint, and includes comprehensive specification documentation. Changes
Estimated code review effort🎯 3 (Moderate) | ⏱️ ~25 minutes Possibly related issues
Possibly related PRs
Poem
🚥 Pre-merge checks | ✅ 3✅ Passed checks (3 passed)
✏️ Tip: You can configure your own custom pre-merge checks in the settings. ✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
Follow-up to ea312ad (R-1 Option A: copilot-cli v1.0.21 pin via gh-aw v0.68.1 PR #25689). Post-compile validation run 24280821490 surfaced that Option A alone was INSUFFICIENT: Copilot CLI 1.0.21 still hallucinated the tavily MCP tool as missing even though the gateway logged "tavily: connected". Rather than wait for a downstream CLI fix, this commit removes the tavily-mcp dependency entirely and redesigns the workflow around built-in tooling, then fixes a strict-mode compilation error surfaced by `gh aw compile --strict`. Changes in feature-research.md: - Remove tavily-mcp server block and its TAVILY_API_KEY secret surface. Replace with `web-fetch` (built-in, zero third-party dep, zero secret to manage) + github MCP `toolsets: [repos]` for structured release data. Eliminates the Copilot CLI 1.0.21 allowlist-hallucination bug path and removes a third-party attack surface. - Narrow network.allowed from `["all"]` to a curated vendor list: defaults, github, cursor.com, anthropic.com, openai.com, code.visualstudio.com, modelcontextprotocol.io, zed.dev. Each entry is justified inline. - Remove `sandbox.agent: false` and its 8-line SECURITY TRADE-OFF comment block. `gh aw compile --strict` (gh-aw v0.68.1) refuses this setting because it disables the AWF firewall and removes hard security isolation. With the firewall ON, the curated network.allowed list IS the egress allowlist; the github MCP covers any GitHub-hosted research target without needing that domain on the list. Prompt text rewritten to reflect allowlisted (not "open") web access and to instruct the agent to fall back to github MCP or pick a different source when a host is firewall-blocked. - Add workflow_dispatch input `focus_area` for manual override of the weekly rotation, with regex-validated identifier. - Add safe-outputs `noop:` entry so the agent can signal "no candidate passed the quality gate" without failing FR-001. - Replace the ~60-line prompt with a ~260-line structured prompt: Mission, 10-area focus rotation, 6-criterion quality gate, 4-step research methodology, Output Structure schema, and Hard Rules including "do NOT treat a firewall-blocked host as a blocker". Changes in feature-research.lock.yml: regenerated via `gh aw compile feature-research --strict` (0 errors, 0 warnings). TAVILY_API_KEY removed; ALLOWED_DOMAINS narrowed; prompt/config hash IDs rotated; frontmatter_hash advances accordingly. Changes in spec tree: - research.md: Status line rewritten to document the full three-change resolution (Option A pin + Tavily removal + strict-mode sandbox fix). Open-questions section extended with a "Post-compile follow-up" paragraph explaining the T025 validation discovery and why Option A alone was insufficient. - plan.md: Research pointer + project-structure tree comment updated. - data-model.md: "MCP blocked by policy" validation rule updated to drop `tavily` from the enforced MCP list. - tasks.md: T033 stays [X] (this commit IS T033). T034 flipped back to [ ] because no PR exists yet — PR creation is the explicit next step after this commit lands. FR-001 post-steps assertion from ea312ad is preserved unchanged and continues to enforce: `create_issue_count >= 1 OR noop_count >= 1` on any `success` run. Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
![coderabbitai[bot]](https://avatars.githubusercontent.com/in/347564?s=60&v=4){kind=small}
There was a problem hiding this comment.
Actionable comments posted: 15
🧹 Nitpick comments (1)
specs/20260411-143905-fix-feature-researcher/data-model.md (1)
137-145: Consider collapsing Entity 4 to the selected R-1 decision to avoid ambiguity.Since Option A is already chosen, keeping alternative option branches here can read like unresolved design state.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed. In `@specs/20260411-143905-fix-feature-researcher/data-model.md` around lines 137 - 145, Collapse the ambiguous "Location depends on R-1 Decision" branches down to the chosen Option A: remove alternative entries for "Option B / D", "Option C / E" and update the text to state that location is implicit in the gh-aw framework via the GH_AW_VERSION env var (and that framework v0.68.1+ pins Copilot CLI to 1.0.21). Also remove or adjust references to source frontmatter (feature-research.md engine.version) so Entity 4 only reflects the selected R-1 decision and no unresolved option branches remain.
🤖 Prompt for all review comments with AI agents
Verify each finding against the current code and only fix it if needed.
Inline comments:
In @.github/aw/actions-lock.json:
- Around line 8-12: The lock file .github/aw/actions-lock.json only pins
actions/github-script@v9 to SHA 373c709c69115d41ff229c7e5df9f8788daa9553 but the
compiled workflow feature-research.lock.yml also uses SHA
3a2844b7e9c422d3c10d287c895573f7108da1b3 in many invocations; update
actions-lock.json to include an entry for actions/github-script@v9 with SHA
3a2844b7e9c422d3c10d287c895573f7108da1b3 (or consolidate all workflow usages to
a single pinned SHA) so the lock file covers every SHA referenced by
feature-research.lock.yml.
In @.github/workflows/feature-research.lock.yml:
- Line 2: The workflow uses actions/github-script@v9 with two different SHAs
(373c709c69115d41ff229c7e5df9f8788daa9553 and
3a2844b7e9c422d3c10d287c895573f7108da1b3) but the project action lock only
records 373c709c..., so normalize all steps to a single audited SHA or add the
missing SHA (3a2844b7e9c422d3c10d287c895573f7108da1b3) to the project action
lock; search for occurrences of actions/github-script@v9 in the workflow and
either replace all usages with the audited SHA 373c709c... or update the action
lock to include the alternate SHA so every executed step is covered.
In @.github/workflows/feature-research.md:
- Line 79: Replace the misspelled tool name "travily" with the correct name
"tavily" in the sentence that reads "Do NOT use WebFetch or direct HTTP requests
to external URLs unless travily is unable to retrieve the content you need." so
the operator guidance reads "...unless tavily is unable to retrieve the content
you need."
In `@CLAUDE.md`:
- Around line 12-13: The CLAUDE.md changelog contains truncated entries
"Markdown (gh-aw" in two places; update both occurrences to a consistent,
complete phrase such as "Markdown (gh-aw workflow docs)" so the changelog reads
clearly (e.g., replace "YAML (GitHub Actions schema) + Markdown (gh-aw" and the
second "Markdown (gh-aw" instance with the full "Markdown (gh-aw workflow
docs)"), ensuring punctuation and parenthesis balance.
In `@specs/20260411-143905-fix-feature-researcher/checklists/requirements.md`:
- Around line 42-47: Update the checklist notes in the Background root-cause /
Iteration 2 section to remove the superseded attribution to Copilot CLI issues
`#2479/`#2481 and replace it with the corrected root-cause reference to the
v1.0.22 regression (gh-aw#25689); ensure FR-002's “fallback-policy” question is
replaced by the concrete diagnosis text, and that any mentions in FR-001 /
SC-005 notes referencing the old attribution are updated to reflect the
finalized spec narrative and the gh-aw#25689 identifier.
In `@specs/20260411-143905-fix-feature-researcher/quickstart.md`:
- Around line 206-215: Update the rollback procedure to handle merge commits by
replacing the single-step `git revert <merge-commit-sha>` with `git revert -m 1
--no-commit <merge-commit-sha>` so the revert uses the mainline parent and does
not auto-commit; then run `gh aw compile feature-research --strict` to
regenerate the lockfile and finally commit once (`git commit -am
"revert(workflow): rollback feature-research fix"`) and push (`git push origin
main`) so the revert and lockfile regeneration are included in the same commit.
- Around line 139-166: The instructions currently ask the user to "note the
current value" of TAVILY_API_KEY (in the Dispatch/Restore flow) which is
impossible because GitHub Actions secrets are write-only; update the
quickstart.md flow to avoid overwriting a live secret: change the steps under
"Dispatch" and "Restore" to use a disposable test secret (e.g.,
TAVILY_API_KEY_TEST) or explicitly require the maintainer to first export the
current TAVILY_API_KEY to an external secret manager or document before
proceeding, and adjust the Restore step to restore that external backup
(referencing TAVILY_API_KEY, the Dispatch block that runs
feature-research.lock.yml, and the Restore step text for where to make the
replacement).
In `@specs/20260411-143905-fix-feature-researcher/research.md`:
- Around line 34-38: Replace the malformed release URL in the markdown link for
the Copilot CLI version: update the link text/target
`([1.0.19](https://github.com/github/copilot-cli/releases1.0.19))` to point to
the correct GitHub release page (e.g.
`https://github.com/github/copilot-cli/releases/tag/v1.0.19`), ensuring the
bracketed link text `[1.0.19]` remains the same and only the URL is corrected.
- Around line 182-190: The documentation for FR-001 is incorrect: the post-steps
assertion for the agent job (which reads /tmp/gh-aw/agent_output.json written by
the “Write agent output placeholder if missing” step) currently describes
counting only .items[] | select(.type == "create_issue") and failing when that
count is zero; update the text to reflect the actual pass condition which
accepts either a create_issue or a noop result (i.e., the assertion should count
items whose .type is "create_issue" or "noop" and only fail if the combined
count is zero), and adjust the described jq expression and explanatory wording
to mention both types and the noop success path for FR-001 and the post-steps
assertion.
- Line 5: Update the document to mark R-1 as resolved now that option A is
chosen: change the Status line from "DRAFT — one `TODO(human)` decision open
(see R-1 Decision)" to reflect the decision is made, remove or resolve the R-1
TODO in the "Open questions" section, and add a brief resolution note
referencing "R-1 Decision" and the existing "Chosen option: A" entry so all
occurrences (including the sections that currently show "Chosen option: A")
consistently indicate R-1 is closed; ensure the three places that mention
R-1/Open questions are updated to avoid contradicting the recorded choice.
In `@specs/20260411-143905-fix-feature-researcher/spec.md`:
- Around line 420-423: Update the Dependencies section so it no longer suggests
using the upstream `copilot-cli/releases/latest` channel; instead state and
document the intentional framework-level pin to version 1.0.21 used to avoid the
1.0.22 regression, and remove or reword any lines that claim the fix must
tolerate `latest` to reflect the selected mitigation strategy.
- Around line 255-259: Update the FR-001 requirement text so it explicitly
allows the documented noop success path: change the condition from "MUST NOT
finish with overall conclusion `success` while emitting zero `create-issue`
records" to allow success when either a `create-issue` record is emitted or a
`noop` record is emitted (i.e., require `create-issue >= 1 OR noop >= 1`);
ensure the rule references the exact symbol `FR-001` and the output types
`create-issue` (or `create_issue` if canonical elsewhere) and `noop` so readers
and validators accept either as a valid success signal.
In `@specs/20260411-143905-fix-feature-researcher/tasks.md`:
- Line 166: T030's FR-006 verification currently hardcodes the cron literal "0 7
* * 5" which causes false positives; update the check (the step labeled (a) in
T030 / FR-006) to extract the cron expression from the source frontmatter in
.github/workflows/feature-research.md and then assert that the extracted value
appears in .github/workflows/feature-research.lock.yml (rather than matching a
baked-in literal), keeping the other grep checks (COPILOT_MODEL,
max-autopilot-continues, GH_AW_INFO_ENGINE_ID) unchanged.
- Line 140: The task T021 artifact uses incorrect weekend dates; update the item
text for T021 (the checklist line starting with "T021 [US4] Create an
observability tracking artifact") to reference the four Fridays April 17, April
24, May 1, and May 8, 2026 instead of the Sundays 2026-04-12 → 2026-05-10, and
adjust the suggested GitHub issue title and body to list those four Friday dates
and still include the three SC targets (SC-001, SC-002, SC-003) and the
instruction to reference the merge commit SHA; ensure the new title reflects the
corrected date range (e.g., "feature-research post-merge observation window
(2026-04-17 → 2026-05-08)").
- Line 7: Replace the incorrect feature-tree string ".specify/" in the
documentation text with the actual feature-tree path used by this PR
("specs/20260411-143905-fix-feature-researcher/") so the scope check matches the
artifact paths, and update the other occurrences of the same string later in the
document (three additional instances) so all references use the real
feature-tree path.
---
Nitpick comments:
In `@specs/20260411-143905-fix-feature-researcher/data-model.md`:
- Around line 137-145: Collapse the ambiguous "Location depends on R-1 Decision"
branches down to the chosen Option A: remove alternative entries for "Option B /
D", "Option C / E" and update the text to state that location is implicit in the
gh-aw framework via the GH_AW_VERSION env var (and that framework v0.68.1+ pins
Copilot CLI to 1.0.21). Also remove or adjust references to source frontmatter
(feature-research.md engine.version) so Entity 4 only reflects the selected R-1
decision and no unresolved option branches remain.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: defaults
Review profile: CHILL
Plan: Pro
Run ID: 426ff03d-6357-4819-a49a-265725f66ecf
📒 Files selected for processing (13)
.github/aw/actions-lock.json.github/workflows/feature-research.lock.yml.github/workflows/feature-research.mdCLAUDE.mdspecs/20260411-143905-fix-feature-researcher/checklists/requirements.mdspecs/20260411-143905-fix-feature-researcher/contracts/post-check.mdspecs/20260411-143905-fix-feature-researcher/contracts/safe-outputs.mdspecs/20260411-143905-fix-feature-researcher/data-model.mdspecs/20260411-143905-fix-feature-researcher/plan.mdspecs/20260411-143905-fix-feature-researcher/quickstart.mdspecs/20260411-143905-fix-feature-researcher/research.mdspecs/20260411-143905-fix-feature-researcher/spec.mdspecs/20260411-143905-fix-feature-researcher/tasks.md
Comment on lines
+8
to
+12
| "actions/github-script@v9": { | ||
| "repo": "actions/github-script", | ||
| "version": "v9", | ||
| "sha": "373c709c69115d41ff229c7e5df9f8788daa9553" | ||
| }, |
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== actions-lock entry for actions/github-script@v9 =="
jq -r '.entries["actions/github-script@v9"]' .github/aw/actions-lock.json
echo
echo "== all github-script SHAs referenced in compiled workflow =="
rg -n 'uses:\s*actions/github-script@' .github/workflows/feature-research.lock.yml
echo
echo "== gh-aw manifest actions/github-script entries =="
rg -n '"repo":"actions/github-script"' .github/workflows/feature-research.lock.ymlRepository: chrisleekr/agentsync
Length of output: 2929
Lockfile does not cover all actions/github-script@v9 SHAs used by the compiled workflow.
.github/aw/actions-lock.json tracks only SHA 373c709c69115d41ff229c7e5df9f8788daa9553, but feature-research.lock.yml uses two different actions/github-script@v9 SHAs:
- SHA
3a2844b7e9c422d3c10d287c895573f7108da1b3(18 invocations, lines 103, 126, 137, 209, 219, 344, 499, 715, 742, 756, 768, 793, 886, 903, 917, 932, 1034, 1097, 1167) - SHA
373c709c69115d41ff229c7e5df9f8788daa9553(1 invocation, line 362) — tracked in lock
Update the lock file to include the missing SHA, or consolidate to a single pinned version.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/aw/actions-lock.json around lines 8 - 12, The lock file
.github/aw/actions-lock.json only pins actions/github-script@v9 to SHA
373c709c69115d41ff229c7e5df9f8788daa9553 but the compiled workflow
feature-research.lock.yml also uses SHA 3a2844b7e9c422d3c10d287c895573f7108da1b3
in many invocations; update actions-lock.json to include an entry for
actions/github-script@v9 with SHA 3a2844b7e9c422d3c10d287c895573f7108da1b3 (or
consolidate all workflow usages to a single pinned SHA) so the lock file covers
every SHA referenced by feature-research.lock.yml.
| @@ -1,4 +1,5 @@ | |||
| # gh-aw-metadata: {"schema_version":"v3","frontmatter_hash":"6c6c8f502405387ab4add67646de5e48f3d52c20534882ade34a5d89560562c6","compiler_version":"v0.67.0","strict":true,"agent_id":"copilot","agent_model":"gpt-5.4"} | |||
| # gh-aw-metadata: {"schema_version":"v3","frontmatter_hash":"177c370ded7e9892b66ef6a251d5ffc75a31a8600584e2f962a212273c980d0c","compiler_version":"v0.68.1","strict":true,"agent_id":"copilot","agent_model":"gpt-5.4"} | |||
| # gh-aw-manifest: {"version":1,"secrets":["COPILOT_GITHUB_TOKEN","GH_AW_GITHUB_MCP_SERVER_TOKEN","GH_AW_GITHUB_TOKEN","GITHUB_TOKEN","TAVILY_API_KEY"],"actions":[{"repo":"actions/checkout","sha":"de0fac2e4500dabe0009e67214ff5f5447ce83dd","version":"v6.0.2"},{"repo":"actions/download-artifact","sha":"3e5f45b2cfb9172054b4087a40e8e0b5a5461e7c","version":"v8.0.1"},{"repo":"actions/github-script","sha":"373c709c69115d41ff229c7e5df9f8788daa9553","version":"v9"},{"repo":"actions/github-script","sha":"3a2844b7e9c422d3c10d287c895573f7108da1b3","version":"v9"},{"repo":"actions/setup-node","sha":"53b83947a5a98c8d113130e565377fae1a50d02f","version":"v6.3.0"},{"repo":"actions/upload-artifact","sha":"bbbca2ddaa5d8feaa63e36b76fdaad77386f024f","version":"v7"},{"repo":"github/gh-aw-actions/setup","sha":"2fe53acc038ba01c3bbdc767d4b25df31ca5bdfc","version":"v0.68.1"}]} | |||
There was a problem hiding this comment.
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
workflow=".github/workflows/feature-research.lock.yml"
lock=".github/aw/actions-lock.json"
echo "Workflow references:"
rg -n 'actions/github-script@' "$workflow"
echo
echo "Locked SHA(s) for actions/github-script@v9:"
jq -r '
.entries
| to_entries[]
| select(.value.repo=="actions/github-script" and .value.version=="v9")
| .value.sha
' "$lock"
echo
echo "Workflow SHAs missing from the action lock:"
comm -23 \
<(rg -o 'actions/github-script@[0-9a-f]{40}' "$workflow" | sed 's/.*@//' | sort -u) \
<(jq -r '
.entries
| to_entries[]
| select(.value.repo=="actions/github-script" and .value.version=="v9")
| .value.sha
' "$lock" | sort -u) || trueRepository: chrisleekr/agentsync
Length of output: 2153
actions/github-script@v9 is split across two SHAs, but the action lock only covers one.
This workflow references both 373c709c... and 3a2844b7... for actions/github-script@v9, while .github/aw/actions-lock.json only records 373c709c.... Sixteen of the seventeen workflow steps execute 3a2844b7..., which is not audited in the action lock. This means reviewers verify a SHA that differs from the one most steps actually use.
Normalize the workflow to a single audited SHA or update the action lock so every executed SHA is covered.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In @.github/workflows/feature-research.lock.yml at line 2, The workflow uses
actions/github-script@v9 with two different SHAs
(373c709c69115d41ff229c7e5df9f8788daa9553 and
3a2844b7e9c422d3c10d287c895573f7108da1b3) but the project action lock only
records 373c709c..., so normalize all steps to a single audited SHA or add the
missing SHA (3a2844b7e9c422d3c10d287c895573f7108da1b3) to the project action
lock; search for occurrences of actions/github-script@v9 in the workflow and
either replace all usages with the audited SHA 373c709c... or update the action
lock to include the alternate SHA so every executed step is covered.
Comment on lines
+12
to
+13
| - YAML (GitHub Actions schema) + Markdown (gh-aw (20260411-143905-fix-feature-researcher) | ||
| - N/A. Workflow file only; no persistent state. (20260411-143905-fix-feature-researcher) |
There was a problem hiding this comment.
Fix truncated/malformed Markdown entries for the new feature-researcher record.
Line 12 and Line 33 both look cut off (...Markdown (gh-aw) and have unbalanced wording, which makes the changelog ambiguous. Please complete the phrase consistently in both places (e.g., “Markdown (gh-aw workflow docs)”).
✏️ Proposed doc fix
-- YAML (GitHub Actions schema) + Markdown (gh-aw (20260411-143905-fix-feature-researcher)
+- YAML (GitHub Actions schema) + Markdown (gh-aw workflow docs) (20260411-143905-fix-feature-researcher)
- N/A. Workflow file only; no persistent state. (20260411-143905-fix-feature-researcher)
@@
-- 20260411-143905-fix-feature-researcher: Added YAML (GitHub Actions schema) + Markdown (gh-aw
+- 20260411-143905-fix-feature-researcher: Added YAML (GitHub Actions schema) + Markdown (gh-aw workflow docs)As per coding guidelines: **/*.md: Use Markdown for documentation.
Also applies to: 33-33
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@CLAUDE.md` around lines 12 - 13, The CLAUDE.md changelog contains truncated
entries "Markdown (gh-aw" in two places; update both occurrences to a
consistent, complete phrase such as "Markdown (gh-aw workflow docs)" so the
changelog reads clearly (e.g., replace "YAML (GitHub Actions schema) + Markdown
(gh-aw" and the second "Markdown (gh-aw" instance with the full "Markdown (gh-aw
workflow docs)"), ensuring punctuation and parenthesis balance.
Comment on lines
+42
to
+47
| - FR-002 + User Story 2 + Background root-cause section 1 replace the | ||
| fallback-policy question with a concrete diagnosis (Copilot CLI | ||
| personal-account MCP allowlist bug, | ||
| [github/copilot-cli#2479](https://github.com/github/copilot-cli/issues/2479), | ||
| [#2481](https://github.com/github/copilot-cli/issues/2481)). | ||
| - FR-001 + SC-005 operationalise Option A: zero-safe-output runs end in |
There was a problem hiding this comment.
Checklist notes still reflect superseded root-cause attribution.
This section says Iteration 2 replaced fallback policy with the #2479/#2481 diagnosis, which conflicts with the corrected attribution to the v1.0.22 regression / gh-aw#25689. Please update these notes so the checklist matches the finalized spec narrative.
🧰 Tools
🪛 LanguageTool
[uncategorized] ~45-~45: The official name of this software platform is spelled with a capital “H”.
Context: ...ersonal-account MCP allowlist bug, [github/copilot-cli#2479](https://github.com/gi...
(GITHUB)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@specs/20260411-143905-fix-feature-researcher/checklists/requirements.md`
around lines 42 - 47, Update the checklist notes in the Background root-cause /
Iteration 2 section to remove the superseded attribution to Copilot CLI issues
`#2479/`#2481 and replace it with the corrected root-cause reference to the
v1.0.22 regression (gh-aw#25689); ensure FR-002's “fallback-policy” question is
replaced by the concrete diagnosis text, and that any mentions in FR-001 /
SC-005 notes referencing the old attribution are updated to reflect the
finalized spec narrative and the gh-aw#25689 identifier.
Comment on lines
+255
to
+259
| - **FR-001**: The workflow MUST NOT finish with overall conclusion `success` | ||
| while also emitting zero safe-output records of the type the workflow is | ||
| designed to produce (`create-issue` in this case). On a zero-safe-output | ||
| run, the workflow MUST end with overall conclusion `failure` so the | ||
| Actions UI shows a red X. |
There was a problem hiding this comment.
FR-001 conflicts with the documented noop success path.
This requirement currently implies failure whenever create-issue count is zero, but elsewhere the spec/contracts treat noop as a valid success output. FR-001 should explicitly allow create_issue >= 1 OR noop >= 1.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@specs/20260411-143905-fix-feature-researcher/spec.md` around lines 255 - 259,
Update the FR-001 requirement text so it explicitly allows the documented noop
success path: change the condition from "MUST NOT finish with overall conclusion
`success` while emitting zero `create-issue` records" to allow success when
either a `create-issue` record is emitted or a `noop` record is emitted (i.e.,
require `create-issue >= 1 OR noop >= 1`); ensure the rule references the exact
symbol `FR-001` and the output types `create-issue` (or `create_issue` if
canonical elsewhere) and `noop` so readers and validators accept either as a
valid success signal.
Comment on lines
+420
to
+423
| - Upstream Copilot CLI release channel (`copilot-cli/releases/latest`). | ||
| The fix must not pin to a specific buggy version; it must tolerate | ||
| whichever `latest` gh-aw installs on a given run. | ||
|
|
There was a problem hiding this comment.
Dependencies section contradicts the chosen Copilot CLI pinning approach.
This says the fix must tolerate whatever latest installs, which conflicts with the stated framework-level pin to 1.0.21 used to avoid the 1.0.22 regression. Please align this section with the selected mitigation strategy.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@specs/20260411-143905-fix-feature-researcher/spec.md` around lines 420 - 423,
Update the Dependencies section so it no longer suggests using the upstream
`copilot-cli/releases/latest` channel; instead state and document the
intentional framework-level pin to version 1.0.21 used to avoid the 1.0.22
regression, and remove or reword any lines that claim the fix must tolerate
`latest` to reflect the selected mitigation strategy.
| **Input**: Design documents from `/specs/20260411-143905-fix-feature-researcher/` | ||
| **Prerequisites**: plan.md ✅, spec.md ✅, research.md ✅ (R-1: Option A), data-model.md ✅, contracts/ ✅, quickstart.md ✅ | ||
|
|
||
| **Testing Regime**: **Documentation-only exception under Constitution Principle II.** All file changes land in the `.specify/` feature tree, plus `.github/workflows/feature-research.md` (source) and `.github/workflows/feature-research.lock.yml` (regenerated). No new automated tests are generated; instead this task list enforces the three mandatory sub-requirements per the constitution: |
There was a problem hiding this comment.
Use the real feature-tree path in the docs-only exception.
This says the exception covers the .specify/ tree, but the artifact paths in this PR live under specs/20260411-143905-fix-feature-researcher/. Leaving the old path here makes the scope check contradict the path conventions below.
Also applies to: 29-31
🧰 Tools
🪛 LanguageTool
[uncategorized] ~7-~7: The official name of this software platform is spelled with a capital “H”.
Context: ...d in the .specify/ feature tree, plus .github/workflows/feature-research.md (source)...
(GITHUB)
[uncategorized] ~7-~7: The official name of this software platform is spelled with a capital “H”.
Context: ...flows/feature-research.md(source) and.github/workflows/feature-research.lock.yml` (r...
(GITHUB)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@specs/20260411-143905-fix-feature-researcher/tasks.md` at line 7, Replace the
incorrect feature-tree string ".specify/" in the documentation text with the
actual feature-tree path used by this PR
("specs/20260411-143905-fix-feature-researcher/") so the scope check matches the
artifact paths, and update the other occurrences of the same string later in the
document (three additional instances) so all references use the real
feature-tree path.
|
|
||
| ### Implementation for User Story 4 | ||
|
|
||
| - [ ] T021 [US4] Create an observability tracking artifact. Either (a) open a GitHub issue on `chrisleekr/agentsync` titled `[tracking] feature-research post-merge observation window (2026-04-12 → 2026-05-10)` listing the four upcoming Friday dates and the three SC targets (SC-001, SC-002, SC-003), OR (b) add a calendar reminder for each Friday. The artifact must be discoverable by the maintainer without re-reading this spec. Do not create it until the PR is merged — the issue should reference the merge commit SHA. |
There was a problem hiding this comment.
The observation window dates are on Sundays, not the Friday runs this task describes.
2026-04-12 and 2026-05-10 are Sundays. If this is meant to track the first four Friday observations from the current PR timeframe, the concrete run dates are April 17, April 24, May 1, and May 8, 2026.
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@specs/20260411-143905-fix-feature-researcher/tasks.md` at line 140, The task
T021 artifact uses incorrect weekend dates; update the item text for T021 (the
checklist line starting with "T021 [US4] Create an observability tracking
artifact") to reference the four Fridays April 17, April 24, May 1, and May 8,
2026 instead of the Sundays 2026-04-12 → 2026-05-10, and adjust the suggested
GitHub issue title and body to list those four Friday dates and still include
the three SC targets (SC-001, SC-002, SC-003) and the instruction to reference
the merge commit SHA; ensure the new title reflects the corrected date range
(e.g., "feature-research post-merge observation window (2026-04-17 →
2026-05-08)").
|
|
||
| - [X] T029 [P] **FR-005 preservation verification** (closes `/speckit.analyze` F3 gap). After T019 completes, run `grep -n "GH_AW_SAFE_OUTPUTS_CONFIG" .github/workflows/feature-research.lock.yml` and inspect the JSON blob on the matched line. It MUST still contain all five values: `"title-prefix":"[feature-research]"`, `"labels":["feature-research","automated"]`, `"assignees":["chrisleekr"]`, `"close-older-issues":"true"` (or `true`), and `"expires":"7d"`. Any missing or rewritten value is a silent FR-005 violation introduced by the framework upgrade — STOP and investigate before T033 commit. | ||
|
|
||
| - [X] T030 [P] **FR-006 preservation verification** (closes `/speckit.analyze` F4 gap). After T019 completes, verify each of: (a) `grep -n "cron:" .github/workflows/feature-research.lock.yml` still shows the original Friday `0 7 * * 5` (or equivalent UTC+10 Friday 5pm) cron expression from the source `.md` frontmatter; (b) `grep -n "COPILOT_MODEL: gpt-5.4" .github/workflows/feature-research.lock.yml` returns ≥1 hit; (c) `grep -n "max-autopilot-continues 3" .github/workflows/feature-research.lock.yml` returns ≥1 hit; (d) `grep -n 'GH_AW_INFO_ENGINE_ID: "copilot"' .github/workflows/feature-research.lock.yml` returns ≥1 hit. Any deviation from these pre-recompile values is a silent FR-006 violation — STOP and investigate before T033 commit. |
There was a problem hiding this comment.
T030 hardcodes the wrong cron value.
This checklist treats 0 7 * * 5 as the required schedule, but the regenerated workflow in this PR uses 9 6 * * 5. As written, FR-006 verification will flag the current lockfile as a regression even when it still matches the source frontmatter. Compare the lockfile against .github/workflows/feature-research.md instead of a baked-in cron literal.
🧰 Tools
🪛 LanguageTool
[uncategorized] ~166-~166: The official name of this software platform is spelled with a capital “H”.
Context: ...ter T019 completes, verify each of: (a) grep -n "cron:" .github/workflows/feature-research.lock.yml st...
(GITHUB)
[uncategorized] ~166-~166: The official name of this software platform is spelled with a capital “H”.
Context: ... from the source .md frontmatter; (b) grep -n "COPILOT_MODEL: gpt-5.4" .github/workflows/feature-research.lock.yml re...
(GITHUB)
[uncategorized] ~166-~166: The official name of this software platform is spelled with a capital “H”.
Context: ...-research.lock.ymlreturns ≥1 hit; (c)grep -n "max-autopilot-continues 3" .github/workflows/feature-research.lock.yml` re...
(GITHUB)
[uncategorized] ~166-~166: The official name of this software platform is spelled with a capital “H”.
Context: ...-research.lock.ymlreturns ≥1 hit; (d)grep -n 'GH_AW_INFO_ENGINE_ID: "copilot"' .github/workflows/feature-research.lock.yml` re...
(GITHUB)
🤖 Prompt for AI Agents
Verify each finding against the current code and only fix it if needed.
In `@specs/20260411-143905-fix-feature-researcher/tasks.md` at line 166, T030's
FR-006 verification currently hardcodes the cron literal "0 7 * * 5" which
causes false positives; update the check (the step labeled (a) in T030 / FR-006)
to extract the cron expression from the source frontmatter in
.github/workflows/feature-research.md and then assert that the extracted value
appears in .github/workflows/feature-research.lock.yml (rather than matching a
baked-in literal), keeping the other grep checks (COPILOT_MODEL,
max-autopilot-continues, GH_AW_INFO_ENGINE_ID) unchanged.
chrisleekr
changed the title
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Fix the silent-success of
.github/workflows/feature-research.md. The 2026-04-10 scheduled run actions/runs/24232817769 rolled up assuccesswhile producing zero safe-outputs.Branch contains two commits stacking complementary fixes:
ea312ad— framework upgrade (Option A:copilot-cli v1.0.21pin viagh-aw v0.68.1/#25689), FR-001 post-steps assertion, read-only prompt scope, spec attribution correction. Closes spec's original misattribution tocopilot-cli#2479(that bug was fixed in CLI v1.0.19); real culprit is the v1.0.22 "0-byte output" regression.61a1159— Tavily MCP removal, strict-modesandbox.agent: falsefix, full prompt redesign. Necessary because post-compile validation run24280821490showed Option A alone was insufficient: Copilot CLI 1.0.21 still hallucinatedtavilyas a missing tool even though the gateway logged✓ tavily: connected.Architecture
Before — three compounding defects made silent success possible:
flowchart TD TrigB["weekly cron<br/>schedule trigger"]:::neutral ActB["activation job"]:::neutral AgentB["agent job<br/>Copilot CLI v1.0.22"]:::neutral MCPBlockB["MCP gateway<br/>0-byte output regression<br/>gh-aw 25689"]:::badFill ZeroOutB["zero safe-outputs emitted"]:::badFill DetSkipB["detection job skipped<br/>no fail-on-empty in gh-aw"]:::badFill ConcB["conclusion job<br/>rolls up success"]:::badFill SilentGreenB["Actions UI green check<br/>no issue filed<br/>maintainer sees nothing"]:::loudFail TrigB --> ActB --> AgentB --> MCPBlockB --> ZeroOutB --> DetSkipB --> ConcB --> SilentGreenB classDef neutral fill:#1f2937,stroke:#e5e7eb,color:#f9fafb classDef badFill fill:#7f1d1d,stroke:#fecaca,color:#fff1f2 classDef loudFail fill:#991b1b,stroke:#fecaca,color:#fff1f2After — fail-loud, strict-mode compliant, three end-states are indistinguishable in the Actions UI:
flowchart TD TrigA["weekly cron<br/>or workflow_dispatch with focus_area override"]:::neutral ActA["activation job"]:::neutral AgentA["agent job<br/>Copilot CLI v1.0.21<br/>pinned by gh-aw v0.68.1"]:::neutral ToolsA["web-fetch + github MCP<br/>no third-party MCP<br/>AWF firewall ON<br/>network.allowed curated"]:::goodFill EmitA{"agent emits<br/>create_issue OR noop?"}:::decision HappyGap["gap matrix issue created"]:::goodFill HappyNoop["noop tracking issue created"]:::goodFill PostA{"post-steps assertion<br/>items length at least 1?"}:::decision AssertFail["post-step exit 1"]:::badFill ConcFail["conclusion job<br/>rolls up failure"]:::badFill RedX["Actions UI red X<br/>maintainer sees immediately"]:::loudFail ConcOK["conclusion job<br/>rolls up success"]:::goodFill GreenCheck["Actions UI green check<br/>linked issue visible"]:::loudOk TrigA --> ActA --> AgentA --> ToolsA --> EmitA EmitA -- create_issue --> HappyGap --> ConcOK EmitA -- noop --> HappyNoop --> ConcOK EmitA -- zero items --> PostA PostA -- pass --> ConcOK PostA -- fail --> AssertFail --> ConcFail --> RedX ConcOK --> GreenCheck classDef neutral fill:#1f2937,stroke:#e5e7eb,color:#f9fafb classDef decision fill:#1e3a8a,stroke:#dbeafe,color:#f0f9ff classDef goodFill fill:#065f46,stroke:#a7f3d0,color:#ecfdf5 classDef badFill fill:#7f1d1d,stroke:#fecaca,color:#fff1f2 classDef loudOk fill:#14532d,stroke:#bbf7d0,color:#f0fdf4 classDef loudFail fill:#991b1b,stroke:#fecaca,color:#fff1f2Changes
Framework + pin (ea312ad)
gh extension upgrade gh-aw→v0.68.1,gh aw compile feature-research --strict. Regenerated lock file inherits gh-aw's framework-level pin of Copilot CLI tov1.0.21viainstall_copilot_cli.sh 1.0.21. No repo-visible version hardcode; forward-compatible per FR-003.FR-001 silent-success backstop (ea312ad)
post-steps:block in the source.mdfrontmatter reads/tmp/gh-aw/agent_output.json, countscreate_issue+nooprecords,exit 1s when both are zero.if: always(),set -euo pipefail, no secret reads. Makes silent-success impossible per FR-001/SC-005.Read-only prompt scope (ea312ad)
bun run check,bun install,bun test,npm test,npm install,git add,git commit. Stops the agent from improvising build/validation commands on a Bun-less runner (FR-004).Tavily MCP removal (61a1159)
tavily-mcpserver block and itsTAVILY_API_KEYsecret surface. Replaced with built-inweb-fetch+githubMCPtoolsets: [repos]for structured release data. Eliminates the Copilot CLI 1.0.21 tool-hallucination path and removes a third-party attack surface.Strict-mode sandbox fix (61a1159)
sandbox.agent: false+ its 8-lineSECURITY TRADE-OFFcomment block.gh aw compile --strictrefuses thefalsesetting because it disables the AWF firewall. With the firewall ON,network.allowedIS the enforced egress list, not a sanitization hint.network.allowedfrom["all"]to a curated vendor allowlist:defaults,githubcursor.com,*.cursor.comanthropic.com,*.anthropic.comopenai.com,*.openai.comcode.visualstudio.com,*.code.visualstudio.commodelcontextprotocol.io,*.modelcontextprotocol.iozed.dev,*.zed.devgithubMCP for GitHub-hosted research targets, or pick a different source when a host is firewall-blocked. Never attempt to bypass the firewall.Prompt redesign (61a1159)
cursor,claude-code,copilot,codex,vscode,emerging-tools,mcp-ecosystem,vault-mechanics,security,sync-ux), 6-criterion quality gate, 4-step research methodology, Output Structure schema, and Hard Rules.workflow_dispatchinputfocus_area(regex-validated^[a-z][a-z0-9-]{0,31}$) for manual rotation override.safe-outputs.noop:so the agent can signal "no candidate passed the quality gate" without failing FR-001.Spec attribution correction (ea312ad)
spec.mdBackground, FR-002, and Assumptions rewritten to cite thev1.0.22regression andgh-aw#25689as the root-cause fix. The original attribution tocopilot-cli#2479/#2481is labelled as an earlier superseded bug (closed 2026-04-09 / 2026-04-03, fixed in CLI v1.0.19).Test Evidence
24280821490onea312ad. Overallsuccess, 0MCP servers blocked by policymatches, 0bun run checkmatches in agent log. But Copilot CLI 1.0.21 hallucinatedtavilyas a missing tool despite✓ tavily: connected— drove the follow-up61a1159commit.10:45:40Z → 10:47:29Z= 1m 49s (target ≤ 8 min). Total wall-clock 3m 46s.agent_output.jsonfixtures: (1) zero items →exit 1+ FR-001 message, (2)missing_toolonly →exit 1+ FR-001 message (load-bearing — proves the assertion catches the silent-success scenario), (3) onecreate_issue→exit 0, (4) onenoop→exit 0, (5) multiple valid →exit 0. Live tampering ofTAVILY_API_KEYwas unreachable because the secret is now removed.check_workflow_timestamp_api.cjsstep already ran and exitedsuccesson run 24280821490 — the hash guardrail is wired and executing on every dispatch. Live mutation test is a post-merge follow-up on a throwaway branch (see Risks).title-prefix:"[feature-research]",labels:["feature-research","automated"],assignees:["chrisleekr"],close-older-issues:true,expires:7d(serialised as 168h).9 6 * * 5,COPILOT_MODEL: gpt-5.4,--max-autopilot-continues 3,GH_AW_INFO_ENGINE_ID: "copilot"all unchanged by the v0.67.0 → v0.68.1 upgrade.min-integrity: approvedretained; agent-job--allow-domainsunchanged (new list post-61a1159, diffed); detection-job--allow-domainsunchanged; no firewall/egress guardrail silently removed.Security Review
github/gh-aw-actions/setup@v0.68.1— version bumpv0.67.0 → v0.68.1, SHA2fe53acc038ba01c3bbdc767d4b25df31ca5bdfcverified against thegithub/gh-aw-actionsmonorepo tag. Chain of trust: GitHub-owned.actions/github-script@v9— new upstream action; annotated-tag chain verified viagh api .../git/tags/<tag-obj-sha>: tag object SHA373c709c69115d41ff229c7e5df9f8788daa9553dereferences to target commit SHA3a2844b7e9c422d3c10d287c895573f7108da1b3. Diff audit: no newfetch(, noexecSync, no secret exfiltration. One standardgithub-tokenwiring. PASS.copilot_driver.cjswrapper —gh-aw v0.68.1now wraps Copilot CLI through a Node.js driver at${RUNNER_TEMP}/gh-aw/actions/copilot_driver.cjs. Runs inside the existing AWF sandbox — same trust boundary, new middleman. No posture change; disclosed for reviewer awareness.gh-aw-firewall/*:0.25.13 → 0.25.18, standard upstream bump.TAVILY_API_KEYsecret removed entirely — third-party MCP attack surface is gone.sandbox.agent: falseremoved — AWF firewall is ON by default (strict-mode compliant);network.allowedis the enforced egress list, not a sanitization hint. Strict compile output:✓ Compiled 1 workflow(s): 0 error(s), 0 warning(s).Constitution Principle II (docs-only exception)
All changes limited to
.github/workflows/feature-research.{md,lock.yml},.github/aw/actions-lock.json,CLAUDE.md, andspecs/20260411-143905-fix-feature-researcher/**.bun run check— passes (lefthook pre-commit hook confirmed green on both ea312ad and 61a1159; typecheck 1.28s).plan.mdaudited against the global Mermaid rules (WCAG-AA hex pairs,<br/>newlines, no parens, inline:::className, single flowchart, all 16 node IDs ≥ 4 chars). The two diagrams in this PR description follow the same rules.Risks / Follow-ups
git restore→ green-check walkthrough was not run. Explicit follow-up: post-merge test on a throwaway branch.web-fetch+githubMCP) sidesteps the bug. If a future focus area needs a third-party MCP, the same hallucination may re-emerge — add a follow-up issue if/when that happens.frontmatter_hashcovers YAML only, not prompt body. The activation-job's hash check caught zero changes to the body-only## Read-only scopeedit during T020. Worth a separate gh-aw upstream feedback item; not in scope for this PR.spec.mdattribution was corrected in-place rather than via amendment commit — the original misattribution is only recoverable fromgit logon this branch beforeea312ad. Acceptable for a spec that is internal-only, called out for reviewer awareness.Checklist
🤖 Generated with Claude Code
Summary by CodeRabbit
Chores
Bug Fixes
Documentation