ci(#5509): add harness-eval static analysis for agent configurations

[Benkapner]

Summary

Adds a CI workflow that runs harness-eval static analysis on every PR touching agent configuration files.

• Scans both repo root (CLAUDE.md, AGENTS.md, skills/, commands/) and scaffold (internal/scaffold/fullsend-repo/)
• Uses the recommended preset (structural errors block, quality warnings are advisory)
• Security gate is advisory (non-blocking) until the team validates signal quality
• SARIF output enables inline annotations on the PR diff via GitHub Code Scanning
• Summary comment posted on each PR with scan results

No LLM calls, runs in seconds, fully deterministic. Uses the harness-eval GitHub Action with pinned checkout action SHA per repo convention.

Related Issue

#5509

Testing

• Workflow file is valid YAML
• Paths filter limits runs to agent config changes only
• Actions checkout SHA-pinned per repo security convention
• Will validate on this PR once CI runs

Checklist

• PR title follows Conventional Commits
• Commits are signed off (DCO)

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[github-actions]

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

[qodo-code-review]

PR Summary by Qodo

CI: add harness-eval checks for agent configuration changes

⚙️ Configuration changes ✨ Enhancement 🕐 Less than 10 minutes

AI Description

• Add a PR-scoped CI workflow to statically analyze agent configuration files.
• Run harness-eval on repo root and scaffold directories, emitting SARIF for code scanning.
• Post a PR comment with results; block only on structural lint errors.

Diagram

graph TD
  A["PR touches agent configs"] --> B["Workflow: harness-eval.yml"] --> C["Checkout repo"] --> D["harness-eval action"] --> E["Upload SARIF"] --> F["GitHub Code Scanning"]
  D --> G["PR summary comment"]

Loading

High-Level Assessment

The following are alternative approaches to this PR:

1. Pin harness-eval action to a tag/SHA

• ➕ Reduces supply-chain risk vs referencing @main
• ➕ Improves reproducibility of lint/security results over time
• ➖ Requires periodic maintenance to bump the pinned ref
• ➖ May slow adoption of upstream fixes unless updated

2. Run harness-eval via a pinned container/CLI step

• ➕ Full control over exact tool version and execution environment
• ➕ Less coupling to upstream action implementation changes
• ➖ More workflow YAML and setup complexity
• ➖ Need to manage installation/auth and SARIF wiring manually

3. Make security gate blocking after an evaluation period

• ➕ Prevents risky agent-config changes once signal quality is validated
• ➕ Aligns with “shift-left” security expectations for config inputs
• ➖ Potential for developer friction if false positives remain
• ➖ Requires agreement on severity thresholds and remediation workflow

Recommendation: The overall approach (path-filtered PR workflow + SARIF annotations + PR comment) is a good fit and should stay. The main improvement to consider is pinning the harness-eval action reference (currently @main) to a tag or commit SHA to match the repo’s apparent supply-chain hardening (checkout is already SHA-pinned). Keeping the security gate advisory is reasonable initially; revisit making it blocking once false-positive rates are understood.

Files changed (1) +35 / -0

Other (1) +35 / -0

harness-eval.ymlAdd PR workflow to run harness-eval with SARIF + PR commenting +35/-0

Add PR workflow to run harness-eval with SARIF + PR commenting

• Introduces a GitHub Actions workflow triggered only when agent configuration paths change. Runs harness-eval against both the repository root and the scaffold directory using the recommended preset, uploads SARIF for code scanning annotations, and posts a PR comment; lint blocks only on errors and the security gate remains non-blocking.

.github/workflows/harness-eval.yml

[github-actions]

Site preview

Preview: https://089d1e8a-site.fullsend-ai.workers.dev

Commit: 310f4893750468cf99d3207fc8463f85e2e17423

[qodo-code-review]

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. Unpinned action reference ✓ Resolved 🐞 Bug ⛨ Security

Description

The workflow runs a third-party action from the moving @main ref, so upstream changes can silently
change CI behavior or introduce a supply-chain compromise. This also breaks the repo’s established
SHA-pinning convention for Actions.

Code

.github/workflows/harness-eval.yml[25]

+      - uses: redhat-community-ai-tools/harness-eval/.github/actions/harness-eval@main

Relevance

⭐⭐⭐ High

Team has accepted pinning mutable refs to immutable commits for CI supply-chain/reproducibility;
@main is riskier than tags.

PR-#1682
PR-#2069

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new workflow references the harness-eval action via a mutable branch ref, while other workflows
in this repo pin actions by commit SHA.

.github/workflows/harness-eval.yml[23-26]
.github/workflows/lint.yml[15-29]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The workflow uses `redhat-community-ai-tools/harness-eval/...@main`, which is a mutable reference and can change without any change in this repository.

### Issue Context
This repo pins GitHub Actions to commit SHAs (see existing workflows) to reduce supply-chain risk and keep CI deterministic.

### Fix Focus Areas
- .github/workflows/harness-eval.yml[23-26]

### Suggested fix
- Replace `@main` with a specific commit SHA (or a tag + SHA pin if that’s your convention).
- Add a version comment (as done for other actions) so updates are deliberate and reviewable.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

2. Overbroad workflow token perms ✓ Resolved 🐞 Bug ⛨ Security

Description

The workflow grants pull-requests: write and security-events: write to the job that runs an
external action, unnecessarily expanding what that action can do with GITHUB_TOKEN. This conflicts
with the repo’s existing pattern of keeping write-scoped permissions out of jobs that process PR
content.

Code

.github/workflows/harness-eval.yml[R14-26]

+permissions:
+  security-events: write
+  contents: read
+  pull-requests: write
+
+jobs:
+  lint-and-security:
+    runs-on: ubuntu-latest
+    steps:
+      - uses: actions/checkout@11bd71901bbe5b1630ceea73d27597364c9af683 # v4.2.2
+
+      - uses: redhat-community-ai-tools/harness-eval/.github/actions/harness-eval@main
+        with:

Relevance

⭐⭐⭐ High

Repo previously split PR-write perms into separate gate job to avoid untrusted checkout jobs having
write scope.

PR-#2106

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The new workflow grants write permissions broadly, whereas existing CI workflows document and
implement a pattern of isolating pull-requests: write from jobs that handle PR content.

.github/workflows/harness-eval.yml[14-35]
.github/workflows/e2e.yml[66-109]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The workflow-level permissions include `pull-requests: write` and `security-events: write`, and the job runs a third-party action. Least-privilege suggests limiting write-scoped permissions to only the steps/jobs that truly need them.

### Issue Context
In this repo, other workflows explicitly isolate `pull-requests: write` away from jobs that operate on PR content (see e2e.yml comments and job permissions).

### Fix Focus Areas
- .github/workflows/harness-eval.yml[14-35]

### Suggested fix
- Move permissions from workflow-level to job-level.
- If PR comments are optional, drop `pull-requests: write` and set `comment: "false"`.
- If PR comments are required, split into two jobs:
 - **scan job**: `contents: read` + `security-events: write` only.
 - **comment job**: `pull-requests: write` only, and post a summary using a trusted in-repo step/script based on scan outputs/artifacts.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

3. Runner image not pinned ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

The workflow uses ubuntu-latest, which is a moving target and can change the execution environment
over time. Other workflows in this repo pin to ubuntu-24.04, so this introduces avoidable CI drift
and inconsistency.

Code

.github/workflows/harness-eval.yml[21]

+    runs-on: ubuntu-latest

Relevance

⭐⭐ Medium

Repo already uses ubuntu-latest in key workflows; no clear precedent pushing pinned runner images.

PR-#1215
PR-#2106

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

Only this new workflow uses ubuntu-latest; existing CI uses a pinned Ubuntu version, reducing
environment drift.

.github/workflows/harness-eval.yml[19-22]
.github/workflows/lint.yml[13-18]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`runs-on: ubuntu-latest` can change over time (image upgrades), which can introduce unexpected CI behavior changes.

### Issue Context
Other workflows in this repo consistently use `ubuntu-24.04`.

### Fix Focus Areas
- .github/workflows/harness-eval.yml[20-22]

### Suggested fix
- Change `runs-on: ubuntu-latest` to `runs-on: ubuntu-24.04` for consistency and reproducibility.

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

---

[rh-hemartin]

Agent files will disappear soon from this repository, is this still valid with that fact?

[Benkapner]

@rh-hemartin and this repo will not have any harness files at all? agents, skills, commands, claude md hooks? if not so its not relevant but maybe in the new/other repo they will be?

[rh-hemartin]

@rh-hemartin and this repo will not have any harness files at all? agents, skills, commands, claude md hooks? if not so its not relevant but maybe in the new/other repo they will be?

No, they were moved to github.com/fullsend-ai/agents. We will have skills, commands and claude.md hooks here, but for our development usage, not for agents.

[Benkapner]

the main thing this tool does is lint + security checks if the user has committed any change to one of those harness files so could be relevant both here and the fullsend-ai/agents. wont hurt to add it to the CI. also it connects to docs/problems/trustworthiness-evidence.md @rh-hemartin

[rh-hemartin]

Ah, for a moment I was confused as well regarding the concept of harness, for us is a specific thing. Then it could be useful here, yes.

You need to pass pre-commit to your files BTW.

[rh-hemartin]

Incorporate this as a step in the current lint.yaml workflow, as it contains all the CI checks. The file filter can be done within a step, or just run it against everything. Optionally, since this does not require LLM, you could add it as a pre-commit (however this is more involved than the current setup, so don't worry, maybe a followup).

[Benkapner]

moved harness-eval into lint.yml as a step in the test job instead of a separate workflow. uses the checkout v7.0.1 already pinned there. will look into pre-commit integration as a follow-up.

[rh-hemartin]

You need to fix all the commits to ci(#5509) or ci(lint) or something similar. You could also squash them into a single one if you would like.

[Benkapner]

squashed into a single commit with ci(#5509) prefix. should pass commit-lint now.