docs: cross-reference problem docs in roadmap, add applied harness-eval doc by Benkapner · Pull Request #5518 · fullsend-ai/fullsend

Benkapner · 2026-07-23T08:55:45Z

Summary

Two small docs improvements:

Roadmap cross-references

Adds "Related problem docs" links to two roadmap sections:

Security hardening: links to audit-log-integrity, MCP config drift, tool call risk assessment
Testing: links to testing-agents, trustworthiness-evidence, flapping-convergence

These problem docs were merged previously but weren't linked from the roadmap sections they're relevant to.

Applied problem doc

Adds docs/problems/applied/harness-eval/README.md covering how fullsend's problem areas manifest in agent setup evaluation tools (the "who watches the watchmen" angle). Follows the existing applied-problems pattern established by applied/konflux-ci/.

Testing

Docs-only change, no code/lint/test impact

Checklist

PR title follows Conventional Commits
Commits are signed off (DCO)

…al doc - Add "Related problem docs" links in the Security hardening and Testing roadmap sections pointing to relevant problem documents - Add docs/problems/applied/harness-eval/ covering how fullsend's problem areas manifest in agent setup evaluation tools Signed-off-by: Benjamin Kapner <bkapner@redhat.com>

github-actions · 2026-07-23T08:55:58Z

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

github-actions · 2026-07-23T08:55:58Z

E2E tests did not run

E2E tests run automatically for org/repo members and collaborators on pull requests.

For other contributors, a maintainer must add the ok-to-test label after the latest push.

See E2E testing guide for details.

qodo-code-review · 2026-07-23T09:00:28Z

PR Summary by Qodo

Docs: cross-reference problem docs in roadmap and add applied harness-eval note

📝 Documentation 🕐 10-20 Minutes

AI Description

• Add roadmap cross-references to relevant problem documents for Testing and Security hardening.
• Add an applied problem doc on agent setup evaluation tools (“who watches the watchmen”).
• Improve discoverability of existing problem areas from the roadmap.

Diagram

graph TD
  A["docs/roadmap.md"] --> B["Problem docs (existing)"]
  C["docs/problems/applied/harness-eval/README.md"] --> B

High-Level Assessment

The chosen approach (explicit cross-links in roadmap sections plus an applied-problems writeup) is the most direct way to improve discoverability without adding new navigation structure. Alternatives like a centralized index or autogenerated link sections would add maintenance overhead for limited benefit at this scale.

Files changed (2) +35 / -0

Documentation (2) +35 / -0

README.mdAdd applied problem doc for agent setup evaluation tooling +31/-0

Add applied problem doc for agent setup evaluation tooling

• Introduces a new applied-problems README explaining how core fullsend problem areas manifest in agent configuration evaluation tools (linters/scanners). Cross-references relevant existing problem docs and calls out tooling-specific considerations like recursive evaluation and multi-assistant format support.

docs/problems/applied/harness-eval/README.md

roadmap.mdAdd “Related problem docs” cross-links in Testing and Security hardening sections +4/-0

Add “Related problem docs” cross-links in Testing and Security hardening sections

• Adds inline “Related problem docs” link lists to the Testing and Security hardening roadmap sections to connect roadmap initiatives to previously-merged problem documents. This improves navigation between strategic roadmap items and deeper problem-area writeups.

docs/roadmap.md

codecov · 2026-07-23T09:02:59Z

Codecov Report

✅ All modified and coverable lines are covered by tests.

📢 Thoughts on this report? Let us know!

qodo-code-review · 2026-07-23T09:03:48Z

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (1) 📜 Skill insights (0)

Context used

✅ Compliance rules (platform): 54 rules

✅ Skills: writing-user-docs, writing-adrs

1. ~~Audit log link unanchored~~ ✓ Resolved 🐞 Bug ≡ Correctness

Description

In docs/roadmap.md, the “audit-log-integrity” related-doc item links to
problems/security-threat-model.md without the section anchor, so readers don’t land on the audit log
integrity section the label implies. This makes the roadmap cross-reference harder to use and easy
to misinterpret.

Code

docs/roadmap.md[204]

+Related problem docs: [audit-log-integrity](problems/security-threat-model.md), [MCP config drift](problems/mcp-config-drift.md), [tool call risk assessment](problems/tool-call-risk-assessment.md)

Relevance

⭐⭐⭐ High
They frequently accept fixes to docs cross-references/anchors so link targets match the referenced
section.
PR-#226
PR-#770

ⓘ Recommendations generated based on similar findings in past PRs

Evidence

The roadmap’s new related-docs line uses link text “audit-log-integrity” but points only to
security-threat-model.md (no fragment). The threat model contains a specific “audit log integrity”
section with a stable GitHub heading anchor that can be linked directly.

docs/roadmap.md[193-205]
docs/problems/security-threat-model.md[424-426]
PR-#226

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
`docs/roadmap.md` adds a related-doc link labeled `audit-log-integrity`, but the URL points to the top of `problems/security-threat-model.md` rather than the specific “audit log integrity” section.

### Issue Context
The security threat model contains a dedicated heading `## Cross-cutting concern: audit log integrity`, so the roadmap link can deep-link directly to that section.

### Fix Focus Areas
- docs/roadmap.md[193-205]
- docs/problems/security-threat-model.md[424-426]

### Suggested change
Update the roadmap link to include the heading fragment, e.g.:
- `[audit log integrity](problems/security-threat-model.md#cross-cutting-concern-audit-log-integrity)`

(Optionally align the link text casing with other items.)

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

2. ~~Applied index omits doc~~ ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

This PR adds docs/problems/applied/harness-eval/README.md but doesn’t add it to
docs/problems/applied/README.md, leaving the new applied doc undiscoverable from the applied-doc
index referenced elsewhere in the docs. Readers following the applied-docs entry point won’t see the
new document.

Code

docs/problems/applied/harness-eval/README.md[R1-3]

+# Applied: Agent Setup Evaluation Tools
+
+How fullsend's problem areas manifest in tools that evaluate agent configurations.

Relevance

⭐⭐⭐ High
Repo has accepted adding new docs to an index/README to keep them discoverable.
PR-#2549

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The applied index currently lists only konflux-ci, while the PR introduces a new applied doc
directory; vision.md explicitly directs readers to the applied docs entry point, so omitting the new
item makes it harder to find.
docs/problems/applied/README.md[5-8]
docs/problems/applied/harness-eval/README.md[1-3]
docs/vision.md[52-52]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
A new applied doc directory `applied/harness-eval/` is added, but `docs/problems/applied/README.md` is not updated to list it.

### Issue Context
`docs/vision.md` points readers to `problems/applied/` for applied docs, so the applied index should enumerate new entries.

### Fix Focus Areas
- docs/problems/applied/README.md[5-8]
- docs/vision.md[52-52]

### Suggested change
Add a new bullet under “Current consumers” (or rename that section to something more general) such as:
- `- **[harness-eval](harness-eval/)** — Applied considerations for agent setup evaluation tools.`

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

3. ~~Plaintext problem references~~ ✓ Resolved 🐞 Bug ⚙ Maintainability

Description

docs/problems/applied/harness-eval/README.md references related problem docs as plain filenames
(e.g., “(testing-agents.md)”), which aren’t clickable and are inconsistent with the established
applied-doc pattern of using relative Markdown links. This reduces navigability and makes it harder
to jump from the applied doc to the underlying problem docs.

Code

docs/problems/applied/harness-eval/README.md[R11-25]

+### Testing agents (testing-agents.md)
+
+An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others.
+
+### MCP configuration drift (mcp-config-drift.md)
+
+Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current.
+
+### Tool call risk assessment (tool-call-risk-assessment.md)
+
+Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users.
+
+### Trustworthiness evidence (trustworthiness-evidence.md)
+
+For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption.

Relevance

⭐⭐⭐ High
Precedent: convert plain-text doc references into explicit relative Markdown links for
consistency/navigation.
PR-#770

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
The new harness-eval applied doc uses filename-only references in the headings, while an existing
applied doc demonstrates the repo’s preferred pattern: real Markdown links with correct relative
paths to docs/problems/*.md.
docs/problems/applied/harness-eval/README.md[11-25]
docs/problems/applied/konflux-ci/README.md[32-35]
PR-#226

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

### Issue description
The new applied doc names related problem docs as bare filenames in parentheses, rather than actual Markdown links.

### Issue Context
Existing applied docs link to the general problem docs using correct relative paths (e.g. `../../architectural-invariants.md`). From `docs/problems/applied/harness-eval/README.md`, the general problem docs live at `../../<problem>.md`.

### Fix Focus Areas
- docs/problems/applied/harness-eval/README.md[11-25]
- docs/problems/applied/konflux-ci/README.md[32-35]

### Suggested change
Convert headings to links, e.g.:
- `### [Testing agents](../../testing-agents.md)`
- `### [MCP configuration drift](../../mcp-config-drift.md)`
- `### [Tool call risk assessment](../../tool-call-risk-assessment.md)`
- `### [Trustworthiness evidence](../../trustworthiness-evidence.md)`

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

4. harness-eval lacks solution options 📘 Rule violation ⌂ Architecture

Description

The new problem document under docs/problems/ does not present at least two solution options with
clearly labeled trade-offs, which makes it read as descriptive background rather than an
options-and-tradeoffs problem doc.

Code

docs/problems/applied/harness-eval/README.md[R1-31]

+# Applied: Agent Setup Evaluation Tools
+
+How fullsend's problem areas manifest in tools that evaluate agent configurations.
+
+## Context
+
+Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side.
+
+## Relevant problem areas
+
+### Testing agents (testing-agents.md)
+
+An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others.
+
+### MCP configuration drift (mcp-config-drift.md)
+
+Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current.
+
+### Tool call risk assessment (tool-call-risk-assessment.md)
+
+Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users.
+
+### Trustworthiness evidence (trustworthiness-evidence.md)
+
+For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption.
+
+## Unique considerations
+
+- **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues
+- **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality
+- **Multi-tool support:** unlike fullsend (which targets a specific platform), evaluation tools must handle multiple AI assistants (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats

Relevance

⭐ Low
Team has rejected “must add options/trade-offs” structure enforcement for problem docs in similar
cases.
PR-#3063
PR-#2549

ⓘ Recommendations generated based on similar findings in past PRs

Evidence
PR Compliance ID 1062035 requires each new/modified file under docs/problems/ to include at least
two distinct options with explicit trade-offs. The added
docs/problems/applied/harness-eval/README.md contains context and relevant-problem-area
descriptions but no Option 1/Option 2 (or equivalent) sections with pros/cons or risks.
Rule 1062035: Problem docs must present multiple options with trade-offs, not a single prescribed solution
docs/problems/applied/harness-eval/README.md[1-31]

Agent prompt

The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The new problem doc does not describe at least two distinct solution options/approaches and their trade-offs.

## Issue Context
Compliance requires problem docs under `docs/problems/` (including `applied/`) to document multiple options with clearly labeled trade-offs (pros/cons, risks), even if one option is recommended.

## Fix Focus Areas
- docs/problems/applied/harness-eval/README.md[1-31]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

github-actions · 2026-07-23T09:08:19Z

Site preview

Preview: https://d20ea12f-site.fullsend-ai.workers.dev

Commit: d030fa8e47d71a83c8f09b4decabc91b06adb90b

rh-hemartin

Remove the "related problem docs" from the roadmap, the convention does not exist as far as I can see and I don't want to start that convention.

rh-hemartin · 2026-07-23T09:12:19Z

I agree with the document, but I do think that you need to refer to them more specfically as "agent-driven tools", for example here:

# Agent Setup Evaluation Tools

How fullsend's problem areas manifest in tools that evaluate agent configurations.

I would go with

# Agent-drive Evaluation Tools

How Fullsend's own problem areas appear in agent-driven tools  that evaluate agent configurations (or "who watches the watchmen" problem).

…index Address review feedback: - Link audit-log-integrity to section anchor in security-threat-model.md - Use relative markdown links in applied/harness-eval doc - Add harness-eval entry to applied/README.md index Signed-off-by: Benjamin Kapner <bkapner@redhat.com>

Address rh-hemartin feedback: - Remove "Related problem docs" lines from roadmap (convention doesn't exist) - Rename applied doc to "Agent-Driven Evaluation Tools" Signed-off-by: Benjamin Kapner <bkapner@redhat.com>

Benkapner · 2026-07-23T09:33:56Z

addressed all notes @rh-hemartin :

removed the "Related problem docs" lines from the roadmap sections, won't start that convention
renamed the applied doc to "Agent-Driven Evaluation Tools" per your suggestion
qodo findings were already fixed in the previous push (section anchor on audit-log link, relative markdown links, applied index updated)

Benkapner requested a review from a team as a code owner July 23, 2026 08:55

qodo-code-review Bot reviewed Jul 23, 2026

View reviewed changes

Comment thread docs/roadmap.md Outdated

Comment thread docs/problems/applied/harness-eval/README.md Outdated

Comment thread docs/problems/applied/harness-eval/README.md Outdated

github-actions Bot deployed to site-preview July 23, 2026 09:08 View deployment

rh-hemartin requested changes Jul 23, 2026

View reviewed changes

github-actions Bot deployed to site-preview July 23, 2026 09:21 View deployment

fix(docs): remove roadmap cross-refs, rename to agent-driven

d030fa8

Address rh-hemartin feedback: - Remove "Related problem docs" lines from roadmap (convention doesn't exist) - Rename applied doc to "Agent-Driven Evaluation Tools" Signed-off-by: Benjamin Kapner <bkapner@redhat.com>

github-actions Bot deployed to site-preview July 23, 2026 09:35 View deployment

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

docs: cross-reference problem docs in roadmap, add applied harness-eval doc#5518

docs: cross-reference problem docs in roadmap, add applied harness-eval doc#5518
Benkapner wants to merge 3 commits into
fullsend-ai:mainfrom
Benkapner:docs/cross-references-and-applied

Benkapner commented Jul 23, 2026 •

edited

Loading

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

github-actions Bot commented Jul 23, 2026

Uh oh!

qodo-code-review Bot commented Jul 23, 2026

Uh oh!

codecov Bot commented Jul 23, 2026

Uh oh!

qodo-code-review Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 23, 2026 •

edited

Loading

Uh oh!

rh-hemartin left a comment

Uh oh!

rh-hemartin commented Jul 23, 2026

Uh oh!

Benkapner commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Uh oh!

Conversation

Benkapner commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Summary

Roadmap cross-references

Applied problem doc

Testing

Checklist

Uh oh!

github-actions Bot commented Jul 23, 2026

E2E tests did not run

Uh oh!

github-actions Bot commented Jul 23, 2026

E2E tests did not run

Uh oh!

qodo-code-review Bot commented Jul 23, 2026

PR Summary by Qodo

Uh oh!

codecov Bot commented Jul 23, 2026

Codecov Report

Uh oh!

qodo-code-review Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Code Review by Qodo

Uh oh!

Uh oh!

Uh oh!

Uh oh!

github-actions Bot commented Jul 23, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Site preview

Uh oh!

rh-hemartin left a comment

Choose a reason for hiding this comment

Uh oh!

rh-hemartin commented Jul 23, 2026

Uh oh!

Benkapner commented Jul 23, 2026

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Benkapner commented Jul 23, 2026 •

edited

Loading

qodo-code-review Bot commented Jul 23, 2026 •

edited

Loading

github-actions Bot commented Jul 23, 2026 •

edited

Loading