Skip to content
Open
Show file tree
Hide file tree
Changes from 1 commit
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
31 changes: 31 additions & 0 deletions docs/problems/applied/harness-eval/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,31 @@
# Applied: Agent Setup Evaluation Tools

How fullsend's problem areas manifest in tools that evaluate agent configurations.
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Outdated

## Context

Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side.

## Relevant problem areas

### Testing agents (testing-agents.md)

An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others.

### MCP configuration drift (mcp-config-drift.md)

Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current.

### Tool call risk assessment (tool-call-risk-assessment.md)

Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users.

### Trustworthiness evidence (trustworthiness-evidence.md)

For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption.
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Outdated

## Unique considerations

- **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues
- **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality
- **Multi-tool support:** unlike fullsend (which targets a specific platform), evaluation tools must handle multiple AI assistants (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats
4 changes: 4 additions & 0 deletions docs/roadmap.md
Original file line number Diff line number Diff line change
Expand Up @@ -112,6 +112,8 @@ Examples of work that could move this forward:
- E2e test improvements — bot authorization fixes, auth alignment ([#2641](https://github.com/fullsend-ai/fullsend/issues/2641), [#2772](https://github.com/fullsend-ai/fullsend/issues/2772), [#2489](https://github.com/fullsend-ai/fullsend/issues/2489))
- Trustworthiness evidence — rework rate tracking, review outcome analysis ([#295](https://github.com/fullsend-ai/fullsend/issues/295))

Related problem docs: [testing-agents](problems/testing-agents.md), [trustworthiness-evidence](problems/trustworthiness-evidence.md), [flapping-convergence](problems/flapping-convergence.md)

### External Partnerships

Making fullsend visible, understandable, and usable by teams outside the core group. This category combines documentation improvements with active partnership engagement — recognizing that docs quality and external adoption are tightly linked.
Expand Down Expand Up @@ -199,6 +201,8 @@ Ongoing work informed by the [security threat model](problems/security-threat-mo
- Separate permission profiles per run phase ([#2826](https://github.com/fullsend-ai/fullsend/issues/2826))
- Privileged operations only in deterministic automation ([#2828](https://github.com/fullsend-ai/fullsend/issues/2828))

Related problem docs: [audit-log-integrity](problems/security-threat-model.md), [MCP config drift](problems/mcp-config-drift.md), [tool call risk assessment](problems/tool-call-risk-assessment.md)
Comment thread
qodo-code-review[bot] marked this conversation as resolved.
Outdated

### Human factors and governance

As autonomous contribution scales, the organizational questions become unavoidable: domain ownership shifts, review fatigue, contributor motivation, and who has authority to make binding decisions about agent behavior.
Expand Down
Loading