-
Notifications
You must be signed in to change notification settings - Fork 74
docs: cross-reference problem docs in roadmap, add applied harness-eval doc #5518
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Open
Benkapner
wants to merge
3
commits into
fullsend-ai:main
Choose a base branch
from
Benkapner:docs/cross-references-and-applied
base: main
Could not load branches
Branch not found: {{ refName }}
Loading
Could not load tags
Nothing to show
Loading
Are you sure you want to change the base?
Some commits from the old base branch may be removed from the timeline,
and old review comments may become outdated.
+32
−0
Open
Changes from 1 commit
Commits
Show all changes
3 commits
Select commit
Hold shift + click to select a range
File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| # Applied: Agent Setup Evaluation Tools | ||
|
|
||
| How fullsend's problem areas manifest in tools that evaluate agent configurations. | ||
|
|
||
| ## Context | ||
|
|
||
| Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side. | ||
|
|
||
| ## Relevant problem areas | ||
|
|
||
| ### Testing agents (testing-agents.md) | ||
|
|
||
| An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others. | ||
|
|
||
| ### MCP configuration drift (mcp-config-drift.md) | ||
|
|
||
| Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current. | ||
|
|
||
| ### Tool call risk assessment (tool-call-risk-assessment.md) | ||
|
|
||
| Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users. | ||
|
|
||
| ### Trustworthiness evidence (trustworthiness-evidence.md) | ||
|
|
||
| For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption. | ||
|
qodo-code-review[bot] marked this conversation as resolved.
Outdated
|
||
|
|
||
| ## Unique considerations | ||
|
|
||
| - **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues | ||
| - **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality | ||
| - **Multi-tool support:** unlike fullsend (which targets a specific platform), evaluation tools must handle multiple AI assistants (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats | ||
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.