From 379639bdb0d6018661486258632da737c0d92449 Mon Sep 17 00:00:00 2001 From: Benjamin Kapner Date: Thu, 23 Jul 2026 11:55:28 +0300 Subject: [PATCH 1/6] docs: cross-reference problem docs in roadmap, add applied harness-eval doc - Add "Related problem docs" links in the Security hardening and Testing roadmap sections pointing to relevant problem documents - Add docs/problems/applied/harness-eval/ covering how fullsend's problem areas manifest in agent setup evaluation tools Signed-off-by: Benjamin Kapner --- docs/problems/applied/harness-eval/README.md | 31 ++++++++++++++++++++ docs/roadmap.md | 4 +++ 2 files changed, 35 insertions(+) create mode 100644 docs/problems/applied/harness-eval/README.md diff --git a/docs/problems/applied/harness-eval/README.md b/docs/problems/applied/harness-eval/README.md new file mode 100644 index 000000000..f8fac5d31 --- /dev/null +++ b/docs/problems/applied/harness-eval/README.md @@ -0,0 +1,31 @@ +# Applied: Agent Setup Evaluation Tools + +How fullsend's problem areas manifest in tools that evaluate agent configurations. + +## Context + +Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side. + +## Relevant problem areas + +### Testing agents (testing-agents.md) + +An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others. + +### MCP configuration drift (mcp-config-drift.md) + +Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current. + +### Tool call risk assessment (tool-call-risk-assessment.md) + +Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users. + +### Trustworthiness evidence (trustworthiness-evidence.md) + +For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption. + +## Unique considerations + +- **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues +- **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality +- **Multi-tool support:** unlike fullsend (which targets a specific platform), evaluation tools must handle multiple AI assistants (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats diff --git a/docs/roadmap.md b/docs/roadmap.md index afc432eed..ec1f8596d 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -112,6 +112,8 @@ Examples of work that could move this forward: - E2e test improvements — bot authorization fixes, auth alignment ([#2641](https://github.com/fullsend-ai/fullsend/issues/2641), [#2772](https://github.com/fullsend-ai/fullsend/issues/2772), [#2489](https://github.com/fullsend-ai/fullsend/issues/2489)) - Trustworthiness evidence — rework rate tracking, review outcome analysis ([#295](https://github.com/fullsend-ai/fullsend/issues/295)) +Related problem docs: [testing-agents](problems/testing-agents.md), [trustworthiness-evidence](problems/trustworthiness-evidence.md), [flapping-convergence](problems/flapping-convergence.md) + ### External Partnerships Making fullsend visible, understandable, and usable by teams outside the core group. This category combines documentation improvements with active partnership engagement — recognizing that docs quality and external adoption are tightly linked. @@ -199,6 +201,8 @@ Ongoing work informed by the [security threat model](problems/security-threat-mo - Separate permission profiles per run phase ([#2826](https://github.com/fullsend-ai/fullsend/issues/2826)) - Privileged operations only in deterministic automation ([#2828](https://github.com/fullsend-ai/fullsend/issues/2828)) +Related problem docs: [audit-log-integrity](problems/security-threat-model.md), [MCP config drift](problems/mcp-config-drift.md), [tool call risk assessment](problems/tool-call-risk-assessment.md) + ### Human factors and governance As autonomous contribution scales, the organizational questions become unavoidable: domain ownership shifts, review fatigue, contributor motivation, and who has authority to make binding decisions about agent behavior. From 613a4cc126e951fa0c7cdcf62e25bb46172b7ea9 Mon Sep 17 00:00:00 2001 From: Benjamin Kapner Date: Thu, 23 Jul 2026 12:19:46 +0300 Subject: [PATCH 2/6] fix(docs): anchor audit-log link, use relative links, update applied index Address review feedback: - Link audit-log-integrity to section anchor in security-threat-model.md - Use relative markdown links in applied/harness-eval doc - Add harness-eval entry to applied/README.md index Signed-off-by: Benjamin Kapner --- docs/problems/applied/README.md | 1 + docs/problems/applied/harness-eval/README.md | 8 ++++---- docs/roadmap.md | 2 +- 3 files changed, 6 insertions(+), 5 deletions(-) diff --git a/docs/problems/applied/README.md b/docs/problems/applied/README.md index 72f20d34a..a9ae5a6ed 100644 --- a/docs/problems/applied/README.md +++ b/docs/problems/applied/README.md @@ -5,6 +5,7 @@ This directory contains organization-specific considerations for applying fullse ## Current consumers - **[konflux-ci](konflux-ci/)** — Kubernetes-native CI/CD platform. The original proving ground for fullsend. +- **[harness-eval](harness-eval/)** — Agent setup evaluation tools. How fullsend's problem areas manifest in tools that lint and security-scan agent configurations. ## Adding a new consumer diff --git a/docs/problems/applied/harness-eval/README.md b/docs/problems/applied/harness-eval/README.md index f8fac5d31..ff4b9ed56 100644 --- a/docs/problems/applied/harness-eval/README.md +++ b/docs/problems/applied/harness-eval/README.md @@ -8,19 +8,19 @@ Agent setup evaluation tools (linters, security scanners for skills/commands/age ## Relevant problem areas -### Testing agents (testing-agents.md) +### [Testing agents](../../testing-agents.md) An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others. -### MCP configuration drift (mcp-config-drift.md) +### [MCP configuration drift](../../mcp-config-drift.md) Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current. -### Tool call risk assessment (tool-call-risk-assessment.md) +### [Tool call risk assessment](../../tool-call-risk-assessment.md) Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users. -### Trustworthiness evidence (trustworthiness-evidence.md) +### [Trustworthiness evidence](../../trustworthiness-evidence.md) For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption. diff --git a/docs/roadmap.md b/docs/roadmap.md index ec1f8596d..ccd3bfca8 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -201,7 +201,7 @@ Ongoing work informed by the [security threat model](problems/security-threat-mo - Separate permission profiles per run phase ([#2826](https://github.com/fullsend-ai/fullsend/issues/2826)) - Privileged operations only in deterministic automation ([#2828](https://github.com/fullsend-ai/fullsend/issues/2828)) -Related problem docs: [audit-log-integrity](problems/security-threat-model.md), [MCP config drift](problems/mcp-config-drift.md), [tool call risk assessment](problems/tool-call-risk-assessment.md) +Related problem docs: [audit-log-integrity](problems/security-threat-model.md#cross-cutting-concern-audit-log-integrity), [MCP config drift](problems/mcp-config-drift.md), [tool call risk assessment](problems/tool-call-risk-assessment.md) ### Human factors and governance From b4954181d9505edaa46f3985961077ca75361f52 Mon Sep 17 00:00:00 2001 From: Benjamin Kapner Date: Thu, 23 Jul 2026 12:32:56 +0300 Subject: [PATCH 3/6] fix(docs): remove roadmap cross-refs, rename to agent-driven Address rh-hemartin feedback: - Remove "Related problem docs" lines from roadmap (convention doesn't exist) - Rename applied doc to "Agent-Driven Evaluation Tools" Signed-off-by: Benjamin Kapner --- docs/problems/applied/README.md | 2 +- docs/problems/applied/harness-eval/README.md | 4 ++-- docs/roadmap.md | 4 ---- 3 files changed, 3 insertions(+), 7 deletions(-) diff --git a/docs/problems/applied/README.md b/docs/problems/applied/README.md index a9ae5a6ed..59b0fcc43 100644 --- a/docs/problems/applied/README.md +++ b/docs/problems/applied/README.md @@ -5,7 +5,7 @@ This directory contains organization-specific considerations for applying fullse ## Current consumers - **[konflux-ci](konflux-ci/)** — Kubernetes-native CI/CD platform. The original proving ground for fullsend. -- **[harness-eval](harness-eval/)** — Agent setup evaluation tools. How fullsend's problem areas manifest in tools that lint and security-scan agent configurations. +- **[harness-eval](harness-eval/)** — Agent-driven evaluation tools. How fullsend's problem areas appear in tools that lint and security-scan agent configurations. ## Adding a new consumer diff --git a/docs/problems/applied/harness-eval/README.md b/docs/problems/applied/harness-eval/README.md index ff4b9ed56..0ab826ebd 100644 --- a/docs/problems/applied/harness-eval/README.md +++ b/docs/problems/applied/harness-eval/README.md @@ -1,6 +1,6 @@ -# Applied: Agent Setup Evaluation Tools +# Agent-Driven Evaluation Tools -How fullsend's problem areas manifest in tools that evaluate agent configurations. +How fullsend's own problem areas appear in agent-driven tools that evaluate agent configurations (or "who watches the watchmen"). ## Context diff --git a/docs/roadmap.md b/docs/roadmap.md index ccd3bfca8..afc432eed 100644 --- a/docs/roadmap.md +++ b/docs/roadmap.md @@ -112,8 +112,6 @@ Examples of work that could move this forward: - E2e test improvements — bot authorization fixes, auth alignment ([#2641](https://github.com/fullsend-ai/fullsend/issues/2641), [#2772](https://github.com/fullsend-ai/fullsend/issues/2772), [#2489](https://github.com/fullsend-ai/fullsend/issues/2489)) - Trustworthiness evidence — rework rate tracking, review outcome analysis ([#295](https://github.com/fullsend-ai/fullsend/issues/295)) -Related problem docs: [testing-agents](problems/testing-agents.md), [trustworthiness-evidence](problems/trustworthiness-evidence.md), [flapping-convergence](problems/flapping-convergence.md) - ### External Partnerships Making fullsend visible, understandable, and usable by teams outside the core group. This category combines documentation improvements with active partnership engagement — recognizing that docs quality and external adoption are tightly linked. @@ -201,8 +199,6 @@ Ongoing work informed by the [security threat model](problems/security-threat-mo - Separate permission profiles per run phase ([#2826](https://github.com/fullsend-ai/fullsend/issues/2826)) - Privileged operations only in deterministic automation ([#2828](https://github.com/fullsend-ai/fullsend/issues/2828)) -Related problem docs: [audit-log-integrity](problems/security-threat-model.md#cross-cutting-concern-audit-log-integrity), [MCP config drift](problems/mcp-config-drift.md), [tool call risk assessment](problems/tool-call-risk-assessment.md) - ### Human factors and governance As autonomous contribution scales, the organizational questions become unavoidable: domain ownership shifts, review fatigue, contributor motivation, and who has authority to make binding decisions about agent behavior. From 8882ae00031cd37739a2438041595b50d807be88 Mon Sep 17 00:00:00 2001 From: Benjamin Kapner Date: Tue, 28 Jul 2026 09:03:50 +0300 Subject: [PATCH 4/6] fix(docs): rename to agent-eval-tools, fix terminology, add tech landscape Address waynesun09 review: - Rename directory from harness-eval/ to agent-eval-tools/ to avoid collision with fullsend's own "harness" terminology - Fix "anti-jailbreak patterns" to "jailbreak patterns" - Fix "phantom MCP detection" to describe the capability without asserting an unverified term - Soften "must handle" to "may need to support" for multi-assistant - Add Technology landscape section per applied-doc template - Name and link harness-eval explicitly as the concrete example Signed-off-by: Benjamin Kapner --- docs/problems/applied/README.md | 2 +- .../applied/agent-eval-tools/README.md | 37 +++++++++++++++++++ docs/problems/applied/harness-eval/README.md | 31 ---------------- 3 files changed, 38 insertions(+), 32 deletions(-) create mode 100644 docs/problems/applied/agent-eval-tools/README.md delete mode 100644 docs/problems/applied/harness-eval/README.md diff --git a/docs/problems/applied/README.md b/docs/problems/applied/README.md index 59b0fcc43..d33e927d9 100644 --- a/docs/problems/applied/README.md +++ b/docs/problems/applied/README.md @@ -5,7 +5,7 @@ This directory contains organization-specific considerations for applying fullse ## Current consumers - **[konflux-ci](konflux-ci/)** — Kubernetes-native CI/CD platform. The original proving ground for fullsend. -- **[harness-eval](harness-eval/)** — Agent-driven evaluation tools. How fullsend's problem areas appear in tools that lint and security-scan agent configurations. +- **[agent-eval-tools](agent-eval-tools/)** — Agent-driven evaluation tools. How fullsend's problem areas appear in tools that lint and security-scan agent configurations. ## Adding a new consumer diff --git a/docs/problems/applied/agent-eval-tools/README.md b/docs/problems/applied/agent-eval-tools/README.md new file mode 100644 index 000000000..9b68cd007 --- /dev/null +++ b/docs/problems/applied/agent-eval-tools/README.md @@ -0,0 +1,37 @@ +# Agent-Driven Evaluation Tools + +How fullsend's own problem areas appear in agent-driven tools that evaluate agent configurations. + +## Context + +Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side. + +An example is [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval), an open-source linter and security scanner for AI agent configurations. It runs deterministic rules against skills, commands, agents, hooks, and MCP configs, with optional LLM-based review. + +## Technology landscape + +These tools typically operate as CLI utilities or CI integrations (GitHub Actions, pre-commit hooks). They parse agent configuration files (markdown with YAML frontmatter, JSON schemas, shell scripts) and run pattern-based or AST-based analysis without executing the agent. The evaluation happens at the configuration layer, not the runtime layer, which means they complement behavioral testing rather than replacing it. + +## Relevant problem areas + +### [Testing agents](../../testing-agents.md) + +An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others. + +### [MCP configuration drift](../../mcp-config-drift.md) + +Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. Static analysis can catch structural mismatches (e.g., a skill references an MCP server not declared in the config), but runtime drift (servers configured but never used in practice) requires observability beyond what a linter provides. + +### [Tool call risk assessment](../../tool-call-risk-assessment.md) + +Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users. + +### [Trustworthiness evidence](../../trustworthiness-evidence.md) + +For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption. + +## Unique considerations + +- **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues +- **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality +- **Multi-assistant support:** an evaluation tool aiming for broad adoption may need to support multiple AI coding-agent runtimes (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats, unlike fullsend which is built around a specific coding-agent runtime diff --git a/docs/problems/applied/harness-eval/README.md b/docs/problems/applied/harness-eval/README.md deleted file mode 100644 index 0ab826ebd..000000000 --- a/docs/problems/applied/harness-eval/README.md +++ /dev/null @@ -1,31 +0,0 @@ -# Agent-Driven Evaluation Tools - -How fullsend's own problem areas appear in agent-driven tools that evaluate agent configurations (or "who watches the watchmen"). - -## Context - -Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side. - -## Relevant problem areas - -### [Testing agents](../../testing-agents.md) - -An evaluation tool's own agent setup (skills, commands, hooks) needs the same static analysis it provides to others. This is the "who watches the watchmen" problem. The tool must dogfood its own checks, and its CI must gate on its own lint and security rules. Failure to do this means the tool's own configuration can drift into the patterns it flags for others. - -### [MCP configuration drift](../../mcp-config-drift.md) - -Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. The tool's own cross-component analysis (phantom MCP detection) directly addresses this problem for downstream users, but the tool itself must also keep its own MCP config current. - -### [Tool call risk assessment](../../tool-call-risk-assessment.md) - -Evaluation tools that run security scans must not themselves become attack vectors. A malicious skill under evaluation could contain patterns designed to influence the evaluator's behavior (anti-jailbreak patterns, evaluator-targeted prompt injection). The tool needs its own defense against adversarial inputs, which is distinct from the defenses it provides to users. - -### [Trustworthiness evidence](../../trustworthiness-evidence.md) - -For an evaluation tool, trustworthiness evidence takes a specific form: false positive rate, false negative rate, and rule accuracy over time. If the tool flags too many false positives, teams disable it. If it misses real issues, teams lose trust. Tracking these metrics is how the tool earns continued adoption. - -## Unique considerations - -- **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues -- **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality -- **Multi-tool support:** unlike fullsend (which targets a specific platform), evaluation tools must handle multiple AI assistants (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats From 10805b4debfffc9c0ba946f5f363d5c8fd361f95 Mon Sep 17 00:00:00 2001 From: Benjamin Kapner Date: Wed, 29 Jul 2026 09:00:04 +0300 Subject: [PATCH 5/6] fix(docs): address review feedback on agent-eval-tools doc - Add disambiguation between harness-eval and agent-eval-harness (ADR-0051) - Fix MCP claim to describe harness-eval as analyzing MCP configs, not integrating via MCP - Correct runtime claim to reflect fullsend's pluggable runtime.Runtime interface with Claude Code as the current production default Co-Authored-By: Claude Opus 4.6 (1M context) Signed-off-by: Benjamin Kapner --- docs/problems/applied/agent-eval-tools/README.md | 6 +++--- 1 file changed, 3 insertions(+), 3 deletions(-) diff --git a/docs/problems/applied/agent-eval-tools/README.md b/docs/problems/applied/agent-eval-tools/README.md index 9b68cd007..cb56a7275 100644 --- a/docs/problems/applied/agent-eval-tools/README.md +++ b/docs/problems/applied/agent-eval-tools/README.md @@ -6,7 +6,7 @@ How fullsend's own problem areas appear in agent-driven tools that evaluate agen Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side. -An example is [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval), an open-source linter and security scanner for AI agent configurations. It runs deterministic rules against skills, commands, agents, hooks, and MCP configs, with optional LLM-based review. +An example is [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval), an open-source linter and security scanner for AI agent configurations (not to be confused with agent-eval-harness, the dynamic test-execution framework adopted in ADR-0051; harness-eval performs static analysis and does not execute agents). It runs deterministic rules against skills, commands, agents, hooks, and MCP configs, with optional LLM-based review. ## Technology landscape @@ -20,7 +20,7 @@ An evaluation tool's own agent setup (skills, commands, hooks) needs the same st ### [MCP configuration drift](../../mcp-config-drift.md) -Evaluation tools that integrate with MCP servers (for LLM-based review) face config drift when new MCP servers are added or removed. Static analysis can catch structural mismatches (e.g., a skill references an MCP server not declared in the config), but runtime drift (servers configured but never used in practice) requires observability beyond what a linter provides. +Evaluation tools that consume `.mcp.json` or similar MCP configs as analysis targets face config drift when servers are added or removed. Static analysis can catch structural mismatches (e.g., a skill references an MCP server not declared in the config), but runtime drift (servers configured but never used in practice) requires observability beyond what a linter provides. ### [Tool call risk assessment](../../tool-call-risk-assessment.md) @@ -34,4 +34,4 @@ For an evaluation tool, trustworthiness evidence takes a specific form: false po - **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues - **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality -- **Multi-assistant support:** an evaluation tool aiming for broad adoption may need to support multiple AI coding-agent runtimes (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats, unlike fullsend which is built around a specific coding-agent runtime +- **Multi-assistant support:** an evaluation tool aiming for broad adoption may need to support multiple AI coding-agent runtimes (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats, unlike fullsend, which defaults production orgs to a single runtime (Claude Code) today despite having a pluggable `runtime.Runtime` interface (docs/runtimes.md), with OpenCode noted as another candidate backend From b9b43238beaf100b9eb792a387e02edbfacfa009 Mon Sep 17 00:00:00 2001 From: Benjamin Kapner Date: Thu, 30 Jul 2026 08:16:05 +0300 Subject: [PATCH 6/6] fix(docs): address follow-up review on agent-eval-tools doc - Rename title from "Agent-Driven" to "Agent Setup" to accurately reflect that the core mechanism is deterministic static analysis - Add harness-eval as a concrete fullsend consumer (citing .fullsend/config.yaml and bot-authored PR #8) - Replace "pre-commit hooks" with "Tekton tasks" per actual install docs - Convert plain-text ADR-0051 and runtimes.md to proper relative links - Match index entry in applied/README.md to new title Signed-off-by: Benjamin Kapner --- docs/problems/applied/README.md | 2 +- docs/problems/applied/agent-eval-tools/README.md | 12 ++++++------ 2 files changed, 7 insertions(+), 7 deletions(-) diff --git a/docs/problems/applied/README.md b/docs/problems/applied/README.md index d33e927d9..de34a162e 100644 --- a/docs/problems/applied/README.md +++ b/docs/problems/applied/README.md @@ -5,7 +5,7 @@ This directory contains organization-specific considerations for applying fullse ## Current consumers - **[konflux-ci](konflux-ci/)** — Kubernetes-native CI/CD platform. The original proving ground for fullsend. -- **[agent-eval-tools](agent-eval-tools/)** — Agent-driven evaluation tools. How fullsend's problem areas appear in tools that lint and security-scan agent configurations. +- **[agent-eval-tools](agent-eval-tools/)** — Agent setup evaluation tools. How fullsend's problem areas appear in tools that lint and security-scan agent configurations. ## Adding a new consumer diff --git a/docs/problems/applied/agent-eval-tools/README.md b/docs/problems/applied/agent-eval-tools/README.md index cb56a7275..77e6fc973 100644 --- a/docs/problems/applied/agent-eval-tools/README.md +++ b/docs/problems/applied/agent-eval-tools/README.md @@ -1,16 +1,16 @@ -# Agent-Driven Evaluation Tools +# Agent Setup Evaluation Tools -How fullsend's own problem areas appear in agent-driven tools that evaluate agent configurations. +How fullsend's own problem areas appear in tools that evaluate agent configurations. ## Context -Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) are themselves agent-driven systems. They face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side. +Agent setup evaluation tools (linters, security scanners for skills/commands/agents/hooks) face a subset of the same challenges fullsend faces, viewed from the tooling side rather than the platform side. Their core mechanism is deterministic static analysis; some optionally offer LLM-based review modes. -An example is [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval), an open-source linter and security scanner for AI agent configurations (not to be confused with agent-eval-harness, the dynamic test-execution framework adopted in ADR-0051; harness-eval performs static analysis and does not execute agents). It runs deterministic rules against skills, commands, agents, hooks, and MCP configs, with optional LLM-based review. +An example is [harness-eval](https://github.com/redhat-community-ai-tools/harness-eval), an open-source linter and security scanner for AI agent configurations (not to be confused with agent-eval-harness, the dynamic test-execution framework adopted in [ADR 0051](../../../ADRs/0051-agent-eval-harness-for-test-infrastructure.md); harness-eval performs static analysis and does not execute agents). It runs deterministic rules against skills, commands, agents, hooks, and MCP configs, with optional LLM-based review. Notably, harness-eval is itself a fullsend consumer: its repo includes a `.fullsend/config.yaml` installation config, and fullsend's own coding bot authored [PR #8](https://github.com/redhat-community-ai-tools/harness-eval/pull/8) (merged June 2026). ## Technology landscape -These tools typically operate as CLI utilities or CI integrations (GitHub Actions, pre-commit hooks). They parse agent configuration files (markdown with YAML frontmatter, JSON schemas, shell scripts) and run pattern-based or AST-based analysis without executing the agent. The evaluation happens at the configuration layer, not the runtime layer, which means they complement behavioral testing rather than replacing it. +These tools typically operate as CLI utilities or CI integrations (GitHub Actions, Tekton tasks). They parse agent configuration files (markdown with YAML frontmatter, JSON schemas, shell scripts) and run pattern-based or AST-based analysis without executing the agent. The evaluation happens at the configuration layer, not the runtime layer, which means they complement behavioral testing rather than replacing it. ## Relevant problem areas @@ -34,4 +34,4 @@ For an evaluation tool, trustworthiness evidence takes a specific form: false po - **Recursive evaluation:** the tool must be able to evaluate its own setup without circular dependency issues - **Rule accuracy feedback loop:** users who suppress findings or override verdicts generate signal about rule quality -- **Multi-assistant support:** an evaluation tool aiming for broad adoption may need to support multiple AI coding-agent runtimes (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats, unlike fullsend, which defaults production orgs to a single runtime (Claude Code) today despite having a pluggable `runtime.Runtime` interface (docs/runtimes.md), with OpenCode noted as another candidate backend +- **Multi-assistant support:** an evaluation tool aiming for broad adoption may need to support multiple AI coding-agent runtimes (Claude Code, Cursor, Copilot, Gemini, OpenCode) with different configuration formats, unlike fullsend, which defaults production orgs to a single runtime (Claude Code) today despite having a pluggable `runtime.Runtime` interface ([runtimes.md](../../../runtimes.md)), with OpenCode noted as another candidate backend