Skip to content

M3: Agent + approval gates - #3

Merged
mikelord007 merged 6 commits into
mainfrom
feat/m3-agent-approval-gates
Aug 27, 2026
Merged

mikelord007 merged 6 commits into
mainfrom
feat/m3-agent-approval-gates

Conversation

@mikelord007

Copy link
Copy Markdown
Owner

Summary

Implements M3: a TrueForge agent manifest that turns M2's 7 MCP chaos tools into an actual agent, with human approval structurally required before any destructive action.

  • agent/chaos-notary.json — the agent manifest. System prompt describes the agent's actual capabilities (no fabricated blast-radius prediction or automated metrics access — those are M4/M5, not built). require_approval_for_tools gates the 5 destructive tools (pause_container, stop_container, kill_container, inject_latency, inject_packet_loss) by exact name; list_targets (read-only) and clear_fault (a corrective/safety-restoring action) are ungated.
  • agent/README.md — setup (registering the mcp-server MCP connector, loading the manifest) and a manual verification walkthrough that specifically checks the approval prompt fires before a destructive tool executes.
  • Root README.md updated: M3 marked done, with wording that doesn't overclaim what this session actually verified (TrueForge is external — verification is manual, by you).

Design doc: docs/superpowers/specs/2026-08-27-m3-agent-approval-gates-design.md
Implementation plan: docs/superpowers/plans/2026-08-27-m3-agent-approval-gates.md

Built via subagent-driven development: fresh implementer + independent reviewer per task, then a final whole-branch review (Opus) that cross-checked every factual claim in the system prompt against M2's actual source (tool names, allowlist, duration bounds, endpoint paths — all matched exactly) and caught 3 real documentation issues, since fixed:

  1. Setup guidance that could've led a reader to re-expose the unauthenticated /mcp endpoint PR M2: MCP server exposing chaos tools #2's Qodo review already locked down to loopback-only.
  2. An "Unverified" caveat that only warned about a loud failure (a schema error) and missed the dangerous silent one — TrueForge silently ignoring an unrecognized field, leaving the approval gate unwired with no error.
  3. The root README's "done" bullet overclaiming a verified runtime property this session couldn't actually confirm.

Known gaps — this repo has no way to verify these

TrueForge is a real, very recently launched project (researched via web search/fetch, not prior training knowledge). Two manifest details couldn't be confirmed against a live instance and are explicitly flagged as "Unverified" in agent/README.md, not asserted with false confidence:

  • The exact mcp_servers[0] field for referencing a Connector-registered custom MCP server (vs. TrueFoundry's own catalog).
  • The exact model catalog string (anthropic/claude-sonnet-5 in the manifest).
  • The approval gate itself has never fired against a real TrueForge instance — agent/README.md's manual verification walkthrough needs to actually be run.

Test plan

  • node -e "require('./agent/chaos-notary.json')" — valid JSON
  • services/mcp-server test suite unaffected — 56/56 passing
  • Register the mcp-server connector and load the manifest into a real TrueForge instance (needs Docker + TrueForge, unavailable in this sandbox — see agent/README.md)
  • Run through agent/README.md's manual verification walkthrough, confirming the approval prompt actually fires before a destructive tool executes

- Make the /mcp loopback binding and its no-auth rationale explicit in
  agent/README.md's Prerequisites and Setup step 1, replacing vague
  "adjust the host" guidance with the actual safe path (put TrueForge
  on this project's Docker network, reach the server via
  http://chaos-mcp-server:3100) and an explicit warning against
  widening 127.0.0.1:3100:3100.
- Extend the connector-reference-field caveat to flag the silent-failure
  risk: a clean manifest load is not proof the approval gate is wired
  if TrueForge ignores an unrecognized field instead of erroring.
- Note that the TrueForge quickstart command follows TrueForge's own
  published docs, consistent with the caveat discipline used elsewhere.
- Soften the root README's M3 status bullet so it no longer asserts the
  approval gate as a confirmed runtime property; matches
  agent/README.md's own framing that the manual walkthrough is the
  actual proof.
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Add TrueForge chaos agent with human approval gates

✨ Enhancement 📝 Documentation ⚙️ Configuration changes 🕐 20-40 Minutes

Grey Divider

AI Description

• Adds a TrueForge agent manifest connecting all seven MCP chaos tools.
• Gates five destructive tools while preserving ungated inspection and recovery actions.
• Documents secure connector setup, schema caveats, and manual approval verification.
Diagram

sequenceDiagram
  actor Human
  participant TF as TrueForge
  participant Agent as Chaos Agent
  participant MCP as MCP Server
  participant Stack as Target Stack
  participant Grafana
  Human->>TF: Request experiment
  TF->>Agent: Run request
  Agent->>MCP: list_targets
  MCP->>Stack: Inspect target
  Stack-->>Agent: Current status
  Agent-->>TF: Propose destructive tool
  TF-->>Human: Request approval
  Human->>TF: Approve action
  TF->>Agent: Release tool call
  Agent->>MCP: Inject bounded fault
  MCP->>Stack: Apply and auto-revert
  Human->>Grafana: Observe impact
  Agent->>MCP: Confirm recovery
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Gate by MCP destructive annotations
  • ➕ Automatically covers newly added destructive tools
  • ➕ Centralizes safety metadata with each MCP tool
  • ➖ Requires modifying the already-delivered M2 tool surface
  • ➖ Incorrect or missing annotations could silently weaken approval coverage
  • ➖ Depends on TrueForge tag-selector compatibility
2. Add an in-repository approval proxy
  • ➕ Allows automated integration testing under repository control
  • ➕ Avoids dependence on uncertain external manifest fields
  • ➖ Duplicates TrueForge approval functionality
  • ➖ Introduces a new security-sensitive runtime service
  • ➖ Substantially expands M3 scope and maintenance

Recommendation: Keep the explicit destructive-tool allowlist for M3 because it is auditable, narrowly scoped, and independent of unimplemented MCP annotations. After validating TrueForge behavior against a live instance, consider annotation-based selection only if schema support and regression tests can guarantee that new destructive tools remain gated.

Files changed (4) +349 / -2

Enhancement (1) +30 / -0
chaos-notary.jsonDefine the approval-gated Chaos Notary agent +30/-0

Define the approval-gated Chaos Notary agent

• Adds the TrueForge agent manifest, bounded execution settings, and capability-accurate operating instructions. Explicitly requires approval for five destructive tools while leaving target inspection and fault clearing ungated.

agent/chaos-notary.json

Documentation (3) +319 / -2
README.mdMark M3 agent milestone complete +2/-2

Mark M3 agent milestone complete

• Updates project status to describe the new TrueForge manifest and directs users to the agent setup and manual verification guide. It distinguishes declared approval configuration from automated verification.

README.md

README.mdDocument secure agent setup and approval verification +96/-0

Document secure agent setup and approval verification

• Explains how to register the loopback-only MCP endpoint with TrueForge without exposing its unauthenticated interface. Documents unverified connector and model schema assumptions, silent-failure risk, and a manual walkthrough proving approval occurs before execution.

agent/README.md

2026-08-27-m3-agent-approval-gates.mdPlan the M3 manifest and documentation rollout +221/-0

Plan the M3 manifest and documentation rollout

• Records the implementation tasks, safety constraints, manifest validation commands, documentation requirements, and root status update. The plan explicitly prevents capability overclaims and preserves manual verification for the external TrueForge integration.

docs/superpowers/plans/2026-08-27-m3-agent-approval-gates.md

@qodo-code-review

qodo-code-review Bot commented Aug 27, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. MCP server type omitted ✓ Resolved 🐞 Bug ≡ Correctness
Description
The mcp_servers entry has only name, but TrueForge defines this object as a discriminated union
and requires type: "truefoundry-mcp-registry" for a registered connector. The manifest therefore
cannot reliably bind the registered MCP server, leaving the agent unusable or without the intended
approval-gated tools.
Code

agent/chaos-notary.json[R14-15]

+    {
+      "name": "mcp-server",
Evidence
The checked-in entry contains no type, while TrueForge's manifest reference says every
mcp_servers[] entry is a discriminated union and marks type: "truefoundry-mcp-registry" as
required for a platform-registered server.

agent/chaos-notary.json[13-24]
🌐 The mcp_servers[] schema is a discriminated union; its registered-server variant requires type: "truefoundry-mcp-registry" and a registered server name.

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The TrueForge MCP server entry is missing its required source discriminator, so the registered connector cannot be resolved according to the current manifest schema.

## Issue Context
For a server registered in TrueForge's MCP registry, add `"type": "truefoundry-mcp-registry"` alongside the connector name. Update the setup caveat so it no longer presents this known schema field as unverified.

## Fix Focus Areas
- agent/chaos-notary.json[13-16]
- agent/README.md[42-56]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools



Remediation recommended

2. Health status is invented ✓ Resolved 🐞 Bug ≡ Correctness
Description
The verification requires the agent to report the recovered container as “healthy,” but
list_targets exposes only Docker status and active-fault state, not container or application
health. This encourages the agent and operator to treat a running, unfaulted container as recovered
even when the workload remains unhealthy.
Code

agent/README.md[R90-92]

+6. After 30+ seconds, ask the agent to confirm the container recovered.
+   Confirm it calls `list_targets` again and reports the container as
+   healthy with no active fault.
Evidence
handleListTargets returns only container, dockerStatus, and fault; Docker inspection does
not feed any healthcheck result into that response. The new instructions nevertheless require a
“healthy” conclusion from this tool.

agent/README.md[90-92]
agent/chaos-notary.json[12-12]
services/mcp-server/src/tools.ts[33-49]
services/mcp-server/src/dockerClient.ts[10-19]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The manual test and agent prompt claim `list_targets` can establish health, although its response has no health signal.

## Issue Context
Restrict recovery claims to the values actually exposed: Docker status and absence of an active registered fault. Direct the human to the dashboard or another real health signal for application recovery.

## Fix Focus Areas
- agent/README.md[90-92]
- agent/chaos-notary.json[12-12]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Verification order contradicts manifest ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
The walkthrough says a passing agent calls list_targets before stating intent, while the
manifest's numbered procedure tells it to state intent first and call list_targets second. An
agent that follows the manifest exactly can therefore fail the documented acceptance test.
Code

agent/README.md[R79-81]

+3. Confirm the agent calls `list_targets` first, then states its intent
+   (which container, how long, what it expects) before proposing
+   `pause_container`.
Evidence
The README explicitly expects list_targets first and intent afterward, but the manifest's “How to
run an experiment” lists intent as step 1 and list_targets as step 2.

agent/README.md[79-81]
agent/chaos-notary.json[12-12]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The manual acceptance sequence and the manifest prescribe opposite ordering for the intent statement and `list_targets` call.

## Issue Context
Choose one sequence and make both the system instructions and verification walkthrough require it consistently.

## Fix Focus Areas
- agent/README.md[79-81]
- agent/chaos-notary.json[12-12]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


4. M3 status overstates verification ✓ Resolved 🐞 Bug ⚙ Maintainability
Description
The root status says the gate was verified manually against a live TrueForge instance, while the new
setup document says the connector field was not confirmed live and treats the manifest as ungated
until a future verification step succeeds. This marks M3 done based on a runtime property the PR's
own documentation says remains unverified.
Code

README.md[16]

+- **M3 — Agent + approval gates**: done. TrueForge agent manifest declares human approval for every destructive chaos tool; the gate itself is verified manually against a live TrueForge instance, not by an automated script. See [`agent/README.md`](agent/README.md) for setup and manual verification.
Evidence
The root README asserts live manual verification; the agent README says the connector-reference
field was not confirmed against a live instance, warns to treat the manifest as ungated until step
4, and calls that walkthrough the actual proof required before considering M3 done.

README.md[16-16]
agent/README.md[45-57]
agent/README.md[70-74]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The root status claims completed live verification that the agent documentation explicitly says has not yet occurred.

## Issue Context
Describe the manifest as implemented, but keep live approval-gate verification pending until the walkthrough has actually been run successfully.

## Fix Focus Areas
- README.md[16-16]
- agent/README.md[45-57]
- agent/README.md[70-74]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Web pages:
  +7 more
Review mode: ⚖️ Balanced: This is a localized manifest and documentation change, but it affects agent tool permissions and human approval behavior; the unverified external schema and security-sensitive gating warrant a careful single-pass review.

Grey Divider

Tip of the day
💡 Did you know, you can ask Qodo to dismiss a finding you disagree with, with your reason on record

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread agent/chaos-notary.json Resolved
Comment thread agent/README.md Resolved
Comment thread agent/README.md Resolved
Comment thread README.md Resolved
- Add required mcp_servers[0].type ("truefoundry-mcp-registry") to the
  manifest: TrueForge's manifest schema is a discriminated union and
  requires this field even for a Connector-registered server, not just
  a platform-catalog one — confirmed against TrueForge's own agent
  manifest reference docs, resolving one of the two items previously
  flagged "Unverified" in agent/README.md.
- Reorder "How to run an experiment" in the system prompt so
  list_targets is called before stating intent, not after — the
  original numbering had the agent describe what it expects to happen
  before checking whether the target was already faulted, and
  contradicted agent/README.md's own verification walkthrough, which
  already expected list_targets first.
- Stop claiming list_targets can report a container as "healthy" —
  it only returns Docker status and fault-registry state, no
  application-health signal. Both the system prompt and the manual
  verification walkthrough now say "Docker status running, no active
  fault" and point at the dashboard for actual workload health.
- Fix the root README's M3 bullet, which read as though the approval
  gate had already been verified against a live TrueForge instance.
  It hasn't — that verification is manual and up to the user; the
  bullet now says so plainly instead of using ambiguous present tense.
@mikelord007

Copy link
Copy Markdown
Owner Author

@qodo Addressed all 4 findings from the review in 380b76a:

  1. MCP server type omitted (High) — added the required mcp_servers[0].type: "truefoundry-mcp-registry". I verified this against TrueForge's actual agent manifest reference docs before applying it — type is required by the discriminated-union schema even for a Connector-registered server, not just a platform-catalog one. This resolves one of the two items previously flagged "Unverified" in agent/README.md.
  2. Health status is invented (Medium) — list_targets only returns Docker status and fault-registry state, never application health. Both the system prompt and the manual verification walkthrough now say "Docker status running, no active fault" instead of "healthy," and point at the Grafana dashboard for actual workload confirmation.
  3. Verification order contradicts manifest (Medium) — reordered the system prompt's "How to run an experiment" so list_targets is called before stating intent (matching agent/README.md's walkthrough, which already expected that order) rather than after.
  4. M3 status overstates verification (Medium) — reworded the root README's M3 bullet so it doesn't read as though the approval gate has already been verified against a live TrueForge instance. It hasn't — that's explicitly a manual step for the user, and the bullet now says so plainly.

Ready for a follow-up review.

@qodo-code-review

Copy link
Copy Markdown

Addressed all 4 findings from the review in 380b76a:

  1. MCP server type omitted (High) — added the required mcp_servers[0].type: "truefoundry-mcp-registry". I verified this against TrueForge's actual agent manifest reference docs before applying it — type is required by the discriminated-union schema even for a Connector-registered server, not just a platform-catalog one. This resolves one of the two items previously flagged "Unverified" in agent/README.md.
  2. Health status is invented (Medium) — list_targets only returns Docker status and fault-registry state, never application health. Both the system prompt and the manual verification walkthrough now say "Docker status running, no active fault" instead of "healthy," and point at the Grafana dashboard for actual workload confirmation.
  3. Verification order contradicts manifest (Medium) — reordered the system prompt's "How to run an experiment" so list_targets is called before stating intent (matching agent/README.md's walkthrough, which already expected that order) rather than after.
  4. M3 status overstates verification (Medium) — reworded the root README's M3 bullet so it doesn't read as though the approval gate has already been verified against a live TrueForge instance. It hasn't — that's explicitly a manual step for the user, and the bullet now says so plainly.

Ready for a follow-up review.

The changes in 380b76a address all four review findings:

  • finding 1: mcp_servers[0].type is now set to "truefoundry-mcp-registry" in agent/chaos-notary.json:15.
  • finding 2: The manifest and verification guide now distinguish Docker/fault state from application health (agent/chaos-notary.json:12, agent/README.md:93-98).
  • finding 3: The manifest now calls list_targets before stating intent (agent/chaos-notary.json:12), matching the walkthrough at agent/README.md:80-84.
  • finding 4: The root status explicitly says live approval-gate verification has not yet been performed (README.md:16) and links to the manual walkthrough.

The remaining validation is intentionally manual: load the manifest in a live TrueForge instance and confirm the approval prompt appears before a destructive tool executes. No further code-review issue is apparent from the updated diff.

@mikelord007
mikelord007 merged commit 4c3ca3e into main Aug 27, 2026
1 check passed
mikelord007 added a commit that referenced this pull request Aug 27, 2026
Adds the per-PR row for M3's Qodo review loop: 4 bugs found (1 High, 3
Medium), all fixed in 380b76a, follow-up review confirmed 0 remaining
findings before merge.
@mikelord007 mikelord007 mentioned this pull request Aug 28, 2026
4 of 5 tasks
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant