Repository navigation
M3: Agent + approval gates #3
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.
Already on GitHub? Sign in to your account
Merged
Merged
Changes from all commits
Commits
Show all changes
6 commits
Select commit
Hold shift + click to select a range
322782a
docs: M3 agent + approval gates implementation plan
mikelord007 3708250
agent: add chaos-notary TrueForge manifest
mikelord007 66712b9
agent: add setup and manual verification docs
mikelord007 bc4d004
README: document M3 agent manifest
mikelord007 5ead0bf
docs: fix M3 review findings in agent/README.md and root README.md
mikelord007 380b76a
agent: fix 4 Qodo-flagged bugs from PR #3 review
mikelord007 File filter
Filter by extension
Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
There are no files selected for viewing
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,102 @@ | ||
| # Chaos Notary agent | ||
|
|
||
| The TrueForge agent manifest for Chaos Notary lives at | ||
| [`chaos-notary.json`](./chaos-notary.json). This is a static config file — | ||
| TrueForge is a separate, external agent harness this repo doesn't run or | ||
| control. Getting the agent working requires one manual setup step this repo | ||
| can't automate. | ||
|
|
||
| ## Prerequisites | ||
|
|
||
| - The M1+M2 stack running: `docker compose up -d --build` from the repo | ||
| root (see the root [`README.md`](../README.md) for details). | ||
| - A TrueForge instance you can reach: locally via | ||
| `npx @truefoundry/trueforge@latest` (SQLite-backed, single command, per | ||
| TrueForge's published quickstart — check TrueForge's current docs if | ||
| that's changed), or a shared deployment. It needs network access to | ||
| `mcp-server`. `docker-compose.yml` publishes that service as | ||
| `127.0.0.1:3100:3100` — loopback-only, on purpose, because `/mcp` has no | ||
| authentication and can pause/stop/kill the whole stack (see the comment | ||
| above that port line in `docker-compose.yml`, and the root | ||
| [`README.md`](../README.md)'s Qodo evidence table for PR #2, where this | ||
| was one of 7 real bugs found and fixed). If TrueForge runs as a process | ||
| on the same host as the Docker stack, it can reach the server at | ||
| `http://localhost:3100`. If it can't — a remote or shared TrueForge | ||
| deployment — put TrueForge on this project's Docker network instead (add | ||
| it to `docker-compose.yml`'s network, or run it with `--network` pointing | ||
| at this stack's compose network) and reach the server via its compose | ||
| service DNS name, `http://chaos-mcp-server:3100`, rather than widening | ||
| the published host port. | ||
|
|
||
| **Do not widen the `127.0.0.1:3100:3100` binding in `docker-compose.yml` | ||
| to expose `/mcp` more broadly — it has no authentication.** | ||
|
|
||
| ## Setup | ||
|
|
||
| 1. **Register `mcp-server` as a Connector.** In TrueForge's UI, go to | ||
| Settings → Connectors, add a new MCP connector pointing at | ||
| `http://localhost:3100/mcp` if TrueForge runs on the same host as the | ||
| Docker stack, or `http://chaos-mcp-server:3100/mcp` if TrueForge is | ||
| instead running on this project's Docker network (see Prerequisites | ||
| above — do not point it at a widened `3100:3100` binding), and name it | ||
| `mcp-server` — the manifest's `mcp_servers[0].name` must match whatever | ||
| you name it here. | ||
|
|
||
| The manifest's `mcp_servers[0]` entry sets `"type": "truefoundry-mcp-registry"` | ||
| alongside `"name": "mcp-server"` — per TrueForge's | ||
| [agent manifest reference](https://www.truefoundry.com/docs/agent-platform/agent-harness/sdk/agent-manifest-reference), | ||
| `mcp_servers[]` is a discriminated union and `type` is required even for | ||
| a Connector-registered server, not just a platform-catalog one; `name` | ||
| is "the name of the MCP server as registered in the platform," which is | ||
| exactly what step 1 above sets when you name the connector `mcp-server`. | ||
|
|
||
| Even so, a clean manifest load is not proof the approval gate is | ||
| wired — if this schema detail is still off in some way this repo | ||
| couldn't check, TrueForge may silently ignore an unrecognized field | ||
| rather than erroring. Treat the manifest as ungated until verification | ||
| step 4 below (in "Manually verifying the approval gate") actually | ||
| confirms the approval prompt fires before the tool executes. | ||
|
|
||
| 2. **Load the manifest.** Via TrueForge's Agent Playground (paste/import | ||
| `chaos-notary.json`) or its SDK (`type: "truefoundry-agent"` manifests | ||
| can be defined in code and saved programmatically — see TrueForge's | ||
| docs). Save it to the Agent Registry. | ||
|
|
||
| **Unverified:** the exact model catalog string | ||
| (`anthropic/claude-sonnet-5` in the manifest) may not match what your | ||
| TrueForge instance's model catalog actually calls it. If the agent | ||
| fails to load or run because of the `model.name` field, check your | ||
| instance's available models and update the manifest. | ||
|
|
||
| ## Manually verifying the approval gate | ||
|
|
||
| This is the actual proof M3 works — there's no automated test for it (see | ||
| the design spec's Testing section for why). Run through this once against | ||
| a real TrueForge instance before considering M3 done: | ||
|
|
||
| 1. Start a conversation with the loaded `chaos-notary` agent. | ||
| 2. Ask it: "Pause chaos-pg-replica for 30 seconds and tell me what you | ||
| expect to happen." | ||
| 3. Confirm the agent calls `list_targets` first, then states its intent | ||
| (which container, how long, what it expects) before proposing | ||
| `pause_container` — this order matches the manifest's own "How to run | ||
| an experiment" instructions (list_targets, then state intent, then | ||
| propose the call). | ||
| 4. **Confirm the approval prompt fires before the tool call executes** — | ||
| this is the core thing being verified. If `pause_container` runs | ||
| without a prompt, the manifest's `require_approval_for_tools` isn't | ||
| wired correctly; check the Connector registration and the manifest's | ||
| `mcp_servers[0].name` match. | ||
| 5. Approve it. Confirm the agent points you at | ||
| `http://localhost:3001/d/chaos-notary` to watch the effect, rather than | ||
| claiming to have checked it itself. | ||
| 6. After 30+ seconds, ask the agent to confirm the container recovered. | ||
| Confirm it calls `list_targets` again and reports the container's | ||
| Docker status as running with no active fault — not "healthy," since | ||
| `list_targets` has no application-health signal to report (only Docker | ||
| status and fault-registry state). Check the Grafana dashboard yourself | ||
| for actual workload recovery. | ||
| 7. Repeat step 2 asking it to target a container NOT on the allowlist | ||
| (e.g. "pause chaos-mcp-server") — confirm the tool call is rejected and | ||
| the agent reports that clearly rather than retrying or working around | ||
| it. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
| Original file line number | Diff line number | Diff line change |
|---|---|---|
| @@ -0,0 +1,31 @@ | ||
| { | ||
| "type": "truefoundry-agent", | ||
| "name": "chaos-notary", | ||
| "description": "Runs bounded, reversible chaos experiments against the Chaos Notary target stack, gated by human approval before any destructive action.", | ||
| "model": { | ||
| "name": "anthropic/claude-sonnet-5", | ||
| "params": { | ||
| "max_tokens": 4096, | ||
| "temperature": 0.3 | ||
| } | ||
| }, | ||
| "instructions": "You are the Chaos Notary agent: you run bounded, reversible resilience\nexperiments against a live demo stack (the \"Chaos Notary target stack\") to\nreveal how it fails, and you always stop for explicit human approval\nbefore taking any action that could disrupt the system.\n\n## What you can do\n\nYou have access to 7 tools on the `mcp-server` connector, restricted to a\nfixed 5-container allowlist (chaos-pg-primary, chaos-pg-replica,\nchaos-checkout-api, chaos-prometheus, chaos-grafana):\n\n- `list_targets` — read-only. Reports each allowlisted container's current\n Docker state and whether a fault is currently active on it — not\n application health, only Docker/fault state. Always safe to call. Call\n it first before proposing any experiment, and again after one.\n- `pause_container`, `stop_container`, `kill_container` — freeze, stop, or\n kill a container's process for a bounded duration_seconds (5-300s),\n after which it automatically recovers on its own even if nothing calls\n you back.\n- `inject_latency`, `inject_packet_loss` — add network latency or packet\n loss to a container's traffic for the same bounded duration.\n- `clear_fault` — manually and immediately revert whatever fault is\n currently active on a container. Safe to call any time; it only ever\n restores safety, so it does not require approval.\n\nEvery fault-injection tool above (pause_container, stop_container,\nkill_container, inject_latency, inject_packet_loss) is destructive and\nrequires human approval before you're allowed to invoke it — you will be\nprompted automatically. Do not try to work around this or suggest the\nhuman disable it.\n\n## How to run an experiment\n\n1. Call `list_targets` first to confirm the target isn't already faulted.\n2. State what you're about to do and why, in plain language, before\n proposing the tool call: which container, which fault type, how long,\n and what you expect to happen (e.g. \"pausing chaos-pg-replica for 60s\n should make GET /products start returning 503s, since it has no\n fallback to the primary\").\n3. Propose the fault-injection tool call. Wait for approval.\n4. After the call returns, tell the human where to watch for the effect:\n the Grafana dashboard at http://localhost:3001/d/chaos-notary\n (provisioned by the M1 target stack) shows the live error-rate panel.\n You do not have a tool to query Prometheus directly yet — reading\n metrics automatically is planned for a later milestone and isn't built.\n Don't claim to have observed impact you can't actually see.\n5. Once the fault's duration_seconds has elapsed, call `list_targets`\n again to confirm the container's Docker status is running with no\n active fault. list_targets has no application-health signal — for\n confirmation the workload itself recovered, point the human at the\n dashboard. If Docker/fault state isn't clean, say so plainly and call\n `clear_fault` — that call never needs approval.\n\n## What you must never do\n\n- Never call a fault-injection tool without first stating your intent and\n getting the approval prompt to fire.\n- Never target a container outside the fixed allowlist — the tools will\n reject it, but don't try anyway or suggest working around it.\n- Never claim you've verified an experiment's outcome by a means you don't\n actually have (e.g. don't claim to have \"checked the metrics\" or that a\n container is \"healthy\" — list_targets only tells you Docker/fault state;\n report exactly that, and point the human at the dashboard for anything\n beyond it).\n", | ||
| "mcp_servers": [ | ||
| { | ||
| "type": "truefoundry-mcp-registry", | ||
| "name": "mcp-server", | ||
| "enable_tools": ["@all"], | ||
| "require_approval_for_tools": [ | ||
| "pause_container", | ||
| "stop_container", | ||
| "kill_container", | ||
| "inject_latency", | ||
| "inject_packet_loss" | ||
| ] | ||
| } | ||
| ], | ||
| "config": { | ||
| "iteration_limit": 15, | ||
| "timeout_seconds": 600 | ||
| } | ||
| } | ||
Oops, something went wrong.
Add this suggestion to a batch that can be applied as a single commit.
This suggestion is invalid because no changes were made to the code.
Suggestions cannot be applied while the pull request is closed.
Suggestions cannot be applied while viewing a subset of changes.
Only one suggestion per line can be applied in a batch.
Add this suggestion to a batch that can be applied as a single commit.
Applying suggestions on deleted lines is not supported.
You must change the existing code in this line in order to create a valid suggestion.
Outdated suggestions cannot be applied.
This suggestion has been applied or marked resolved.
Suggestions cannot be applied from pending reviews.
Suggestions cannot be applied on multi-line comments.
Suggestions cannot be applied while the pull request is queued to merge.
Suggestion cannot be applied right now. Please check back later.
Uh oh!
There was an error while loading. Please reload this page.