Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
22 commits
Select commit Hold shift + click to select a range
6581534
metrics-watcher: scaffold package
mikelord007 Aug 28, 2026
6461e49
metrics-watcher: add Prometheus query client
mikelord007 Aug 28, 2026
7b01917
metrics-watcher: add severity classification and verdict logic
mikelord007 Aug 28, 2026
7db48a5
metrics-watcher: add observe_impact MCP tool
mikelord007 Aug 28, 2026
30af1e5
metrics-watcher: add server entrypoint
mikelord007 Aug 28, 2026
e63ef2b
metrics-watcher: add Dockerfile
mikelord007 Aug 28, 2026
14885e9
docker-compose: add metrics-watcher service, isolate on observability…
mikelord007 Aug 28, 2026
3831d97
add M5 acceptance test
mikelord007 Aug 28, 2026
6498f85
fix: mark scripts/verify-m5.sh executable (100755)
mikelord007 Aug 28, 2026
02d8f3e
agent: wire in the metrics-watcher connector
mikelord007 Aug 28, 2026
4736a98
README: document M5 metrics-watcher
mikelord007 Aug 28, 2026
20710a8
README: fix self-contradictory verify-m5.sh description
mikelord007 Aug 28, 2026
c3b377c
fix: filter observe_impact to route-shaped targets and buffer its win…
mikelord007 Aug 28, 2026
ad59eb1
fix: correct latency aggregation, add fetch timeout, guard non-finite…
mikelord007 Aug 28, 2026
0aee567
test: make observe_impact's fake fetcher route-aware (I5)
mikelord007 Aug 28, 2026
c7d9039
docs: correct M5 status caveat, network-isolation claim, and opening …
mikelord007 Aug 28, 2026
90e5f6b
fix: ground severity classification in latency, api-health, and time-…
mikelord007 Aug 28, 2026
14a8b4f
fix: wire fault_ended_at time-anchoring and apiWasDown through observ…
mikelord007 Aug 28, 2026
0829bc5
test: cover latency-hard-threshold, offset anchoring, and apiWasDown …
mikelord007 Aug 28, 2026
2c9d312
test: add short-fault and checkout-api-outage acceptance cases to ver…
mikelord007 Aug 28, 2026
a00aa58
Merge branch 'main' into feat/m5-metrics-watcher
mikelord007 Aug 28, 2026
a5f4eb7
fix: create a fresh MCP transport+server per request in metrics-watcher
mikelord007 Aug 28, 2026
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
48 changes: 37 additions & 11 deletions README.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,9 @@
# Chaos Notary

A chaos-engineering agent that runs destructive resilience experiments
against a live system, predicts blast radius before acting, observes impact
in real time, and stops for human approval before any irreversible action.
against a live system, predicts blast radius before acting, checks real
impact against its own prediction after the fact, and stops for human
approval before any irreversible action.

The agent runs on [TrueForge](https://truefoundry.com) (an existing agent
harness). This repo builds everything around it: the target stack to break,
Expand All @@ -15,7 +16,9 @@ sandbox for blast-radius computation.
- **M2 — MCP server**: done. Exposes chaos tools (container pause/stop/kill, network latency/loss injection) over Streamable HTTP, wrapping Pumba, with bounded duration per fault and guaranteed auto-revert.
- **M3 — Agent + approval gates**: done. TrueForge agent manifest declares human approval for every destructive chaos tool; whether the gate actually fires is confirmed by manually running it against a live TrueForge instance, a check not yet performed in this repo — see [`agent/README.md`](agent/README.md) for setup and that verification walkthrough.
- **M4 — Blast-radius sandbox**: done. Sealed read-only service for predicting chaos experiment blast radius, wired as a second MCP connector in the agent manifest, with zero Docker access to ensure it cannot affect the live stack.
- M5 (metrics-watcher subagent), M6 (hardening) are not yet built.
- **M5 — Metrics-watcher**: done. Reports what Prometheus actually recorded after a chaos fault reverts and compares it against M4's prediction, wired as a third MCP connector in the agent manifest, closing the predict-then-observe loop. The unit test suite passes and the service typechecks cleanly, but the Docker image build, the docker-compose network wiring, and `scripts/verify-m5.sh`'s live run are unverified against a real Docker daemon — a check not yet performed in this repo (Docker was unavailable throughout this milestone's implementation environment).

M6 (hardening) is not yet built.

## M1 — Target stack

Expand Down Expand Up @@ -86,6 +89,17 @@ non-allowlisted container.
bash scripts/verify-m4.sh
```

`scripts/verify-m5.sh` brings up the full stack and exercises the metrics-watcher
service, checking that baseline observed severity is "none," a real pause-triggered
fault produces "hard" observed severity with a "matched" verdict, invalid input
(empty affected_routes) is rejected, and verifying metrics-watcher's network isolation
from mcp-server — placed on its own `observability-net` to prevent reaching
mcp-server, the same topology-isolation guard M4 established via `sandbox-net`.

```bash
bash scripts/verify-m5.sh
```

### Services

| Service | Container | Port | Role |
Expand All @@ -98,6 +112,7 @@ bash scripts/verify-m4.sh
| `loadgen` | `chaos-loadgen` | — | Fire-and-forget traffic generator, ~10rps, 70% reads / 30% writes |
| `mcp-server` | `chaos-mcp-server` | 3100 | MCP server exposing chaos tools: pause/stop/kill containers, inject network latency/packet loss, with bounded duration and guaranteed auto-revert |
| `blast-radius-sandbox` | `chaos-blast-radius-sandbox` | 3200 | Read-only prediction service: computes expected blast radius of chaos experiments using static topology model, no Docker access |
| `metrics-watcher` | `chaos-metrics-watcher` | 3300 | Observes and reports actual impact severity from live Prometheus data after chaos faults, wired as third MCP connector alongside MCP server and blast-radius-sandbox |

All credentials in `docker-compose.yml` are dev-only placeholders
(`chaos_dev_only_not_a_secret`, `chaos_dev_replica_not_a_secret`) — obviously
Expand All @@ -109,14 +124,25 @@ self-contained non-production demo stack, not an oversight. The `/mcp`
endpoint itself has no authentication — any client that can reach it can
pause, stop, or kill the allowlisted containers — so the published port is
bound to `127.0.0.1` only (see `docker-compose.yml`), not exposed to the
wider network. `blast-radius-sandbox` is placed on its own dedicated
Compose network (`sandbox-net`), separate from the implicit default network
every other service shares — it makes zero outbound calls and depends on
nothing, so this costs it no reachability it actually needs, while ensuring
it cannot reach `mcp-server` (or anything else) by Docker's internal
service-name DNS even if a compromised or buggy dependency inside it tried
to. That keeps the "sealed sandbox" claim true at the network layer, not
just at the application layer (no Docker socket, read-only computation).
wider network. Three services now have deliberately restricted network
membership, each isolated on its own dedicated Compose network rather than
the implicit default network: `mcp-server` itself has none of these
restrictions — it needs full default-network access to reach the Docker
socket and the containers it manages — but `blast-radius-sandbox` is placed
on its own network (`sandbox-net`), and `metrics-watcher` is placed on its
own network (`observability-net`, shared only with `prometheus`, which is
dual-homed onto both `default` and `observability-net` so it can still be
scraped from and reached by `metrics-watcher`). `blast-radius-sandbox`
makes zero outbound calls and depends on nothing, so `sandbox-net`
isolation costs it no reachability it actually needs; `metrics-watcher`'s
only legitimate outbound call is to `prometheus`, so `observability-net`
isolation costs it nothing either. Both arrangements ensure neither service
can reach `mcp-server` (or anything else outside its own dedicated network)
by Docker's internal service-name DNS even if a compromised or buggy
dependency inside either tried to. That keeps the "sealed sandbox" claim
true at the network layer for `blast-radius-sandbox`, and the equivalent
claim true for `metrics-watcher`, not just at the application layer (no
Docker socket, read-only computation).

## M2 — MCP server tool surface

Expand Down
61 changes: 59 additions & 2 deletions agent/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -45,6 +45,20 @@ can't automate.

**Do not widen the `127.0.0.1:3200:3200` binding either.**

- The same TrueForge instance also needs network access to the M5
`metrics-watcher` service, for the same reachability reasons —
`docker-compose.yml` publishes it as `127.0.0.1:3300:3300`, loopback-only.
Its `/mcp` endpoint has no authentication either; the stakes are lower
since `observe_impact` is a read-only Prometheus query with no Docker
access, but the same reachability pattern applies. If TrueForge runs as
a process on the same host as the Docker stack, it can reach the server
at `http://localhost:3300`. If it can't, put TrueForge on this project's
Docker network (as above) and reach it via its compose service DNS name,
`http://chaos-metrics-watcher:3300`, rather than widening the published
host port.

**Do not widen the `127.0.0.1:3300:3300` binding either.**

## Setup

1. **Register `mcp-server` as a Connector.** In TrueForge's UI, go to
Expand Down Expand Up @@ -82,7 +96,18 @@ can't automate.
`require_approval_for_tools`: `predict_blast_radius` is read-only
computation with no Docker access, so nothing on it needs gating.

3. **Load the manifest.** Via TrueForge's Agent Playground (paste/import
3. **Register `metrics-watcher` as a Connector.** Same UI, add a third MCP
connector pointing at `http://localhost:3300/mcp` if TrueForge runs on
the same host as the Docker stack, or `http://chaos-metrics-watcher:3300/mcp`
if TrueForge is instead running on this project's Docker network (see
Prerequisites above — do not point it at a widened `3300:3300`
binding), and name it `metrics-watcher` — the manifest's
`mcp_servers[2].name` must match whatever you name it here. Like
`blast-radius-sandbox`, this entry has no `require_approval_for_tools`:
`observe_impact` is a read-only Prometheus query with no Docker access,
so nothing on it needs gating.

4. **Load the manifest.** Via TrueForge's Agent Playground (paste/import
`chaos-notary.json`) or its SDK (`type: "truefoundry-agent"` manifests
can be defined in code and saved programmatically — see TrueForge's
docs). Save it to the Agent Registry.
Expand Down Expand Up @@ -131,7 +156,39 @@ a real TrueForge instance before considering M3 done:
`list_targets` has no application-health signal to report (only Docker
status and fault-registry state). Check the Grafana dashboard yourself
for actual workload recovery.
8. Repeat step 2 asking it to target a container NOT on the allowlist
8. **Confirm the closing report reflects `observe_impact`'s real verdict,
not a restated prediction.** Let the same experiment run through to
completion. For `chaos-pg-replica` + `pause`, `predict_blast_radius`'s
only `affected` target is `GET /products`, which strips down to the
known route `/products`, so `observe_impact` applies here and the agent
should call it once Docker/fault state is confirmed clean in step 7:
passing the `severity` from its earlier `predict_blast_radius` call as
`predicted_severity`, `["/products"]` as `affected_routes`, the
`expiresAt` value captured from the earlier `pause_container` response as
`fault_ended_at`, and `window_seconds` set to **exactly** the fault's
`duration_seconds` (e.g. `30` for this 30-second pause — no buffer
needed, since `fault_ended_at` lets the tool anchor its query window to
the fault's own active period precisely, using its own server-side
clock, regardless of how much reasoning/tool-call time has passed since
the fault reverted). Separately, call `observe_impact` yourself with
those same arguments (predicted_severity `"hard"`, affected_routes
`["/products"]`, the same `fault_ended_at`, and `window_seconds: 30`) and
compare: the agent's closing report should state the same real
`observedSeverity` and `verdict` your direct call returns, not just a
repeat of what `predict_blast_radius` predicted earlier. If the agent
skips calling `observe_impact` when it does apply, fails to pass
`fault_ended_at` when it captured a real `expiresAt`, or its closing
report only restates the prediction instead of a real observed outcome,
the wiring from Task 9 of the M5 plan (and the fix waves on top of it)
isn't working.

Not every fault reaches this branch. If you instead run an experiment
whose `predict_blast_radius affected` list contains no route-shaped
targets (e.g. pausing `chaos-grafana`, or `inject_packet_loss` at
`percent: 0`), confirm the agent does NOT call `observe_impact` at all,
and instead says plainly that no route-level observation applies to
that fault.
9. Repeat step 2 asking it to target a container NOT on the allowlist
(e.g. "pause chaos-mcp-server") — confirm the tool call is rejected and
the agent reports that clearly rather than retrying or working around
it.
Loading