Repository navigation
fix: repair real bugs found by first live Docker verification - #6
Conversation
Design for the metrics-watcher service: an MCP tool that reports real Prometheus-observed impact after a fault reverts and compares it against M4's prediction. Confirmed TrueForge has no manifest-level subagent construct, so this is built the same way M2/M4 were - a new MCP tool the existing single agent calls, not a second declared agent.
9 tasks + Final Step, matching M4's shape: scaffold, Prometheus query client, severity classification, observe_impact MCP tool, server entrypoint, Dockerfile, docker-compose wiring (with network isolation designed in from the start), acceptance test, agent wiring, README.
…ication Every prior milestone deferred live Docker verification (never available in the working environment); this is the first time it's actually run. Three real, previously-undetectable bugs surfaced immediately: - services/mcp-server/Dockerfile pinned gaiaadm/pumba:0.10.5, a tag that does not exist on Docker Hub (confirmed via the registry API) - the whole stack has never been buildable. Verified all of pumbaCommands.ts's CLI flags (pause --duration, stop, kill --signal, netem delay --time/--jitter, netem loss --percent) against a real 1.2.1 binary's --help output; every flag already matches, so bumping the tag is the complete fix. - docker-compose.yml's checkout-api healthcheck probed http://localhost:3000, but this container's /etc/hosts resolves "localhost" to ::1 first while Fastify only binds IPv4 - every healthcheck attempt failed with "connection refused" even though the server was genuinely healthy and serving real traffic (reached via the container's actual IP, not loopback). checkout-api being permanently "unhealthy" then blocked mcp-server and loadgen from ever starting at all (both depend_on checkout-api: service_healthy). Fixed by probing 127.0.0.1 instead of localhost. - scripts/verify-m1.sh's replication check never passed a password to psql, so it failed with "no password supplied" on every single run since M1 was written - unrelated to whether replication was actually working (it is). All three verified against a live stack: full docker compose up succeeds, every service reports healthy, and scripts/verify-m1.sh now passes end to end with real numbers (baseline 0%, fault-window 67.7%, recovered 0%).
…instance Real, previously-undetectable Critical bug found by this project's first ever live multi-call test: the installed @modelcontextprotocol/sdk version enforces "one transport per request" in stateless mode at two levels - StreamableHTTPServerTransport.handleRequest() throws "Stateless transport cannot be reused across requests" on its second invocation, and Server.connect() throws "Already connected to a transport" if the same McpServer instance is still attached to a prior transport. Both mcp-server's and blast-radius-sandbox's server.ts created ONE McpServer+transport pair at startup and reused it for the process's entire lifetime - meaning the FIRST /mcp request ever received would succeed, and every single request after that would fail with a bare, empty-body 500. Every unit test suite in this project uses InMemoryTransport.createLinkedPair() (a fresh pair per test, in-process) - a pattern that structurally cannot exercise this bug, which only manifests over real repeated HTTP calls to a long-running server. Docker was unavailable throughout every prior milestone's development, so nothing could ever call these services more than zero times for real until now. Fixed by creating a fresh McpServer + StreamableHTTPServerTransport pair per incoming /mcp request in both services (registerTools() is cheap - zod schemas and closures, no I/O - so re-registering per request costs nothing meaningful). Verified against a live stack: 6 sequential raw requests and 2 separate cliCall.ts invocations against mcp-server both now succeed (previously request #2 onward always failed); 2 separate calls against blast-radius-sandbox both now succeed with correct predictions. Unit tests still pass unchanged (56/56 mcp-server, 29/29 blast-radius-sandbox), and scripts/verify-m2.sh and scripts/verify-m4.sh - which make multiple real sequential MCP calls - both now pass end to end against a live Docker daemon for the first time in this project's history. metrics-watcher (M5, not yet merged) has the identical bug in its own server.ts - fixed directly on its own branch, not here.
PR Summary by QodoFix live Docker stack and stateless MCP request handling
AI Description
Diagram
High-Level Assessment
Files changed (7)
|
Code Review by Qodo
1. Plan reintroduces transport reuse
|
| const transport = new StreamableHTTPServerTransport({ | ||
| sessionIdGenerator: undefined, | ||
| }); | ||
| await mcpServer.connect(transport); |
There was a problem hiding this comment.
1. Plan reintroduces transport reuse 🐞 Bug ⚙ Maintainability
The required M5 plan constructs one McpServer and stateless transport at startup and reuses them for every /mcp request, so implementing it will make metrics-watcher fail after its first HTTP request for the same SDK reason fixed in both existing services. This also conflicts with the SDK 1.30.0 canonical stateless example, which creates and connects a fresh server/transport inside each POST handler.
Agent Prompt
## Issue description
The M5 implementation plan reintroduces the process-wide stateless MCP server/transport reuse bug repaired by this PR. Update the prescribed `metrics-watcher` server entrypoint so every `/mcp` request creates, registers, connects, handles, and closes its own `McpServer` and `StreamableHTTPServerTransport`.
## Issue Context
The plan is explicitly required input for agentic implementation. Both existing server implementations now document that SDK 1.30.0 stateless transports cannot be reused across requests, and the SDK's canonical stateless example uses per-request instances.
## Fix Focus Areas
- docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md[631-663]
ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools
Identical bug to mcp-server/blast-radius-sandbox (fixed in #6, merged into this branch): one shared McpServer+transport pair at startup made every /mcp request after the first fail with a bare 500, since the installed SDK enforces "one transport per request" in stateless mode. Fixed the same way: a fresh McpServer + StreamableHTTPServerTransport pair per request. Verified against a live stack for the first time ever: bash scripts/verify-m5.sh passes end to end, including the two scenarios Qodo's review specifically flagged as broken (short-fault offset-anchoring via fault_ended_at, and chaos-checkout-api full-outage detection via Prometheus's up{} scrape-health signal) - both previously verified only by code reading and unit tests, now confirmed with real data from a real fault injection. Unit tests unchanged (33/33 pass).
* metrics-watcher: scaffold package
* metrics-watcher: add Prometheus query client
* metrics-watcher: add severity classification and verdict logic
* metrics-watcher: add observe_impact MCP tool
* metrics-watcher: add server entrypoint
* metrics-watcher: add Dockerfile
* docker-compose: add metrics-watcher service, isolate on observability-net
* add M5 acceptance test
* fix: mark scripts/verify-m5.sh executable (100755)
Task 8 brief specified chmod +x scripts/verify-m5.sh, but the file was
committed with mode 100644. core.fileMode=false in this repo means a
filesystem chmod alone doesn't update the git index, so this uses
git update-index --chmod=+x to fix the tracked mode.
* agent: wire in the metrics-watcher connector
* README: document M5 metrics-watcher
* README: fix self-contradictory verify-m5.sh description
* fix: filter observe_impact to route-shaped targets and buffer its window (C1, C2)
The mandatory observe_impact workflow step reliably failed for 3 of 5
allowlisted containers: predict_blast_radius's `affected` list includes
non-route targets (Grafana dashboard descriptions, Prometheus scrape/query
descriptions) that don't strip down to a known route, and zod's fixed
route enum (kept as-is, deliberately) rejects the whole call outright.
agent/chaos-notary.json's workflow now filters `affected` down to
route-shaped targets before calling observe_impact, and skips the call
entirely (with a plain explanation) when no route-shaped target remains.
Separately, observe_impact had no time anchor: a window_seconds equal to
just the fault's duration_seconds risks landing entirely in the
already-recovered period once real reasoning/tool-call delay is accounted
for. The workflow now asks for duration_seconds + ~60s of buffer instead.
scripts/verify-m5.sh gains a regression assertion that calls observe_impact
well after the fault has reverted, with a buffered window, and asserts it
still reports "hard"/"matched" — the test that would have caught this.
Bundled in the same files: the observe_impact tool description and the
agent's own capability bullet now note observedSeverity/verdict are
error-rate-only (I3), and verify-m5.sh gains a positive-control wget check
before its network-isolation negative control so that assertion can't
false-pass on a broken container/exec rather than real isolation (I4).
agent/README.md's manual verification walkthrough is updated to match.
* fix: correct latency aggregation, add fetch timeout, guard non-finite values (I1, I2, M3)
- Wrap the avgLatencyMs PromQL query's numerator and denominator in sum(),
matching the error-rate/request-count queries. http_request_duration_seconds
is labeled by both route and status, so during a fault the un-summed query
returned multiple series and result[0] picked an arbitrary status code's
latency rather than an aggregate (I1).
- Add a 5s AbortSignal.timeout to the Prometheus fetch call, mirroring
verify-m1.sh's curl --max-time 5, so a paused chaos-prometheus (an
allowlisted target) can't hang the tool indefinitely (I2). Verified the
test fixtures' single-argument fake fetchers remain structurally
compatible with typeof fetch under strict mode; no test changes needed.
- Guard against Prometheus returning non-finite value strings ("NaN",
"+Inf"): check Number.isFinite before returning, falling back to the
same null "no data" semantics instead of letting a NaN silently become
a contradictory "degraded" classification downstream (M3).
* test: make observe_impact's fake fetcher route-aware (I5)
The fake fetcher in tools.test.ts ignored its `route` argument and always
returned the same metrics object, so no test could actually verify that
observedSeverity is scoped to affected_routes only, or that the response's
routes field reports all 4 known routes' real metrics rather than just the
affected ones. Replaced it with a dispatch-on-route helper and added:
- a scoping test (an unrelated route failing hard must not affect the
verdict for a different, healthy affected_routes selection)
- an all-4-routes test (routes field always has real per-route data)
- a worse_than_predicted end-to-end case
- an explicit error-propagation test through the real
registerTools/McpServer/Client round trip
Test count in this file: 2 -> 6 (net +4, matching the fix wave's target of
15 -> ~19 across the whole suite).
* docs: correct M5 status caveat, network-isolation claim, and opening tagline (I6, I7, M5)
- M5's status bullet now discloses the same class of gap M3's already
does: unit suite passes and the service typechecks cleanly, but the
Docker image build, compose network wiring, and verify-m5.sh's live run
are unverified against a real Docker daemon (Docker was unavailable
throughout this milestone's implementation environment) (I6).
- The network-isolation paragraph, written when only blast-radius-sandbox
was isolated, is now factually wrong now that metrics-watcher is also
off the default network and prometheus is dual-homed onto
observability-net. Extended it to describe metrics-watcher's identical
isolation and replaced "every other service shares" with an accurate
three-way breakdown (mcp-server: none, needs full default access;
blast-radius-sandbox and metrics-watcher: each on their own dedicated
network) (I7).
- Reworded the opening tagline's "observes impact in real time" — M5
explicitly rules out live/streaming observation as a non-goal; it
samples Prometheus once, after the fact — to "checks real impact against
its own prediction after the fact" (M5).
* fix: ground severity classification in latency, api-health, and time-anchored queries (Qodo PR#5 findings 1-3)
classifyRoute only ever examined errorRatePercent, so a latency-only fault
below checkout-api's 2000ms timeout produced no 5xx errors and reported
observedSeverity "none" even when M4 predicted "degraded" (finding 1). Add
a grounded OBSERVED_HARD_LATENCY_MS=2000 escalation (matches checkout-api's
own connectionTimeoutMillis/query_timeout and M4's LATENCY_HARD_THRESHOLD_MS)
so latency at/above that threshold now forces "hard" regardless of error
rate. Deliberately no sub-2000ms "degraded latency" threshold is added —
there's no grounded constant for that in this codebase.
queryRouteMetrics built PromQL windows ending at "now" with no time anchor;
a prior fix added a fixed +60s buffer, which mathematically dilutes short
faults below the hard-error-rate threshold (a 30s fault in a 90s window is
only 33% fault-period) — the one prior regression test used a 90s fault,
masking this for shorter, equally valid experiments (finding 2). Add an
optional offsetSeconds parameter to queryRouteMetrics (and the shared query()
helper) that appends a PromQL `offset Ns` modifier to every range-vector
selector, letting a caller anchor the window to a real elapsed time instead
of guessing with a buffer.
checkout-api's only request counter increments in Fastify's onResponse hook,
which cannot fire while the process is paused/stopped/killed, so Prometheus's
scrape fails entirely during that window and queryRouteMetrics returns nulls
that classifyRoute reads as "none" — inverted for the single most common
chaos scenario (finding 3). Add checkoutApiWasDown(), which checks
Prometheus's own up{job="checkout-api"} scrape-health signal, and thread a
new apiWasDown parameter through classifyRoute: null error-rate data now
escalates to "hard" when the API was genuinely unreachable, rather than
being silently read as idle traffic.
* fix: wire fault_ended_at time-anchoring and apiWasDown through observe_impact
Add an optional fault_ended_at (ISO-8601) input to observe_impact. When
provided (the fault tool's own expiresAt, passed verbatim), the server
computes offsetSeconds = max(0, round((Date.now() - Date.parse(fault_ended_at))
/ 1000)) using its own clock and applies it to every route-metrics query and
a single per-call checkoutApiWasDown() check, whose result now feeds every
classifyRoute() call in the affected-routes computation. This lets
window_seconds mean exactly the fault's own duration_seconds — no more
manual buffer guessing, and it works correctly for short faults, not just
long ones.
Update the tool description to document fault_ended_at and the new
hard-latency/api-outage escalations, keeping the old buffer guidance only as
a fallback for callers with no timestamp to anchor with.
Update the chaos-notary agent manifest's mandatory observe_impact workflow
step to capture the fault tool's expiresAt and pass it as fault_ended_at
with window_seconds set to exactly duration_seconds, removing the now-
obsolete "+60s buffer" instruction (kept only as an explicit fallback).
Update agent/README.md's manual verification walkthrough to match.
* test: cover latency-hard-threshold, offset anchoring, and apiWasDown escalation
severity.test.ts: avgLatencyMs at/above 2000ms classifies as hard regardless
of error rate; just below 2000ms does not escalate via latency alone (exact
boundary); null error rate + apiWasDown true classifies as hard; apiWasDown
false/omitted preserves existing "none" behavior.
prometheus.test.ts: queryRouteMetrics includes an `offset Ns` clause on every
range-vector query when offsetSeconds > 0 and omits it when 0/omitted;
checkoutApiWasDown returns true/false/false for up{} values of 0/1/no-data,
and its own query carries the offset clause correctly.
tools.test.ts: updated the fake route-metrics fetcher and every call site to
the new (route, offsetSeconds) signature, added a fake api-health fetcher to
every handleObserveImpact/registerTools call (the real default would
otherwise hit the network in tests), and added cases proving fault_ended_at
is parsed into the correct offsetSeconds and threaded through to every route
fetch plus the single per-call api-health check, that a full outage
(apiWasDown true) escalates null route metrics to hard, and that zod rejects
a malformed fault_ended_at.
19 tests -> 33 tests, all passing.
* test: add short-fault and checkout-api-outage acceptance cases to verify-m5.sh
Add extract_expires_at(), following this script's established convention of
piping raw MCP tool JSON responses through python3 on stdin rather than
shell-interpolating JSON, to pull expiresAt out of a fault-injection tool's
response.
New case 1: pauses chaos-pg-replica for a SHORT 12s fault (well under the
existing 90s test's 60% fault-fraction), waits well past its own auto-revert
plus a real extra delay, then calls observe_impact with fault_ended_at set to
the captured expiresAt and window_seconds equal to the fault's own exact
duration_seconds — asserts observedSeverity stays "hard"/"matched" despite
the real elapsed delay. This is the regression case the prior fixed +60s
buffer could not have passed for a fault this short (its own regression test
only covered a 90s fault).
New case 2: pauses chaos-checkout-api itself (a full API outage, not just a
DB dependency) and verifies observe_impact reports "hard" for the routes
M4's topology predicts as hard-affected (/products, /orders, /health), even
though checkout-api's own request counters cannot increment while the
process is paused — proving the up{job="checkout-api"} scrape-health
escalation actually works.
Existing baseline/mid-fault/buffered-window assertions are unchanged; they
still validate the no-fault_ended_at fallback path.
Docker is unavailable in this environment, so this script could not be run
live as part of this fix — bash -n syntax-checked only.
* fix: create a fresh MCP transport+server per request in metrics-watcher
Identical bug to mcp-server/blast-radius-sandbox (fixed in #6, merged into
this branch): one shared McpServer+transport pair at startup made every
/mcp request after the first fail with a bare 500, since the installed SDK
enforces "one transport per request" in stateless mode. Fixed the same way:
a fresh McpServer + StreamableHTTPServerTransport pair per request.
Verified against a live stack for the first time ever: bash
scripts/verify-m5.sh passes end to end, including the two scenarios Qodo's
review specifically flagged as broken (short-fault offset-anchoring via
fault_ended_at, and chaos-checkout-api full-outage detection via Prometheus's
up{} scrape-health signal) - both previously verified only by code reading
and unit tests, now confirmed with real data from a real fault injection.
Unit tests unchanged (33/33 pass).
…ication fixes) Also updates the M5 status bullet to reflect that scripts/verify-m5.sh now passes end to end against a real Docker daemon, superseding the "unverified" caveat written before Docker became available in this environment.
Summary
The project got Docker working in its WSL environment for the first time ever this session — every prior milestone's Dockerfiles, docker-compose wiring, and acceptance scripts were written and reviewed but never actually executed against a live daemon (disclosed as a known gap in every prior PR). Running the stack for real immediately surfaced 4 real, previously-undetectable bugs, all now fixed and verified live.
1.
services/mcp-server/Dockerfile: broken Pumba image tagPinned
gaiaadm/pumba:0.10.5, a tag that doesn't exist on Docker Hub (docker.io/gaiaadm/pumba:0.10.5: not found, confirmed via the registry API) — the whole stack has never been buildable. Bumped to1.2.1. Also used this as an opportunity to verifypumbaCommands.ts's CLI flag construction (pause --duration,stop,kill --signal,netem delay --time/--jitter,netem loss --percent) against the real1.2.1binary's--helpoutput — every flag already matches exactly, so no other changes were needed there.2.
docker-compose.yml: checkout-api healthcheck probes the wrong loopback addressProbed
http://localhost:3000/health, but this container's/etc/hostsresolves"localhost"to::1(IPv6) first, while Fastify only binds IPv4 (0.0.0.0) — every healthcheck attempt failed with "connection refused" even though the server was genuinely healthy and serving real traffic (reached via the container's actual IP, not loopback).checkout-apibeing permanently "unhealthy" then silently blockedmcp-serverandloadgenfrom ever starting at all, since bothdepends_on: checkout-api: condition: service_healthy. Fixed by probing127.0.0.1instead.3.
scripts/verify-m1.sh: replication check never passed a passworddocker exec chaos-pg-primary psql -U checkout ...never suppliedPGPASSWORD, so it failed with "no password supplied" on every single run since M1 was written — unrelated to whether replication was actually working (it is; confirmedselect count(*) from pg_stat_replicationreturns1once authenticated).4.
mcp-server/blast-radius-sandbox: shared MCP transport breaks every call after the firstThe most significant finding. Both services'
server.tscreated oneMcpServer+StreamableHTTPServerTransportpair at startup and reused it for the process's entire lifetime. The installed@modelcontextprotocol/sdkversion enforces "one transport per request" in stateless mode at two levels — the transport itself throws"Stateless transport cannot be reused across requests"on its second call, andServer.connect()throws"Already connected to a transport"if the sameMcpServeris still attached to a prior transport. The result: the first/mcprequest a freshly-started container ever received would succeed, and every single request after that would fail with a bare, empty-body 500 — reproduced directly (6 sequential raw requests:200, 500, 500, 500, 500) before the fix.Every unit test in this project uses
InMemoryTransport.createLinkedPair()(a fresh in-process pair per test) — a pattern that structurally cannot exercise this bug, since it only manifests over real repeated HTTP calls to a long-running server. That's exactly why 56/56 and 29/29 unit tests passed throughout M2 and M4 while this sat broken the whole time.Fixed by creating a fresh
McpServer+ transport pair per incoming/mcprequest in both services (registerTools()is cheap — zod schemas and closures, no I/O — so re-registering per request costs nothing meaningful).Verification (all live, against a real Docker daemon, for the first time)
docker compose up -d --build: full 9-service stack comes up, every service healthy.bash scripts/verify-m1.sh: PASSED — baseline 0%, fault-window 67.7%, recovered 0%.bash scripts/verify-m2.sh: PASSED — realpause_containercall, auto-revert, allowlist rejection, conflict detection, all real.bash scripts/verify-m4.sh: PASSED — real predictions, severity thresholds, allowlist rejection.services/mcp-server:npm test56/56 pass (unchanged).services/blast-radius-sandbox:npm test29/29 pass (unchanged).cliCall.tsinvocations againstmcp-server: all succeed (previously request M2: MCP server exposing chaos tools #2+ always failed).cliCall.tsinvocations againstblast-radius-sandbox: both succeed with correct predictions.Scope note
services/metrics-watcher(M5, PR #5, not yet merged) has the identical shared-transport bug in its ownserver.ts— fixed directly on that branch, not here, since it's still an open PR.Test plan
verify-m1.sh,verify-m2.sh,verify-m4.sh), described abovemcp-server56/56,blast-radius-sandbox29/29)scripts/verify-m5.sh— blocked on PR M5: Metrics-watcher #5 merging its own identical fix first