Skip to content

fix: repair real bugs found by first live Docker verification - #6

Merged
mikelord007 merged 4 commits into
mainfrom
fix/live-verification-stack-bugs
Aug 28, 2026
Merged

mikelord007 merged 4 commits into
mainfrom
fix/live-verification-stack-bugs

Conversation

@mikelord007

Copy link
Copy Markdown
Owner

Summary

The project got Docker working in its WSL environment for the first time ever this session — every prior milestone's Dockerfiles, docker-compose wiring, and acceptance scripts were written and reviewed but never actually executed against a live daemon (disclosed as a known gap in every prior PR). Running the stack for real immediately surfaced 4 real, previously-undetectable bugs, all now fixed and verified live.

1. services/mcp-server/Dockerfile: broken Pumba image tag

Pinned gaiaadm/pumba:0.10.5, a tag that doesn't exist on Docker Hub (docker.io/gaiaadm/pumba:0.10.5: not found, confirmed via the registry API) — the whole stack has never been buildable. Bumped to 1.2.1. Also used this as an opportunity to verify pumbaCommands.ts's CLI flag construction (pause --duration, stop, kill --signal, netem delay --time/--jitter, netem loss --percent) against the real 1.2.1 binary's --help output — every flag already matches exactly, so no other changes were needed there.

2. docker-compose.yml: checkout-api healthcheck probes the wrong loopback address

Probed http://localhost:3000/health, but this container's /etc/hosts resolves "localhost" to ::1 (IPv6) first, while Fastify only binds IPv4 (0.0.0.0) — every healthcheck attempt failed with "connection refused" even though the server was genuinely healthy and serving real traffic (reached via the container's actual IP, not loopback). checkout-api being permanently "unhealthy" then silently blocked mcp-server and loadgen from ever starting at all, since both depends_on: checkout-api: condition: service_healthy. Fixed by probing 127.0.0.1 instead.

3. scripts/verify-m1.sh: replication check never passed a password

docker exec chaos-pg-primary psql -U checkout ... never supplied PGPASSWORD, so it failed with "no password supplied" on every single run since M1 was written — unrelated to whether replication was actually working (it is; confirmed select count(*) from pg_stat_replication returns 1 once authenticated).

4. mcp-server/blast-radius-sandbox: shared MCP transport breaks every call after the first

The most significant finding. Both services' server.ts created one McpServer + StreamableHTTPServerTransport pair at startup and reused it for the process's entire lifetime. The installed @modelcontextprotocol/sdk version enforces "one transport per request" in stateless mode at two levels — the transport itself throws "Stateless transport cannot be reused across requests" on its second call, and Server.connect() throws "Already connected to a transport" if the same McpServer is still attached to a prior transport. The result: the first /mcp request a freshly-started container ever received would succeed, and every single request after that would fail with a bare, empty-body 500 — reproduced directly (6 sequential raw requests: 200, 500, 500, 500, 500) before the fix.

Every unit test in this project uses InMemoryTransport.createLinkedPair() (a fresh in-process pair per test) — a pattern that structurally cannot exercise this bug, since it only manifests over real repeated HTTP calls to a long-running server. That's exactly why 56/56 and 29/29 unit tests passed throughout M2 and M4 while this sat broken the whole time.

Fixed by creating a fresh McpServer + transport pair per incoming /mcp request in both services (registerTools() is cheap — zod schemas and closures, no I/O — so re-registering per request costs nothing meaningful).

Verification (all live, against a real Docker daemon, for the first time)

  • docker compose up -d --build: full 9-service stack comes up, every service healthy.
  • bash scripts/verify-m1.sh: PASSED — baseline 0%, fault-window 67.7%, recovered 0%.
  • bash scripts/verify-m2.sh: PASSED — real pause_container call, auto-revert, allowlist rejection, conflict detection, all real.
  • bash scripts/verify-m4.sh: PASSED — real predictions, severity thresholds, allowlist rejection.
  • services/mcp-server: npm test 56/56 pass (unchanged).
  • services/blast-radius-sandbox: npm test 29/29 pass (unchanged).
  • 6 sequential raw HTTP requests + 2 separate cliCall.ts invocations against mcp-server: all succeed (previously request M2: MCP server exposing chaos tools #2+ always failed).
  • 2 separate cliCall.ts invocations against blast-radius-sandbox: both succeed with correct predictions.

Scope note

services/metrics-watcher (M5, PR #5, not yet merged) has the identical shared-transport bug in its own server.ts — fixed directly on that branch, not here, since it's still an open PR.

Test plan

  • Full stack live verification (verify-m1.sh, verify-m2.sh, verify-m4.sh), described above
  • Unit test suites unaffected (mcp-server 56/56, blast-radius-sandbox 29/29)
  • scripts/verify-m5.sh — blocked on PR M5: Metrics-watcher #5 merging its own identical fix first

Design for the metrics-watcher service: an MCP tool that reports real
Prometheus-observed impact after a fault reverts and compares it against
M4's prediction. Confirmed TrueForge has no manifest-level subagent
construct, so this is built the same way M2/M4 were - a new MCP tool the
existing single agent calls, not a second declared agent.
9 tasks + Final Step, matching M4's shape: scaffold, Prometheus query
client, severity classification, observe_impact MCP tool, server
entrypoint, Dockerfile, docker-compose wiring (with network isolation
designed in from the start), acceptance test, agent wiring, README.
…ication

Every prior milestone deferred live Docker verification (never available in
the working environment); this is the first time it's actually run. Three
real, previously-undetectable bugs surfaced immediately:

- services/mcp-server/Dockerfile pinned gaiaadm/pumba:0.10.5, a tag that
  does not exist on Docker Hub (confirmed via the registry API) - the whole
  stack has never been buildable. Verified all of pumbaCommands.ts's CLI
  flags (pause --duration, stop, kill --signal, netem delay --time/--jitter,
  netem loss --percent) against a real 1.2.1 binary's --help output; every
  flag already matches, so bumping the tag is the complete fix.

- docker-compose.yml's checkout-api healthcheck probed http://localhost:3000,
  but this container's /etc/hosts resolves "localhost" to ::1 first while
  Fastify only binds IPv4 - every healthcheck attempt failed with "connection
  refused" even though the server was genuinely healthy and serving real
  traffic (reached via the container's actual IP, not loopback). checkout-api
  being permanently "unhealthy" then blocked mcp-server and loadgen from ever
  starting at all (both depend_on checkout-api: service_healthy). Fixed by
  probing 127.0.0.1 instead of localhost.

- scripts/verify-m1.sh's replication check never passed a password to psql,
  so it failed with "no password supplied" on every single run since M1 was
  written - unrelated to whether replication was actually working (it is).

All three verified against a live stack: full docker compose up succeeds,
every service reports healthy, and scripts/verify-m1.sh now passes end to
end with real numbers (baseline 0%, fault-window 67.7%, recovered 0%).
…instance

Real, previously-undetectable Critical bug found by this project's first
ever live multi-call test: the installed @modelcontextprotocol/sdk version
enforces "one transport per request" in stateless mode at two levels -
StreamableHTTPServerTransport.handleRequest() throws "Stateless transport
cannot be reused across requests" on its second invocation, and
Server.connect() throws "Already connected to a transport" if the same
McpServer instance is still attached to a prior transport. Both mcp-server's
and blast-radius-sandbox's server.ts created ONE McpServer+transport pair
at startup and reused it for the process's entire lifetime - meaning the
FIRST /mcp request ever received would succeed, and every single request
after that would fail with a bare, empty-body 500.

Every unit test suite in this project uses InMemoryTransport.createLinkedPair()
(a fresh pair per test, in-process) - a pattern that structurally cannot
exercise this bug, which only manifests over real repeated HTTP calls to a
long-running server. Docker was unavailable throughout every prior
milestone's development, so nothing could ever call these services more
than zero times for real until now.

Fixed by creating a fresh McpServer + StreamableHTTPServerTransport pair
per incoming /mcp request in both services (registerTools() is cheap - zod
schemas and closures, no I/O - so re-registering per request costs nothing
meaningful). Verified against a live stack: 6 sequential raw requests and
2 separate cliCall.ts invocations against mcp-server both now succeed
(previously request #2 onward always failed); 2 separate calls against
blast-radius-sandbox both now succeed with correct predictions. Unit tests
still pass unchanged (56/56 mcp-server, 29/29 blast-radius-sandbox), and
scripts/verify-m2.sh and scripts/verify-m4.sh - which make multiple real
sequential MCP calls - both now pass end to end against a live Docker
daemon for the first time in this project's history.

metrics-watcher (M5, not yet merged) has the identical bug in its own
server.ts - fixed directly on its own branch, not here.
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Fix live Docker stack and stateless MCP request handling

🐞 Bug fix 📝 Documentation 🧪 Tests ⚙️ Configuration changes 🕐 20-40 Minutes

Grey Divider

AI Description

• Repair Docker image, healthcheck, and authenticated replication verification failures.
• Create fresh stateless MCP server transports for every HTTP request.
• Document the planned Prometheus-backed M5 metrics-watcher service.
Diagram

graph TD
  BUILD["MCP Image Build"] --> PUMBA["Pumba 1.2.1"]
  COMPOSE["Docker Compose"] --> HEALTH["IPv4 Healthcheck"] --> DEPS["Dependent Services"]
  CLIENT["MCP Client"] --> HTTP["HTTP Server"] --> PAIR["Fresh MCP Pair"]
  PAIR --> TOOLS["Registered Tools"]
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Maintain stateful MCP sessions
  • ➕ Reuses server and transport instances across related requests.
  • ➕ Supports longer-lived clients without rebuilding tool registrations.
  • ➖ Requires session IDs, transport lookup, cleanup, and concurrency handling.
  • ➖ Does not match the current one-call client lifecycle.
  • ➖ Adds unnecessary state to otherwise stateless endpoints.

Recommendation: Keep the request-scoped McpServer and transport approach. It directly matches the SDK's stateless transport contract and the repository's short-lived clients; a stateful session registry would add lifecycle complexity without a current requirement.

Files changed (7) +1439 / -47

Bug fix (4) +71 / -46
docker-compose.ymlProbe checkout-api through IPv4 loopback +7/-1

Probe checkout-api through IPv4 loopback

• Changes the checkout-api healthcheck from localhost to 127.0.0.1. This avoids IPv6-first localhost resolution against a Fastify server bound only to IPv4, allowing health-dependent services to start.

docker-compose.yml

server.tsScope stateless MCP instances to each request +33/-22

Scope stateless MCP instances to each request

• Moves McpServer creation, tool registration, transport creation, and connection into the /mcp request handler. This prevents all requests after the first from failing due to SDK transport reuse restrictions.

services/blast-radius-sandbox/src/server.ts

DockerfileUse an available Pumba image release +1/-1

Use an available Pumba image release

• Updates the Pumba build stage from the nonexistent 0.10.5 tag to 1.2.1, restoring image buildability while retaining the existing binary-copy layout.

services/mcp-server/Dockerfile

server.tsCreate a fresh MCP server and transport per call +30/-22

Create a fresh MCP server and transport per call

• Creates and connects a request-scoped McpServer and stateless HTTP transport for each /mcp request while retaining shared Docker and fault-registry state. This fixes second-and-later MCP calls failing with transport reuse errors.

services/mcp-server/src/server.ts

Tests (1) +1 / -1
verify-m1.shAuthenticate the replication readiness query +1/-1

Authenticate the replication readiness query

• Passes the development PostgreSQL password into the primary container's psql process. The replication wait now measures replication state instead of repeatedly failing authentication.

scripts/verify-m1.sh

Documentation (2) +1367 / -0
2026-08-27-m5-metrics-watcher.mdAdd the M5 metrics-watcher implementation plan +1074/-0

Add the M5 metrics-watcher implementation plan

• Defines a task-by-task plan for a sealed MCP service that queries Prometheus, classifies observed severity, and compares outcomes with blast-radius predictions. It also plans isolated Compose networking, tests, agent integration, and acceptance verification.

docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md

2026-08-27-m5-metrics-watcher-design.mdSpecify the M5 metrics-watcher architecture +293/-0

Specify the M5 metrics-watcher architecture

• Documents the metrics-watcher goals, security boundaries, PromQL data model, severity rules, MCP tool contract, and M3 agent integration. The design explicitly rejects streaming observation and Docker access.

docs/superpowers/specs/2026-08-27-m5-metrics-watcher-design.md

@qodo-code-review

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (1) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Remediation recommended

1. Plan reintroduces transport reuse 🐞 Bug ⚙ Maintainability
Description
The required M5 plan constructs one McpServer and stateless transport at startup and reuses them
for every /mcp request, so implementing it will make metrics-watcher fail after its first HTTP
request for the same SDK reason fixed in both existing services. This also conflicts with the SDK
1.30.0 canonical stateless example, which creates and connects a fresh server/transport inside each
POST handler.
Code

docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md[R640-643]

+  const transport = new StreamableHTTPServerTransport({
+    sessionIdGenerator: undefined,
+  });
+  await mcpServer.connect(transport);
Evidence
The plan requires agentic workers to execute its steps and its server snippet creates/connects the
pair before the HTTP handler, then invokes the same transport from every request. The two server
implementations repaired by this PR explicitly create both objects inside /mcp because shared
instances fail on subsequent requests, and the SDK 1.30.0 canonical example confirms the per-request
construction pattern.

docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md[3-7]
docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md[631-655]
services/mcp-server/src/server.ts[27-48]
services/blast-radius-sandbox/src/server.ts[17-43]
🌐 The SDK 1.30.0 stateless example creates a fresh McpServer and StreamableHTTPServerTransport inside each POST /mcp handler, connects them, handles that request, and closes them when the response closes.

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The M5 implementation plan reintroduces the process-wide stateless MCP server/transport reuse bug repaired by this PR. Update the prescribed `metrics-watcher` server entrypoint so every `/mcp` request creates, registers, connects, handles, and closes its own `McpServer` and `StreamableHTTPServerTransport`.

## Issue Context
The plan is explicitly required input for agentic implementation. Both existing server implementations now document that SDK 1.30.0 stateless transports cannot be reused across requests, and the SDK's canonical stateless example uses per-request instances.

## Fix Focus Areas
- docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md[631-663]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
✅ Web pages:
  +5 more
Review mode: ⚖️ Balanced: Downgraded extended -> standard: change is below the extended eligibility bar (hunks 7/18, lines 1486/200; both must reach the floor). Router rationale: The PR combines several independent runtime, Docker/Compose, shell verification, and MCP transport changes, including a new large implementation plan with subtle cross-path and integration risks.

Grey Divider

Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment on lines +640 to +643
const transport = new StreamableHTTPServerTransport({
sessionIdGenerator: undefined,
});
await mcpServer.connect(transport);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Remediation recommended

1. Plan reintroduces transport reuse 🐞 Bug ⚙ Maintainability

The required M5 plan constructs one McpServer and stateless transport at startup and reuses them
for every /mcp request, so implementing it will make metrics-watcher fail after its first HTTP
request for the same SDK reason fixed in both existing services. This also conflicts with the SDK
1.30.0 canonical stateless example, which creates and connects a fresh server/transport inside each
POST handler.
Agent Prompt
## Issue description
The M5 implementation plan reintroduces the process-wide stateless MCP server/transport reuse bug repaired by this PR. Update the prescribed `metrics-watcher` server entrypoint so every `/mcp` request creates, registers, connects, handles, and closes its own `McpServer` and `StreamableHTTPServerTransport`.

## Issue Context
The plan is explicitly required input for agentic implementation. Both existing server implementations now document that SDK 1.30.0 stateless transports cannot be reused across requests, and the SDK's canonical stateless example uses per-request instances.

## Fix Focus Areas
- docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md[631-663]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools

@mikelord007
mikelord007 merged commit 68b0567 into main Aug 28, 2026
1 check passed
@mikelord007
mikelord007 deleted the fix/live-verification-stack-bugs branch August 28, 2026 07:46
mikelord007 added a commit that referenced this pull request Aug 28, 2026
Identical bug to mcp-server/blast-radius-sandbox (fixed in #6, merged into
this branch): one shared McpServer+transport pair at startup made every
/mcp request after the first fail with a bare 500, since the installed SDK
enforces "one transport per request" in stateless mode. Fixed the same way:
a fresh McpServer + StreamableHTTPServerTransport pair per request.

Verified against a live stack for the first time ever: bash
scripts/verify-m5.sh passes end to end, including the two scenarios Qodo's
review specifically flagged as broken (short-fault offset-anchoring via
fault_ended_at, and chaos-checkout-api full-outage detection via Prometheus's
up{} scrape-health signal) - both previously verified only by code reading
and unit tests, now confirmed with real data from a real fault injection.
Unit tests unchanged (33/33 pass).
@mikelord007 mikelord007 mentioned this pull request Aug 28, 2026
4 of 5 tasks
mikelord007 added a commit that referenced this pull request Aug 28, 2026
* metrics-watcher: scaffold package

* metrics-watcher: add Prometheus query client

* metrics-watcher: add severity classification and verdict logic

* metrics-watcher: add observe_impact MCP tool

* metrics-watcher: add server entrypoint

* metrics-watcher: add Dockerfile

* docker-compose: add metrics-watcher service, isolate on observability-net

* add M5 acceptance test

* fix: mark scripts/verify-m5.sh executable (100755)

Task 8 brief specified chmod +x scripts/verify-m5.sh, but the file was
committed with mode 100644. core.fileMode=false in this repo means a
filesystem chmod alone doesn't update the git index, so this uses
git update-index --chmod=+x to fix the tracked mode.

* agent: wire in the metrics-watcher connector

* README: document M5 metrics-watcher

* README: fix self-contradictory verify-m5.sh description

* fix: filter observe_impact to route-shaped targets and buffer its window (C1, C2)

The mandatory observe_impact workflow step reliably failed for 3 of 5
allowlisted containers: predict_blast_radius's `affected` list includes
non-route targets (Grafana dashboard descriptions, Prometheus scrape/query
descriptions) that don't strip down to a known route, and zod's fixed
route enum (kept as-is, deliberately) rejects the whole call outright.
agent/chaos-notary.json's workflow now filters `affected` down to
route-shaped targets before calling observe_impact, and skips the call
entirely (with a plain explanation) when no route-shaped target remains.

Separately, observe_impact had no time anchor: a window_seconds equal to
just the fault's duration_seconds risks landing entirely in the
already-recovered period once real reasoning/tool-call delay is accounted
for. The workflow now asks for duration_seconds + ~60s of buffer instead.
scripts/verify-m5.sh gains a regression assertion that calls observe_impact
well after the fault has reverted, with a buffered window, and asserts it
still reports "hard"/"matched" — the test that would have caught this.

Bundled in the same files: the observe_impact tool description and the
agent's own capability bullet now note observedSeverity/verdict are
error-rate-only (I3), and verify-m5.sh gains a positive-control wget check
before its network-isolation negative control so that assertion can't
false-pass on a broken container/exec rather than real isolation (I4).

agent/README.md's manual verification walkthrough is updated to match.

* fix: correct latency aggregation, add fetch timeout, guard non-finite values (I1, I2, M3)

- Wrap the avgLatencyMs PromQL query's numerator and denominator in sum(),
  matching the error-rate/request-count queries. http_request_duration_seconds
  is labeled by both route and status, so during a fault the un-summed query
  returned multiple series and result[0] picked an arbitrary status code's
  latency rather than an aggregate (I1).
- Add a 5s AbortSignal.timeout to the Prometheus fetch call, mirroring
  verify-m1.sh's curl --max-time 5, so a paused chaos-prometheus (an
  allowlisted target) can't hang the tool indefinitely (I2). Verified the
  test fixtures' single-argument fake fetchers remain structurally
  compatible with typeof fetch under strict mode; no test changes needed.
- Guard against Prometheus returning non-finite value strings ("NaN",
  "+Inf"): check Number.isFinite before returning, falling back to the
  same null "no data" semantics instead of letting a NaN silently become
  a contradictory "degraded" classification downstream (M3).

* test: make observe_impact's fake fetcher route-aware (I5)

The fake fetcher in tools.test.ts ignored its `route` argument and always
returned the same metrics object, so no test could actually verify that
observedSeverity is scoped to affected_routes only, or that the response's
routes field reports all 4 known routes' real metrics rather than just the
affected ones. Replaced it with a dispatch-on-route helper and added:

- a scoping test (an unrelated route failing hard must not affect the
  verdict for a different, healthy affected_routes selection)
- an all-4-routes test (routes field always has real per-route data)
- a worse_than_predicted end-to-end case
- an explicit error-propagation test through the real
  registerTools/McpServer/Client round trip

Test count in this file: 2 -> 6 (net +4, matching the fix wave's target of
15 -> ~19 across the whole suite).

* docs: correct M5 status caveat, network-isolation claim, and opening tagline (I6, I7, M5)

- M5's status bullet now discloses the same class of gap M3's already
  does: unit suite passes and the service typechecks cleanly, but the
  Docker image build, compose network wiring, and verify-m5.sh's live run
  are unverified against a real Docker daemon (Docker was unavailable
  throughout this milestone's implementation environment) (I6).
- The network-isolation paragraph, written when only blast-radius-sandbox
  was isolated, is now factually wrong now that metrics-watcher is also
  off the default network and prometheus is dual-homed onto
  observability-net. Extended it to describe metrics-watcher's identical
  isolation and replaced "every other service shares" with an accurate
  three-way breakdown (mcp-server: none, needs full default access;
  blast-radius-sandbox and metrics-watcher: each on their own dedicated
  network) (I7).
- Reworded the opening tagline's "observes impact in real time" — M5
  explicitly rules out live/streaming observation as a non-goal; it
  samples Prometheus once, after the fact — to "checks real impact against
  its own prediction after the fact" (M5).

* fix: ground severity classification in latency, api-health, and time-anchored queries (Qodo PR#5 findings 1-3)

classifyRoute only ever examined errorRatePercent, so a latency-only fault
below checkout-api's 2000ms timeout produced no 5xx errors and reported
observedSeverity "none" even when M4 predicted "degraded" (finding 1). Add
a grounded OBSERVED_HARD_LATENCY_MS=2000 escalation (matches checkout-api's
own connectionTimeoutMillis/query_timeout and M4's LATENCY_HARD_THRESHOLD_MS)
so latency at/above that threshold now forces "hard" regardless of error
rate. Deliberately no sub-2000ms "degraded latency" threshold is added —
there's no grounded constant for that in this codebase.

queryRouteMetrics built PromQL windows ending at "now" with no time anchor;
a prior fix added a fixed +60s buffer, which mathematically dilutes short
faults below the hard-error-rate threshold (a 30s fault in a 90s window is
only 33% fault-period) — the one prior regression test used a 90s fault,
masking this for shorter, equally valid experiments (finding 2). Add an
optional offsetSeconds parameter to queryRouteMetrics (and the shared query()
helper) that appends a PromQL `offset Ns` modifier to every range-vector
selector, letting a caller anchor the window to a real elapsed time instead
of guessing with a buffer.

checkout-api's only request counter increments in Fastify's onResponse hook,
which cannot fire while the process is paused/stopped/killed, so Prometheus's
scrape fails entirely during that window and queryRouteMetrics returns nulls
that classifyRoute reads as "none" — inverted for the single most common
chaos scenario (finding 3). Add checkoutApiWasDown(), which checks
Prometheus's own up{job="checkout-api"} scrape-health signal, and thread a
new apiWasDown parameter through classifyRoute: null error-rate data now
escalates to "hard" when the API was genuinely unreachable, rather than
being silently read as idle traffic.

* fix: wire fault_ended_at time-anchoring and apiWasDown through observe_impact

Add an optional fault_ended_at (ISO-8601) input to observe_impact. When
provided (the fault tool's own expiresAt, passed verbatim), the server
computes offsetSeconds = max(0, round((Date.now() - Date.parse(fault_ended_at))
/ 1000)) using its own clock and applies it to every route-metrics query and
a single per-call checkoutApiWasDown() check, whose result now feeds every
classifyRoute() call in the affected-routes computation. This lets
window_seconds mean exactly the fault's own duration_seconds — no more
manual buffer guessing, and it works correctly for short faults, not just
long ones.

Update the tool description to document fault_ended_at and the new
hard-latency/api-outage escalations, keeping the old buffer guidance only as
a fallback for callers with no timestamp to anchor with.

Update the chaos-notary agent manifest's mandatory observe_impact workflow
step to capture the fault tool's expiresAt and pass it as fault_ended_at
with window_seconds set to exactly duration_seconds, removing the now-
obsolete "+60s buffer" instruction (kept only as an explicit fallback).
Update agent/README.md's manual verification walkthrough to match.

* test: cover latency-hard-threshold, offset anchoring, and apiWasDown escalation

severity.test.ts: avgLatencyMs at/above 2000ms classifies as hard regardless
of error rate; just below 2000ms does not escalate via latency alone (exact
boundary); null error rate + apiWasDown true classifies as hard; apiWasDown
false/omitted preserves existing "none" behavior.

prometheus.test.ts: queryRouteMetrics includes an `offset Ns` clause on every
range-vector query when offsetSeconds > 0 and omits it when 0/omitted;
checkoutApiWasDown returns true/false/false for up{} values of 0/1/no-data,
and its own query carries the offset clause correctly.

tools.test.ts: updated the fake route-metrics fetcher and every call site to
the new (route, offsetSeconds) signature, added a fake api-health fetcher to
every handleObserveImpact/registerTools call (the real default would
otherwise hit the network in tests), and added cases proving fault_ended_at
is parsed into the correct offsetSeconds and threaded through to every route
fetch plus the single per-call api-health check, that a full outage
(apiWasDown true) escalates null route metrics to hard, and that zod rejects
a malformed fault_ended_at.

19 tests -> 33 tests, all passing.

* test: add short-fault and checkout-api-outage acceptance cases to verify-m5.sh

Add extract_expires_at(), following this script's established convention of
piping raw MCP tool JSON responses through python3 on stdin rather than
shell-interpolating JSON, to pull expiresAt out of a fault-injection tool's
response.

New case 1: pauses chaos-pg-replica for a SHORT 12s fault (well under the
existing 90s test's 60% fault-fraction), waits well past its own auto-revert
plus a real extra delay, then calls observe_impact with fault_ended_at set to
the captured expiresAt and window_seconds equal to the fault's own exact
duration_seconds — asserts observedSeverity stays "hard"/"matched" despite
the real elapsed delay. This is the regression case the prior fixed +60s
buffer could not have passed for a fault this short (its own regression test
only covered a 90s fault).

New case 2: pauses chaos-checkout-api itself (a full API outage, not just a
DB dependency) and verifies observe_impact reports "hard" for the routes
M4's topology predicts as hard-affected (/products, /orders, /health), even
though checkout-api's own request counters cannot increment while the
process is paused — proving the up{job="checkout-api"} scrape-health
escalation actually works.

Existing baseline/mid-fault/buffered-window assertions are unchanged; they
still validate the no-fault_ended_at fallback path.

Docker is unavailable in this environment, so this script could not be run
live as part of this fix — bash -n syntax-checked only.

* fix: create a fresh MCP transport+server per request in metrics-watcher

Identical bug to mcp-server/blast-radius-sandbox (fixed in #6, merged into
this branch): one shared McpServer+transport pair at startup made every
/mcp request after the first fail with a bare 500, since the installed SDK
enforces "one transport per request" in stateless mode. Fixed the same way:
a fresh McpServer + StreamableHTTPServerTransport pair per request.

Verified against a live stack for the first time ever: bash
scripts/verify-m5.sh passes end to end, including the two scenarios Qodo's
review specifically flagged as broken (short-fault offset-anchoring via
fault_ended_at, and chaos-checkout-api full-outage detection via Prometheus's
up{} scrape-health signal) - both previously verified only by code reading
and unit tests, now confirmed with real data from a real fault injection.
Unit tests unchanged (33/33 pass).
mikelord007 added a commit that referenced this pull request Aug 28, 2026
…ication fixes)

Also updates the M5 status bullet to reflect that scripts/verify-m5.sh now
passes end to end against a real Docker daemon, superseding the "unverified"
caveat written before Docker became available in this environment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant