Skip to content

M5: Metrics-watcher - #5

Merged
mikelord007 merged 22 commits into
mainfrom
feat/m5-metrics-watcher
Aug 28, 2026
Merged

mikelord007 merged 22 commits into
mainfrom
feat/m5-metrics-watcher

Conversation

@mikelord007

@mikelord007 mikelord007 commented Aug 28, 2026 •

Copy link
Copy Markdown
Owner

Summary

Adds services/metrics-watcher, a sealed, read-only MCP service that reports what Prometheus actually recorded after a chaos fault reverts and compares it against M4's prediction — closing the predict-then-observe loop this project's README has claimed since M1 but never built.

  • New observe_impact MCP tool: takes (predicted_severity, affected_routes, window_seconds, fault_ended_at?), queries Prometheus for checkout-api's real per-route error rate/latency, and returns { observedSeverity, verdict, routes } — an explicit "matched" / "milder_than_predicted" / "worse_than_predicted" comparison against the prediction, not just raw numbers.
  • Zero Docker access by design, same sealed posture as M4 — its only external dependency is a read-only HTTP call to Prometheus's query API.
  • Network-isolated from the start (not after a review catches it, unlike M4): its own dedicated observability-net Compose network, shared only with a dual-homed prometheus, so it cannot reach mcp-server's unauthenticated destructive endpoint even if compromised — confirmed at the DNS level in live testing (metrics-watcher can't even resolve mcp-server).
  • Severity classification factors in both error rate and latency (a fault at/above checkout-api's real 2000ms timeout is hard regardless of error rate), and escalates to hard when Prometheus's own up{job="checkout-api"} signal shows the API itself was unreachable — distinguishing a genuine outage from "no traffic."
  • Query windows are time-anchored via an optional fault_ended_at (the fault tool's own expiresAt timestamp, passed through verbatim), so calling observe_impact any real amount of time after a fault reverts still correctly measures the fault's actual window via a server-computed PromQL offset, not a fragile fixed buffer.
  • Wired into M3's agent manifest as a third mcp_servers[] connector; the experiment workflow's final step is now mandatory — call observe_impact and report the real outcome instead of stopping at a prediction.
  • scripts/verify-m5.sh: brings up the full stack and asserts baseline "none" severity, a real pause-triggered fault reaching "hard"/"matched" mid-fault, after a buffered post-revert delay, and via offset-anchoring on a short fault; invalid-input rejection; a real chaos-checkout-api outage correctly detected as "hard"; and the mcp-server network-isolation guard (with a positive control proving the check itself is meaningful).

Process notes

Built via subagent-driven development: 9 tasks + final step, each with a fresh implementer + task reviewer. A final whole-branch review (Opus) then found 2 Critical + 7 Important + 7 Minor issues that per-task review couldn't have caught (agent-instructions/schema mismatch, an unanchored query window), all independently reproduced and fixed in one wave, re-reviewed clean.

Qodo's review on this PR then found 3 more real bugs (all High/Correctness): latency-only faults getting a false verdict, the query-window buffer having a genuine mathematical flaw for short faults, and chaos-checkout-api outages being misreported as healthy since the API's own request counters can't increment while it's frozen. Fixed properly (not with another heuristic) by discovering mid-verification that M2's fault tools already return a real expiresAt timestamp — observe_impact now uses that for genuine PromQL time-anchoring, plus a new up{}-based outage check. All 3 independently re-verified, confirmed by Qodo.

This session also got Docker working for the first time in this project's history (every prior PR disclosed live verification as an unrun gap). Running the full stack live immediately surfaced a 4th, previously invisible Critical bug: metrics-watcher's server.ts (like mcp-server's and blast-radius-sandbox's) shared one McpServer+transport pair across the process's whole lifetime — the installed SDK enforces "one transport per request" in stateless mode, so the first /mcp request ever received succeeded and every request after that failed with a bare 500. No unit test could ever have caught this (they all use an in-process InMemoryTransport pair, not real repeated HTTP calls). Fixed by creating a fresh transport+server pair per request; the identical fix for mcp-server/blast-radius-sandbox merged separately in #6 and is included here via a merge from main.

Live verification (new — first time ever for this project)

  • docker compose up -d --build: all 9 services healthy.
  • bash scripts/verify-m1.sh, verify-m2.sh, verify-m4.sh: all pass against a live daemon (see fix: repair real bugs found by first live Docker verification #6).
  • bash scripts/verify-m5.sh: passes end to end, including the exact scenarios Qodo's review flagged — the short-fault offset-anchored case and the chaos-checkout-api outage case, both now confirmed with real fault-injection data, not just code reading.
  • services/metrics-watcher: npm test 33/33 pass.

Known gaps

  • The manifest connector registration in TrueForge (Settings → Connectors) remains a manual step for whoever runs the agent, same as M3/M4.

Test plan

  • services/metrics-watcher: npm run build (zero TS errors), npm test (33/33 pass)
  • bash -n scripts/verify-m5.sh (syntax valid)
  • node -e "require('./agent/chaos-notary.json')" + the established no-overclaim guard (both pass)
  • Live docker compose up -d --build + bash scripts/verify-m5.sh against a real Docker daemon — passes
  • Manual TrueForge connector registration + agent run confirming observe_impact is actually called and its real verdict shapes the closing report (same manual step M3/M4 already flagged as pending)

Task 8 brief specified chmod +x scripts/verify-m5.sh, but the file was
committed with mode 100644. core.fileMode=false in this repo means a
filesystem chmod alone doesn't update the git index, so this uses
git update-index --chmod=+x to fix the tracked mode.
…dow (C1, C2)

The mandatory observe_impact workflow step reliably failed for 3 of 5
allowlisted containers: predict_blast_radius's `affected` list includes
non-route targets (Grafana dashboard descriptions, Prometheus scrape/query
descriptions) that don't strip down to a known route, and zod's fixed
route enum (kept as-is, deliberately) rejects the whole call outright.
agent/chaos-notary.json's workflow now filters `affected` down to
route-shaped targets before calling observe_impact, and skips the call
entirely (with a plain explanation) when no route-shaped target remains.

Separately, observe_impact had no time anchor: a window_seconds equal to
just the fault's duration_seconds risks landing entirely in the
already-recovered period once real reasoning/tool-call delay is accounted
for. The workflow now asks for duration_seconds + ~60s of buffer instead.
scripts/verify-m5.sh gains a regression assertion that calls observe_impact
well after the fault has reverted, with a buffered window, and asserts it
still reports "hard"/"matched" — the test that would have caught this.

Bundled in the same files: the observe_impact tool description and the
agent's own capability bullet now note observedSeverity/verdict are
error-rate-only (I3), and verify-m5.sh gains a positive-control wget check
before its network-isolation negative control so that assertion can't
false-pass on a broken container/exec rather than real isolation (I4).

agent/README.md's manual verification walkthrough is updated to match.
… values (I1, I2, M3)

- Wrap the avgLatencyMs PromQL query's numerator and denominator in sum(),
  matching the error-rate/request-count queries. http_request_duration_seconds
  is labeled by both route and status, so during a fault the un-summed query
  returned multiple series and result[0] picked an arbitrary status code's
  latency rather than an aggregate (I1).
- Add a 5s AbortSignal.timeout to the Prometheus fetch call, mirroring
  verify-m1.sh's curl --max-time 5, so a paused chaos-prometheus (an
  allowlisted target) can't hang the tool indefinitely (I2). Verified the
  test fixtures' single-argument fake fetchers remain structurally
  compatible with typeof fetch under strict mode; no test changes needed.
- Guard against Prometheus returning non-finite value strings ("NaN",
  "+Inf"): check Number.isFinite before returning, falling back to the
  same null "no data" semantics instead of letting a NaN silently become
  a contradictory "degraded" classification downstream (M3).
The fake fetcher in tools.test.ts ignored its `route` argument and always
returned the same metrics object, so no test could actually verify that
observedSeverity is scoped to affected_routes only, or that the response's
routes field reports all 4 known routes' real metrics rather than just the
affected ones. Replaced it with a dispatch-on-route helper and added:

- a scoping test (an unrelated route failing hard must not affect the
  verdict for a different, healthy affected_routes selection)
- an all-4-routes test (routes field always has real per-route data)
- a worse_than_predicted end-to-end case
- an explicit error-propagation test through the real
  registerTools/McpServer/Client round trip

Test count in this file: 2 -> 6 (net +4, matching the fix wave's target of
15 -> ~19 across the whole suite).
…tagline (I6, I7, M5)

- M5's status bullet now discloses the same class of gap M3's already
  does: unit suite passes and the service typechecks cleanly, but the
  Docker image build, compose network wiring, and verify-m5.sh's live run
  are unverified against a real Docker daemon (Docker was unavailable
  throughout this milestone's implementation environment) (I6).
- The network-isolation paragraph, written when only blast-radius-sandbox
  was isolated, is now factually wrong now that metrics-watcher is also
  off the default network and prometheus is dual-homed onto
  observability-net. Extended it to describe metrics-watcher's identical
  isolation and replaced "every other service shares" with an accurate
  three-way breakdown (mcp-server: none, needs full default access;
  blast-radius-sandbox and metrics-watcher: each on their own dedicated
  network) (I7).
- Reworded the opening tagline's "observes impact in real time" — M5
  explicitly rules out live/streaming observation as a non-goal; it
  samples Prometheus once, after the fact — to "checks real impact against
  its own prediction after the fact" (M5).
@qodo-code-review

Copy link
Copy Markdown

PR Summary by Qodo

Add isolated metrics-watcher for post-fault impact verification

✨ Enhancement 🧪 Tests 📝 Documentation ⚙️ Configuration changes 🕐 40+ Minutes

Grey Divider

AI Description

• Adds a read-only MCP service comparing Prometheus observations with blast-radius predictions.
• Isolates metrics-watcher to Prometheus and integrates observation into the agent workflow.
• Adds unit and acceptance coverage for severity, validation, buffering, and isolation.
Diagram

graph TD
  P["Blast Predictor"] -->|prediction| A["Chaos Agent"] -->|post-fault call| W["Metrics Watcher"] -->|read-only queries| M[(Prometheus)]
  W --> C["Severity Classifier"] -->|comparison| V["Verdict Report"] -->|observed outcome| A
Loading
High-Level Assessment

The following are alternative approaches to this PR:

1. Extend the existing chaos MCP server
  • ➕ Avoids deploying and registering another MCP connector.
  • ➕ Could reuse existing workflow and service scaffolding.
  • ➖ Combines read-only observation with destructive Docker capabilities.
  • ➖ Weakens least-privilege isolation and increases compromise impact.
2. Pass anchored fault timestamps
  • ➕ Queries the exact fault interval regardless of post-revert delay.
  • ➕ Avoids dilution from oversized buffered windows.
  • ➖ Requires changing contracts across fault execution, prediction, and observation.
  • ➖ Introduces timestamp propagation and retention complexity beyond this milestone.

Recommendation: Keep the separate, Prometheus-only MCP service because it preserves a clear least-privilege boundary while closing the prediction loop. The buffered lookback is pragmatic for M5; explicit fault start/end timestamps are the strongest future refinement if observation precision becomes important.

Files changed (20) +3942 / -14

Enhancement (4) +225 / -0
prometheus.tsQuery per-route metrics from Prometheus +68/-0

Query per-route metrics from Prometheus

• Adds timeout-bounded Prometheus instant queries for request count, error rate, and average latency. Distinguishes no traffic and non-finite values from valid zero measurements and propagates query failures.

services/metrics-watcher/src/prometheus.ts

server.tsExpose metrics-watcher over stateless MCP HTTP +55/-0

Expose metrics-watcher over stateless MCP HTTP

• Creates health and MCP endpoints on port 3300 using stateless Streamable HTTP transport. Handles request failures without crashing the service.

services/metrics-watcher/src/server.ts

severity.tsClassify observed impact and compute verdicts +29/-0

Classify observed impact and compute verdicts

• Defines none, degraded, and hard thresholds from existing M1 error-rate criteria. Ranks affected-route severity and compares it with the predicted level.

services/metrics-watcher/src/severity.ts

tools.tsImplement the observe_impact MCP tool +73/-0

Implement the observe_impact MCP tool

• Validates predicted severity, known affected routes, and positive windows before querying all checkout-api routes. Computes severity only from affected routes and returns the prediction verdict with the complete route breakdown.

services/metrics-watcher/src/tools.ts

Tests (4) +383 / -0
verify-m5.shAdd live M5 acceptance and isolation checks +127/-0

Add live M5 acceptance and isolation checks

• Builds the stack and verifies baseline, active-fault, and buffered post-revert observations. Also tests invalid input and proves metrics-watcher can reach Prometheus but not mcp-server.

scripts/verify-m5.sh

prometheus.test.tsTest Prometheus query behavior and failures +64/-0

Test Prometheus query behavior and failures

• Covers numeric metric conversion, no-traffic nulls, HTTP errors, and unsuccessful Prometheus response bodies with injected fetch implementations.

services/metrics-watcher/test/prometheus.test.ts

severity.test.tsTest severity boundaries and verdict ordering +47/-0

Test severity boundaries and verdict ordering

• Covers threshold boundaries, no-traffic classification, worst-route aggregation, and matched, milder, or worse verdicts.

services/metrics-watcher/test/severity.test.ts

tools.test.tsTest observe_impact semantics and MCP validation +145/-0

Test observe_impact semantics and MCP validation

• Verifies affected-route scoping, complete per-route output, verdict calculation, propagated query errors, registration, and schema rejection through real in-memory MCP round trips.

services/metrics-watcher/test/tools.test.ts

Documentation (4) +1461 / -13
README.mdDocument M5 completion and security topology +37/-11

Document M5 completion and security topology

• Marks metrics-watcher complete, adds its service and acceptance-test instructions, and explains the Prometheus-only observability network. It also discloses that Docker-based validation remains outstanding in the implementation environment.

README.md

README.mdDocument metrics-watcher connector setup and verification +57/-2

Document metrics-watcher connector setup and verification

• Adds TrueForge registration and reachability guidance for the third MCP connector. Extends manual verification to confirm buffered, route-level observations match direct tool results.

agent/README.md

2026-08-27-m5-metrics-watcher.mdAdd the task-by-task M5 implementation plan +1074/-0

Add the task-by-task M5 implementation plan

• Defines the service scaffold, Prometheus client, classification logic, MCP wiring, deployment, acceptance testing, agent integration, and documentation tasks. Records constraints and verification expectations for each step.

docs/superpowers/plans/2026-08-27-m5-metrics-watcher.md

2026-08-27-m5-metrics-watcher-design.mdSpecify the M5 metrics-watcher design +293/-0

Specify the M5 metrics-watcher design

• Documents goals, non-goals, architecture, PromQL behavior, severity thresholds, tool schema, network isolation, and acceptance criteria. Explains why the capability is implemented as an MCP service rather than a declared subagent.

docs/superpowers/specs/2026-08-27-m5-metrics-watcher-design.md

Other (8) +1873 / -1
chaos-notary.jsonRequire post-fault observation in the agent workflow +6/-1

Require post-fault observation in the agent workflow

• Registers the metrics-watcher connector and teaches the agent to filter predicted targets into known routes. Makes observe_impact mandatory when applicable, uses buffered windows, and constrains claims to actual tool output.

agent/chaos-notary.json

docker-compose.ymlDeploy metrics-watcher on an isolated observability network +30/-0

Deploy metrics-watcher on an isolated observability network

• Adds the metrics-watcher service with loopback-only port publishing and health checks. Dual-homes Prometheus onto observability-net while keeping metrics-watcher separated from the destructive MCP server.

docker-compose.yml

.dockerignoreExclude generated Node artifacts from image context +3/-0

Exclude generated Node artifacts from image context

• Omits dependencies, compiled output, and logs from the metrics-watcher Docker build context.

services/metrics-watcher/.dockerignore

DockerfilePackage metrics-watcher as a production Node image +16/-0

Package metrics-watcher as a production Node image

• Adds a multi-stage Node 22 Alpine build that compiles TypeScript and installs production-only dependencies. Exposes port 3300 and starts the MCP server.

services/metrics-watcher/Dockerfile

package-lock.jsonLock metrics-watcher dependency graph +1753/-0

Lock metrics-watcher dependency graph

• Pins the MCP SDK, Zod, TypeScript tooling, and their transitive dependencies for reproducible installs.

services/metrics-watcher/package-lock.json

package.jsonDefine the metrics-watcher Node package +20/-0

Define the metrics-watcher Node package

• Adds build, start, and test scripts with MCP, schema-validation, TypeScript, and test-runtime dependencies.

services/metrics-watcher/package.json

cliCall.tsAdd a command-line MCP tool caller +30/-0

Add a command-line MCP tool caller

• Provides a short-lived Streamable HTTP client for invoking metrics-watcher tools from acceptance scripts. Returns a failing exit status for MCP tool errors.

services/metrics-watcher/src/cliCall.ts

tsconfig.jsonConfigure strict TypeScript compilation +15/-0

Configure strict TypeScript compilation

• Adds NodeNext ES2022 compilation from src into dist with strict type checking and consistent module resolution.

services/metrics-watcher/tsconfig.json

@qodo-code-review

qodo-code-review Bot commented Aug 28, 2026 •

Copy link
Copy Markdown

Code Review by Qodo

🐞 Bugs (0) 📘 Rule violations (0) 📜 Skill insights (0)

Grey Divider


Action required

1. Latency faults get false verdicts ✓ Resolved 🐞 Bug ≡ Correctness
Description
handleObserveImpact derives severity and verdict exclusively from error rate, ignoring the
measured avgLatencyMs, so a latency injection below the API's 2000ms timeout returns
none/milder_than_predicted despite the predicted and observed latency degradation. The prose
warning does not prevent the structured verdict consumed by the agent from being wrong.
Code

services/metrics-watcher/src/tools.ts[R40-42]

+  const affectedSeverities = args.affected_routes.map((route) => classifyRoute(routes[route]));
+  const observedSeverity = worstOf(affectedSeverities);
+  const verdict = computeVerdict(args.predicted_severity, observedSeverity);
Evidence
The metrics query returns average latency, but classifyRoute examines only errorRatePercent.
M4's topology explicitly predicts increased response time for degraded database latency and says
only requests exceeding the 2000ms timeout fail, which the checkout API configuration confirms.

services/metrics-watcher/src/prometheus.ts[57-65]
services/metrics-watcher/src/severity.ts[10-14]
services/blast-radius-sandbox/src/topology.ts[34-39]
services/blast-radius-sandbox/src/topology.ts[53-58]
services/checkout-api/src/server.ts[10-29]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
Observed severity and verdict ignore latency even though the tool measures it and M4 predicts latency-only degradation.

## Issue Context
Database latency below the 2000ms timeout can substantially degrade response time without causing 5xx responses. Define latency thresholds or signal-specific verdicts so the structured result reflects that observation.

## Fix Focus Areas
- services/metrics-watcher/src/tools.ts[40-42]
- services/metrics-watcher/src/severity.ts[10-14]
- services/metrics-watcher/src/prometheus.ts[57-65]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


2. API outages appear healthy ✓ Resolved 🐞 Bug ≡ Correctness
Description
When chaos-checkout-api is paused, stopped, or killed, it cannot run the onResponse hook that
increments http_requests_total, so failed requests are absent and the resulting flat or missing
series is converted to zero or null and classified as none. Because the watcher never queries an
external availability signal such as Prometheus up, a predicted hard outage across all routes is
incorrectly reported after recovery as none or milder_than_predicted.
Code

services/metrics-watcher/src/prometheus.ts[R50-53]

+
+  const errorRatePercent = await query(
+    baseUrl,
+    `100 * (sum(rate(http_requests_total{route="${route}",status=~"5.."}[${windowSeconds}s])) or vector(0)) / sum(rate(http_requests_total{route="${route}"}[${windowSeconds}s]))`,
Evidence
The only checkout-api request counter is incremented in Fastify's post-response onResponse hook,
which cannot execute while the API container is frozen, stopped, or otherwise down. Prometheus
scrapes that same service every five seconds and provides target availability, while the topology
declares checkout-api outages hard for /products, /orders, and /health; however, the
implementation reads only request and duration metric families, converts absent request data to zero
or null, and classifies null as none, producing a false healthy result.

services/checkout-api/src/server.ts[34-58]
observability/prometheus/prometheus.yml[1-8]
services/blast-radius-sandbox/src/topology.ts[66-87]
services/metrics-watcher/src/prometheus.ts[41-67]
services/metrics-watcher/src/severity.ts[10-14]
agent/chaos-notary.json[12-12]
services/blast-radius-sandbox/src/topology.ts[66-80]
services/metrics-watcher/src/prometheus.ts[41-65]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description
The observer infers severity solely from checkout-api's self-reported response counters. A process-level outage produces no application responses and therefore no counter increments; because missing or flat data is treated as no impact, the tool labels a predicted hard outage as `none` or `milder_than_predicted` after recovery.

## Issue Context
M4 and the topology model pause, stop, or kill of `chaos-checkout-api` as a hard outage for `/products`, `/orders`, and `/health`, and the agent filters those route targets before requiring `observe_impact`. Use an external availability signal—such as Prometheus `up` for checkout-api over the same experiment interval, load-generator request outcomes, or a blackbox probe—to detect unavailable scraping/API time, and represent genuinely unavailable data separately from healthy zero-error data.

## Fix Focus Areas
- services/metrics-watcher/src/prometheus.ts[41-67]
- services/metrics-watcher/src/severity.ts[10-14]
- services/metrics-watcher/src/tools.ts[35-49]
- services/metrics-watcher/test/prometheus.test.ts[29-47]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


3. Buffered window dilutes hard faults ✓ Resolved 🐞 Bug ≡ Correctness
Description
observe_impact averages the entire prescribed duration_seconds + ~60s trailing window, mixing
recovered traffic with the fault, so valid short hard replica pauses—such as 30 seconds of
/products 503s in a 90-second mixed-traffic window—fall below the 40% hard threshold and are
reported degraded/milder_than_predicted instead of matching M4. The current verification uses
only a 90-second fault with a 150-second window, where the approximately 70% affected read mix
remains just above the threshold, masking this failure for shorter allowed experiments.
Code

services/metrics-watcher/src/prometheus.ts[60]

+    fetchImpl,
Evidence
The route error rate is calculated using PromQL rate over the complete caller-provided range,
while the tool documentation directs callers to add about 60 seconds of recovered time to the fault
duration. M4 predicts pause/stop/kill faults as hard regardless of duration, but the classifier
requires at least a 40% error rate and the chaos API accepts durations from 5 through 300 seconds;
because only approximately 70% of generated requests target the affected read route, healthy
recovery traffic dominates shorter prescribed windows—for example, even a 30-second 100%-error fault
is only about 33% of a 90-second range—and changes the classification outcome.

services/metrics-watcher/src/prometheus.ts[51-55]
services/metrics-watcher/src/prometheus.ts[63-67]
agent/README.md[159-180]
services/blast-radius-sandbox/src/topology.ts[164-181]
loadgen/loadgen.mjs[39-47]
scripts/verify-m5.sh[63-80]
services/metrics-watcher/src/tools.ts[60-66]
services/metrics-watcher/src/prometheus.ts[51-65]
services/metrics-watcher/src/severity.ts[3-14]
services/mcp-server/src/tools.ts[264-264]

Agent prompt
The issue below was found during a code review. Follow the provided context and guidance below and implement a solution

## Issue description

`observe_impact` computes severity from a trailing range that includes the required post-revert recovery buffer. For short valid faults, healthy recovered traffic dilutes the 5xx percentage below the 40% hard threshold, causing hard M4 predictions to be reported as `degraded` or `milder_than_predicted` during the mandatory post-revert comparison.

## Issue Context

The agent is instructed to invoke the tool after recovery with a lookback of `duration_seconds + roughly 60 seconds`, and all PromQL rates are calculated over that complete range. Fault durations can be as short as 5 seconds, but the tool receives no fault start/end time and therefore cannot isolate the experiment interval; preserve safe post-revert invocation while making the queried interval represent the fault period, for example by accepting an explicit fault interval or end timestamp, or otherwise deriving and exposing an interval that excludes the recovery buffer. Update verification to cover shorter valid durations rather than relying only on the existing 90-second fault and 150-second window that masks the issue.

## Fix Focus Areas

- services/metrics-watcher/src/prometheus.ts[41-66]
- services/metrics-watcher/src/tools.ts[31-66]
- agent/chaos-notary.json[12-12]
- services/metrics-watcher/test/tools.test.ts[20-77]

ⓘ Copy this prompt and use it to remediate the issue with your preferred AI generation tools


Grey Divider

Context sources
Review mode: 🧠 Deep: This is a dense cross-cutting integration change spanning a new Prometheus-backed service, security/network isolation, workflow contracts, and end-to-end verification, with multiple independent seams likely to hide subtle defects.

Grey Divider

Tip of the day
💡 Did you know, you can reply 'qodo' on any finding to push back, ask questions, or dig deeper

More tips ↗ | Customize Qodo ↗ | Qodo docs ↗

Grey Divider

Qodo Logo

Comment thread services/metrics-watcher/src/tools.ts Resolved
…anchored queries (Qodo PR#5 findings 1-3)

classifyRoute only ever examined errorRatePercent, so a latency-only fault
below checkout-api's 2000ms timeout produced no 5xx errors and reported
observedSeverity "none" even when M4 predicted "degraded" (finding 1). Add
a grounded OBSERVED_HARD_LATENCY_MS=2000 escalation (matches checkout-api's
own connectionTimeoutMillis/query_timeout and M4's LATENCY_HARD_THRESHOLD_MS)
so latency at/above that threshold now forces "hard" regardless of error
rate. Deliberately no sub-2000ms "degraded latency" threshold is added —
there's no grounded constant for that in this codebase.

queryRouteMetrics built PromQL windows ending at "now" with no time anchor;
a prior fix added a fixed +60s buffer, which mathematically dilutes short
faults below the hard-error-rate threshold (a 30s fault in a 90s window is
only 33% fault-period) — the one prior regression test used a 90s fault,
masking this for shorter, equally valid experiments (finding 2). Add an
optional offsetSeconds parameter to queryRouteMetrics (and the shared query()
helper) that appends a PromQL `offset Ns` modifier to every range-vector
selector, letting a caller anchor the window to a real elapsed time instead
of guessing with a buffer.

checkout-api's only request counter increments in Fastify's onResponse hook,
which cannot fire while the process is paused/stopped/killed, so Prometheus's
scrape fails entirely during that window and queryRouteMetrics returns nulls
that classifyRoute reads as "none" — inverted for the single most common
chaos scenario (finding 3). Add checkoutApiWasDown(), which checks
Prometheus's own up{job="checkout-api"} scrape-health signal, and thread a
new apiWasDown parameter through classifyRoute: null error-rate data now
escalates to "hard" when the API was genuinely unreachable, rather than
being silently read as idle traffic.
…e_impact

Add an optional fault_ended_at (ISO-8601) input to observe_impact. When
provided (the fault tool's own expiresAt, passed verbatim), the server
computes offsetSeconds = max(0, round((Date.now() - Date.parse(fault_ended_at))
/ 1000)) using its own clock and applies it to every route-metrics query and
a single per-call checkoutApiWasDown() check, whose result now feeds every
classifyRoute() call in the affected-routes computation. This lets
window_seconds mean exactly the fault's own duration_seconds — no more
manual buffer guessing, and it works correctly for short faults, not just
long ones.

Update the tool description to document fault_ended_at and the new
hard-latency/api-outage escalations, keeping the old buffer guidance only as
a fallback for callers with no timestamp to anchor with.

Update the chaos-notary agent manifest's mandatory observe_impact workflow
step to capture the fault tool's expiresAt and pass it as fault_ended_at
with window_seconds set to exactly duration_seconds, removing the now-
obsolete "+60s buffer" instruction (kept only as an explicit fallback).
Update agent/README.md's manual verification walkthrough to match.
…escalation

severity.test.ts: avgLatencyMs at/above 2000ms classifies as hard regardless
of error rate; just below 2000ms does not escalate via latency alone (exact
boundary); null error rate + apiWasDown true classifies as hard; apiWasDown
false/omitted preserves existing "none" behavior.

prometheus.test.ts: queryRouteMetrics includes an `offset Ns` clause on every
range-vector query when offsetSeconds > 0 and omits it when 0/omitted;
checkoutApiWasDown returns true/false/false for up{} values of 0/1/no-data,
and its own query carries the offset clause correctly.

tools.test.ts: updated the fake route-metrics fetcher and every call site to
the new (route, offsetSeconds) signature, added a fake api-health fetcher to
every handleObserveImpact/registerTools call (the real default would
otherwise hit the network in tests), and added cases proving fault_ended_at
is parsed into the correct offsetSeconds and threaded through to every route
fetch plus the single per-call api-health check, that a full outage
(apiWasDown true) escalates null route metrics to hard, and that zod rejects
a malformed fault_ended_at.

19 tests -> 33 tests, all passing.
…ify-m5.sh

Add extract_expires_at(), following this script's established convention of
piping raw MCP tool JSON responses through python3 on stdin rather than
shell-interpolating JSON, to pull expiresAt out of a fault-injection tool's
response.

New case 1: pauses chaos-pg-replica for a SHORT 12s fault (well under the
existing 90s test's 60% fault-fraction), waits well past its own auto-revert
plus a real extra delay, then calls observe_impact with fault_ended_at set to
the captured expiresAt and window_seconds equal to the fault's own exact
duration_seconds — asserts observedSeverity stays "hard"/"matched" despite
the real elapsed delay. This is the regression case the prior fixed +60s
buffer could not have passed for a fault this short (its own regression test
only covered a 90s fault).

New case 2: pauses chaos-checkout-api itself (a full API outage, not just a
DB dependency) and verifies observe_impact reports "hard" for the routes
M4's topology predicts as hard-affected (/products, /orders, /health), even
though checkout-api's own request counters cannot increment while the
process is paused — proving the up{job="checkout-api"} scrape-health
escalation actually works.

Existing baseline/mid-fault/buffered-window assertions are unchanged; they
still validate the no-fault_ended_at fallback path.

Docker is unavailable in this environment, so this script could not be run
live as part of this fix — bash -n syntax-checked only.
@mikelord007

Copy link
Copy Markdown
Owner Author

@qodo Fixed in 90e5f6b. classifyRoute now escalates to "hard" whenever avgLatencyMs >= 2000ms (a new OBSERVED_HARD_LATENCY_MS constant, matching checkout-api's own real connectionTimeoutMillis/query_timeout and M4's own latency threshold), checked independently of error rate — so a latency-only fault at or above the timeout is now correctly reflected in the structured verdict, not just in prose. I deliberately did not invent a sub-2000ms "degraded latency" threshold, since no real constant in this codebase grounds one — that remains a disclosed limitation (visible via the raw avgLatencyMs field, documented in the tool description), not silently papered over with an arbitrary number. New boundary tests confirm 1999ms stays governed by error rate and 2000ms exactly triggers hard.

@qodo-code-review

Copy link
Copy Markdown

Fixed in 90e5f6b. classifyRoute now escalates to "hard" whenever avgLatencyMs >= 2000ms (a new OBSERVED_HARD_LATENCY_MS constant, matching checkout-api's own real connectionTimeoutMillis/query_timeout and M4's own latency threshold), checked independently of error rate — so a latency-only fault at or above the timeout is now correctly reflected in the structured verdict, not just in prose. I deliberately did not invent a sub-2000ms "degraded latency" threshold, since no real constant in this codebase grounds one — that remains a disclosed limitation (visible via the raw avgLatencyMs field, documented in the tool description), not silently papered over with an arbitrary number. New boundary tests confirm 1999ms stays governed by error rate and 2000ms exactly triggers hard.

Commit 90e5f6b addresses the timeout-level case in finding 1: classifyRoute now checks avgLatencyMs >= 2000 independently of error rate, and the boundary tests cover both 1999ms and 2000ms (services/metrics-watcher/src/severity.ts:21-26, test/severity.test.ts:49-59).

The remaining limitation is explicit and consistent: latency-only degradation below 2000ms still does not affect the structured severity/verdict, so those cases remain visible only through routes[*].avgLatencyMs.

Identical bug to mcp-server/blast-radius-sandbox (fixed in #6, merged into
this branch): one shared McpServer+transport pair at startup made every
/mcp request after the first fail with a bare 500, since the installed SDK
enforces "one transport per request" in stateless mode. Fixed the same way:
a fresh McpServer + StreamableHTTPServerTransport pair per request.

Verified against a live stack for the first time ever: bash
scripts/verify-m5.sh passes end to end, including the two scenarios Qodo's
review specifically flagged as broken (short-fault offset-anchoring via
fault_ended_at, and chaos-checkout-api full-outage detection via Prometheus's
up{} scrape-health signal) - both previously verified only by code reading
and unit tests, now confirmed with real data from a real fault injection.
Unit tests unchanged (33/33 pass).
@mikelord007

Copy link
Copy Markdown
Owner Author

@qodo Update: this branch got its first ever live Docker verification this session (Docker was unavailable in the original development environment for the entire project, disclosed as a known gap in every PR through #4 — that's now fixed).

Running the full stack live surfaced a new Critical bug beyond the 3 findings already fixed on this PR: metrics-watcher/src/server.ts (like mcp-server and blast-radius-sandbox) created one McpServer + StreamableHTTPServerTransport pair at startup and reused it for the process's entire lifetime. The installed @modelcontextprotocol/sdk version enforces "one transport per request" in stateless mode at two levels — the transport itself throws "Stateless transport cannot be reused across requests" on its second call, and Server.connect() throws "Already connected to a transport" if the same McpServer is still attached to a prior transport. Result: the first /mcp request a freshly-started container ever received succeeded, and every request after that failed with a bare, empty-body 500. Every unit test in this project uses InMemoryTransport.createLinkedPair() (a fresh in-process pair per test) — structurally incapable of catching this, since it only manifests over real repeated HTTP calls to a long-running server.

Fixed in a5f4eb7 by creating a fresh McpServer + transport pair per incoming /mcp request (registerTools() is cheap — zod schemas and closures, no I/O). The identical fix for mcp-server/blast-radius-sandbox already merged separately in #6.

This branch also merged in #6's other fixes (Pumba image tag, checkout-api healthcheck, verify-m1.sh auth) via a merge from main, so the full stack now builds and runs cleanly end to end.

Full live verification, all real, all passing:

  • docker compose up -d --build: all 9 services healthy.
  • bash scripts/verify-m1.sh, verify-m2.sh, verify-m4.sh: all pass against a live daemon.
  • bash scripts/verify-m5.sh: passes end to end, including the exact two scenarios your findings M2: MCP server exposing chaos tools #2 and M3: Agent + approval gates #3 flagged — the short-fault (12s) offset-anchored fault_ended_at case still correctly reports hard/matched despite real elapsed delay, and a real chaos-checkout-api pause is correctly detected as hard via the up{job="checkout-api"} scrape-health signal (not via request counters, which are genuinely absent while the API process is frozen) — both previously verified only by code reading, now confirmed with real fault-injection data. Network isolation confirmed at the DNS level (wget: bad address 'mcp-server:3100' — metrics-watcher can't even resolve mcp-server, let alone connect to it), while it correctly reaches prometheus.
  • services/metrics-watcher: npm test 33/33 pass (unchanged).

@qodo-code-review

Copy link
Copy Markdown

Update: this branch got its first ever live Docker verification this session (Docker was unavailable in the original development environment for the entire project, disclosed as a known gap in every PR through #4 — that's now fixed).

Running the full stack live surfaced a new Critical bug beyond the 3 findings already fixed on this PR: metrics-watcher/src/server.ts (like mcp-server and blast-radius-sandbox) created one McpServer + StreamableHTTPServerTransport pair at startup and reused it for the process's entire lifetime. The installed @modelcontextprotocol/sdk version enforces "one transport per request" in stateless mode at two levels — the transport itself throws "Stateless transport cannot be reused across requests" on its second call, and Server.connect() throws "Already connected to a transport" if the same McpServer is still attached to a prior transport. Result: the first /mcp request a freshly-started container ever received succeeded, and every request after that failed with a bare, empty-body 500. Every unit test in this project uses InMemoryTransport.createLinkedPair() (a fresh in-process pair per test) — structurally incapable of catching this, since it only manifests over real repeated HTTP calls to a long-running server.

Fixed in a5f4eb7 by creating a fresh McpServer + transport pair per incoming /mcp request (registerTools() is cheap — zod schemas and closures, no I/O). The identical fix for mcp-server/blast-radius-sandbox already merged separately in #6.

This branch also merged in #6's other fixes (Pumba image tag, checkout-api healthcheck, veri******sh auth) via a merge from main, so the full stack now builds and runs cleanly end to end.

Full live verification, all real, all passing:

  • docker compose up -d --build: all 9 services healthy.
  • bash scripts/verify-m1.sh, verify-m2.sh, verify-m4.sh: all pass against a live daemon.
  • bash scripts/verify-m5.sh: passes end to end, including the exact two scenarios your findings M2: MCP server exposing chaos tools #2 and M3: Agent + approval gates #3 flagged — the short-fault (12s) offset-anchored fault_ended_at case still correctly reports hard/matched despite real elapsed delay, and a real chaos-checkout-api pause is correctly detected as hard via the up{job="checkout-api"} scrape-health signal (not via request counters, which are genuinely absent while the API process is frozen) — both previously verified only by code reading, now confirmed with real fault-injection data. Network isolation confirmed at the DNS level (wget: bad address 'mcp-server:3100' — metrics-watcher can't even resolve mcp-server, let alone connect to it), while it correctly reaches prometheus.
  • services/metrics-watcher: npm test 33/33 pass (unchanged).

The live verification closes the main remaining confidence gap: the full stack and verify-m5.sh now pass against Docker, including the scenarios behind finding 2 and finding 3.

  • The short-fault fault_ended_at path correctly reports hard/matched after real elapsed delay.
  • A paused checkout-api is correctly classified as hard through up{job="checkout-api"}, despite missing request-counter data.
  • Repeated HTTP calls now work because server.ts creates a fresh McpServer and transport per request; the previous stateless-transport lifecycle bug is fixed in a5f4eb7.
  • Network isolation and Prometheus reachability were both verified live, and all 33 metrics-watcher tests pass.

The findings index still lists finding 1 as active, so it remains formally open here even though the current diff includes latency-based hard classification. I have not dismissed it because no explicit dismissal request was made.

@mikelord007
mikelord007 merged commit 4fade77 into main Aug 28, 2026
1 check passed
@mikelord007
mikelord007 deleted the feat/m5-metrics-watcher branch August 28, 2026 08:19
mikelord007 added a commit that referenced this pull request Aug 28, 2026
…ication fixes)

Also updates the M5 status bullet to reflect that scripts/verify-m5.sh now
passes end to end against a real Docker daemon, superseding the "unverified"
caveat written before Docker became available in this environment.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant