Repository navigation
M5: Metrics-watcher - #5
Conversation
Task 8 brief specified chmod +x scripts/verify-m5.sh, but the file was committed with mode 100644. core.fileMode=false in this repo means a filesystem chmod alone doesn't update the git index, so this uses git update-index --chmod=+x to fix the tracked mode.
…dow (C1, C2) The mandatory observe_impact workflow step reliably failed for 3 of 5 allowlisted containers: predict_blast_radius's `affected` list includes non-route targets (Grafana dashboard descriptions, Prometheus scrape/query descriptions) that don't strip down to a known route, and zod's fixed route enum (kept as-is, deliberately) rejects the whole call outright. agent/chaos-notary.json's workflow now filters `affected` down to route-shaped targets before calling observe_impact, and skips the call entirely (with a plain explanation) when no route-shaped target remains. Separately, observe_impact had no time anchor: a window_seconds equal to just the fault's duration_seconds risks landing entirely in the already-recovered period once real reasoning/tool-call delay is accounted for. The workflow now asks for duration_seconds + ~60s of buffer instead. scripts/verify-m5.sh gains a regression assertion that calls observe_impact well after the fault has reverted, with a buffered window, and asserts it still reports "hard"/"matched" — the test that would have caught this. Bundled in the same files: the observe_impact tool description and the agent's own capability bullet now note observedSeverity/verdict are error-rate-only (I3), and verify-m5.sh gains a positive-control wget check before its network-isolation negative control so that assertion can't false-pass on a broken container/exec rather than real isolation (I4). agent/README.md's manual verification walkthrough is updated to match.
… values (I1, I2, M3)
- Wrap the avgLatencyMs PromQL query's numerator and denominator in sum(),
matching the error-rate/request-count queries. http_request_duration_seconds
is labeled by both route and status, so during a fault the un-summed query
returned multiple series and result[0] picked an arbitrary status code's
latency rather than an aggregate (I1).
- Add a 5s AbortSignal.timeout to the Prometheus fetch call, mirroring
verify-m1.sh's curl --max-time 5, so a paused chaos-prometheus (an
allowlisted target) can't hang the tool indefinitely (I2). Verified the
test fixtures' single-argument fake fetchers remain structurally
compatible with typeof fetch under strict mode; no test changes needed.
- Guard against Prometheus returning non-finite value strings ("NaN",
"+Inf"): check Number.isFinite before returning, falling back to the
same null "no data" semantics instead of letting a NaN silently become
a contradictory "degraded" classification downstream (M3).
The fake fetcher in tools.test.ts ignored its `route` argument and always returned the same metrics object, so no test could actually verify that observedSeverity is scoped to affected_routes only, or that the response's routes field reports all 4 known routes' real metrics rather than just the affected ones. Replaced it with a dispatch-on-route helper and added: - a scoping test (an unrelated route failing hard must not affect the verdict for a different, healthy affected_routes selection) - an all-4-routes test (routes field always has real per-route data) - a worse_than_predicted end-to-end case - an explicit error-propagation test through the real registerTools/McpServer/Client round trip Test count in this file: 2 -> 6 (net +4, matching the fix wave's target of 15 -> ~19 across the whole suite).
…tagline (I6, I7, M5) - M5's status bullet now discloses the same class of gap M3's already does: unit suite passes and the service typechecks cleanly, but the Docker image build, compose network wiring, and verify-m5.sh's live run are unverified against a real Docker daemon (Docker was unavailable throughout this milestone's implementation environment) (I6). - The network-isolation paragraph, written when only blast-radius-sandbox was isolated, is now factually wrong now that metrics-watcher is also off the default network and prometheus is dual-homed onto observability-net. Extended it to describe metrics-watcher's identical isolation and replaced "every other service shares" with an accurate three-way breakdown (mcp-server: none, needs full default access; blast-radius-sandbox and metrics-watcher: each on their own dedicated network) (I7). - Reworded the opening tagline's "observes impact in real time" — M5 explicitly rules out live/streaming observation as a non-goal; it samples Prometheus once, after the fact — to "checks real impact against its own prediction after the fact" (M5).
PR Summary by QodoAdd isolated metrics-watcher for post-fault impact verification
AI Description
Diagram
High-Level Assessment
Files changed (20)
|
Code Review by Qodo
1.
|
…anchored queries (Qodo PR#5 findings 1-3)
classifyRoute only ever examined errorRatePercent, so a latency-only fault
below checkout-api's 2000ms timeout produced no 5xx errors and reported
observedSeverity "none" even when M4 predicted "degraded" (finding 1). Add
a grounded OBSERVED_HARD_LATENCY_MS=2000 escalation (matches checkout-api's
own connectionTimeoutMillis/query_timeout and M4's LATENCY_HARD_THRESHOLD_MS)
so latency at/above that threshold now forces "hard" regardless of error
rate. Deliberately no sub-2000ms "degraded latency" threshold is added —
there's no grounded constant for that in this codebase.
queryRouteMetrics built PromQL windows ending at "now" with no time anchor;
a prior fix added a fixed +60s buffer, which mathematically dilutes short
faults below the hard-error-rate threshold (a 30s fault in a 90s window is
only 33% fault-period) — the one prior regression test used a 90s fault,
masking this for shorter, equally valid experiments (finding 2). Add an
optional offsetSeconds parameter to queryRouteMetrics (and the shared query()
helper) that appends a PromQL `offset Ns` modifier to every range-vector
selector, letting a caller anchor the window to a real elapsed time instead
of guessing with a buffer.
checkout-api's only request counter increments in Fastify's onResponse hook,
which cannot fire while the process is paused/stopped/killed, so Prometheus's
scrape fails entirely during that window and queryRouteMetrics returns nulls
that classifyRoute reads as "none" — inverted for the single most common
chaos scenario (finding 3). Add checkoutApiWasDown(), which checks
Prometheus's own up{job="checkout-api"} scrape-health signal, and thread a
new apiWasDown parameter through classifyRoute: null error-rate data now
escalates to "hard" when the API was genuinely unreachable, rather than
being silently read as idle traffic.
…e_impact Add an optional fault_ended_at (ISO-8601) input to observe_impact. When provided (the fault tool's own expiresAt, passed verbatim), the server computes offsetSeconds = max(0, round((Date.now() - Date.parse(fault_ended_at)) / 1000)) using its own clock and applies it to every route-metrics query and a single per-call checkoutApiWasDown() check, whose result now feeds every classifyRoute() call in the affected-routes computation. This lets window_seconds mean exactly the fault's own duration_seconds — no more manual buffer guessing, and it works correctly for short faults, not just long ones. Update the tool description to document fault_ended_at and the new hard-latency/api-outage escalations, keeping the old buffer guidance only as a fallback for callers with no timestamp to anchor with. Update the chaos-notary agent manifest's mandatory observe_impact workflow step to capture the fault tool's expiresAt and pass it as fault_ended_at with window_seconds set to exactly duration_seconds, removing the now- obsolete "+60s buffer" instruction (kept only as an explicit fallback). Update agent/README.md's manual verification walkthrough to match.
…escalation
severity.test.ts: avgLatencyMs at/above 2000ms classifies as hard regardless
of error rate; just below 2000ms does not escalate via latency alone (exact
boundary); null error rate + apiWasDown true classifies as hard; apiWasDown
false/omitted preserves existing "none" behavior.
prometheus.test.ts: queryRouteMetrics includes an `offset Ns` clause on every
range-vector query when offsetSeconds > 0 and omits it when 0/omitted;
checkoutApiWasDown returns true/false/false for up{} values of 0/1/no-data,
and its own query carries the offset clause correctly.
tools.test.ts: updated the fake route-metrics fetcher and every call site to
the new (route, offsetSeconds) signature, added a fake api-health fetcher to
every handleObserveImpact/registerTools call (the real default would
otherwise hit the network in tests), and added cases proving fault_ended_at
is parsed into the correct offsetSeconds and threaded through to every route
fetch plus the single per-call api-health check, that a full outage
(apiWasDown true) escalates null route metrics to hard, and that zod rejects
a malformed fault_ended_at.
19 tests -> 33 tests, all passing.
…ify-m5.sh
Add extract_expires_at(), following this script's established convention of
piping raw MCP tool JSON responses through python3 on stdin rather than
shell-interpolating JSON, to pull expiresAt out of a fault-injection tool's
response.
New case 1: pauses chaos-pg-replica for a SHORT 12s fault (well under the
existing 90s test's 60% fault-fraction), waits well past its own auto-revert
plus a real extra delay, then calls observe_impact with fault_ended_at set to
the captured expiresAt and window_seconds equal to the fault's own exact
duration_seconds — asserts observedSeverity stays "hard"/"matched" despite
the real elapsed delay. This is the regression case the prior fixed +60s
buffer could not have passed for a fault this short (its own regression test
only covered a 90s fault).
New case 2: pauses chaos-checkout-api itself (a full API outage, not just a
DB dependency) and verifies observe_impact reports "hard" for the routes
M4's topology predicts as hard-affected (/products, /orders, /health), even
though checkout-api's own request counters cannot increment while the
process is paused — proving the up{job="checkout-api"} scrape-health
escalation actually works.
Existing baseline/mid-fault/buffered-window assertions are unchanged; they
still validate the no-fault_ended_at fallback path.
Docker is unavailable in this environment, so this script could not be run
live as part of this fix — bash -n syntax-checked only.
|
@qodo Fixed in 90e5f6b. classifyRoute now escalates to "hard" whenever avgLatencyMs >= 2000ms (a new OBSERVED_HARD_LATENCY_MS constant, matching checkout-api's own real connectionTimeoutMillis/query_timeout and M4's own latency threshold), checked independently of error rate — so a latency-only fault at or above the timeout is now correctly reflected in the structured verdict, not just in prose. I deliberately did not invent a sub-2000ms "degraded latency" threshold, since no real constant in this codebase grounds one — that remains a disclosed limitation (visible via the raw avgLatencyMs field, documented in the tool description), not silently papered over with an arbitrary number. New boundary tests confirm 1999ms stays governed by error rate and 2000ms exactly triggers hard. |
Commit The remaining limitation is explicit and consistent: latency-only degradation below |
Identical bug to mcp-server/blast-radius-sandbox (fixed in #6, merged into this branch): one shared McpServer+transport pair at startup made every /mcp request after the first fail with a bare 500, since the installed SDK enforces "one transport per request" in stateless mode. Fixed the same way: a fresh McpServer + StreamableHTTPServerTransport pair per request. Verified against a live stack for the first time ever: bash scripts/verify-m5.sh passes end to end, including the two scenarios Qodo's review specifically flagged as broken (short-fault offset-anchoring via fault_ended_at, and chaos-checkout-api full-outage detection via Prometheus's up{} scrape-health signal) - both previously verified only by code reading and unit tests, now confirmed with real data from a real fault injection. Unit tests unchanged (33/33 pass).
|
@qodo Update: this branch got its first ever live Docker verification this session (Docker was unavailable in the original development environment for the entire project, disclosed as a known gap in every PR through #4 — that's now fixed). Running the full stack live surfaced a new Critical bug beyond the 3 findings already fixed on this PR: Fixed in a5f4eb7 by creating a fresh This branch also merged in #6's other fixes (Pumba image tag, checkout-api healthcheck, verify-m1.sh auth) via a merge from Full live verification, all real, all passing:
|
The live verification closes the main remaining confidence gap: the full stack and
The findings index still lists finding 1 as active, so it remains formally open here even though the current diff includes latency-based hard classification. I have not dismissed it because no explicit dismissal request was made. |
…ication fixes) Also updates the M5 status bullet to reflect that scripts/verify-m5.sh now passes end to end against a real Docker daemon, superseding the "unverified" caveat written before Docker became available in this environment.
Summary
Adds
services/metrics-watcher, a sealed, read-only MCP service that reports what Prometheus actually recorded after a chaos fault reverts and compares it against M4's prediction — closing the predict-then-observe loop this project's README has claimed since M1 but never built.observe_impactMCP tool: takes(predicted_severity, affected_routes, window_seconds, fault_ended_at?), queries Prometheus for checkout-api's real per-route error rate/latency, and returns{ observedSeverity, verdict, routes }— an explicit "matched" / "milder_than_predicted" / "worse_than_predicted" comparison against the prediction, not just raw numbers.observability-netCompose network, shared only with a dual-homedprometheus, so it cannot reachmcp-server's unauthenticated destructive endpoint even if compromised — confirmed at the DNS level in live testing (metrics-watchercan't even resolvemcp-server).hardregardless of error rate), and escalates tohardwhen Prometheus's ownup{job="checkout-api"}signal shows the API itself was unreachable — distinguishing a genuine outage from "no traffic."fault_ended_at(the fault tool's ownexpiresAttimestamp, passed through verbatim), so callingobserve_impactany real amount of time after a fault reverts still correctly measures the fault's actual window via a server-computed PromQLoffset, not a fragile fixed buffer.mcp_servers[]connector; the experiment workflow's final step is now mandatory — callobserve_impactand report the real outcome instead of stopping at a prediction.scripts/verify-m5.sh: brings up the full stack and asserts baseline "none" severity, a real pause-triggered fault reaching "hard"/"matched" mid-fault, after a buffered post-revert delay, and via offset-anchoring on a short fault; invalid-input rejection; a realchaos-checkout-apioutage correctly detected as "hard"; and the mcp-server network-isolation guard (with a positive control proving the check itself is meaningful).Process notes
Built via subagent-driven development: 9 tasks + final step, each with a fresh implementer + task reviewer. A final whole-branch review (Opus) then found 2 Critical + 7 Important + 7 Minor issues that per-task review couldn't have caught (agent-instructions/schema mismatch, an unanchored query window), all independently reproduced and fixed in one wave, re-reviewed clean.
Qodo's review on this PR then found 3 more real bugs (all High/Correctness): latency-only faults getting a false verdict, the query-window buffer having a genuine mathematical flaw for short faults, and
chaos-checkout-apioutages being misreported as healthy since the API's own request counters can't increment while it's frozen. Fixed properly (not with another heuristic) by discovering mid-verification that M2's fault tools already return a realexpiresAttimestamp —observe_impactnow uses that for genuine PromQL time-anchoring, plus a newup{}-based outage check. All 3 independently re-verified, confirmed by Qodo.This session also got Docker working for the first time in this project's history (every prior PR disclosed live verification as an unrun gap). Running the full stack live immediately surfaced a 4th, previously invisible Critical bug:
metrics-watcher'sserver.ts(likemcp-server's andblast-radius-sandbox's) shared oneMcpServer+transport pair across the process's whole lifetime — the installed SDK enforces "one transport per request" in stateless mode, so the first/mcprequest ever received succeeded and every request after that failed with a bare 500. No unit test could ever have caught this (they all use an in-processInMemoryTransportpair, not real repeated HTTP calls). Fixed by creating a fresh transport+server pair per request; the identical fix formcp-server/blast-radius-sandboxmerged separately in #6 and is included here via a merge frommain.Live verification (new — first time ever for this project)
docker compose up -d --build: all 9 services healthy.bash scripts/verify-m1.sh,verify-m2.sh,verify-m4.sh: all pass against a live daemon (see fix: repair real bugs found by first live Docker verification #6).bash scripts/verify-m5.sh: passes end to end, including the exact scenarios Qodo's review flagged — the short-fault offset-anchored case and thechaos-checkout-apioutage case, both now confirmed with real fault-injection data, not just code reading.services/metrics-watcher:npm test33/33 pass.Known gaps
Test plan
services/metrics-watcher:npm run build(zero TS errors),npm test(33/33 pass)bash -n scripts/verify-m5.sh(syntax valid)node -e "require('./agent/chaos-notary.json')"+ the established no-overclaim guard (both pass)docker compose up -d --build+bash scripts/verify-m5.shagainst a real Docker daemon — passesobserve_impactis actually called and its real verdict shapes the closing report (same manual step M3/M4 already flagged as pending)