Skip to content

feat(worker): publish doctor results per host, shown in the console and panel (#712) - #723

Merged
edgehero merged 1 commit into
mainfrom
feat/doctor-results-712c
Oct 21, 2026
Merged

edgehero merged 1 commit into
mainfrom
feat/doctor-results-712c

Conversation

@edgehero

Copy link
Copy Markdown
Owner

Part 3 of #712. Part 1 (#722) is merged. The console used to say "run pi-dispatch doctor for the reason". It now shows each host's latest doctor result.

How results are published

  • Every doctor run publishes. Each pi-dispatch doctor run, including the doctor step of up, writes one record per host to doctor:h:<host>. It is JSON, written with SET PX for 7 days, and indexed in doctor:hosts.
    • Publishing is best-effort. It never changes doctor's exit code or its stdout.
    • --no-publish turns it off.
  • The worker reruns doctor. It spawns doctor a minute after boot, then every PI_DOCTOR_INTERVAL_HOURS (default 24, 0 turns it off, maximum 168).
    • It never passes --live, --secrets or --fix, and runs never overlap.
    • It kills the doctor process group at 120 s, and at shutdown.
  • The record says who ran it (by: { uid, worker }), so a sudo run is labelled and cannot pass for the worker's own.
  • Nothing reads doctor:* to decide anything, so deleting those keys changes nothing.

What a published check contains

A passing check is just its id and its level. Failing and warning checks also carry a label and a fix, redacted before the length cap:

  • paths, file URLs, scp-style and UNC forms become <path>;
  • a URL keeps only its scheme, host and port;
  • the input that a JSON.parse error quotes becomes <content>;
  • these all become [redacted]: credential query parameters, assignments, flags and headers; the value of any secret-named env variable; known token shapes;
  • invisible characters are deleted first, so they cannot split a token.

Tests seed a provider key and a Valkey password and check that neither appears in the published record.

What the console and panel show

  • Fleet page: a new Doctor panel shows, per host, its failing checks first. It marks a stale result, and names a live host that has never published.
  • "Does not load" sentences quote this host's own check first, and another host's only when this host has none.
  • /dispatch hosts view: shows the same doctor lines.

Specs

  • New: DES-DOCTOR-RESULTS-PUBLISHED, INT-DOCTOR-RESULTS-KEY.
  • Amended: DES-OPERATOR-CONSOLE; DES-CLI-SURFACE, since doctor now makes one write, its own results; INT-HOST-REGISTRY-CONTRACT; REQ-OPERATOR-CONSOLE-SCREENS; REQ-DEPLOYMENT-BOOTSTRAP.
  • UNCHANGED, checked: REQ-CONSOLE-SECURITY, REQ-ADMIN-VIA-PI-EXTENSION, DES-RETENTION-SWEEPS-ON-A-TIMER, INT-CONSOLE-STATE-API.

Residuals (named in the DES entry)

  • Fullwidth slashes and relative paths are kept as written.
  • The user name in an scp-style path is kept.
  • A doctor child started before a worker crash outlives the worker until its own time limits stop it.

Verification

  • Review: two adversarial rounds, every finding fixed.
  • Full suite with Valkey: 9628 tests, 2 failures, both known (the macOS mcp test, and a service.test.mjs SIGTERM timing test under load).
  • Build and checks: node admin/build.mjs builds, and the revision-row, test-count, temp-dir and dated-fixture checks pass.

…nd panel (#712)

Part 3 of #712. The console used to say "run pi-dispatch doctor for the
reason". It now shows each host's latest doctor result.

- Every `pi-dispatch doctor` run, including up's doctor step, publishes one
  record per host to doctor:h:<host> (JSON, SET PX 7 days) and indexes
  it in doctor:hosts. Publishing is best-effort: it never changes doctor's
  exit code or stdout. --no-publish turns it off.
- A published check is its id and level. Only warn and fail checks carry a
  label and fix, both redacted before the length cap:
  - paths, file URLs, scp and UNC forms become <path>;
  - URLs keep only scheme, host and port;
  - JSON.parse snippets become <content>;
  - credential query parameters, assignments, flags and headers, any
    secret-named env value and known token shapes become [redacted];
  - invisible characters are deleted first.
- The record says who ran it (by: { uid, worker }), and records the real
  interval, so a sudo run is labelled and cannot pass as the worker's.
- The worker spawns `doctor --by-worker=<hours>` a minute after boot and
  every PI_DOCTOR_INTERVAL_HOURS (default 24, 0 off, max 168). It never
  passes --live, --secrets or --fix, never overlaps runs, and kills the
  process group at 120 s and on shutdown.
- Console: a Doctor panel on the Fleet page, listing failing checks first.
  It marks a stale record and names a live host that never published. The
  "does not load" sentences quote this host's own check first.
- The /dispatch hosts view shows the same doctor lines.
- Nothing reads doctor:* to decide anything, so deleting it changes
  nothing.

Specs: DES-DOCTOR-RESULTS-PUBLISHED and INT-DOCTOR-RESULTS-KEY are new.
DES-OPERATOR-CONSOLE, DES-CLI-SURFACE (doctor's single write),
INT-HOST-REGISTRY-CONTRACT, REQ-OPERATOR-CONSOLE-SCREENS and
REQ-DEPLOYMENT-BOOTSTRAP are amended. UNCHANGED, checked:
REQ-CONSOLE-SECURITY, REQ-ADMIN-VIA-PI-EXTENSION,
DES-RETENTION-SWEEPS-ON-A-TIMER and INT-CONSOLE-STATE-API.

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
edgehero merged commit f204208 into main Oct 21, 2026
8 of 10 checks passed
edgehero added a commit that referenced this pull request Oct 21, 2026
… frame (#725)

The copy-mode steering case ran with maxCostMicros 0 and a real model, so
the runner hit its cost cap within milliseconds of the first ack. The
worker ends the exec's stdin on the verified exit line, so the second
message's frame was never written, and the case failed whenever the
relay had not written it first: on the #723 rerun and on #724.

The case now runs on a zero-rated, keyless fake model served by the
stand-in (POST /v1/chat/completions, streamed):
- The first request is held until the conformance script signals, by
  touching STEER_SIGNAL_FILE, that the relay has written the second
  frame. It waits at most 30 s. It then asks for a `read` of the job's
  prompt file.
- Any request carrying a tool result answers "done".
- The reply is chosen from the request body, not a counter, so a retry of
  the first request is held again. The body is never logged, and every
  other route answers exactly as before.

The assertion stays strict: exit 0 with a verified exit line naming both
messages, both delivered, at least 2 frames written and 2 acks verified,
no cost refusal, and a cost of 0.

Rejected: a relay change that reruns a tick requested mid-pass. Nothing
in production ticks from a mark, the failure was the closed sink, and in
production it would only add back-to-back Valkey passes when Valkey is
slow.

Specs: DES-OPENSHELL-VENUE (the CI bullet, with the residual that this is
the first model stream through OpenShell 0.1.4's rest layer).
UNCHANGED, checked: DES-STEER-VIA-READONLY-INBOX-MOUNT and
INT-STEER-INBOX-CONTRACT.

Signed-off-by: Rob Boerman <robboerman@live.nl>
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant