Repository navigation
feat(worker): publish doctor results per host, shown in the console and panel (#712) - #723
Merged
Merged
Conversation
…nd panel (#712) Part 3 of #712. The console used to say "run pi-dispatch doctor for the reason". It now shows each host's latest doctor result. - Every `pi-dispatch doctor` run, including up's doctor step, publishes one record per host to doctor:h:<host> (JSON, SET PX 7 days) and indexes it in doctor:hosts. Publishing is best-effort: it never changes doctor's exit code or stdout. --no-publish turns it off. - A published check is its id and level. Only warn and fail checks carry a label and fix, both redacted before the length cap: - paths, file URLs, scp and UNC forms become <path>; - URLs keep only scheme, host and port; - JSON.parse snippets become <content>; - credential query parameters, assignments, flags and headers, any secret-named env value and known token shapes become [redacted]; - invisible characters are deleted first. - The record says who ran it (by: { uid, worker }), and records the real interval, so a sudo run is labelled and cannot pass as the worker's. - The worker spawns `doctor --by-worker=<hours>` a minute after boot and every PI_DOCTOR_INTERVAL_HOURS (default 24, 0 off, max 168). It never passes --live, --secrets or --fix, never overlaps runs, and kills the process group at 120 s and on shutdown. - Console: a Doctor panel on the Fleet page, listing failing checks first. It marks a stale record and names a live host that never published. The "does not load" sentences quote this host's own check first. - The /dispatch hosts view shows the same doctor lines. - Nothing reads doctor:* to decide anything, so deleting it changes nothing. Specs: DES-DOCTOR-RESULTS-PUBLISHED and INT-DOCTOR-RESULTS-KEY are new. DES-OPERATOR-CONSOLE, DES-CLI-SURFACE (doctor's single write), INT-HOST-REGISTRY-CONTRACT, REQ-OPERATOR-CONSOLE-SCREENS and REQ-DEPLOYMENT-BOOTSTRAP are amended. UNCHANGED, checked: REQ-CONSOLE-SECURITY, REQ-ADMIN-VIA-PI-EXTENSION, DES-RETENTION-SWEEPS-ON-A-TIMER and INT-CONSOLE-STATE-API. Signed-off-by: Rob Boerman <robboerman@live.nl>
This was referenced Oct 21, 2026
edgehero
added a commit
that referenced
this pull request
Oct 21, 2026
… frame (#725) The copy-mode steering case ran with maxCostMicros 0 and a real model, so the runner hit its cost cap within milliseconds of the first ack. The worker ends the exec's stdin on the verified exit line, so the second message's frame was never written, and the case failed whenever the relay had not written it first: on the #723 rerun and on #724. The case now runs on a zero-rated, keyless fake model served by the stand-in (POST /v1/chat/completions, streamed): - The first request is held until the conformance script signals, by touching STEER_SIGNAL_FILE, that the relay has written the second frame. It waits at most 30 s. It then asks for a `read` of the job's prompt file. - Any request carrying a tool result answers "done". - The reply is chosen from the request body, not a counter, so a retry of the first request is held again. The body is never logged, and every other route answers exactly as before. The assertion stays strict: exit 0 with a verified exit line naming both messages, both delivered, at least 2 frames written and 2 acks verified, no cost refusal, and a cost of 0. Rejected: a relay change that reruns a tick requested mid-pass. Nothing in production ticks from a mark, the failure was the closed sink, and in production it would only add back-to-back Valkey passes when Valkey is slow. Specs: DES-OPENSHELL-VENUE (the CI bullet, with the residual that this is the first model stream through OpenShell 0.1.4's rest layer). UNCHANGED, checked: DES-STEER-VIA-READONLY-INBOX-MOUNT and INT-STEER-INBOX-CONTRACT. Signed-off-by: Rob Boerman <robboerman@live.nl>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Part 3 of #712. Part 1 (#722) is merged. The console used to say "run pi-dispatch doctor for the reason". It now shows each host's latest doctor result.
How results are published
pi-dispatch doctorrun, including the doctor step ofup, writes one record per host todoctor:h:<host>. It is JSON, written withSET PXfor 7 days, and indexed indoctor:hosts.--no-publishturns it off.PI_DOCTOR_INTERVAL_HOURS(default 24, 0 turns it off, maximum 168).--live,--secretsor--fix, and runs never overlap.by: { uid, worker }), so a sudo run is labelled and cannot pass for the worker's own.doctor:*to decide anything, so deleting those keys changes nothing.What a published check contains
A passing check is just its id and its level. Failing and warning checks also carry a label and a fix, redacted before the length cap:
<path>;<content>;[redacted]: credential query parameters, assignments, flags and headers; the value of any secret-named env variable; known token shapes;Tests seed a provider key and a Valkey password and check that neither appears in the published record.
What the console and panel show
/dispatchhosts view: shows the same doctor lines.Specs
DES-DOCTOR-RESULTS-PUBLISHED,INT-DOCTOR-RESULTS-KEY.DES-OPERATOR-CONSOLE;DES-CLI-SURFACE, since doctor now makes one write, its own results;INT-HOST-REGISTRY-CONTRACT;REQ-OPERATOR-CONSOLE-SCREENS;REQ-DEPLOYMENT-BOOTSTRAP.REQ-CONSOLE-SECURITY,REQ-ADMIN-VIA-PI-EXTENSION,DES-RETENTION-SWEEPS-ON-A-TIMER,INT-CONSOLE-STATE-API.Residuals (named in the DES entry)
Verification
service.test.mjsSIGTERM timing test under load).node admin/build.mjsbuilds, and the revision-row, test-count, temp-dir and dated-fixture checks pass.