Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions .env.example
Original file line number Diff line number Diff line change
Expand Up @@ -182,6 +182,10 @@ PI_JOB_IMAGE=pi-job:latest
# The window is a FLOOR, not a ceiling: a file dies on the first sweep AFTER its window closes, so the real ceiling is its window plus this interval (up to 48h for a sandbox at both defaults)
# Refused above 168 (one week): setInterval clamps a longer delay to 1ms, and a sweep slower than a week is 0 with extra steps
# PI_SWEEP_INTERVAL_HOURS=
# default 24; how often the worker runs `pi-dispatch doctor --publish` itself (the first run a minute after boot), so the console and the /dispatch panel show each host's checks (issue #712)
# What is published to Valkey is each check's level, and for a warning or a failure its label and fix with paths and URLs cut to <path> and scheme://host and secret-named values and known token shapes redacted; kept 7 days per host
# A doctor you run by hand publishes the same way (pass --no-publish to skip it). 0 = the worker runs no doctor of its own; refused above 168 (one week)
# PI_DOCTOR_INTERVAL_HOURS=
# where a finished run's directory is kept so you can re-open it (default: <PI_JOBS_DIR>/sandboxes, mode 0700)
# `pi-dispatch sandbox <jobId>` starts a fresh container from the same image with the same workspace and NO credentials (docs/sandbox.md)
# A retained directory holds the run's clone plus its prompt.md/event.json -- so issue text. Host-only; never mounted into a job
Expand Down
2 changes: 1 addition & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -311,7 +311,7 @@ wait for its budget, the jobs running there now, and how busy it was over the la
`pi-dispatch capacity`, `pi-dispatch doctor`, the insights page and the `dispatch_capacity` tool read the same report. It counts jobs
only: a machine busy with other work reads as idle.

![The panel's hosts view: three machines, each with its slots in use of its limit, the memory and CPU its budget has promised against that budget (or that it has none), its running jobs with project, size and age, a stale host with an orphaned container, and the last 7 days as busy share, average and peak jobs at once, time at the slot limit, the 95th percentile wait and the projects that used it most, ending with the note that only jobs are counted](docs/images/dispatch-hosts.svg?v=2026-10-14)
![The panel's hosts view: three machines, each with its slots in use of its limit, the memory and CPU its budget has promised against that budget (or that it has none), its running jobs with project, size and age, a stale host with an orphaned container, and the last 7 days as busy share, average and peak jobs at once, time at the slot limit, the 95th percentile wait and the projects that used it most, each host's last doctor run with its failing and warning checks and their fixes (one result stale, one host that has published none), ending with the note that only jobs are counted](docs/images/dispatch-hosts.svg?v=2026-10-21)

## Flows: the custom prompt a trigger runs

Expand Down
2 changes: 1 addition & 1 deletion admin/README.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,7 +195,7 @@ One command puts a live terminal view over the whole deployment:
It counts jobs only: a machine busy with other work reads as idle.

<p align="center">
<img src="https://raw.githubusercontent.com/edgehero/pi-dispatch/main/docs/images/dispatch-hosts.png?v=2026-10-14" alt="The panel's hosts view: three machines, each with its slots in use, the memory and CPU its budget has promised, its running jobs with project, size and age, a stale host with an orphaned container, and the last 7 days as busy share, jobs at once, time full, the wait and the top projects" width="820">
<img src="https://raw.githubusercontent.com/edgehero/pi-dispatch/main/docs/images/dispatch-hosts.png?v=2026-10-21" alt="The panel's hosts view: three machines, each with its slots in use, the memory and CPU its budget has promised, its running jobs with project, size and age, a stale host with an orphaned container, and the last 7 days as busy share, jobs at once, time full, the wait and the top projects, then each host's last doctor run with its failing and warning checks" width="820">
</p>

- **The budget split.** With an allocation envelope (a dollar total per window and a floor per project), `b`
Expand Down
3 changes: 2 additions & 1 deletion admin/src/console/pages.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -37,6 +37,7 @@ import {
fleetFailed,
fleetHeld,
fleetHosts,
fleetDoctor,
fleetQueues,
fleetSchedulers,
INSIGHTS_DEFAULT_RANGE,
Expand Down Expand Up @@ -98,7 +99,7 @@ const PAGE_PANELS = Object.freeze({
run: [["p-progress", runProgress], ["p-chat", runChat], ["p-inbox", runInbox], ["p-run", runSummary], ["p-cost", runCostPanel], ["p-resources", runResources], ["p-session", runSession], ["p-chain", runChain]],
parked: [["p-parked", parkedInbox]],
insights: [["p-range", insightsRange], ["p-whatif", insightsWhatIf], ["p-insights", insightsPanel]],
fleet: [["p-queues", fleetQueues], ["p-active", fleetActive], ["p-held", fleetHeld], ["p-hosts", fleetHosts], ["p-schedulers", fleetSchedulers], ["p-failed", fleetFailed]],
fleet: [["p-queues", fleetQueues], ["p-active", fleetActive], ["p-held", fleetHeld], ["p-hosts", fleetHosts], ["p-doctor", fleetDoctor], ["p-schedulers", fleetSchedulers], ["p-failed", fleetFailed]],
config: [["p-triggers", configTriggers], ["p-settings", configSettings], ["p-pause-windows", configPauseWindows], ["p-scoped-limits", configScopedLimits], ["p-projects", configProjects], ["p-mcp", configMcp], ["p-packages", configPackages], ["p-secret-profiles", configSecretProfiles], ["p-secret-bundles", configSecretBundles]],
audit: [["p-audit", auditLog]],
});
Expand Down
64 changes: 57 additions & 7 deletions admin/src/console/panels.mjs
Original file line number Diff line number Diff line change
Expand Up @@ -61,6 +61,7 @@ import { RUN_FILTER_FIELDS } from "../run-filter.mjs";
import { sessionKeyOfRecord } from "@edgehero/pi-dispatch/session-key";
import { parkedVerdictOf } from "@edgehero/pi-dispatch/chat-mode";
import { html, trustedRenderer } from "./html.mjs";
import { DOCTOR_CHECKS_SHOWN, DOCTOR_FILE_CHECKS, doctorByText, doctorCountsText, doctorHostsOf, doctorProblems, doctorQuote, doctorStale } from "../doctor-view.mjs";

const SEC = 1000;
const MIN = 60 * SEC;
Expand Down Expand Up @@ -1170,6 +1171,42 @@ export const fleetHosts = fromSnapshot("Hosts", (s, view) => {
);
});

/**
* The Doctor panel (issue #712, `DES-DOCTOR-RESULTS-PUBLISHED`): each host's last published `pi-dispatch doctor` run, its
* age and counts, then its failing and warning checks with their fix, failing first. A result older than two of its
* host's doctor intervals is flagged stale; a live host that has published nothing is named, so "no problems listed"
* never reads as "healthy" for a host that was never checked. The labels are doctor's redacted sentences (no path, no
* URL beyond its host, secret-named values and token shapes redacted, file content quoted by a parse error cut), shown
* to both roles and escaped like every value here.
*/
export const fleetDoctor = fromSnapshot("Doctor", (s, view) => {
const d = s.doctor;
if (!d || d.unreachable) return html`<p class="warn">The published doctor results could not be read.</p>`;
const rows = doctorHostsOf(s);
const published = new Set(rows.map((r) => r.name));
const silent = (Array.isArray(s.hostBudgets) ? s.hostBudgets : []).map((h) => h?.name).filter((n) => typeof n === "string" && !published.has(n));
const block = (r) => {
if (r.unreadable) return html`<h3>${r.name}</h3><p class="warn">Its published result cannot be read (a worker of another version may have written it).</p>`;
const problems = doctorProblems(r);
const shown = problems.slice(0, DOCTOR_CHECKS_SHOWN);
const hidden = problems.length - shown.length + (Number.isSafeInteger(r.more) ? r.more : 0);
const stale = doctorStale(r);
const head = html`<p${r.counts?.fail > 0 ? html` class="bad"` : r.counts?.warn > 0 ? html` class="warn"` : null}>Doctor ran ${ageText(r.ageMs)} ago${r.version ? ` (version ${r.version})` : ""}, ${doctorByText(r)}: ${doctorCountsText(r)}.${r.openshell ? " The openshell venue was judged." : ""}</p>`;
// A run by another account over the worker's (sudo, a second login) is published like any run, and said: it shows
// that account's view of the host, not the worker's.
const replaced = r.workerRun && r.by?.worker !== true ? html`<p class="warn">It replaced the worker's own run of ${ageText(view.now - r.workerRun.at)} ago${Number.isSafeInteger(r.workerRun.uid) ? ` (uid ${r.workerRun.uid})` : ""}, so it shows another account's view of the host.</p>` : null;
const staleNote = stale ? html`<p class="warn">Stale: older than two of this host's doctor runs (every ${r.every}h), so it may no longer describe the host.</p>` : null;
const list = table(
`Doctor checks on ${r.name}`,
["Level", "Check", "What doctor said", "Fix"],
shown.map((c) => html`<tr><td class="${c.level === "fail" ? "bad" : "warn"}">${c.level === "fail" ? "failing" : "warning"}</td><td class="mono">${c.id}</td><td>${cell(c.label)}</td><td>${cell(c.fix)}</td></tr>`),
"Every check passed.",
);
return html`<h3>${r.name}</h3>${head}${replaced}${staleNote}${list}${hidden > 0 ? html`<p class="note">${count(hidden)} more not shown here; pi-dispatch doctor on that host prints them all.</p>` : null}`;
};
return html`${rows.length === 0 ? html`<p class="note">No host has published doctor results yet. Each worker runs doctor a minute after it starts and then every PI_DOCTOR_INTERVAL_HOURS (24 by default), and pi-dispatch doctor publishes too.</p>` : rows.map(block)}${silent.length > 0 ? html`<p class="note">No doctor result published by: ${silent.join(", ")}.</p>` : null}${d.more > 0 ? html`<p class="note">${count(d.more)} more hosts published results; not every one is read.</p>` : null}`;
});

export const fleetSchedulers = fromSnapshot("Schedules", (s, view) => {
const list = Array.isArray(s.schedulers) ? s.schedulers : [];
const stalls = s.schedulerStalls ?? {};
Expand Down Expand Up @@ -1214,12 +1251,25 @@ export const fleetFailed = fromSnapshot("Failed jobs", (s, view) => {

// ---- configuration ---------------------------------------------------------------------------------------------------

/**
* Why a file does not load, as one sentence: the newest published doctor check about that file where one was published
* (issue #712, `doctorQuote`), naming the host and its age, else the pointer to run doctor. Never the parser's own
* message, which names this host's path; a published label had its paths cut before it left the host that wrote it.
*/
function doctorReason(s, what) {
const id = DOCTOR_FILE_CHECKS[what];
const q = id ? doctorQuote(s, id) : null;
if (!q) return `The ${what} does not load; run pi-dispatch doctor for the reason.`;
const where = q.own ? `on this host (${q.host}, ${ageText(q.ageMs)} ago)` : `on ${q.host} (its own copy of the file, ${ageText(q.ageMs)} ago)`;
return `The ${what} does not load. pi-dispatch doctor ${where} says: ${q.label}${q.fix ? `. Fix: ${q.fix}` : ""}`;
}

/** A file view's trouble as one sentence (no parser message: it names the host's path), or null when it loaded. */
function fileTrouble(v, what) {
function fileTrouble(v, what, s = null) {
if (!v) return `The ${what} could not be read.`;
if (v.unset) return `No ${what} is configured.`;
if (v.missing) return `The ${what} does not exist.`;
if (v.invalid) return `The ${what} does not load; run pi-dispatch doctor for the reason.`;
if (v.invalid) return doctorReason(s, what);
if (v.unreadable) return `The ${what} cannot be read (${String(v.unreadable)}).`;
return null;
}
Expand Down Expand Up @@ -1276,7 +1326,7 @@ function triggersEditor(view) {

export const configTriggers = fromSnapshot("Triggers", (s, view) => {
const v = s.triggers;
const trouble = fileTrouble(v, "triggers file");
const trouble = fileTrouble(v, "triggers file", s);
if (trouble) return html`<p class="warn">${trouble}</p>${triggersEditor(view)}`;
const list = Array.isArray(v.triggers) ? v.triggers : [];
return html`${table(
Expand All @@ -1303,7 +1353,7 @@ function settingsEditor(view) {

export const configSettings = fromSnapshot("Settings overlay", (s, view) => {
const v = s.settings;
if (!v || v.invalid) return html`<p class="warn">The settings overlay does not load; run pi-dispatch doctor for the reason.</p>${settingsEditor(view)}`;
if (!v || v.invalid) return html`<p class="warn">${doctorReason(s, "settings overlay")}</p>${settingsEditor(view)}`;
const overlay = v.overlay ?? {};
const shown = SETTINGS_SHOWN.filter((k) => ["string", "number", "boolean"].includes(typeof overlay[k]));
const others = Object.keys(overlay).filter((k) => !SETTINGS_SHOWN.includes(k)).sort();
Expand All @@ -1322,7 +1372,7 @@ function pauseEditor(view) {

export const configPauseWindows = fromSnapshot("Pause windows", (s, view) => {
const v = s.pauseWindows;
const trouble = fileTrouble(v, "pause-windows file");
const trouble = fileTrouble(v, "pause-windows file", s);
if (trouble) return html`<p class="note">${trouble}</p>${pauseEditor(view)}`;
const list = Array.isArray(v.windows) ? v.windows : [];
return html`${table(
Expand All @@ -1345,7 +1395,7 @@ function limitsEditor(view) {

export const configScopedLimits = fromSnapshot("Scoped limits", (s, view) => {
const v = s.scopedLimits;
const trouble = fileTrouble(v, "scoped-limits file");
const trouble = fileTrouble(v, "scoped-limits file", s);
if (trouble) return html`<p class="note">${trouble}</p>${limitsEditor(view)}`;
const list = Array.isArray(v.limits) ? v.limits : [];
return html`${table(
Expand All @@ -1365,7 +1415,7 @@ function projectsEditor(view) {

export const configProjects = fromSnapshot("Projects", (s, view) => {
const v = s.projects;
const trouble = fileTrouble(v, "projects file");
const trouble = fileTrouble(v, "projects file", s);
if (trouble) return html`<p class="note">${trouble}</p>${projectsEditor(view)}`;
const list = Array.isArray(v.projects) ? v.projects : [];
return html`${table(
Expand Down
46 changes: 44 additions & 2 deletions admin/src/dashboard.ts
Original file line number Diff line number Diff line change
Expand Up @@ -40,6 +40,7 @@ import { WORKER_NAME_RE } from "@edgehero/pi-dispatch/config";
import { LIVE_MAX_ENTRIES } from "@edgehero/pi-dispatch/live-transcript";
import { sizeBits, renderStatus, renderBudget, renderHeldJobs, renderScopedLimits, renderTriggers, renderSettingsView, commandSlashLabel, scrubTrigger, skillsBasename, allocAt, allocHostsShown, allocPlanId, outsideEdit, outsideEditText, splitTotalMicros, fileTotalText, allocationRowIds, SPLIT_ONLY_MARK } from "./render.mjs";
import { matchesKey } from "./keys.mjs";
import { doctorByText, doctorCountsText, doctorHostsOf, doctorProblems, doctorStale } from "./doctor-view.mjs";
import { box, clip, clipData, cutUnits, escapeInterpreted, fmtCost, hasControls, makeLineInput, meter, scrubControls, scrubKeepingStyle, sliceColumns } from "./panel.mjs";
import { makeStyler, frame, RULE } from "./style.mjs";

Expand Down Expand Up @@ -3211,14 +3212,26 @@ function hostsView(snapshot: any, info: any, iw: number, styler: any): { title:
return out;
};

// Issue #712: each host's last published doctor run under its other lines, and a host that published one but has no
// live row and no history drawn after them, so a host that went quiet still says what was wrong with it.
const doctorRead = snapshot?.doctor && !snapshot.doctor.unreachable;
const doctorByName = new Map<string, any>(doctorHostsOf(snapshot).map((d: any) => [d.name, d]));
const doctorOf = (name: any): string[] => (doctorRead ? doctorLines(doctorByName.get(name), iw, styler) : []);
if (snapshot?.doctor?.unreachable) lines.push(...wrapped(`doctor results unreadable (${cellOf(snapshot.doctor.unreachable)})`, "warning"));
for (const r of rows) {
if (lines.length > 0) lines.push(styler.cell("", iw));
lines.push(...liveLines(r), ...historyLines(byName.get(r?.name), r?.name));
lines.push(...liveLines(r), ...historyLines(byName.get(r?.name), r?.name), ...doctorOf(r?.name));
}
for (const h of historyOnly) {
if (lines.length > 0) lines.push(styler.cell("", iw));
lines.push(fitLine(styler.bold(styler.fg("accent", clip(hostName(h?.name), Math.max(8, iw - 14)))) + " " + styler.fg("dim", "no live row"), iw, styler));
lines.push(...historyLines(h, h?.name));
lines.push(...historyLines(h, h?.name), ...doctorOf(h?.name));
}
const drawn = new Set<any>([...rows.map((r: any) => r?.name), ...historyOnly.map((h: any) => h?.name)]);
for (const d of doctorRead ? [...doctorByName.values()].filter((d: any) => !drawn.has(d.name)) : []) {
if (lines.length > 0) lines.push(styler.cell("", iw));
lines.push(fitLine(styler.bold(styler.fg("accent", clip(hostName(d.name), Math.max(8, iw - 14)))) + " " + styler.fg("dim", "no live row"), iw, styler));
lines.push(...doctorOf(d.name));
}
if (report) {
lines.push(styler.cell("", iw));
Expand All @@ -3230,6 +3243,35 @@ function hostsView(snapshot: any, info: any, iw: number, styler: any): { title:
return { title, lines };
}

/** The warn and fail checks a host's doctor lines list before they count the rest. */
const DOCTOR_TUI_SHOWN = 4;

/**
* A host's doctor lines in the HOSTS view (issue #712, `INT-DOCTOR-RESULTS-KEY`): its last published run's age and counts
* (and `stale` once it is older than two of its own intervals), then up to `DOCTOR_TUI_SHOWN` failing and warning checks,
* failing first, each with its fix. A host that published nothing says so, so an empty list never reads as healthy.
* Labels are another host's redacted text, escaped and gated (`textCell`), and every line is fitted to `iw`.
*/
function doctorLines(row: any, iw: number, styler: any): string[] {
const wrapped = (t: string, color: string, indent: string) =>
wrapColumns(t, Math.max(1, iw - indent.length), styler).map((l) => styler.cell(indent + l, iw, { color }));
if (!row) return wrapped("doctor: nothing published by this host yet", "dim", " ");
if (row.unreadable) return wrapped("doctor: its published result cannot be read", "warning", " ");
const age = Number.isSafeInteger(row.ageMs) ? `${durationText(row.ageMs)} ago` : "at an unknown time";
const stale = doctorStale(row);
const failing = Number.isSafeInteger(row.counts?.fail) && row.counts.fail > 0;
const head = `doctor ${age}, ${doctorByText(row)}: ${doctorCountsText(row)}${stale ? ` · stale (runs every ${row.every}h)` : ""}${row.workerRun && row.by?.worker !== true ? " · replaced the worker's run" : ""}`;
const out = wrapped(head, failing ? "error" : stale || row.counts?.warn > 0 ? "warning" : "muted", " ");
const problems = doctorProblems(row);
for (const c of problems.slice(0, DOCTOR_TUI_SHOWN)) {
out.push(...wrapped(`${c.level === "fail" ? "✗" : "⚠"} ${textCell(c.label ?? c.id)}`, c.level === "fail" ? "error" : "warning", " "));
if (c.fix) out.push(...wrapped(`→ ${textCell(c.fix)}`, "dim", " "));
}
const hidden = problems.length - Math.min(problems.length, DOCTOR_TUI_SHOWN) + (Number.isSafeInteger(row.more) ? row.more : 0);
if (hidden > 0) out.push(...wrapped(`+${hidden} more: the console's Fleet page lists them, and doctor on that host prints them`, "dim", " "));
return out;
}

/**
* `RUNS`, or `RUNS · 2 hosts`, or `RUNS · this host only`.
*
Expand Down
Loading
Loading