Skip to content

feat(worker,admin): suggest a job size per project from its runs, capped at the host, never applied automatically (#596) - #601

Merged
edgehero merged 7 commits into
mainfrom
feat/596-suggest
Oct 7, 2026
Merged

feat(worker,admin): suggest a job size per project from its runs, capped at the host, never applied automatically (#596)#601
edgehero merged 7 commits into
mainfrom
feat/596-suggest

Conversation

@edgehero

@edgehero edgehero commented Oct 7, 2026

Copy link
Copy Markdown
Owner

Phase 3 of #596: suggest a job size per project from its own runs, and show it where an operator decides.

What changes

  • Suggestions from the runs. For each project, the last 30 days or 50 runs (whichever is fewer) that carry resources and size give a suggestion, with its evidence and the exact dispatch_limit_edit (or dispatch_limit_add) call to apply it. Nothing is ever applied automatically: the numbers are produced inside the job.
  • Memory is raised only after a confirmed out of memory kill, by one 1.5x step over the largest size that was killed. It is lowered only with 10 or more runs, to 1.25x the 95th percentile peak, and never below the largest recent peak with a margin or 1.5x any size that was killed. A job whose page cache fills its limit is not raised: memory.peak counts page cache, measured at exactly the limit for a job that only read files.
  • CPU is only ever lowered (runs that use far less than their size). A job held back by the host's CPU ceiling is shown as a fact with no call, because a job's CPU size is a weight and raising it does not lift the ceiling (measured: the same throttling at 1024 and 4096 shares).
  • Capped at the host. A suggestion never goes beyond what the host offers (its budget, or its memory and CPUs when the budget is off). When a project needs more, the line says so and never advises raising the host's budget.
  • Where it shows: a line per project in doctor; size, peaks, the suggestion and the call in the panel's PROJECTS view (the call wraps, never cut); the peaks in the dispatch_limit_edit preview; and a "job sizes" section on the insights page with peak memory over time against the size line.
  • Bounded reading. Records are read once per render, from the last 31 days, newest 50 per project, and files over 256 KiB are skipped and counted.

Specs

New DES-SIZE-SUGGESTIONS and REQ-SIZE-SUGGESTIONS. INT-RUN-HISTORY-FILE-CONTRACT and INT-HOST-REGISTRY-CONTRACT name their new readers; both shapes are unchanged.

Checked

  • Full suite with live Valkey: 7247 tests, 0 failures; the new tests also pass with the clock 399 days ahead. Every new pin mutation checked.
  • Review scenarios: a page cache heavy job gets no raise; a repeatedly killed job climbs one step per kill and stops at the host's cap; a busy CPU job gets no raise; 48 light runs and 2 heavy ones never lower below the heavy peak; a project killed at 2g and raised to 3g is not lowered back; crafted record values are ignored or clamped; no record text reaches the suggested call or the screens.

Refs #596

@edgehero
feat(worker): suggest a job size per project from its runs, in doctor (…
e3cf6b8 
…#596)

Phase 3 of issue #596. A project's memory and CPUs are suggested from what its
recent runs used, by one pure function, worker/src/size-suggest.mjs
(suggestSize), which doctor, the admin panel and the insights page all call.

It reads the project's runs of the last 30 days or its last 50 runs, only those
that carry both resources and size, and in each dimension only the runs given
at least the current size, so the OOM that caused a raise does not ask for the
next one. Memory: an oom-killed run raises to max(1.5x, 1.25x p95); more than
10% of the runs at 90% of their limit raises to 1.5x; under 10 runs is not
enough; 1.25x p95 at most 0.75x the size lowers to it; steps of 256m to 2g,
512m to 8g, then 1g, floor 512m. CPUs, from 10 runs: the median run throttled
over 25% of its wall raises by 50%; p95 cores below 0.4x lowers to 1.25x p95,
in 0.25 steps. Every boundary is decided in integers. Every record is judged
again as untrusted: non-integers are ignored, a peak is clamped to its run's
limit, CPU time to 256 cores and throttled time to the wall.

Doctor prints one line per project: a raise or a size above this host's budget
is a warning whose fix is the exact dispatch_limit_edit call
(dispatch_limit_add for a project with no row); a lowering is a fact line that
carries the call; otherwise "fits" or "not enough runs". The records are read
from the logs directory by mtime through a readRunRecords seam; the doctor test
helper defaults it to no runs so no test reads a developer's real logs.

The worker package exports ./size-suggest and ./host-budget for the admin.

Spec IDs: DES-SIZE-SUGGESTIONS, REQ-SIZE-SUGGESTIONS (new, next commit),
INT-RUN-HISTORY-FILE-CONTRACT (readers). DES-HOST-BUDGET and the job size
rules are UNCHANGED, checked: nothing here admits or sizes a job.

Refs #596

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
feat(admin): show size suggestions in the panel, the limit edit and i…
b42be52 
…nsights (#596)

Phase 3 of issue #596, the admin surfaces of the one suggestSize.

- PROJECTS view: under each project its size, the p95 memory peak and p95
  cores of its runs, the suggestion with its reason, then the exact
  dispatch_limit_edit call on a line of its own, and a warning when the
  suggestion is above every live host's budget. The budget is the largest each
  live host publishes per dimension, from the registry rows the panel's tick
  already reads (snapshot.hostBudgets), handed to the projectsInfo seam.
- dispatch_limit_edit: when a size field of a project row is edited, the
  confirm adds the project's size now, its peaks and what they suggest. An
  edit of a count, or of a row that is not a project's, adds nothing.
- insights: a "job sizes" section with each live host's budget and use, and
  per project a chart of each run's peak memory over the window against the
  size line (each run's size, then the current one), the suggestion and the
  call. The payload is allowlisted like every other slice; the page restates
  the worker's reason lists and spellings, pinned by tests.

read-model gains readSizeSuggestions (the worker's function over this host's
run records) and largestHostBudget. Nothing applies a suggestion.

Spec IDs: REQ-SIZE-SUGGESTIONS, DES-SIZE-SUGGESTIONS,
INT-HOST-REGISTRY-CONTRACT (readers of the budget fields).
REQ-INSIGHTS-HTML-EXPORT UNCHANGED, checked: a section under its posture.

Refs #596

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
docs(specs): record the size suggestion rules and where they show (#596)
ec02dec 
Phase 3 of issue #596.

- NEW DES-SIZE-SUGGESTIONS: the rules, the untrusted-input judging, the budget
  flag, and why it is a separate entry rather than an amendment of
  DES-HOST-BUDGET or the job size rules (those decide admission from operator
  files; this decides advice from job-produced numbers), with what was
  rejected (applying automatically, counting runs at smaller sizes, trusting
  the record's validation, float boundaries, p99).
- NEW REQ-SIZE-SUGGESTIONS, newest-first row.
- INT-RUN-HISTORY-FILE-CONTRACT and INT-HOST-REGISTRY-CONTRACT amended with
  their new readers; both shapes UNCHANGED.
- docs/scoped-limits.md: "Choosing a size from the runs".
- docs/insights.md: the job sizes section, and the advisory note now says how
  the suggestions treat the numbers.

Checked and UNCHANGED: DES-HOST-BUDGET, DES-SCOPED-LIMITS-AND-FOLDER-MUTEX,
DES-CONCURRENCY-3, DES-HOST-REGISTRY, REQ-HOST-BUDGET, REQ-SCOPED-LIMITS,
REQ-INSIGHTS-HTML-EXPORT, INT-SCOPED-LIMITS-FILE-CONTRACT,
INT-RUNNER-EXIT-CODE-PROTOCOL, INT-CONFIG-OVERLAY-CONTRACT, the constitution.

Refs #596

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
fix(sizing): raise memory only on an OOM, never raise CPUs, cap every…
136471c 
… suggestion at the host (#596)

Two measurements refute the first rules. Page cache alone drives memory.peak
to exactly the limit with no OOM kill (a 512m container reading 1.2 GB of
files), so raising on runs "at the limit" walked an I/O-heavy project up to the
budget one confirm at a time. And a job's throttled time is the same at 1024
and 4096 CPU shares, because --cpus is the host's ceiling for every job and the
size only its weight, so the throttled raise could never help.

The rules are now smaller:

- memory raises only on a confirmed oom-killed run at the current size or
  larger, one 1.5x step of the larger of the size and the largest OOM-killed
  size in the window; pressure at the limit (memFullUsec above 1% of the wall)
  is a fact line with no call;
- memory lowers as before, but never below 1.25x the window's largest peak,
  1.5x its largest OOM-killed size, nor 512m (the flip-flop after an OOM and
  the p95 dropping two heavy runs of fifty are both pinned);
- CPUs only lower, never below 1.25x the busiest run; throttling over 25% is
  a fact about the host's CPU ceiling;
- every raise is capped at this host's budget, or its own memory and CPU
  count where the budget is off or unknown, and offers no call where neither
  is known; a size already at the cap says "already at the largest size this
  host offers"; nothing advises growing a host's budget;
- the panel and the insights page cap per live host, each judged on its own
  pair of budgets against the size the other dimension will have;
- one shared record reader (worker/src/size-records.mjs, exported for the
  admin): an mtime prefilter, a 256 KiB cap per file with the skipped files
  counted and shown, the newest 50 per project, once per render;
- doctor prints no size line while the scoped-limits file does not load,
  flags a suggestion above the project's hostShare, says "1 core" and
  "1 CPU", and the SIZE_LABEL_PS_ARGS comment is back above its constant;
- the PROJECTS view wraps the call (after commas where it can) and the
  verdict onto continuation lines, never clipping them; width tests at 140,
  80 and 60 columns.

DES-SIZE-SUGGESTIONS, REQ-SIZE-SUGGESTIONS, INT-RUN-HISTORY-FILE-CONTRACT and
INT-HOST-REGISTRY-CONTRACT are corrected in the next commit. Checked and
unchanged: DES-HOST-BUDGET (the cap reads it), the record and registry shapes.

Refs #596

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
docs(specs): record the simpler size suggestion rules and the cap (#596)
644f4dd 
DES-SIZE-SUGGESTIONS CORRECTED: the two measured facts (page cache drives a
peak to the limit with no OOM; throttling is the same at any CPU share), the
OOM-only memory raise, the CPU lowering only, the lowering floors (largest
peak / 0.8, 1.5x the largest OOM-killed size, 512m), the cap at what the host
offers (never advising a larger budget), per-host pair judging, the shared
bounded reader and doctor's limits-file guard; the refuted rules join the
rejected list. Revision row appended.

REQ-SIZE-SUGGESTIONS CORRECTED: statement and acceptance rewritten around the
cap, the page-cache and throttle non-raises, the deflation and flip-flop
floors and the 256 KiB cap. Revision row at the top.

INT-HOST-REGISTRY-CONTRACT CORRECTED (readers paragraph: per-host pairs, not
the largest per field across hosts) and INT-RUN-HISTORY-FILE-CONTRACT AMENDED
(readers paragraph: memFullUsec, OOM-only raise, the bounded reader). Shapes
unchanged. Revision row appended.

docs/scoped-limits.md and docs/insights.md say the same in plain words.

Checked and unchanged: DES-HOST-BUDGET, DES-SCOPED-LIMITS-AND-FOLDER-MUTEX,
DES-HOST-REGISTRY, REQ-HOST-BUDGET, REQ-SCOPED-LIMITS, REQ-INSIGHTS-HTML-EXPORT,
INT-SCOPED-LIMITS-FILE-CONTRACT.

Refs #596

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
fix(size): cap suggestions at the project's hostShare and keep refusa…
f4805e5 
…ls out of the window (#596)

Review of the size suggestions found five defects, fixed here.

A raise was capped at the whole host budget, not at the project's
hostShare of it. Doctor then said the size would never fit the share and
still offered the call that applies it. Every raise is now capped at
floor(budget x hostShare / 100) per dimension (the worker's own
largestFit rule) in doctor and across live hosts in the panel and the
insights page. A suggestion that would still never fit (a lowering of a
size already above the cap, or a pair whose other dimension is above the
share) carries no call on any surface.

The record reader kept the newest 50 records per project before looking
for measurements, so fifty refusals written before any container started
crowded every measured run out. It now drops a record without an object
`resources` and a `size` before the cut.

The panel and the insights page said "no host budget read" also when
budgets were read but no live host holds the project's size, or every
live host's budget is off. The suggestion now carries `capMissing`
(unread, none-holds, off) and each surface words the three apart.

The panel's wrapped call could break inside a JSON string. It now breaks
only outside one (after the tool name, a `{` or a `,`, and after a `:`
only where a whole field is wider than the frame), lets its indent give
way before a string would break, and adds a dim line "join the lines
before running it" under a wrapped call. Printing the call outside the
box was rejected: every view is drawn inside one frame.

Doctor now says a size above the cap is above it (naming the cap)
rather than "already at the largest size", and the CPU over-cap text
names its unit ("2 CPUs").

Specs: DES-SIZE-SUGGESTIONS and REQ-SIZE-SUGGESTIONS corrected (the cap,
the reader, the no-cap reasons, the call wrapping), with revision rows.
INT-HOST-REGISTRY-CONTRACT readers paragraph corrected and rewrapped;
INT-RUN-HISTORY-FILE-CONTRACT readers paragraph corrected (filter before
the cut). docs/scoped-limits.md and docs/insights.md follow.

Checked and unchanged: DES-HOST-BUDGET (largestFit is the rule the cap
restates), REQ-HOST-BUDGET, INT-SCOPED-LIMITS-FILE-CONTRACT, the run
record and registry row shapes.

Refs #596

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
fix(size): offer the apply call exactly when admission would accept t…
a567a84 
…he suggested job (#596)

Doctor, the PROJECTS view and the insights page each decided on their own
whether a size suggestion carries its apply call, and the three tests
disagreed: doctor offered a call whose kept dimension was above the whole
budget, the panel offered one whose kept CPUs were above the project's
hostShare of every live host, and doctor withheld a lowering above the
runtime's total memory while the budget was off, which admission accepts.

Now there is one rule, in one exported pure helper (sizeRefusal in
worker/src/size-suggest.mjs), used by doctor, the panel, the insights page
and the edit preview alike. The pair judged is the suggested size per
dimension, else the current one, against the worker's own never-fits
arithmetic (restated in the leaf, held equal to neverFits over a grid).

- One host (doctor): withheld when its budget is an integer in some
  dimension and refuses the pair. With the budget off or unknown nothing is
  refused, so the call is offered and a size above the host's own memory or
  CPU count is noted beside it.
- Live hosts (panel, insights): withheld when at least one host published an
  integer budget and every such host refuses the pair; with none, offered.
- The edit preview judges no host and still says the budget was not checked.

A withheld call names the dimension that does not fit ("its 4 CPUs are
above its hostShare (40%) of this host's budget (3.2 CPUs)"), and the
no-cap wording for unread budgets is now "no host budget read as a number".

Specs: DES-SIZE-SUGGESTIONS and REQ-SIZE-SUGGESTIONS corrected to state the
one rule, with revision rows; docs/scoped-limits.md says the same.
UNCHANGED, checked: DES-HOST-BUDGET and REQ-HOST-BUDGET (admission itself is
untouched), the suggestion rules and the caps.

Tests: the helper against neverFits over a grid, the fleet rule, the words,
doctor withholding a pair whose kept CPUs exceed the budget, doctor offering
a lowering with the budget off or unknown, the panel withholding a pair whose
kept CPUs exceed the share (rendered), and doctor and the panel agreeing on
every pair both suggest. Mutation checked: 28 of 28 mutants of the helper and
its call sites killed.

Refs #596

Signed-off-by: Rob Boerman <robboerman@live.nl>
@edgehero
edgehero merged commit f2f0e8a into main Oct 7, 2026
7 checks passed
This was referenced Oct 7, 2026
Merged
Closed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@edgehero