feat(cli): run batches in chunks so large sweeps stay readable - #773
Open
Quenty wants to merge 3 commits into
Open
feat(cli): run batches in chunks so large sweeps stay readable#773Quenty wants to merge 3 commits into
Quenty wants to merge 3 commits into
Conversation
A single execution task covering every package loses the ones that ran first: Open Cloud retains only the tail of a task's log. Measured across Nevermore's 73 packages, the first 14 came back unreadable while the rest were fine — and those 14 pass with real counts in a smaller run. Execute in chunks of 16 against the same uploaded place and merge the results. Each chunk's log stays inside the window (36-130KB here, against the 272KB that lost 14 packages), and each gets its own execution budget rather than sharing one 300s ceiling. --chunk-size tunes it. Sized by package count because output volume is not knowable before a run and the retained window is not a fixed size either. Result on the same 73 packages: 72 passed, up from 59, in 5m24s against 4m45s — about 8s of queue overhead per extra task. Also: - Document which command is the gate, that tracebacks fail a run even when every test passed, and why jest cannot see them. - Print how long each phase took. Group output flushes at once, so a five-minute step otherwise reads as instantaneous. - Name reordering as the cause when a section cannot be closed because a marker arrived out of order, rather than implying output was lost. Claude-Session: https://claude.ai/code/session_014KaJaDsPyPWQ3whHaav2r8
Aggregated mode broadcasts every phase change to every package, so the first version printed 1095 lines across a 73-package sweep — nearly all of them "waiting took 0ms". Noise added to CI output by a change meant to make CI output legible. Report the transition rather than each recipient, and drop anything under a second. The same sweep shape now yields the breakdown that was the point: building took 6.7s combining took 1.1s uploading took 2.1s executing took 13.3s Claude-Session: https://claude.ai/code/session_014KaJaDsPyPWQ3whHaav2r8
Records what is settled with the evidence behind it, so the next person does not re-derive it: the log-window root cause and what was ruled out, the probe showing Open Cloud's unused return channel, the probe showing ScriptContext.Error catches deferred crashes, and the one attribution case still unverified. Also the traps. Nevermore has no real test failures — a large count on an --all sweep means unreadable, not broken. Two bugs this session passed the unit suite and were caught only against a real place. Temporary; delete once the follow-ups land or become issues. Claude-Session: https://claude.ai/code/session_014KaJaDsPyPWQ3whHaav2r8
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follows #772.
Why
Open Cloud retains only the tail of a task's log, so one execution task covering every package loses the ones that ran first. On Nevermore's 73 packages that meant 14 unreadable — and correctly failed, post-#772 — while the other 59 passed. All 14 pass with real counts in a smaller run.
Chunking
Execute in chunks of 16 against the same uploaded place, merging results. Each chunk's log stays inside the window, and each gets its own execution budget rather than 73 packages sharing one 300 s ceiling.
72 passing, up from 59. Wall clock 5m24s vs 4m45s — roughly 8 s of queue overhead per extra task.
--chunk-sizetunes it. Sized by package count rather than bytes because output volume isn't knowable before a run, and the retained window isn't a fixed size either.Also
batch test --cloudis the authority;test --clouditerates on one package. Both documented as applying one rule — including that tracebacks fail a run even when every test passed, and why jest can't see them.Known residual
@quenty/steputilsstill fails, in a chunk where all 9BEGINmarkers were found. ItsENDalmost certainly arrived out of order, and the guard refuses to let a strayENDclaim a section rather than risk taking another package's output. Failing closed is correct, but the cost is a flaky false-red that will move between packages run to run.Tracked as a follow-up. The durable fix is carrying per-package results inside
BATCH_TEST_SUMMARY, which survives by construction and would make both chunking and marker ordering irrelevant — worth weighing against this rather than building both.Not included
Threading jest config overrides to the in-engine run, and the baseline comparison that depends on it. Both need a config channel that doesn't exist at any hop today.
125 tests in
nevermore-cli, 875 across all tools packages, prettier clean.https://claude.ai/code/session_014KaJaDsPyPWQ3whHaav2r8