[TheGreatAxios]

Chain scenarios no longer wait on production-sized windows. Every wait the hub, listener and manager share is now one named timing. Each has its production value as default and a TRIAGE_*_MS override. The chain starts its hub with short values, and each scenario waits on the state it names.

What changed

• Timings. triage-contracts defines the waits, with defaults unchanged, and one timingsFrom(env) reader. It throws when the manager start grace is shorter than the ready window.
  • The hub reads the timings through loadHubEnv like every other variable: a blank value keeps its default, and a bad value shows up in its combined error. Boot sets them once.
  • The hub writes the listener's and manager's share into their release.json, because a workflow's process gets none of the hub's environment. A tree without them runs on the defaults.
  • The deploy hold is still derived from the same ready window the listener waits.
  • The existing reconcile interval, coalesce, liveness and stalled variables move into the same place. The new variables are documented in .env.example and SELF_HOST.
• Harness. The short timings are TIMINGS in tests/chain/stack.ts: ready and start grace 60 s, manager poll 1 s, wake poll 100 ms, reconcile retry 1 s, coalesce 500 ms. The reconcile interval stays at main's 15 s. Polling is every 100 ms, with one database handle per stack.
• Scenarios assert state, not time.
  • woken waits for its manager to poll again with its judge still out.
  • unset reads the hub's needs-setup answer to the delivery.
  • unrouted keeps the reconcile pass out of reach instead of timing the retry.
  • local-sidecars checks each cause in its own test.
  • sidecar-cap keeps the production ready window and start grace, because its second manager's wake waits out the first manager's whole pass.
• Lint. A nested tests/chain/.oxlintrc.json fails bun run lint on a literal sleep or timer of 1 s or more. The root config stays data-only, so the code-review lint step still runs on this repo.
• Docs. AGENTS.md says a failed chain shard is read, never rerun. docs/DEV.md describes the timings.

On deploy

On the first deploy after this, every workspace's listener and manager asset hash changes, because their release.json now always carries the waits. Each workspace rolls its listener once.

Left out

• The hub tests' real sleeps, and a lint rule wider than tests/chain/: CL-11664.
• Getting the slowest shard under three minutes: CL-11665.
• The real waits left: stock's one-minute sidecar ping timeout, in disconnected and unrouted, and unwoken's two 60 s ready windows.

Verification

• bun run lint and bun run check pass. The unit tests for triage-manager, triage-contracts, the hub's env, manager state and workflow-release pass.
• Every chain scenario passed locally, run one at a time.
• CI is green on c73adb7.

Scenario times

All columns are CI test time, without the stack's setup, one run each.

• Before: main at 1b7eaf8.
• First: this PR at be34a95, reconcile interval 2 s.
• Now: c73adb7, reconcile interval back to 15 s and liveness back to its default.

Scenario	Before	First	Now
unwoken	299 s	174 s	175 s
woken	146 s	47 s	44 s
disconnected	100 s	74 s	88 s
unrouted	98 s	80 s	81 s
waiting	65 s	39 s	40 s
held	60 s	33 s	37 s
local-sidecars	56 s	46 s	43 s
reread	49 s	25 s	49 s
always	48 s	70 s	69 s
pushed	44 s	63 s	64 s
unset	32 s	20 s	20 s
sidecar-cap	32 s	48 s	48 s
judged	31 s	24 s	40 s
deterministic	27 s	22 s	20 s

Shard wall	Before	First	Now
1/5	344 s	211 s	218 s
2/5	259 s	132 s	139 s
3/5	145 s	109 s	165 s
4/5	201 s	135 s	130 s
5/5	134 s	186 s	184 s

What changed between the two PR runs:

• reread, judged and disconnected went back up, so shard 3/5 too. reread waits on the reconcile pass, which is 15 s again.
• Shard 5/5 did not change: always, pushed and sidecar-cap are still slower than on main. The 2 s reconcile interval was not the cause. The cause is still unknown.

Closes CL-11645

[linear-code]

CL-11645

CL-11609

[TheGreatAxios]

Independent review at be34a95. bun run lint, bun run check and the unit tests for triage-manager, triage-contracts and the hub's workflow-release pass (353). Chain scenarios were not run.

Production defaults match main for every wait that moved: ready 120000, start grace 120000, manager poll 30000, wake poll 2000, reconcile 300000, retry 15000, coalesce 30000, liveness 60000, stalled 300000. The deploy hold is still 2 * ready + 30000, and PASSES stays 720. A tree without timings falls back to the defaults. Nothing outside timingsFrom reads a TRIAGE_* timing.

Findings, most important first:

1. A blank timing variable now crashes the hub at boot. On main, loadHubEnv drops blank values through presentValues ("Blank values count as missing"), so TRIAGE_RECONCILE_INTERVAL_MS= ran on the default. timingsFrom reads process.env directly and throws on "", " " and " 300000". I reproduced this with timingsFrom({ TRIAGE_RECONCILE_INTERVAL_MS: "" }). The four variables that already existed (reconcile interval, coalesce quiet, liveness, stalled) changed behaviour this way. Separately, apps/hub/src/timings.ts reads process.env when the module is imported, outside loadHubEnv, so bad values no longer show up in its combined error and every test that imports manager-state.ts reads the real environment. Suggested fix: read the timings from the env loadHubEnv has already filtered and pass TIMINGS in as a dependency, not as a module-level constant.
2. The PR closes CL-11645, but two of its outcomes are not met. (a) Hub tests still sleep in real time: apps/hub/src/server.postgres.test.ts:688 (Bun.sleep(5_000)) and apps/hub/src/github/lock-key.postgres.test.ts:34 (1.5 s). The plan put the sleep check outside the chain harness, but the lint rule only covers tests/chain/. (b) The slowest shard is still over three minutes (1/5 at 211 s, 5/5 at 186 s). Either finish both or narrow the issue and keep it open.
3. Shard 5/5 regression (unverified hypothesis). It is not a layout change: main also ran --parallel=2 with pushed and always together (117 s total, now 163 s). It is not only two stacks sharing a runner either: sidecar-cap mostly ran alone after pushed finished and still went from 32 s to 48 s. pushed.test.ts and always.test.ts did not change, so the harness timings are the cause. The likeliest hot path is reconcileIntervalMs: 2_000. repo-activity.ts treats a repository with no webhook as quiet, so it is stale and listed again on every pass. That means all 12 chain repositories are listed every 2 s instead of every 15 s. A pushed repository is also relisted for 60 s of grace. The 1 s manager poll and 2 s liveness add to it. A 100 ms harness poll is unlikely to matter, since it now reuses one DB handle. To confirm, rerun shard 5 locally with reconcileIntervalMs back at 15000 and compare.
4. sidecar-cap raises only managerReadyMs. Its start grace stays at 60 s while the listener waits 120 s. On main both values were one 2-minute window, and manager-state.ts describes the grace as "the listener waits this long for a manager to start". Production now accepts settings where the grace is shorter than the ready window, with nothing to stop it. Either enforce grace >= ready the way triageCoalesceWindow checks its limit, or override both in sidecar-cap. The why comment on sidecar-cap itself is fine.
5. The lint rule only catches literal numbers. It works as described: Bun.sleep(1_000) and setTimeout(r, 2000) fail and Bun.sleep(100) passes. It misses Bun.sleep(SETTLE) with a named constant, Bun.sleep(2 * 1000), and the promise setTimeout(5_000) from node:timers/promises imported under another name. Keeping it out of the root config is justified: the root stays plain data for the code-review lint step.
6. unrouted lost its timing guarantee. With the reconcile interval at 60 min, only the retry or the pass when the listener starts can mail the head. expect(msOf(sent)).toBeGreaterThanOrEqual(msOf(live)) checks the order but not which path sent the mail, yet the test title says "by the short retry rather than the reconcile pass". Either assert the retry's log line or loosen the title.
7. Release republish on first deploy (informational). withRelease now always writes release.json with timings for chain workflows, so every listener and manager asset hash changes and every workspace rolls once on deploy. State this in the PR.
8. Nits.
   • The Published schema in packages/triage-manager/src/release.ts hardcodes github? and bitbucket-datacenter? where it could derive them from REPO_SOURCES.
   • docs/DEV.md says the ping timeout is "the one real wait left", but unwoken still waits through two 60 s ready windows.
   • docs/DEV.md repeats the AGENTS.md shard rule word for word.

Confirmed fine:

• The AGENTS.md line is in place.
• .env.example and SELF_HOST give one line per variable, with no hedges.
• TIMINGS in tests/chain/stack.ts is documented.
• local-sidecars is split into three tests, one cause each.
• woken waits on a counted take, and unset reads needs-setup.
• Two signed Conventional Commits, no attribution.

[TheGreatAxios]

Replies to the review, fixed in c73adb7:

1. Fixed. The timing variables are back in loadHubEnv's schema, so a blank value keeps its default and a bad value shows up in the combined error. Boot calls timingsFrom(env) once and sets it in apps/hub/src/timings.ts, which no longer reads process.env. A new env test checks that a blank value gives the default and that " 300000" and "abc" are errors.
2. Narrowed. The hub-test sleeps and the wider lint rule move to CL-11664, the three-minute target to CL-11665. The PR description names both under what this PR leaves out.
3. The chain's reconcile interval is back to main's 15 s and liveness to its default. No scenario needs either shorter. The manager poll stays at 1 s. The PR description gets the new CI shard times once this run finishes.
4. Fixed. timingsFrom throws when the start grace is shorter than the ready window, and sidecar-cap overrides both.
5. Kept as is, by decision: literal sleeps only, scoped to tests/chain/.
6. Fixed. The unrouted title now says only what it proves.
7. Added to the PR description: on the first deploy after this, every workspace's listener and manager asset hash changes and rolls once.
8. Fixed. Published derives its host keys from REPO_SOURCES. DEV.md names both remaining real waits and links to the AGENTS.md Tests rules instead of repeating them.

[TheGreatAxios]

Re-review at c73adb7. All eight items are addressed:

• The timings go back through loadHubEnv. The new env test covers a blank value (keeps the default) and " 300000" and "abc" (both fail).
• Boot calls timingsFrom(env) once and passes it to setTimings. apps/hub/src/timings.ts no longer reads process.env, and every timings() read happens inside a function, so none of them sees the defaults before boot sets the values.
• A start grace shorter than the ready window now throws. sidecar-cap overrides both.
• The harness is back to main's 15 s reconcile interval and the default liveness interval.
• Published takes its host keys from REPO_SOURCES.
• The unrouted title and DEV.md are fixed.
• The PR body names CL-11664 and CL-11665 and describes the one-time roll caused by the asset-hash change.

bun run lint and bun run check pass. The env, triage-contracts, triage-manager and workflow-release tests pass (358). CI is green.

One nit: docs/DEV.md says the chain starts its hub with "every Triage wait short", but the reconcile interval (15 s), liveness and stalled now keep their non-short values.

On shard 5: the test step logs only print one line per file, so the extra time can't be traced to a step inside pushed. pushed ended 65.8 s into the step on main and 89.8 s on c73adb7. About 4 s of that difference is in setup and about 19.5 s is in the single test body. Nothing in the logs ties it to the 1 s manager poll, the 100 ms poll, the single DB handle or the release.json timings read, so it is unexplained.

[TheGreatAxios]

DEV.md now says the chain shortens the manager and listener waits, not every wait.

[TheGreatAxios]

No findings.

[corbits-triage]

Makes chain test scenarios assert state instead of waiting on production-sized timings, by moving every shared wait into a named, env-overridable timing set in triage-contracts with short values used by the chain harness. The main risks are a new response-parsing assumption in the test helper and a few assertions that became weaker or no longer catch late asynchronous behaviour.

[corbits-triage]

P2 · deliver now parses every 202 body as JSON

deliver used to ignore the response body but now calls res.json() on every accepted delivery, including push and check_run deliveries whose result no caller uses.

If the hub answers any of those with an empty or non-JSON body, the call throws and the scenario fails even though the hub accepted the delivery. For example, the push in unset.test.ts would fail this way.

Suggested fix: only parse the body when the caller needs status, or guard the parse so a non-JSON 202 is treated as accepted:

const text = await res.text();
return text ? (JSON.parse(text) as { status: string }).status : "";

[corbits-triage]

P2 · Retry timing assertion reduced to a lower bound, so a slow retry now passes

The test used to require the mail within MAILED_MS of the replacement starting. It now only checks that the mail came after the listener started:

expect(msOf(sent)).toBeGreaterThanOrEqual(msOf(live));

A reconcile retry that takes minutes, or fires only on a later pass, still passes, even though the test name promises the mail arrives once the replacement runs. The reconcile interval is 60 min, so the short retry is the only path left to mail the head, and that path is no longer checked.

Suggested fix: keep an upper bound tied to the configured retry, for example expect(msOf(sent) - msOf(live)).toBeLessThan(RETRY_MS * 3), or assert the retry fired through a hub-side signal.

[corbits-triage]

P3 · Retry start not checked against the retry ceiling

timingsFrom cross-checks only the start grace against the ready window. reconcileRetryMs is now configurable, but nothing checks it against the fixed 120 s TRIAGE_RETRY_MAX_MS ceiling.

With TRIAGE_RECONCILE_RETRY_MS set above 120000, the documented 'doubling up to two minutes' behaviour breaks. Depending on createReconcileLoop, the first retry either exceeds the ceiling or is silently clamped to it.

Suggested fix: throw in timingsFrom when reconcileRetryMs exceeds the retry max, alongside the existing start-grace check.

[corbits-triage]

P3 · No-policy check now reads state immediately after the delivery, dropping the wait that caught late deployments

The old 5 s sleep gave an asynchronous manager deployment for a repository without a policy file time to appear before deployed was read. The scenario now calls managers() right after pushed returns:

deployed: (await managers()).slice(before)

A deployment that starts a few seconds later, for example through a reconcile pass, is missed. The answered value only shows the synchronous answer, not that nothing started afterwards.

Suggested fix: poll managers() for a bounded window that is long enough for a reconcile pass to run, and assert it stays empty, instead of reading once.