Fix/389 Harness baseline/opt comparison direction and timeout

[yueliu14]

Fix issue #389

Bug 1 — the isolated speedup could point the wrong way

The isolated unittest timed the candidate against baseline_src/, a second copy of the
source inside the task dir, selected by a meta.baseline_callable string that the
generated harness bound by hand. Both trees held byte-identical, identically-named code,
so "which one is the baseline" was a coin flip — and it failed silently: both legs are
correct implementations, so correctness still passed and only the ratio inverted. A real
1.055x candidate was reported as 0.96x and rejected. The same hole let mode=author time
optimized-HIP against its own naive-HIP seed: 15.7x isolated, ~0% e2e.

Both legs now run the same leg_runner.py over the same cases.py, differing only
in the overlay on PYTHONPATH:

leg	overlay	contents
baseline	<task>/baseline_overlay	frozen CURRENT_OVERLAY = install + every already-accepted kernel (identical to the e2e gate's ref leg)
candidate	<task>/_cand_overlay	that same stack + exactly one entry built from kernel_src/ (meta.candidate_bind)

Direction becomes a property of the environment rather than of a name, so
speedup = baseline_ms / optimized_ms cannot invert. assert_legs_differ refuses to measure
until both legs provably resolve target_callable to different code and the baseline
resolves outside the task dir.

This also closes the ~8% loader gap from comparing a from-file loader against the installed
package — both legs now import through the identical mechanism.

Two consequences worth calling out:

• The extractor finally receives CURRENT_OVERLAY (every other role already did), so
  isolated speedup and e2e_delta_pct share a denominator and the Amdahl cross-check means
  something.
• Two accepted kernels in the same module now compound instead of the second silently
  reverting the first, because kernel_src/ starts from the module as the current stack
  resolves it.

Bug 2 — startup budget graded by purpose, not by evidence

Shape capture for the MiniMax-M3 MoE kernel died at Server not healthy within 360s, so the
most valuable editable target in the run was never authored. bench_e2e.sh graded its
/health budget by "is an overlay set" (72 × 5s = 360s) — a budget that exists to reject a
wedged candidate overlay. Shape capture sets OVERLAY_PYTHONPATH too, so a legitimate
slow TP4 cold start inherited a fail-fast deadline.

Raising the constant per purpose (capture → 1800s) would still be a guess per model, and
would still treat slow as a proxy for wedged. Wedged shows up as silence, not as
elapsed time. So judge on that:

• Keep waiting while the server log keeps growing; declare death only after
  STALL_WINDOW_SEC (600) with no new bytes. Both backends print continuously through dist
  init, shard load and graph capture, so no-new-bytes is wedged. This rejects a wedged
  candidate sooner than the old 360s while letting a legitimately slow cold start run as
  long as it makes progress.
• The absolute ceiling is a single 7200s backstop, overridable via
  SERVER_STARTUP_TIMEOUT_SEC and named in the failure message. It is a backstop against a
  server that spins printing forever, not a prediction — the progress wait is what decides
  liveness.
• Always write OUT_DIR/server_start.json (status, reason, phase_hint, wait_sec,
  ceiling_sec, …), so a failed cold start is a readable reason downstream instead of an
  empty task dir that reads as "authored and found no gain". Classification
  (engine_core_timeout, dist_init_fail, oom, died_early, stalled) is preserved for
  the caller to act on.
• No startup retry: Two kernel-extraction bugs silently discarded both valuable candidates in an e2e run #389 does not ask for it, and the misclassification it would paper over
  is exactly what the stall window fixes. OOM in particular must not be retried at a lower
  MEM_FRACTION — "this candidate does not fit" is a result.

OVERLAY_KIND / CAPTURE_HEALTH_TRIES / OVERLAY_HEALTH_TRIES no longer select a budget;
OVERLAY_KIND survives as a provenance label. That incidentally fixes callers that never set
it — director.md's final validation leg was silently getting the 360s fail-fast budget.

Unit tests

• test_leg_runner.py (new, 423 lines) — the four leg_runner modes against a stub task dir.
• test_harness_lib.py two-leg section — _run_leg PYTHONPATH ordering,
  build_candidate_overlay, and both of assert_legs_differ's refusals.
• test_leg_runner.py is added to the explicit pytest list in ci-l0-checks.yml; that list
  is what decides whether a test file ever runs.

e2e validation on real hardware

Both fixes have since driven a full multi-model e2e batch on gfx950, across both vLLM and
sglang — 17 model × backend workflows run to completion.

Bug 2 — startup

427 server_start.json records were written (378 ok, 49 failed):

evidence	result
successful cold starts with wait_sec > 360	50 — every one would have been killed by the old fail-fast budget
slowest successful start	1738s (MiniMax-M3-MXFP8, sglang); also 1391s same model, 1284s DeepSeek-V4-Pro
the #389 repro	MiniMax-M3 shape capture now completes instead of dying at 360s
stall window firing	1 × reason=stalled at 155s on a candidate overlay — wedged rejected sooner than the old 360s
oom	1, not retried, as designed
died_early	47, each carrying a phase_hint line lifted from the server log

Bug 1 — leg direction

The two-leg path is what actually ran: baseline_overlay + _cand_overlay are present across
the batch and zero baseline_src/ directories remain. leg_runner.py was staged into 196
task dirs and --mode oracle was invoked 843 times. 353 measurement records carry real
baseline_ms / optimized_ms pairs, so measure_legs timing and the oracle .pt roundtrip
are both exercised (including a 1.7 GB reference_io.pt replayed for MiniMax-M3).

The direct evidence that direction is now sound is the Amdahl cross-check — every accepted
kernel's isolated speedup and its measured e2e delta agree in sign and rough magnitude. The
pathology this PR fixes (15.7x isolated against ~0% e2e) does not appear anywhere in the batch:

model	kernel	isolated	e2e Δ%
Qwen3.5-27B-FP8	kernel_gemm_xdl_cshuffle_v3 (dense fp8 a8w8)	3.17x	+16.14
Qwen3-14B-FP8	dense_fp8_a8w8_blockscale_gemm	2.34x	+104.61
Qwen3-14B-FP8	fp8_a8w8_blockscale_bpreshuffle_GEMM (qkv/o)	1.61x	+13.88
DeepSeek-V4-Flash-0731	matmul_ogs_mxfp4_grouped_fused_moe	1.065x	+3.61

End-to-end throughput

Workflows that reached a validated win with output parity:

model	backend	baseline tok/s	final tok/s	speedup	parity
DeepSeek-V4-Pro	vllm	1295.9	1607.3	1.260x	pass
Qwen3-14B-FP8	sglang	1667.5	1719.5	1.175x	pass

The remaining workflows completed as validated_no_win — the harness ran, measured, and
correctly declined to claim a gain, which is the outcome this PR is meant to make trustworthy.

[iraj465]

Reviewed the code, not just the description. The core design is right and I'd like to see this land. Making leg identity a property of the environment rather than of a name is the correct fix, and assert_legs_differ is the part that makes it stick — refusing to measure until both legs provably resolve to different code, and the baseline resolves outside the task dir, turns "don't invert the ratio" from a convention into an invariant.

Worth stating: measure_legs gives serving_weighted_speedup its first production producer. That function has been in harness_lib.py with zero non-test Python callers; the per-bucket per_case this emits is what finally feeds it.

Five things before merge, one of which I think is blocking.

1. Blocking — measure_legs takes n=1 per leg per bucket

b = _run_leg(task, base, "time", bucket=sig, ...)["cases"]
o = _run_leg(task, cand, "time", bucket=sig, ...)["cases"]

One measurement per leg, and — correctly — in a fresh process each time. But fresh-process variance is exactly the variance you cannot average away inside the process.

I measured this today on MI355X gfx950, 5 fresh-process replicates of the same GEMM task:

bucket	CV across fresh processes
prefill M=8192	0.09 – 0.47%
decode M=64	0.12 – 2.42%
decode M=64, aiter	13.7%

The motivating case in the description is a 1.055x candidate. At ~2.4% per-leg CV, a single unpaired B-then-C pair puts 1.055x at roughly 2 sigma — recoverable, but not reliably, and the tail backend sits at 13.7%. The direction bug is fixed; the resolution is not yet enough to trust the number that bug was corrupting.

Suggest reps (default 3) and interleaving B,C,B,C,B,C per bucket rather than one B then one C, then reporting median plus the spread. Interleaving matters more than the count — it makes any drift during the sweep common-mode.

2. _run_leg timeout can leave a process on the GPU

subprocess.run(..., timeout=timeout) kills only the direct child. If the leg is mid-kernel or has spawned any helper, the GPU allocation survives the raise, and the next leg then measures against a busy device — which reads as a slowdown attributable to nothing.

start_new_session=True plus os.killpg on TimeoutExpired closes it. Note os.kill(pid, 0) will report a zombie as alive, so check reaping too.

3. oracle mode assumes the op returns a bare tensor

out = call(shape["make_inputs"](rng))
blob[...] = out.detach().clone().cpu()

Attention kernels commonly return a tuple (out, lse). This raises AttributeError and, via _run_leg, surfaces as a generic non-zero-exit RuntimeError rather than "this op returns a tuple". A small normalize over tuple/list would cover it.

4. unittest.py is declared IMMUTABLE but is still agent-authored

kernel_extractor.md:30 calls the driver immutable, and :314 tells the agent to make exactly one measure_legs call. That is the right contract — but it is enforced by prompt. Since the file is now fully generic (it only needs TASK and meta), shipping it as a real file next to leg_runner.py and copying it in would make the contract structural, and would remove the last place the direction bug could be reintroduced by generation drift.

5. Minor

• Log-byte growth as the liveness token is defeated by a retry loop printing the same error forever. The 7200s ceiling does backstop it, so this is a note, not an objection — but stalled vs ceiling_exceeded will then misreport which one happened.
• The FATAL in the fatal-marker grep is broad enough to match a log-level legend line. Anchoring it would avoid a false fatal_marker.

Relationship to the other open PRs

This is complementary to #415 (issue #417), not overlapping: #416 fixes which leg wins, #415 fixes at what shape it is timed. #416 does not touch op_bench.py, and #415 does not touch the isolated-unittest path. I trial-merged them — the only conflict is an append-collision at the end of tests/test_harness_lib.py, trivially resolved.

Sequenced with #384 (measured m_buckets rather than guessed), the three are one chain: right shapes → timed at those shapes → credited to the right leg. None is sufficient alone.

[yueliu14]

Thanks — 1, 2, 3 and 5 are fixed in 7a7abd0 + 0505c14.

1 (blocking). Took the interleaving — it's free and makes drift common-mode. Didn't take flat reps=3: fresh-per-bucket means every rep pays another cold torch import, so flat reps triples every bucket to buy resolution on the few in doubt. Instead max_reps=3 is a ceiling, and the running median ratio is tested against an undecided band (0.95–1.10) after each pair — a 2.3x bucket costs one pair as before, a 1.05x bucket spends the budget. That puts the spend where your CV data says the noise is (decode 2.4/13.7%) and nothing where it isn't (prefill 0.1–0.5%). Times are the median pair; reps and speedup_spread are in per_case. Say the word if you'd rather have the simpler flat contract.

2. start_new_session=True + SIGTERM→SIGKILL on the group. Good catch on the zombie — death is confirmed by proc.wait(), not kill(pid, 0). Test spawns a real grandchild and asserts it doesn't outlive the timeout.

3. You found the recorder; the same assumption was on the consumer side and worse there — correct() catches its own exceptions, so a tuple-returning candidate wasn't erroring, it was reported incorrect and silently rejected. Fixing only leg_runner would have moved the failure. Both ends now handle tensor/tuple/list/dict/namedtuple component-wise; count mismatch is a failure. kernel_extractor.md now tells the agent to return the output whole.

5. Anchored to [FATAL] / FATAL:. On stalled-vs-ceiling: those are already distinct reasons in server_start.json, so a spinning retry loop reports ceiling_exceeded — accurate for what happened.

4. Agree with the direction, but not this PR: unittest.py isn't yet as generic as it looks — it still reconstructs inputs from reference_io.pt, picks dtype tolerance, and builds the ≥2-boundary replay_bundle. Shipping it as a real file means moving those into cases.py first, which changes the extractor's generation contract. Opening a follow-up issue.

[iraj465]

Re-reviewed at 0505c14. 1, 2, 3 and 5 are all closed — and the adaptive band is the better answer than my flat reps=3: it spends pairs where my CV data says the noise is and nothing where it isn't. Keep it. Deferring 4 to a follow-up is right; unittest.py isn't as generic as it looks yet.

Three things I found on the closer pass. None blocks merge.

1. assert_legs_differ is asymmetric. It proves the baseline resolves outside the task dir, but nothing says the candidate resolves inside it. A candidate whose overlay silently failed but which differs from the baseline for some other reason still passes the guard. ci["file"] is already in hand, so the symmetric check is one line.

2. stat -c %s (:398) is GNU-only and fails to 0. On a box where it fails, _last_tok never changes again, so any cold start slower than STALL_WINDOW_SEC is reported stalled when it was healthy. Low risk in the ROCm containers, but it misdiagnoses rather than erroring — wc -c < "$LOG" is portable and fails loudly.

3. _phase escaping is incomplete. It handles \r, \\ and ", but a tab or other control character in the last log line goes into server_start.json unescaped, which is invalid JSON — so the one artifact written to explain a failed start becomes unparseable. tr -d '\000-\037' after the existing tr covers it.

Merge order with #415

I trial-merged both into current main (post-#408). #416 merges clean. With both applied: 1116 tests pass, and the only conflict is an append-collision at the end of tests/test_harness_lib.py — both PRs add a block at EOF. Resolution is keeping both blocks; I verified the combined suite green afterwards. harness_lib.py itself auto-merges, and the combined file has no duplicate definitions — served_regimes (main), served_buckets (#415) and serving_weighted_speedup (#416) coexist.

Whichever lands second picks up that one-file resolution. Happy for #416 to go first.