fix(journey): fold a return-named acceptance into its overlay entry - #408
Conversation
Cross-workload view: how often the twin actually firesPer-workload breakdown of the duplicate this PR removes, measured 17 Aug 2026 "Claim" below means a
10 of 36 accepted-kernel rows are twins — 28%. They appear in 8 of the 15 Two workloads are worth reading closely:
The replay result already in the PR body — 97 → 87 journey entries, 10 removed, Reproducepython3 - <<'PY'
import glob, json
from collections import defaultdict
per = defaultdict(lambda: {"runs": 0, "keeps": 0, "twins": 0})
seen_runs = set()
for kj in sorted(glob.glob('/shared_nfs/hyperloom-claw/*/*/geak/e2e_cycle*/kernel_journey.json')):
m = kj.split('/')[3]
rows = []
for k in (json.load(open(kj)).get("kernels") or []):
e = k.get("e2e") or {}
if str(e.get("decision") or "").upper() == "KEEP" and e.get("integrated"):
rows.append((k.get("gpu_pct"), e.get("e2e_gain_pct")))
per[m]["keeps"] += len(rows)
groups = defaultdict(list)
for i, (gpu, gain) in enumerate(rows):
groups[round(gain, 3) if isinstance(gain, (int, float)) else i].append(gpu)
for g in groups.values():
meas = [x for x in g if x is not None]; un = [x for x in g if x is None]
if meas and un: per[m]["twins"] += len(un)
for m, d in sorted(per.items(), key=lambda kv: -kv[1]["keeps"]):
if d["keeps"]: print(f"{m:32} keeps={d['keeps']:3} twins={d['twins']:3}")
PY |
Follow-up: what the 26 claimed kernels are worth downstreamExtending the cross-workload view above, over the same 66 GEAK runs across 15 GEAK claims 26 distinct Hyperloom credits 0 of them with an end-to-end gain:
Four runs show GEAK and Hyperloom both positive and within 3x. In all four, This does not weaken the case for this PR. It sharpens it: the claim count is |
build_kernel_journey emits one kernels[] entry per optimization overlay, then
augments with acceptances "named only in the workflow return". It tests that
"only" with the emitted kernel_id — but the two substreams do not name a kernel
the same way. An overlay directory is named for its CANDIDATE TAG
(cand_c0_triton); the workflow return names the KERNEL SYMBOL
(dsa_sparse_attn_prefill_main_kernel). The ids therefore never collide, and one
acceptance is written twice: once measured from integrate_result.json, and once
from the return with gpu_pct null and the same e2e_delta_pct.
integrate_result.json is the file that ties the two spellings together: it
records cand_tag AND short_name. Read the overlay's claim from there and dedupe
pass 2 against it.
* only an INTEGRATED overlay claims anything (gate accepted/stack); a rejected
or cut-off overlay leaves a return acceptance as new information,
* the symbol is compared on the same normalization the profiler match uses,
* when the overlay recorded no usable symbol, the integrated e2e delta folds
it — the return copies that number from the overlay's own A/B rather than
recomputing it — but only when exactly one unconsumed overlay claims it,
* a claim is consumed at most once, so a single overlay can never swallow two
distinct acceptances.
Replayed over all 77 e2e cycles under /shared_nfs/hyperloom-claw: 97 -> 87
journey entries, 10 duplicates removed, 0 added, 0 altered. Every removed entry
carried gpu_pct null and a gain identical to a surviving measured entry. Two
acceptances that decompose one overlay into differently-shaped GEMMs
(Qwen3-14B-FP8/20260816T050457Z, deltas 6.779 and 0.705 against an overlay
delta of 7.53) are correctly left alone.
Tests: interface/ 168 -> 176 passed, 0 failed.
5c891d9 to
7a63544
Compare
3 participants
Branch:
fix/journey-overlay-return-alias→mainBase:
ab9dd2dbCloses: #407
What was wrong
build_kernel_journeyadds acceptances "named only in the workflow return"after it has written one entry per overlay. It tests that "only" with the
emitted
kernel_id. The two substreams do not name a kernel the same way: anoverlay directory carries the candidate tag (
cand_c0_triton) and thereturn carries the kernel symbol (
dsa_sparse_attn_prefill_main_kernel).The ids never collide, so one acceptance is emitted twice — once measured, once
with
gpu_pct: nulland the samee2e_gain_pct, both markedKEEP.Full evidence, worked example, and the affected session list are in the issue.
What this changes
integrate_result.jsonis the file that ties the two spellings together: itrecords
cand_tagandshort_name. The fix reads the overlay's claim fromthere and dedupes pass 2 against it.
Two small helpers, no change to any emitted field:
_overlay_claim(ir)— what an integrated overlay says it optimized, orNone. An overlay whosegateis notaccepted/stackclaims nothing, soa rejected or cut-off overlay still leaves a return acceptance as new
information.
_claim_for(name, gain, claims)— the claim that already covers thisreturn-named acceptance. Symbol first, on the same normalization
_match_profileruses. Integrated e2e delta as the fallback, for the runswhere the overlay recorded no usable symbol: the return copies that number
from the overlay's own A/B rather than recomputing it, so an exact hit is the
same measurement. The delta only folds when exactly one unconsumed
overlay claims it.
A claim is consumed at most once, so one overlay can never absorb two distinct
acceptances.
This is deliberately not a name-similarity heuristic and not a rounded-gain
match. Every join is a value GEAK itself wrote to
integrate_result.json.Why it is safe on real decompositions
/shared_nfs/hyperloom-claw/Qwen3-14B-FP8/20260816T050457Z/geak/e2e_cycle0hasone overlay (
cand_c0_aiter, delta 7.53) and two accepted records — adown_proj-shaped and a gate_up-shaped GEMM, deltas 6.779 and 0.705. Those are a
decomposition, not an alias. The consume-once rule keeps both entries. Verified
in the replay below and pinned by
test_one_overlay_is_consumed_by_at_most_one_return_acceptance.Verification 1 — replay over the live campaign
Rebuilt the journey for every
workflow_return.jsonunder/shared_nfs/hyperloom-claw/*/*/geak/e2e_cycle*/, onab9dd2dband on thisbranch, and diffed the emitted entries.
kernels[]entriesKEEPentriesEvery removed entry carried
gpu_pct: nulland ane2e_gain_pctidentical to asurviving measured entry. Nothing measured was dropped. Ten sessions changed;
they are listed in the issue.
Downstream, Hyperloom's collector credited 8 of these duplicates as real
kernels, so the number on the TOP Model CI dashboard was over-counting GEAK's
kernels by 35 %.
Verification 2 — tests
Eight new tests in the existing
interface/test_run_e2e_dispatch.pyjourneysuite. Each pins one rule, and each is modelled on a real session:
test_return_acceptance_already_on_disk_as_an_overlay_is_not_re_emittedtest_sibling_candidates_for_one_symbol_each_stay_their_own_entrytest_symbol_less_overlay_folds_the_return_on_its_integrated_deltatest_an_ambiguous_delta_never_folds_two_kernelstest_one_overlay_is_consumed_by_at_most_one_return_acceptancetest_a_rejected_overlay_claims_nothingtest_an_incomplete_ab_overlay_claims_nothingintegrate_resultat alltest_folded_return_acceptance_is_absent_from_synthetic_discoveryFull
interface/suite, same interpreter, same machine:Reproduce the replay
Not addressed here
integrate_result.short_nameholds the real profiler symbol. Changing theemitted
kernel_idwould move the fold key the orchestrator's assembler uses,so it belongs in its own change.
8d108b0) stays. It has to:the 77 journeys already written to disk still contain the duplicates, and the
dashboard reads those files. Once this ships, that collapse is a no-op on new
runs.