Publish the throughput pair an e2e delta is a ratio of - #427
Merged
Conversation
A journey e2e block states `e2e_gain_pct` and nothing else. A percentage alone does not compose: two wins measured against different denominators cannot be added. Every consumer holding only percentages is forced to sum them, which is a unit error, not a gain. The two numbers are already measured. `integrate_result.json` carries `ref_med` and `cand_med` from the same A/B that produced the delta, and 25 of the 26 accepted/stack overlays on disk reproduce their own delta from that pair to within 4.74e-03 pp. GEAK measured the denominator and threw it away. Publish it. Both journey builders now emit `base_tput`/`new_tput` alongside the delta, read off that same A/B, and the workflow carries the pair on each accepted head/kernel record so the live path has it too. A run that genuinely has no pair emits null rather than a fabricated denominator. Replaying all 36 KEEP blocks in /shared_nfs/hyperloom-claw through the consumer's real arithmetic: without the pair 34 of 36 steps fall back to summing local percentages; with it 23 take the composable baseline chain. The remaining 12 are live-path KEEPs whose runs predate this change. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
zihaoanllm
approved these changes
Aug 21, 2026
chao-xu-spec
approved these changes
Aug 21, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #426.
The defect
A journey e2e block publishes
e2e_gain_pctand nothing else. A percentage isa ratio; two ratios with different denominators cannot be added. So a consumer
holding only percentages is forced into a unit error.
GEAK already measures both numbers —
ref_medandcand_medinintegrate_result.json, from the same A/B that produced the delta — and dropsthem before publishing.
Evidence
/shared_nfs/hyperloom-claw, measured 2026-08-20, Hyperloom CI only.Every KEEP block is missing the pair, and the naive sum is meaningless:
The denominator was measured and thrown away:
accepted/stackoverlaysref_medandcand_medWhat the consumer can do once it has the pair — all 36 KEEP blocks replayed
through Hyperloom's real recorder and real
collect_recorded_optimizations,counting the arithmetic branch each credited step takes:
gain_methodbaseline_chain— composable, points of one baselinelocal_gain_projected— sums percentagesmissingThe composable arithmetic is already fully implemented on the Hyperloom side.
It never fires because its input is not published. The 12 remaining are
live-path KEEPs from runs predating this change.
The change
One rule, three places:
interface/run_e2e.py—_journey_overlay_entryref_med/cand_medfrom the sameintegrate_result.jsonthe delta comes from, through the existing_ir_getso both flat and nested integrator shapes workinterface/run_e2e.py—_journey_return_entrye2e_workflow/e2e_workflow.jse2eFrom(integ)helper on each accepted head/kernel push, so the live path carries it tooField names are
base_tput/new_tput— the names the consumer alreadyreads. One name end to end, no rename in the middle.
A run with no measured pair emits
null. It never fabricates a denominator.Tests
interface/test_run_e2e_dispatch.py::TestE2EDenominatorIsPublished, 4 tests:..._publishes_the_pair_and_it_reproduces_the_deltanew/basereproducese2e_gain_pct..._reads_the_pair_from_the_nested_shape_too..._return_path_carries_the_pair......_a_missing_pair_is_null_not_fabricatedtest_run_e2e_dispatch.py+test_run_e2e_recovery.pyScope and limits
analysis host, so
e2e_workflow.jswas not executed. The edit is 11mechanical call-site substitutions plus one helper; the diff is deliberately
uniform so it reads at a glance.
history.ledgersites were deliberatelyleft alone.
alias twins across 9 of the 15 sessions still double-count. That is the
producer bug kernel_journey writes one acceptance twice when the overlay tag and the kernel symbol differ #407/fix(journey): fold a return-named acceptance into its overlay entry #408 fixed (merged 2026-08-19); every replayed session
predates it. Not reintroduced or worsened here.
Llama-3.1-8B-Instruct/20260816T114410Z→
cand_pa_ll4mi_c0_hippublishese2e_delta_pct_all_repeats(1.330) ande2e_delta_pct_equal_count(4.298) and no plain delta, soe2e_gain_pctisalready
Nonethere today. Choosing a canonical variant is not something thedata settles. Noted in kernel_journey e2e blocks publish a percentage with no denominator, so gains cannot be composed #426 as its own follow-up.
(fix(breakdown): put GEAK's kernel journey on the canonical streams Hyperloom#1252). Both are needed before the dashboard's
Kernel Agent Gain (GEAK)column is both non-zero and correct.