Skip to content

Publish the throughput pair an e2e delta is a ratio of - #427

Merged
zihaoanllm merged 2 commits into
mainfrom
fix/publish-e2e-denominator
Aug 21, 2026
Merged

Publish the throughput pair an e2e delta is a ratio of#427
zihaoanllm merged 2 commits into
mainfrom
fix/publish-e2e-denominator

Conversation

@iraj465

@iraj465 iraj465 commented Aug 20, 2026

Copy link
Copy Markdown
Collaborator

Closes #426.

The defect

A journey e2e block publishes e2e_gain_pct and nothing else. A percentage is
a ratio; two ratios with different denominators cannot be added. So a consumer
holding only percentages is forced into a unit error.

GEAK already measures both numbers — ref_med and cand_med in
integrate_result.json, from the same A/B that produced the delta — and drops
them before publishing.

Evidence

/shared_nfs/hyperloom-claw, measured 2026-08-20, Hyperloom CI only.

Every KEEP block is missing the pair, and the naive sum is meaningless:

n
KEEP journey e2e blocks on disk 36
...carrying a throughput pair 0
naive sum of their percentages 348.6 pp / 15 sessions

The denominator was measured and thrown away:

n
accepted/stack overlays 26
...carrying ref_med and cand_med 25 (96.2%)
...whose pair reproduces its own delta 25 / 25
worst deviation 4.74e-03 pp

What the consumer can do once it has the pair — all 36 KEEP blocks replayed
through Hyperloom's real recorder and real collect_recorded_optimizations,
counting the arithmetic branch each credited step takes:

gain_method before after
baseline_chain — composable, points of one baseline 0 23
local_gain_projected — sums percentages 34 12
missing 1 1

The composable arithmetic is already fully implemented on the Hyperloom side.
It never fires because its input is not published. The 12 remaining are
live-path KEEPs from runs predating this change.

The change

One rule, three places:

Publish the two numbers the percentage is a ratio of, taken off the same A/B
that produced it.

file change
interface/run_e2e.py_journey_overlay_entry read ref_med/cand_med from the same integrate_result.json the delta comes from, through the existing _ir_get so both flat and nested integrator shapes work
interface/run_e2e.py_journey_return_entry carry the pair off the accepted-kernel record
e2e_workflow/e2e_workflow.js one e2eFrom(integ) helper on each accepted head/kernel push, so the live path carries it too

Field names are base_tput / new_tput — the names the consumer already
reads. One name end to end, no rename in the middle.

A run with no measured pair emits null. It never fabricates a denominator.

Tests

interface/test_run_e2e_dispatch.py::TestE2EDenominatorIsPublished, 4 tests:

test asserts
..._publishes_the_pair_and_it_reproduces_the_delta pair present, and new/base reproduces e2e_gain_pct
..._reads_the_pair_from_the_nested_shape_too nested integrator output is not silently dropped
..._return_path_carries_the_pair... live path carries it
..._a_missing_pair_is_null_not_fabricated no invented denominator
result
new tests without this change 4 / 4 fail
new tests with this change 4 / 4 pass
test_run_e2e_dispatch.py + test_run_e2e_recovery.py 205 passed

Scope and limits

A journey e2e block states `e2e_gain_pct` and nothing else. A percentage
alone does not compose: two wins measured against different denominators
cannot be added. Every consumer holding only percentages is forced to sum
them, which is a unit error, not a gain.

The two numbers are already measured. `integrate_result.json` carries
`ref_med` and `cand_med` from the same A/B that produced the delta, and
25 of the 26 accepted/stack overlays on disk reproduce their own delta
from that pair to within 4.74e-03 pp. GEAK measured the denominator and
threw it away.

Publish it. Both journey builders now emit `base_tput`/`new_tput`
alongside the delta, read off that same A/B, and the workflow carries
the pair on each accepted head/kernel record so the live path has it too.
A run that genuinely has no pair emits null rather than a fabricated
denominator.

Replaying all 36 KEEP blocks in /shared_nfs/hyperloom-claw through the
consumer's real arithmetic: without the pair 34 of 36 steps fall back to
summing local percentages; with it 23 take the composable baseline chain.
The remaining 12 are live-path KEEPs whose runs predate this change.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@zihaoanllm
zihaoanllm merged commit c44529a into main Aug 21, 2026
9 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

kernel_journey e2e blocks publish a percentage with no denominator, so gains cannot be composed

3 participants