Fix/kernel op identity

[chao-xu-spec]

Verify kernel selection against the running server instead of trusting the extraction

Refs #405

Problem

A run picks a GPU kernel out of a profile, then binds an authored replacement at a Python
callable in the live server. Those are two different identities and the orchestrator only ever
checked one. A profile row names a device symbol; a deployable seam is a module:attr that
actually launches it. Nothing checked that the selected callable was on the live path at all,
let alone that it was the innermost one that launches the profiled kernel.

That is how the 20260802 attention run banked a 4.47x isolated speedup that moved the server 0%:
the selected callable was an outer wrapper and the kernel it "replaced" was launched further
down the chain. No static gate can catch this. Of the 210 archived extractions in the 88-run
corpus, 188 carry a syntactically perfect module:attr target — the run that worked (20260720)
and the run that did not (20260802) are indistinguishable on paper. They differ only in what
actually executes.

What this changes

Selection now has to produce runtime evidence, and the gate fails closed without it.

• scripts/parse_profile.py — every Top-N row declares what it is (entity_kind:
  gpu_kernel / memory_op / dispatcher_op / python_launcher / unresolved) together with
  the evidence the decision rests on. Rows confirmed to be dispatcher aggregates are expanded
  into the device kernels they launched, using External-id edges, so an aggregate can no longer
  hide the kernel underneath it.
• scripts/seam_trace.py (new) — installs profiler markers on the declared seam candidates,
  including dotted class-method specs such as Runner.run. Only pure Python callables are
  wrapped; JIT/native objects exposing fn / cache / warmup / run / __torch_dispatch__
  are refused rather than corrupted. Traces are exported per PID/rank.
• scripts/kernel_selection.py (new) — reads the trace and returns a machine verdict: did the
  selected callable run, did the profiled kernel launch inside its span, and is there a live
  candidate nested deeper. Launch causality is established from External-id and launch-correlation
  edges on the marker's own thread inside the marker span, never from timestamp overlap.
• e2e_workflow.js — kernelSelectionVerified gates completion on that verdict. "Deepest" is
  settled by the profiler's observed nesting; a self-reported depth can add a rejection but can
  never buy a pass. A head naming no device symbol is refused at admission. When retries are
  exhausted the run is forced to smoke:'fail' with selection_failed:true.

Evidence

1. Production end-to-end run: the descent happens, on a live server

Full --mode e2e run of Qwen3.5-122B-A10B-FP8 on SGLang (BACKEND=sglang TP=1 GPU=0),
driven by the orchestrator rather than by hand. seam_trace.py in that run is byte-identical to
this branch. The run produced machine verdicts for two different operator families.

MoE — the seam actually descended off the wrapper it used to sit on. Earlier attempts on this
head targeted the triton_utils.fused_moe alias. This run marked all six declared candidates,
including the class-method candidate TritonRunnerCore.run, and the verdict resolved to the
direct fused_moe_kernel[grid] launcher — a target that appears in none of the previously
attempted callables:

Field	Value
target_callable	sglang.kernels.ops.moe.fused_moe_triton_kernels:invoke_fused_moe_kernel
device_kernel	fused_moe_kernel
ok / deepest_verified	true / true
target_marker_calls	128 live marker spans
matched_kernel_calls	64 launches attributed to the seam
correlated_external_ids	514
candidate_targets_tested	6 of 6 declared candidates installed, 4 of them live
deeper_live_candidates / missing_candidate_markers	[] / []

Four of the six declared candidates were live and the verdict still landed on the innermost one,
with nothing observed nested below it. That is the 0802 failure mode resolved in the opposite
direction, in production, on an operator family unrelated to attention.

GEMM — three seam specifications, three certified verdicts. For the CK kernel
kernel_gemm_xdl_cshuffle_v3_multi_d_blockscale_b_preshuffle:

Seam under test	ok / deepest_verified	marker spans	matched launches	External-id edges
aiter.ops.gemm_op_a8w8:gemm_a8w8_blockscale_bpreshuffle	true / true	12	3	161
sglang...fp8_utils:gemm_a8w8_blockscale_bpreshuffle	true / true	12	3	161
aiter.ops.gemm_op_a8w8:gemm_a8w8_blockscale_bpreshuffle_ck	true / true	12	3	134

Entity classification ran on the production profile. profile_topN.json from that run carries
entity_kind_contract: v1 with all 25 of 25 Top-N rows resolved to gpu_kernel, over 13,922
kernel launches and 72 distinct kernels — the head track is fed by rows that declared what they
are, not by rows assumed to be kernels.

2. Controlled hardware validation: the wrapper is refused, the launcher is certified

A second, deliberately adversarial check on one MI355X, Qwen3.5-122B-A10B-FP8 served by vLLM
v0.24.0 (V1 engine, in-process). head_dim=256 forces the Triton paged-attention path — the exact
call chain the 0802 run mis-selected. 60 traces over prefill and decode; capture meta records the
target called 36 times across 3 cases covering both regimes.

Selection put to the contract	Verdict
rocm_attn:chunked_prefill_paged_decode, kernel kernel_paged_attention_2d (decode)	ok:true, deepest_verified:true, 72 live marker spans, 36 matched launches, 447 External-id / 714 launch-correlation edges, no deeper live candidate
same target, kernel _fwd_kernel (prefill)	ok:true, deepest_verified:true, 12 matched launches
the 0802 selection: outer wrapper attention:unified_attention_with_output	ok:false, failed=[capture_target_mismatch, device_kernel_not_under_target, target_marker_missing], 0 matched launches

The selection that produced the 4.47x paper win cannot reach extraction. The launcher that
actually issues the kernel is certified, backed by 48 individual launch edges.

Attribution does not depend on a single correlation channel. Kernel rows in that trace divide
cleanly: all 73 at::native::* rows carried both an External id and a correlation, while all
48 Triton rows — 36 kernel_paged_attention_2d plus 12 _fwd_kernel — carried a correlation
and no External id. Binding attribution to either edge covers both shapes. The correlation edge
is held to the same strictness as the External-id one: the host launch event must sit on the
marker's own thread, inside the marker span. An edge falling outside the span is refused, and
merged multi-process traces prefix correlation so ids from different ranks cannot collide. Both
properties are pinned by test_launch_correlation_outside_the_marker_span_is_not_enough and
test_correlation_ids_do_not_collide_across_merged_call_traces.

3. Regression suites

python3 -m pytest -q e2e_workflow/scripts/tests — 890 passed, 1 skipped on a clean tree.
test_kernel_selection.py alone carries 19 cases pinning each rejection branch, including:

• test_0720_live_inner_launcher_is_selection_success — the live inner launcher passes.
• test_0802_outer_wrapper_fails_when_0720_launcher_is_marked — the wrapper is refused when a
  deeper live candidate is observed.
• test_triton_kernel_without_external_id_is_linked_by_launch_correlation — a device row carrying
  only correlation is still attributed to the seam that launched it.
• test_the_observed_nesting_is_what_settles_deepest_not_the_declared_depth — a candidate
  appended with depth:999 cannot outrank what the profiler observed.
• test_device_projected_annotation_does_not_invert_call_nesting — the profiler re-emits every
  marker on the device timeline, where an outer seam's short span lands inside an inner one's;
  only host spans establish nesting, and the correct launcher still passes.
• test_every_declared_probe_candidate_must_have_an_installed_marker and
  test_a_probed_candidate_that_never_fired_is_reported_not_silently_dropped — the candidate set
  has to be complete, and "never probed" is reported apart from "probed and never observed".
• test_cli_requires_deepest_selection_on_every_capture_process — one TP rank disagreeing sinks
  the verdict.

4. Replay against the 88-Hyperloom-run archive

An out-of-repo harness runs the real predicates — e2e_workflow.js source slices executed
through QuickJS, and the shipped Python scripts driven as subprocesses — over 210 archived
meta.json from 88 Hyperloom production runs. 260 passed / 1130 subtests. On that real archived data:

Check	Result
kernelSelectionVerified passing on archived evidence	0 / 210 — every archived extraction predates the machine verdict, and the gate fails closed on all of them
Prose (non-module:attr) targets refused	11 / 210, with 0 prose targets ever emitted as a machine target by prepareHeadSelection
Extractions the archive proves lacked a frozen baseline	11 / 210, all flagged

Entity classification was run with the shipped parse_profile.py --annotate over every archived
round whose torch trace survived: dispatcher_op resolves to 0, because confirmed dispatcher
aggregates are replaced by the device kernels they launched — precisely the descent step the 0802
head needed and the old behaviour discarded.

[iraj465]

Reviewed at c040738 (15 files, +2616/−37 vs main @ c0433ba). The 319 tests
in the five suites pass locally. The design is right: replacing a static
no_rebind_seam guess with installed-marker + External-id/correlation evidence
is the correct shape of fix for #405, and installed_but_never_live_candidates
being reported rather than read as "not deeper" is the honest call.

One blocking finding. canonical_kernel_name is wrong on real ROCm profile
names, and the tests do not catch it because every canonicalization case in
test_kernel_selection.py is namespaced (void vllm::..., void ns::...),
where the :: split rescues it.

Two mechanical causes, both in the same three lines:

• text.split("(", 1)[0] truncates at the first (. For
  void (anonymous namespace)::clamp_position_kernel<long>(...) that is the (
  of (anonymous namespace), not the argument list, so the token becomes void.
• re.sub(r"[^a-z0-9_]+", "", ...) deletes separators instead of splitting on
  them, so a namespace-free void wvSplitKrc_<...> becomes voidwvsplitkrc_.
  (?:^|_) then cannot anchor, and the head never matches its own kernel.

Measured over every profile_topN.json in /shared_nfs/hyperloom-claw/
(127 profiles, 3003 head rows, 679 distinct kernel names) — for each row I
compared the identity the gate would use (short_name, via
requiredDeviceKernel) against that row's own trace name:

	current	reusing parse_profile.short_name
head fails to match its own kernel	1486 / 3003 (49.5%)	120 / 3003 (4.0%)
distinct degenerate tokens ("" or <6 chars)	8	7
kernels sharing a degenerate token	23	19

Both directions are wrong, and the false positive is the worse one:

• False negative. paged_attention_ll4mi_QKV_mfma4_kernel vs
  void paged_attention_ll4mi_QKV_mfma4_kernel<__hip_bfloat16, ...> →
  voidpaged_attention_ll4mi_qkv_mfma4_kernel → no match →
  device_kernel_not_under_target → after BASELINE_EXTRACT_RETRIES the new
  code sets smoke:'fail', selection_failed:true. A correct seam is killed.
  31 distinct names have void fused on this way; 54 of 127 profiles carry at
  least one degenerate row.
• False positive. 9 distinct kernels all reduce to the token void, and
  kernel_matches returns want == got before the len(want) >= 6 guard
  applies. So clamp_position_kernel and elementwise_kernel_with_index
  mutually certify. That is the gate granting matched_kernel_calls > 0 and
  ok: true for a kernel the seam did not launch — the exact credit this PR
  exists to withhold.

Suggested fix, in the repo's own idiom rather than a new special case:
parse_profile.short_name() already strips ^void\s+, takes [\w:]+ (which
stops at both ( and <), and drops the namespace — and norm_key() already
folds on top of it. Building canonical_kernel_name on short_name gets the
table above, deletes _strip_template_args entirely, and leaves one
canonicalizer in the repo instead of two. Two additions still needed:
strip a leading (anonymous namespace):: (most of the residual 120), and move
the length guard so it also covers the equality branch.

Second, smaller: the JS and Python matchers are not the pair the comment
claims. _strip_template_args says it "must stay identical to
canonicalDeviceKernel in e2e_workflow.js". The canonicalizers do agree, but the
matchers built on them do not — kernelIdentitiesMatch tests containment in
both directions, kernel_matches only want in got:

(x.length >= 6 && rx(x).test(y)) || (y.length >= 6 && rx(y).test(x))   // JS

len(want) >= 6 and re.search(..., got)                                  # PY

The asymmetry may well be deliberate (the JS compares two declared symbols, the
Python compares a declared symbol to trace events, where the reverse direction
would be unsafe). If so, please scope the comment to the canonicalization only —
as written it tells the next maintainer the two are interchangeable. A shared
fixture of real symbol pairs, asserted on both sides, would pin whichever
contract you intend.

Three minor points, none blocking:

1. seam_trace._trace_path: with GEAK_SELECTION_TRACE_UNIQUE=0 it returns
   template un-formatted, so a template containing {pid}/{rank} yields a
   file literally named ...{pid}.... It also drops the pid/rank suffix that
   kernel_selection.main matches on (\.pid-N(?:\.|$)), so every rank races one
   path and the per-process pairing fails to capture_process_trace_missing.
   Returning the formatted path there is the one-line version.
2. kernel_selection.main picks a max(attempts, ...) for
   selected_meta_path, then overwrites nearly every field with an aggregate
   across all attempts and reassigns selected_paths to all paths (L409). The
   emitted capture_meta_file therefore names one process while the counts next
   to it are sums. Either drop the max or label the field
   best_process_meta_file.
3. capture_shapes._wrapper now emits GEAK_TARGET::<target> as well as
   seam_trace.install. When the capture target is also a marker candidate the
   same span name nests inside itself, so target_marker_calls double-counts.
   Only compared against > 0 today, so it is a reporting nit.

_start_profile has one narrow leak: _PROFILE.update(active=True, ...) runs
before _record_install_markers(), so if that raises, the except sets
done=True while leaving active=True and the profiler entered —
_finish_profile then returns early and never exits it. Moving the update after
the marker write closes it.

Happy to re-review once the canonicalizer lands; the rest of the contract reads
well and I could not fault the nesting/causality logic.

Note: no node runtime is available in my environment, so the JS above is
reviewed by reading only; every Python number in this comment came from
executing the branch's own code over the live campaign.

[chao-xu-spec]

Thanks for the detailed review and corpus measurement. I reproduced the reported failure mode and addressed all blocking, secondary, and minor findings.

What changed

Kernel identity canonicalization

The Python and JavaScript canonicalizers now follow the same contract:

• Remove balanced <...>, (...), and [...] groups innermost-first.
• Cut at an opener that never closes instead of treating an elided template fragment as a kernel name.
• Handle (anonymous namespace) anywhere in a qualification.
• Drop the leading void explicitly and take the first identifier, matching parse_profile.short_name.
• Ignore pipeline annotations such as _fwd_grouped_kernel_stage1 [sliding_attention], main_kernel[prefill], hgemm_... [qkv_proj], and Memcpy DtoD (Device -> Device).
• Compare complete template arguments exactly; elided arguments must agree over their visible prefix.
• Preserve separators while folding arguments, so <128, 4, ...> does not match <128, 48, ...>.
• Remove unsafe substring containment, so _fwd_kernel cannot certify _fwd_kernel_stage2.
• Allow prefix matching only at the shared 60-character display truncation boundary.

parse_profile.short_name now strips the anonymous-namespace prefix correctly and exports SHORT_NAME_LIMIT, so Python and JavaScript use the same truncation contract.

I did not canonicalize solely through short_name, because that discards template arguments. The corpus contains 20 distinct vectorized_elementwise_kernel instantiations sharing the same base name and differing only in template functors; stripping those arguments would allow them to certify each other.

Python/JavaScript parity

A shared fixture now contains 15 symbols taken verbatim from ROCm captures, including anonymous namespaces, namespace-free symbols, elided templates, pipeline annotations, and memcpy rows.

It is asserted by:

• Python: test_kernel_selection.py
• JavaScript: test_kernel_canonicalization_parity.js

The JavaScript test extracts the implementation directly from e2e_workflow.js rather than testing a copied implementation, and is wired into the node-regression CI job.

Other review findings

1. seam_trace._trace_path now returns the formatted path when GEAK_SELECTION_TRACE_UNIQUE=0, preserving the PID/rank suffix required for process-local trace pairing.

2. The process-specific metadata field is named best_process_meta_file; aggregate counts are no longer presented as belonging to that one process.

3. Duplicate target-marker accounting is fixed. capture_shapes and seam_trace can both emit the same GEAK_TARGET::<target> marker, nesting one span inside the other for a single invocation. The verdict now counts only the outermost same-thread copy as one logical target_marker_calls entry, while retaining every raw span for launch-causality analysis. Concurrent markers on other threads and later non-nested calls remain distinct.

4. _start_profile records install markers before setting the profiler active. If anything fails after __enter__, the profiler is explicitly exited; a close failure is reported rather than silently swallowed.

Corpus validation

I repeated the review methodology over every current profile_topN.json under /shared_nfs/hyperloom-claw/. The corpus has grown to:

• 136 profiles
• 3,206 head rows
• 690 distinct kernel names

Measurement	Before	Updated PR
Head fails to match its own stored kernel identity	1,589/3,206 (49.6%)	161/3,206 (5.0%)
Degenerate tokens ("" or fewer than 6 characters)	8	6
Kernels sharing a degenerate token	25	14

All 161 residual rows contain a historical short_name that disagrees with the fixed parse_profile.short_name; they were generated by the bug being fixed and are frozen in existing artifacts.

Re-deriving short_name from each row's original name, as newly generated profiles now do, gives 0/3,206 self-match failures (0.0%).

The template-aware matcher also reduces the distinct-symbol collision surface: 158 accepted distinct-symbol pairs versus 942 for a short_name-only matcher. In particular, the 20 different vectorized_elementwise_kernel instantiations no longer mutually certify.

For cross-language parity, I compared Python and JavaScript over:

• 1,101 distinct symbols from the live corpus
• 21,101 symbol-pair verdicts
• 0 canonical-token disagreements
• 0 match-verdict disagreements

All review findings are now addressed. Please re-review when convenient.

[iraj465]

Approving. The blocking canonicalization defect is fixed, and I re-measured it against the live campaign rather than taking the tests' word for it.

A/B on /shared_nfs/hyperloom-claw/ — 3202 profile rows, 20260820

metric	c040738 (reviewed)	2253f32 (head)
head fails to match its own kernel	1486 / 3003 (49.5%)	0 / 3202 (0.0%)
'void' collision bucket	9 kernels	0
degenerate tokens	8	0
M(clamp_position_kernel, elementwise_kernel_with_index)	True (false positive)	False
the three cited false negatives	all False	all True

The 0% is the number that matters, and it is only 0% because the fix landed on both halves. Measuring the new matcher against the stored short_name still leaves 172 residuals (5.4%) — but every one of those is an artifact the old parse_profile.short_name produced: 'StoreKVCacheParams)', 'FastTopKParams, int*, int const*, long)', 'result_type*)', i.e. the trailing parameter type where the kernel name should be. The (anonymous namespace) strip in parse_profile.py fixes them at the producer. Replaying producer + matcher together, the residual is exactly zero.

What I checked

• Full suite: 1076 passed, 2 failed, 4 skipped, 66 subtests. Both failures are in test_bench_e2e_teardown_lookup.py, which this PR does not touch; I checked out origin/main and ran that file alone — 2 failed, 2 passed. Pre-existing, out of scope.
• Parity fixture — this answers my second finding. kernel_symbols.json carries 20 canonical symbols (15 lifted verbatim from ROCm captures) and 17 match verdicts, 10 of them negative, so it pins the false-positive direction too. The Python half (TestTheSharedFixtureHoldsOnThisSide) passes: 3 tests / 37 subtests. ci-l0-checks.yml wires the node half into the existing node-regression job.
• The parity test extracts the block from e2e_workflow.js by source offset rather than reimplementing it. That is the right call — a copy would pass while the shipped code drifted.
• All four minor points addressed: _trace_path under GEAK_SELECTION_TRACE_UNIQUE=0 keeps the per-process suffix; _start_profile now closes a profiler that fails after __enter__; _record_install_markers() has a single call site; no vestigial attempts clamp remains.
• _template_arguments discrimination for the 20-way at::native::vectorized_elementwise_kernel collision was not something I asked for and is a real improvement — the base token alone cannot separate those.

Caveats on this review, stated plainly

1. I could not execute the JS. There is no node, deno, bun, or qjs in this environment. canonicalDeviceKernel and kernelIdentitiesMatch were reviewed by reading only. I walked them step-for-step against canonical_kernel_name / kernel_matches and they agree in order and in every transform. CI's node-regression job is what actually proves it.
2. Non-blocking, for a follow-up: Python's [\w:]+ is Unicode-aware and JS's is ASCII-only, so a non-ASCII identifier character would canonicalize differently. No ROCm symbol will hit this, and the fixture cannot catch it. Worth one ASCII-only probe in the Python regex if you touch this again.

Neither caveat blocks the merge.