WIP: discover kernel-source roots at runtime, and replace the two LLM source tiers with one agent review - #1263
Open
jiaqiang-dot-liu wants to merge 10 commits into
Open
Conversation
KNOWN_SEARCH_ROOTS pinned one container layout (/sgl-workspace plus a python3.10 venv). On a host that installs the frameworks anywhere else every root was absent, so the grep tier searched nothing and resolved no kernel at all. The LLM tiers could not recover: their shortlist comes from the same grep, and their validation gate rejects any path outside those absent roots, so even a correct rewrite was refused. The run still reported success with zero routable kernels -- indistinguishable from a trace with nothing worth optimizing -- and kernel-opt sat idle with no work to dispatch. Resolve the roots through the orchestrator's centralised resolver instead, so this tool agrees with PolicyGate and patch application about where framework source lives, and so a wheel install, an editable checkout and a serving image all work without naming any of them. The pinned layouts stay only as a last-resort fallback for standalone CLI use, and non-existent roots are always dropped. A host with no searchable root now says so, in the log and as a no_framework_source_root trace-health warning, rather than looking healthy. Restoring the roots exposed a latent collision: a vendor-playbook operator such as mori::EpDispatchCombineOp::dispatch reduces to the keyword "dispatch" and greps onto an unrelated vendor header, displacing the task-bundle anchor. A registry match is a curated statement that the operator has no rewritable device source, so let its anchor override the guess -- otherwise a backend is handed the wrong file to rewrite. Verified offline against a recorded MiniMax-M3 run: routable kernels 0 -> 6, covering 13.74% of GPU time, with three resolved by trace-launcher corroboration that previously could not pass cross-validation. Co-authored-by: Cursor <cursoragent@cursor.com>
…te table. Source resolution had two model tiers. The per-kernel fallback picked a path from a grep shortlist for any kernel above 5% of GPU time; the whole-table pass audited the finished resolution table. The first was the weaker of the two and mostly redundant: it fired only where the second already looked, at a higher threshold, and it saw less -- a symbol and eight paths, without the launcher frames the table pass gets. Neither could see what the kernel actually is, because both were tool-free completions fed a prompt assembled in advance. Both are gone. In their place one agent session audits the candidate table itself, and is handed paths rather than contents: the raw table, the resolution audit, the TraceLens report and per-category metrics, the model directory, the framework source roots. It reads what the evidence leads it to instead of what was guessed to be relevant ahead of time, so it can open a file and confirm it defines the kernel it is credited with rather than ranking paths by keyword. The pass is mandatory on the agent route and ungated: the thresholds only ever described what a completion was worth, and the deterministic route keeps its own no-LLM guarantee by not running the stage at all. It retries once, and a definitive failure records an error-severity trace-health warning and leaves the deterministic table standing -- losing the audit costs some candidates, while failing the run would cost the hours of benchmarking behind the trace. Three bounds keep the added freedom from becoming a new failure mode: - Proposals only. The session may revise where a kernel lives and whether it is worth dispatching; it may not touch what the trace measured. Derived state is recomputed by the deterministic stamping pass afterwards, so classify_patchability stays the one gate rather than gaining a model-written second one, and a restrictive routability hint is honoured while a permissive one is not. - A revised path must resolve under a known framework root, and a candidate already resolved by the active finder is not overridable: reading the tree cannot beat knowing which symbol the binary exports. - The session may run shell commands, so the candidate source files are fingerprinted around it and a review that modified them is discarded. The benchmark that follows would otherwise measure an unrecorded edit. Artifacts are now kernel_candidates.raw.json (deterministic), kernel_candidates.json (reviewed) and kernel_candidates_revisions.json (what changed and why), so a bad dispatch can be traced to the stage that caused it. Verified against a recorded MiniMax-M3 run: the deterministic stage is unchanged at 6 of 16 routable covering 13.74% of GPU time. WIP: the new module's own test suite is still to come. Co-authored-by: Cursor <cursoragent@cursor.com>
jiaqiang-dot-liu
requested review from
a team,
devalshahamd and
tsrikris
as code owners
August 21, 2026 15:07
![github-code-quality[bot]](https://avatars.githubusercontent.com/u/9919?s=60&v=4){kind=small}
Comment on lines
+223
to
+224
| "Audit the kernel-candidate table produced by the deterministic " | ||
| "analysis stage and correct it where the evidence disagrees.", |
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Comment on lines
+223
to
+224
| "Audit the kernel-candidate table produced by the deterministic " | ||
| "analysis stage and correct it where the evidence disagrees.", |
The stage is advisory by construction, and it runs at the very end of an analysis that hours of benchmarking paid for. The session driver and the revision loader already report their own failures, but the code around them -- writing the raw table, fingerprinting, applying revisions, re-deriving -- could still raise past the caller and take the whole analysis down with it. Split the entry point from the body so nothing escapes: an unforeseen fault becomes the same error-severity trace-health warning a failed session produces, and the deterministic table stands. Losing the audit costs some candidates; losing the run costs the trace. Co-authored-by: Cursor <cursoragent@cursor.com>
tsrikris
reviewed
Aug 21, 2026
| source_paths = [str(c.get("source_file") or "") for c in candidates if isinstance(c, dict)] | ||
| before = source_fingerprint(source_paths) | ||
|
|
||
| reference_paths = { |
Collaborator
There was a problem hiding this comment.
@jiaqiang-dot-liu Not all that these files are not meant to be read in practice. The only source of truth is analysis.md. If you continue to keep reading that we will add a rm -rf to clear out all files execept analysis.md
CI E2E report — ❌ Timeout
|
…t are absent. Two findings from an 8k/1k MiniMax-M3 run whose batch dispatcher never selected a single candidate. The 10% GPU-share floor assumed hot kernels are concentrated. A 60-layer sparse-MoE decoder spreads its work so thin that nothing but a graph-launch wrapper reached double digits: the largest real operator sat at 9.47% and missed by half a point, so every batch pass logged "0 selected" for six hours while the orchestrator picked candidates one at a time. Two of five routable kernels were reached before the run ended. The floor is now 5%, which admits the operators this model actually has; ``HYPERLOOM_KERNEL_OPT_MIN_GPU_PCT`` still overrides it. The curated harness table pinned absolute ``/sgl-workspace`` paths, and the lookup dropped its existence check whenever no repo root was known -- which is always, for a wheel install with no ``.git`` above it. The hottest candidate therefore shipped three harness paths that do not exist on this host, into kernel_candidates.json and on into the dispatch prompt. The comment explaining the exemption asked for stable test fixtures in containers without ``/sgl-workspace``; that is a reason to build fixtures in the test, not to let production fabricate paths, because a non-empty list reads downstream as a harness someone can run. Hints are now checkout-relative and resolved against the same runtime-discovered roots the source tiers use, so a serving image and a wheel install both work without either being named, and only files actually present are returned. On this host that turns three fabricated paths into an honest empty list. The review session may also revise benchmark_files. The table is keyed by coarse name markers, so it offers a harness for a whole kernel family rather than the member in hand -- a session that can open the tree is better placed to judge, and its answer is filtered to paths that exist before it is applied. Co-authored-by: Cursor <cursoragent@cursor.com>
The batch filter drops a candidate below the GPU-share floor and says so in its log, then discards the reason. When the orchestrator had named that kernel, the handler canonicalized the id and fell through to the validation guards, whose failure result was recorded as an attempt. The 9.47% kernel of the last 8k/1k run therefore shows two attempts and two failures against a backend that never ran, and reads in the report as a technical failure when the cause was a threshold. UNATTEMPTED_BELOW_MIN_GPU_PCT already exists to say this properly. The filter now hands its skip reasons back, and a named kernel dropped for one that means "no backend ran" -- below the floor, its group exhausted, merged into an op-fanout sibling -- returns skipped with that reason. The clean skipped exit existed already but sat behind an ``elif`` only reachable when no kernel was named, which is the one case that cannot happen when the batch queue is empty and the orchestrator is picking targets by hand. Two identity fixes come with it, and have to: the ledger keys on kernel_id plus source_file, and the empty-candidate branch never passed the source through even though it had the full candidate list in hand. Every rejection was filed under an empty source, which is also why the retry quota went to an empty-string bucket and the two failures never blocked redispatch. Backfilling the source without the change above would have turned that harmless accident into a real block, since a single recorded attempt exhausts the default cap of one -- so the guards now carry the resolved source, and the branch backfills the candidate it already loaded. Verified against the recorded run: naming the gate-rejected kernel returns status=skipped with reason=below_min_gpu_pct instead of a failure, and a guard rejection now carries its source_file. Co-authored-by: Cursor <cursoragent@cursor.com>
The review is handed a tool-enabled session because the deterministic tiers fail by being confidently wrong, and that freedom is what these tests bound. Three properties carry the weight, each with a concrete failure behind it: a measured field overwritten by a model would corrupt the impact ranking and the tuning harness computed from it; an invented path would hand a backend the wrong file to rewrite, which is the failure the pipeline exists to prevent; and a session that edited the framework tree would leave the benchmark that follows measuring an unrecorded change. Also covers the stage boundary in tracelens_analysis: an unforeseen fault must cost the audit rather than the run, a restrictive routability hint is honoured while a permissive one is not, and a harness list the session verified survives the restamping pass that would otherwise recompute it from the coarse marker table.
…rded. A graph replay has no CPU-side parent op, so the profiler records no arguments for a graph-launched kernel and `shapes` arrives empty. On this model that is every hot kernel: both 8k/1k runs resolved 0 of 13 and 2 of 12 candidates with dims. Empty is not a neutral state. With no shapes the tuning backend picks its own and cannot see the serving configuration, so a prefill kernel serving an 8192-token input was tuned at sequence lengths 512 and 1024, measured 1.1155x, and lost 3.05% end to end once integrated. The decode kernel of the same pair guessed close to the real batch and sequence and reported an honest 1.028x. So the choice was never measured dims against invented ones -- it was which side invents. The review already reads the model config, the serving arguments and the run's own artifacts, and the backend reads none of them. `shapes` and `input_dtypes` therefore leave IMMUTABLE_FIELDS, which now holds only what cannot be revised for a reason: the trace's own measurements, which feed the dispatch floor and the closing gain figure, and the keys the row is joined by. The alternate shape representations move to DERIVED_SHAPE_FIELDS and are rebuilt, since a harness assembled from a mix of old and new dims still benchmarks cleanly. Dims carry their own provenance and a session cannot claim `torch_trace`: collapsing recovered and computed would remove the one signal that says whether a disappointing integration is worth blaming on the shape. The forge prompt stops asserting a measurement it no longer has. Two things had to be fixed for any of it to fire. `apply_revisions` only read proposal fields on a rewrite that moved the path, so the rows that most need dims -- the ones the deterministic tiers already located -- would have had their shapes dropped at the `keep` branch; 10 of this run's 13 candidates take that path. And review dims had to enter DISPATCHABLE_SHAPE_PROVENANCE, or the gate would refuse them as untrusted, which is worse than empty: an empty shape has an override flag and an untrusted provenance has none. Replayed against the real 13-candidate table, all 7 routable candidates now carry dims, clear the gate and reach the backend's shape block, against 0 before. Also drops `recommended_backends` from the prompt. The stamping pass recomputes it, so inviting a revision there spent the session's effort on a value that was silently overwritten. And repairs a `_batch_kernel_candidates` monkeypatch left stale by the gate-rejection change, which the aborted full regressions hid. Co-authored-by: Cursor <cursoragent@cursor.com>
The correction only matters if it reaches invocation_spec_*.json, which is what forge reads to construct the test driver. It does: build_invocation_spec takes `shapes` as the fallback for `input_shapes`, so the review's dims arrive parsed into invocation.arguments and `inputs` leaves missing_fields. Nothing covered that link, and it is the one that decides whether the driver benchmarks the serving workload or a shape the backend chose for itself. Also pins that only the reviewed table can be resolved as the candidate source. The pre-review baseline sits in the same directory under kernel_candidates.raw .json, and resolving that would hand the backend exactly the dims the review corrected -- the one outcome the two-artifact split exists to prevent. Co-authored-by: Cursor <cursoragent@cursor.com>
The re-derivation is skipped for candidates that did not change, and the check asked only whether `source_file` moved. That was complete while a path was the only thing a revision could alter. It no longer is: operand dims are supplied most often for a kernel the deterministic tiers already located, so the rows carrying the hardest-won proposals are precisely the ones the check passed over. Two analyses in one 8k/1k run staged `review_backfill` dims on `keep` revisions and shipped a table with none -- the revisions file recorded the proposals and the candidates never took them. The dims that did land came from rows whose path happened to move for unrelated reasons. The condition now names every field a revision can stage. Pinned at the stage boundary rather than on the re-derivation itself: the unit tests called that function directly, which is why they passed while the pipeline dropped the data. Co-authored-by: Cursor <cursoragent@cursor.com>
The review was handed `category_data/` and `priority_data.json` alongside `analysis.md`. Only the last of those is a supported TraceLens output; the rest of that directory is internal and the maintainer has said it may be cleared. Building on it earns a dependency that can disappear between releases. Almost nothing is given up. For every operator the sidecars describe, `analysis.md` carries the same operand dims and the same `file(line): function` launcher in its own table -- the parser already reads both from there, and the CSV lookups only run once that came back empty. Checked against the current 8k/1k run: 4 of 14 rows carry dims, the same 4 the sidecars can account for, and for the other 10 neither has anything. A graph replay has no CPU-side parent op, so nothing recorded those arguments anywhere; withdrawing the sidecars does not make that worse. The backfill route survives, re-aimed at the contract. A composite operator that kept its module attribution names the device kernels it launches in its Kernel Name cell while carrying the arguments those kernels' own rows lack, so the dims are recoverable from `analysis.md` alone -- 2 of the 10 empty rows in this run. The prompt now asks for that, warns that a neighbouring instantiation of the same kernel family is a different problem size, and tells the session not to go looking for the intermediate files. Deterministic-stage readers of the sidecars are untouched here; they are a larger question than this stage. Co-authored-by: Cursor <cursoragent@cursor.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
On a host that installs the frameworks anywhere other than the one layout
KNOWN_SEARCH_ROOTSnamed, every search root was absent. The grep tier searchednothing and resolved no kernel at all, the LLM tiers got an empty shortlist and
a validation gate that rejected every path outside those absent roots, and the
run still reported success — with zero routable kernels, which reads exactly
like a trace with nothing worth optimizing. kernel-opt sat idle for hours with
no work to dispatch and nothing in the artifacts said why.
Measured on a recorded MiniMax-M3 session: 0 of 16 hot kernels routable.
After the first commit, 6 of 16, covering 13.74% of GPU time.
Commit 1 — discover the search roots at runtime
resolve_kernel_search_roots()joins the orchestrator's centralised resolver inframework/paths.py, so this tool agrees with PolicyGate and patch applicationabout where framework source lives, and a wheel install, an editable checkout
and a serving image all resolve without naming any of them. It is deliberately
narrower than
resolve_source_file_allowlist(), which also reports the baresite/dist-packages parents: right for a containment test, wrong for a recursive
grep. Non-existent roots are always dropped, and a host with no searchable root
now says so in the log and as a
no_framework_source_roottrace-health warninginstead of looking healthy.
Restoring the roots exposed a latent collision: a vendor-playbook operator such
as
mori::EpDispatchCombineOp::dispatchreduces to the keyworddispatchandgreps onto an unrelated vendor header, displacing the task-bundle anchor. A
registry match is a curated statement that the operator has no rewritable
device source, so its anchor now overrides the guess.
Commit 2 — one agent review instead of two completion tiers
Source resolution had two model tiers. The per-kernel fallback picked a path
from a grep shortlist for any kernel above 5% of GPU time; the whole-table pass
audited the finished resolution table. The first was the weaker and mostly
redundant: it fired only where the second already looked, at a higher threshold,
and it saw less — a symbol and eight paths, without the launcher frames the
table pass gets. Neither could see what the kernel actually is, because both
were tool-free completions fed a prompt assembled in advance.
Both are gone. One agent session now audits the candidate table itself and is
handed paths rather than contents: the raw table, the resolution audit, the
TraceLens report and per-category metrics, the model directory, the framework
source roots. It reads what the evidence leads it to instead of what was guessed
to be relevant, so it can open a file and confirm it defines the kernel it is
credited with rather than ranking paths by keyword.
The pass is mandatory on the agent route and ungated — the thresholds only ever
described what a completion was worth, and the deterministic route keeps its
no-LLM guarantee by not running the stage at all. It retries once; a definitive
failure records an error-severity trace-health warning and leaves the
deterministic table standing. Losing the audit costs some candidates, while
failing the run would cost the hours of benchmarking behind the trace.
Bounds on the added freedom
is worth dispatching. It may not touch what the trace measured — GPU share,
durations, shapes, argument specs — because the impact ranking, the tuning
harness and the final report are all computed from those. Derived state is
recomputed afterwards by the deterministic stamping pass, so
classify_patchabilitystays the one gate rather than gaining amodel-written second one. A restrictive routability hint is honoured; a
permissive one is not.
root, and a candidate already resolved by the active finder is not
overridable: reading the tree cannot beat knowing which symbol the binary
exports.
source files are fingerprinted around it and a review that modified them is
discarded — the benchmark that follows would otherwise measure an unrecorded
edit.
Artifacts
kernel_candidates.raw.json(deterministic),kernel_candidates.json(reviewed) and
kernel_candidates_revisions.json(what changed and why), so abad dispatch can be traced to the stage that caused it.
Test plan
framework/paths.py+ patch-safety + e2e source suites greenfor drop-if-absent, package-dir-only, empty, and explicit-checkout cases
routable, three of them corroborated by the trace launcher
_candidate_review_agent