feat(kernel_workflow): warm-start from a local kb_artifacts - #401
Draft
yueliu14 wants to merge 17 commits into
Draft
feat(kernel_workflow): warm-start from a local kb_artifacts#401yueliu14 wants to merge 17 commits into
yueliu14 wants to merge 17 commits into
Conversation
…e store
Close the read/write loop on the machine-produced, code-carrying KB so a
kernel lane can start from its own best historical patch instead of cold.
- scripts/experience_store.py: the store itself (stdlib + PyYAML, GPU-free).
`write` stores one measured win under
kb_artifacts/<gfx>/<kernel_class>/<slug>/<exp_id>/ (meta.yaml + patch.diff +
report.md) behind its own gate (missing_arch / no_improvement / empty_diff);
`resolve` enumerates a slug's solutions, keeps the SAME gfx only, ranks by
speedup and mirrors every candidate's prose into <eval>/kb_references so a
rejected warm start is still auditable. Neither subcommand ever raises — a
store failure prints {"written": false, ...} and exits 0.
- kernel_lane.js: new WarmStart phase between Profile and Optimize. Reads the
top-3 same-arch patches, validates EACH through the same verify_engineer
gate as a round winner, and adopts the first that passes; the recorded
speedup only ranks, adoption is decided by a fresh on-box measurement. After
Validate the run writes its own win back. The return value splits
total_speedup from incremental_speedup so a KB-derived gain is never
reported as this run's own work.
- kernel_workflow.js: thread warm_start / kb_artifacts_dir down to each
bake-off lane (the lane invocation spreads specific keys, not ...A).
- warm_start=off is a cold start, byte-identical to pre-feature behavior.
- kb_artifacts/ is gitignored: runtime-accumulated and unbounded.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…gnment Condense the verbose comment blocks in experience_store.py / kernel_lane.js / kernel_workflow.js (remove internal plan/KernelForge references and restated prose) and remove the dead `warm_start.total_speedup` write — the return value reports total from finalPrimary, never from that field. No behavior change; syntax + a write/resolve round-trip smoke test pass. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
KernelForge already ships the two-plane design this needs: one RewriteRecordStore protocol with LocalRewriteRecords on disk and KBStoreRewriteRecords over HTTP, selected by KNOWLEDGE_STORE_MODE. kb_store_local.py is that same on-disk shape, so the whole read/apply/optimize/write-back loop can be proven offline and moving to the service later is a change of backend, not of behaviour. Three properties are copied from upstream deliberately and must not drift: ranking is `speedup` descending and nothing else (the store does not know what a bench key is — comparability is the caller's job); candidates() reads knowledge documents only, so a 240KB patch is not paid for until it is selected; and every mutation lands by atomic rename, with a repeated session id meaning overwrite, because session ids are content-addressed and one port must stay one candidate. The id and path regexes are copied rather than widened: a record this plane accepts and the service rejects is exactly the failure it exists to catch early. kb_remote_upload.py gains --local DIR, which takes the SAME records the service path takes, byte for byte. That is the only thing supporting "proven locally = correct remotely", and it needs no KB_STORE_URL or token. Cross-checked against upstream's own reader: LocalRewriteRecords lists, ranks and materializes this tree identically (skipped when no KernelForge checkout is importable, so it catches drift without adding a dependency). Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The store was lossless and unranked: every recorded win was offered, including retired duplicates and near-ties, and nothing said which measurements were even comparable. This adds the curation the read path needs and the export/resolve/ write path that reaches the same experience through a KB Store key. Curation (the directory plane): a `retained:false` gate, one rank per optimization `direction` with runners-up riding along as `alternates` rather than costing a second verify slot, a `--min-speedup` floor, and bench-key comparability — an imported `b:` measurement and an on-box `b2:` one are ranked together but never claimed to be comparable, because they are not. Re-recording code the store already holds counts a reproduction instead of importing our own output as a fresh win. tech_lead now emits closed directions as a machine-readable block, so the next run does not spend a round re-funding a dead end that has evidence against it. The store plane: `export-remote` maps an entry onto the record shape the service uses, under the seven-segment key `kernel:geak:<name>:rocm:<major.minor>:<triton|hip|ck>:mi355x` — framework stays `rocm` for all three languages because one container image supplies them, and the language is the `backend` dimension. `resolve-remote` and `write-remote` read and write through that key while printing the same JSON as `resolve`/`write`, so the lane needs no branch. What lands under a key is decided by the patch, not the caller: new code appends a session, the same code remeasured replaces its own. `--framework-version` exists because a box with no /opt/rocm measures no stack, and every record would then file under `unspecified` — splitting one kernel's history across two keys. It overrides the key segment only, never the recorded stack. On the read path a store root that is not there is a hard miss rather than an empty store: a typo must not quietly cold-start a run with experience waiting. Verified offline on the real 248-entry store: 80 records over 20 keys, all valid against upstream's own identity regexes and record_id; read a key, land its top patch on a workspace whose layout it was never recorded against, optimize on top, write back, and read the improvement out again. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`--kb_mode local|store` picks which plane warm start reads and writes. Only the two command strings change: the phases, the schemas, the verify gate, the remap and the adopt decision are untouched, because the store subcommands were built to print the same JSON as the directory ones. `local` stays the default, so an unparameterized run is unchanged. Writing in store mode records BOTH planes in one call. The directory tree stays the source of truth a curation pass edits and the KB record is derived from it, so the two cannot drift into disagreeing about what was measured. In store mode the lane logs the canonical id it read from and, on write, which of the two outcomes it got — appended a candidate under the key, or updated this patch's own. Both fields are declared in the schemas rather than left to additionalProperties, so the agent relaying the JSON has no reason to drop them. kernel_workflow and e2e forward the knobs the same way they forward the rest, so every recursive lane in a run uses the plane the run was launched with. The CI job that runs these tests needs pyyaml and the three new test files added to its explicit file list; that edit touches .github/workflows and is left out of this commit because pushing it needs a token with `workflow` scope. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`resolve` and `resolve-remote` had grown the same ~90 lines twice: collapse to one rank per direction, render reference_NN.md, build the candidate dict, write index.md. Two copies of prose that is supposed to read identically whichever plane served it is a slow drift, not a saving. Extracted `_collapse_by_direction`, `_render_references` and `_candidate`. Only the genuinely per-plane bits stay behind: the address in the page header (slug vs canonical id), the origin line (source eval dir vs session id + champion flag), and the extra candidate keys the store plane carries. Also drops three LocalKBStore members nothing calls (`configured`, `Candidate.as_dict`, `read_bytes`); the one test that used `read_bytes` now checks the same thing through `materialize`, which is the path production actually takes. No behaviour change: 1105 passed, and a read of the real probe store returns the same two ranks, the same alternates and the same index text as before. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Wires the e2e KB into e2e_workflow.js as a front read+validate module and a final write module, adds the remote plane to both lanes, and adds the one thing a store with no DELETE needs: a way to take a record back. Retraction is a rewrite to a tombstone (mode="replace" + deterministic session ids), and it does three things at once because doing two is worse than doing none: mark the document, zero the ranking scalars, re-point the champion. Shared by both lanes in kb_retract.py. 54 tests pass. Validated end-to-end against the real service on a scratch canonical id. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t-kb-artifacts # Conflicts: # kernel_workflow/kernel_lane.js
- e2e_store.py build_record: carry a `comparability` block (schema v2) so a stored speedup travels with the basis it was measured on (client, workload points, measured-on config) instead of being rediscovered per run. - e2e_workflow.js: template-literal → string-concat polish in the KB warm-start config path (no behavior change). - knowledge/learned: record the 2026-08-19 real-run confirms (gpt-oss-120b mxfp4 grouped-MoE +26.9% byte-exact after corrective re-author; Qwen3-14B-FP8 a8w8 swap-only 1.513× serving-wtd with prefill-regression note; new moe-fp8-blockscale-tune-gfx950 lever) and a roofline-prior calibration line for the launch-overhead-invisible failure mode. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
| ...(kernelVerdicts.length ? kernelVerdicts.map(v => | ||
| `| ${v.name} | ${v.kind || '?'} | ${v.isolated_speedup ? v.isolated_speedup + 'x' : '—'} | ` + | ||
| `${v.measured_delta_pct != null ? (v.measured_delta_pct >= 0 ? '+' : '') + v.measured_delta_pct + '%' : '—'} | ` + | ||
| `**${v.outcome}** | ${String(v.why || '').replace(/\|/g, '\\|').slice(0, 160)} |`) |
| """ | ||
| views = list(views) | ||
| key = "|".join(v["key"] for v in views).encode("utf-8", "replace") | ||
| set_dir = os.path.join(refs_dir, "sets", hashlib.sha1(key).hexdigest()[:7]) |
…f kernel_workflow Both the kernel lane and the e2e serving lane warm-start from the same machine-produced KB, but the shared machinery lived under kernel_workflow/scripts/ with each lane carrying its own near-duplicate of open_plane / collapse-by-direction / ladder-publish. Hoist that into a single kb/ package both lanes import: kb/plane.py open_plane per-metric (primary, mirror, why) kb/curate.py collapse_by_direction one rung per idea, alternates ride along kb/ladder.py publish write one rung all-or-none, champion promote kb/identity.py kb/retract.py kb/store_local.py kb/store_remote.py kb/store_client.py kb/remote_upload.py (moved from kernel_workflow/scripts) e2e_store.py and its test move from kernel_workflow/scripts/ to e2e_workflow/scripts/ — it is the e2e lane's CLI, used only by e2e_workflow.js, so the reference is now same-subtree instead of reaching across into the kernel dir. e2e_store.py and experience_store.py both drop their private copies and call the kb/ helpers. Fix: retract --result recompute crashed (AttributeError) because the retract subparser has no --file; build_record now reads it via getattr. The move pulls e2e_store.py (~313 stmts) into the coverage tree; new test_e2e_store.py brings it to 99.68% and both e2e_store tests are added to ci-l0-checks.yml. .gitignore ignores the on-disk store root (kb_store_local/). Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
… backups These ci/ changes (SPUR_PROBE_TIME/-t UNLIMITED tweaks plus their *.bak.20260817-wall auto-backups) were branch-local and not wanted on the shared branch. Revert ci/config.sh and ci/lib.sh to main and remove the backups so the branch introduces no changes under ci/. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
…ible records Four gaps in what the KB gave a reader back, and one new module both lanes share. Recall ordering. Only the finest e2e rung ranked on throughput; the two coarse rungs — the ones a reader on a different workload point actually lands on — ranked on speedup, so "the offer is ordered by throughput" was true on a third of the pages. The read path now opens every rung on throughput (--sort-by restores the old order). This had to move to the store metric rather than a client-side re-sort: RemoteKBStore.candidates() pages sessions/top?metric=self.metric, so re-sorting a speedup-ranked sample ranks a biased sample. Writes still crown per-rung; the two metrics are reported as separate fields rather than collapsed into one word. Attestation (kb/attest.py). A record's `validated` flag is a judgement its own writer made about its own measurement, once. It cannot answer the question that decides whether the record is worth keeping: has anyone since pulled it out, run it, and had it work. The new value.attestations ledger counts that — recalls / validations / failures / not_reproduced plus a bounded history — in one vocabulary both lanes use. `recalls` counts attempts ON HARDWARE, not reads, or the only ratio a retire pass can act on would decay for records nobody ever doubted. Unlike retraction it moves no ranking scalar and re-points no champion: one failure on one box is evidence, not a verdict, and collapsing the two would make the command too dangerous to run automatically, which would mean it never ran. retire_hint() is advisory, a string naming which pattern fired, and nothing filters on it. Writes carry the ledger forward, because session ids are content-addressed off the config and exclude the measurement — re-benching one config lands on the SAME session and would otherwise silently reset its whole history while looking well-formed. Reproducibility. An e2e record could be recalled and consist of nothing you could run. Now value.repro is structured, launch.sh is synthesized against bench_e2e.sh's env contract when none was captured (and says plainly that it was), kernels without their patch are counted rather than omitted — three kernels and two patches otherwise reads either as a no-op or as lost bytes, and those point opposite ways — and --kernel-store fetches patches from the kernel lane, whose scratch is usually gone by the time an e2e run finalizes. A result with no script, no flags, no env, no patch and no overlay is refused: this store has no delete. Not reproduced != rejected. The warm-start verdict was binary, so "would not run" and "ran but did not win" were the same word despite meaning opposite things to a retire pass. It is three-way now, benched candidates attest back (non-fatal), and the ones that did not reproduce get their own REFERENCE ONLY section carrying the launch script and patch paths, as leads for the optimization flow rather than as discards. Verification: 190 passed, 1 skipped over the CI set; node --check clean. Nine failures in test_experience_store.py's export-remote cluster are pre-existing drift, confirmed by stashing this work and re-running. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
…t-kb-artifacts # Conflicts: # .github/workflows/ci-l0-checks.yml # .gitignore # docs/reference/api-reference.md # kernel_workflow/kernel_lane.js # kernel_workflow/kernel_workflow.js
| """Mirror the offer into prose the Director can read, or return "" and let the read stand.""" | ||
| try: | ||
| os.makedirs(refs_dir, exist_ok=True) | ||
| key = hashlib.sha1(("|".join(v["session_id"] for v in views)).encode()).hexdigest()[:7] |
…rust The KB_ENV_PRELUDE no longer probes for /shared_nfs/hyperloom/ca and exports SSL_CERT_FILE/REQUESTS_CA_BUNDLE/CURL_CA_BUNDLE/NODE_EXTRA_CA_CERTS itself. Container TLS trust (AMD CA + DNS) is a launch-time concern, injected by the run harness via `docker run --add-host` + a read-only CA mount and the four CA env vars (warmstart_run/node_docker.sh -> kb_net_docker_args). Keeping a copy here baked a /shared_nfs host path into the repo for a job the launcher already does, so remove it. The prelude is back to just KB_STORE_URL + KB_STORE_TOKEN. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Most callers run the e2e/kernel workflows OUTSIDE the warm-start launcher (node_docker.sh), which is the only place that injects AMD-CA trust at `docker run`. GEAK's own CI (ci/node/run_geak_e2e.sh) is one such caller: it drives run_e2e.py with the default kb_mode=both (remote KB) and sets no CA of its own, so without an in-workflow fallback its warm-start silently degrades to a cold start on any node where the gateway's internal AMD CA is untrusted. So KB_ENV_PRELUDE again DETECTS then heals: only when SSL_CERT_FILE is unset does it point urllib/requests/curl/node at the first readable AMD-root bundle (KB_CA_BUNDLE override, else the shared Hyperloom bundle). It is a strict no-op when the caller already set SSL_CERT_FILE (warm-start lane) or no bundle is readable (CI / already-trusting images stay byte-identical). Path-only, no CA content or secret in the repo. Supersedes d6fd2ee. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Close the read/write loop on the machine-produced, code-carrying KB so a kernel lane can start from its own best historical patch instead of cold.
writestores one measured win under kb_artifacts//<kernel_class>//<exp_id>/ (meta.yaml + patch.diff + report.md) behind its own gate (missing_arch / no_improvement / empty_diff);resolveenumerates a slug's solutions, keeps the SAME gfx only, ranks by speedup and mirrors every candidate's prose into /kb_references so a rejected warm start is still auditable. Neither subcommand ever raises — a store failure prints {"written": false, ...} and exits 0.