[Stepping] Correctness sweep across the optimization loop: measurement integrity, phase budgets, resume safety, and a large dead-code removal

[ZhengGong-amd]

Summary

This branch is an end-to-end audit pass over the orchestrator. It started from
one question — is every number the loop promotes actually measured? — and the
answer pulled in four adjacent areas: the promotion path into current_best,
the phase budget gates, the resume/session identity path, and the benchmark
environment contract. Along the way it removes ~10k lines of code that nothing
reaches: an undeployed robustness-server data path, an orphaned kernel harness,
and a long tail of symbols, artifacts and state fields with no reader.

The net is negative by 8k lines, but this is not a cleanup PR. Roughly half
the commits fix live defects, several of which silently corrupted the session's
reported gain. The removals are what made those defects visible.

---

1. Measurement integrity: nothing enters current_best unmeasured

The central defect class: several paths wrote a throughput onto current_best
that no benchmark ever produced, which then raised the grading anchor for every
later candidate and rewrote the session's reported gain.

• b4ff1b297 — GEMM tuning KEEPs are now measured end-to-end. The GEAK
  branch derived a throughput as baseline_tput * best_speedup and wrote it
  straight onto current_best. A GEMM speedup of 1.1 would overwrite a
  current_best already holding an explore win of 1.5×, and rewrite the
  session's validated cumulative gain as that single GEMM's speedup. Both
  result shapes now reduce to a common candidate list and route through the
  per-tuner validator (previously forge-only), which measures each candidate
  against current_best via integrate_handler.
• 045f7d08b — profiler-on measurements no longer promote. A profile run
  benches with the torch profiler attached, so its throughput is not comparable
  with un-profiled measurements. Promoting it on a private +1% rule bypassed the
  anchor gate, inflated the anchor for every later candidate, and left no
  optimization_stack entry — so the stale-stack warning never fired and a
  genuinely better KEEP measured below the inflated anchor was silently refused.
• 227471e3d — one writer for current_best. integrate, geak_e2e,
  forge GEMM, fusion and warm_replay each hand-built their own current_best
  and stack entry; four replaced current_best outright, so a KEEP could drop
  cumulative server args or resurrect an env an earlier ablation had removed.
  All five now hand a winner dict to _lift_to_current_best. Two behaviour
  changes fall out: integrate and fusion gain the anchor guard they lacked
  (a KEEP measured below the established recipe is now refused rather than
  lowering current_best), and warm_replay stops hand-appending a rounded
  percentage to gain_per_stack_entry.
• cc9f8843f, 82318eabe, 6389b5df4 — the stack replay is gone. Resume
  rebuilt current_best from an optimization_stack replay and overwrote it on
  any disagreement. The replay unioned every entry's extra_envs without
  applying unset_envs, so resume "repaired" a correct current_best into a
  wrong one. With every config KEEP going through the lift, the two are written
  together and cannot drift. The GEAK handoff and post-resume rebench now read
  current_best directly; the launch normalization that survives also had a
  bug where a --prefixed env key routed to server args replaced the arg
  string instead of appending to it.
• de10702d2 — target_reached gates on the validated gain.
  TargetGainObjective read cumulative_gain, which any promotion could move
  without a stack rebench behind it, so a run could stop on a number no
  full-stack measurement supported. Every other loop decision already read
  cumulative_gain_validated.
• eee047f26, b81dc3b35, cc30655cb — cumulative_gain removed. It and
  cumulative_gain_validated computed the same expression; the only way they
  could differ was a write path recording a gain it should not have. The
  documented "per-round sum" never existed in any revision. One of the three
  removed "validated or raw" fallbacks published the raw value under the key
  validated_e2e_gain into the cross-session recipe KB whenever the validated
  gain was exactly 0.0 — a legal result, not a missing one.
• beadc4c8a — a refused promotion journals as no_promote. The lift can
  decline a KEEP the executor granted; the audit row said no_promote but the
  journal keyed only on executor status and wrote KEEP, so a session audit
  counted an adoption that never happened.

2. Phase budgets: stop evicting phases that have done no work

• fdadb6500 fixes three defects that let a phase be skipped or starved
  without spending any of its own budget:
  • normalize_budget_pct dropped 0.0 as missing, so the sentinel written for
    a disabled phase was overlaid with the library default and its revived share
    stayed in the charge-back denominator — shrinking exactly the phases the
    redistribution was meant to fund. Measured: under --no-kernel, EXPLORE was
    allotted 324.8 min where 498.8 min was intended.
  • The EXPLORE force-exit compared session wall-clock against a fixed 3 h, so
    any run of 3 h or less force-exited EXPLORE on its first tick with
    phase_remaining_pct == 1.0. The surviving budget-fraction arm reproduces
    every long-run case the hours arm caught.
  • The FRAMEWORK force-exit compared time-left against a fraction of the
    session total, so every macro-cycle reloop into FRAMEWORK — the preferred
    reloop target — entered and force-exited on the same tick.
  • Also retires --explore-force-exit-hours-remaining from the 3 h demo, which
    could only run because it disabled the gate.
• 504bb245d puts framework_agent KEEP on the decaying threshold curve;
  _materialize_approved_proposal returned before the injection ran, so it
  always fell back to the static 1.0% default.
• 27ad67737 routes all four dispatch sites through one
  resolve_keep_threshold(state), replacing an except Exception: pass wrapper
  and a baked-in literal dict. Prompt copy no longer states a number that drifts
  from the live curve.

3. Resume and session identity

• 8f5532572 — TERMINAL_STATES derived from the transition table. The
  failed -> running edge was removed when retries moved to a fresh idempotency
  key, but the hand-listed set kept its old membership, so a failed row
  reported as still live. Three guards wedged on this: the watermark roofline
  gate never released (and the marker is persisted, so the next resume inherits
  the wedge), resume recovery never cleared
  enablement_validation_pending, and the FRAMEWORK authoring pump re-selected
  a dead candidate every tick.
• c29169601 — never roll back a pending integrate on an unreadable event
  log. "No KEEP found" and "could not read the event log" were the same
  outcome, so a transient DB error during the first resume after a hard crash
  would reverse-apply every recorded patch, discard the attempt venv, report the
  loss as a fix, and clear the sentinel so no later resume could retry.
• 2d7667a02 — session dirs are unique within a UTC second. The name is the
  de-facto session id (it feeds KB fact writes, critic requests and per-session
  sinks), so a same-second collision merged two runs' identity, not just their
  artifacts.
• 493e801db — bare --resume removed (BREAKING). It scanned every model
  subtree under $USER_DATA_PATH, attached to the newest timestamp, and
  re-exported that session's model as MODEL_PATH without a word — silently
  continuing someone else's run on a different model, which SKILL.md already
  forbids. --resume-from is the only entry point; c70e31548 re-anchors the
  crash-recovery monitor, 7c13a720a pins that legacy command lines exit 2
  (argparse resolves --resume as a prefix of --resume-from), and 7f615df2b
  fixes the docs that documented timestamp-guessing in the same file that
  forbade it.
• ab9568ca7 — a relative $USER_DATA_PATH is absolutised at the launcher
  boundary. Subprocesses spawn with their own cwds, so parent and child
  resolved different workspace roots while both believed they agreed.
• a4620b32f, 78189a8ea, b9a4b09d5 — blank path-id components are
  refused outright instead of mapping onto a shared unknown bucket that had no
  producer and forced an early return past the traversal check.

4. Benchmark environment contract

• 76b03786b — the bypass server gets the materialized envs. The bypass
  serving path hand-picked ROCR_VISIBLE_DEVICES out of benchmark.envs and
  dropped the rest, so a candidate whose only difference from baseline was an env
  launched a byte-identical server — same argv, same environment. The
  measured delta was run-to-run noise, the 1% keep threshold promoted it, and the
  env landed in current_best and in the reference script handed to the user.
  Kernel bundles and framework levers are activated by exactly such switches, so
  their attribution was measuring nothing. Behaviour change: configs carrying a
  static PATH under benchmark.envs now have it reach the server, matching
  Magpie.
• 8aa97021d — per-variant env overrides hardened. GridVariant.extra_envs
  was filtered for control-plane credentials only, so a proposed variant could
  carry LD_PRELOAD / PATH / PYTHONPATH into the benchmark config — inert
  while bypass exported nothing, a live injection surface the moment it does.
  An allowlist isn't usable (grids legitimately set CONC/ISL/TP/…), so
  shell and loader hijacks plus control-plane credentials are blocked instead.
• 6e0740cdf — unset_envs cannot drop pinned workload envs. It popped any
  name given, so a variant could remove TP, CONC, ISL, RUN_EVAL or the GPU mask.
  That doesn't toggle a knob, it retargets the benchmark — dropping TP also
  shrinks the Ray lease to a single GPU, since the lease size is read back out of
  benchmark.envs.
• 8167b8b72 — the breakdown recorder stopped dropping workload fields. A
  recorder fragment replaces its whole section rather than merging, and the
  workload fragment wrote framework where both renderers expect
  framework_name, omitted framework_version and objective, and coerced
  unset tp/conc/isl/osl from None to 0 (which reads as a real value). The
  recorder is on by default.

5. Robustness agent

• e01bbcca1 (BREAKING) — the remote cluster data path is removed. No
  robustness-server is deployed, ROBUSTNESS_SERVER_URL is never set, and the
  five workload-uid env keys the hierarchy lookup depends on have read sites but
  no writers. The client was GET-only with no registration endpoint, so the
  server could not have learned this session either way — yet Config.discover()
  paid two doomed health probes every tick. Removed with it: the Source
  adapter, the cluster payload decoder, and the cluster_fault /
  pod_not_running symptoms that had no other producer. Four CLI flags go with
  it (9d700952a). ReactorBundle.aclose() now closes the RCA engine's HTTP
  client, which was never released.
• 9d700952a — multi-node runs get the real agent, not the heartbeat mock.
  The downgrade guarded against LocalProbe false positives, but
  disable_local_probe already defaults to True on multi-node and swaps in a
  silent stub — so the downgrade was discarding every signal read straight off
  the Coordinator prompt and inbox, which cannot false-fire on a worker node.
  Multi-node produced no robustness symptoms at all, including
  deadline_imminent, the one that drives the delegate(report) wind-down.
  Expect alerts where there were none; --robustness-mock restores the old
  behaviour.
• 95d4646f0 — a tick that harvested nothing no longer counts as a healthy
  primary, so the router degrades to the local probe instead of reporting a clean
  bill of health.
• 96d89e51e — the runtime CLI fallback that backfilled a shadow
  session_id rebuilt the context by hand-listing members; that list was
  complete at 7 fields and the snapshot has since grown to 19, so a tick taking
  the fallback silently lost stop_reason, the wall-clock budget, validated
  gain, tick and macro_cycle — muting postmortem finalization and the deadline
  wind-down.
• b70177075 — the SKILL.md symptom table listed ~20 of the 52 symptoms the
  rules emit. It is now a complete inventory keyed on the SignalSpec.name the
  classifier and factory both read (verified by set-comparison against the
  name= literals in signals/), so it cannot drift. cd7616103 drops the six
  families of milestone codes that said when a feature was built rather than
  what it does — three of which pointed at nothing.

6. Dead code and unread artifacts

• 02162bce8 retires parallel_e2e_runner.py (538 lines), the
  self-validation harness from before a KERNEL phase existed. Its baseline step
  was removed in May and --backends defaults empty, so a plain invocation only
  raised. Its deletion also closes an aggregation bug rather than fixing it: every
  attempt's exception flattened to a row-level failed while the aggregate
  printed "status": "succeeded" and returned 0 (measured: 2/2 failed, exit 0).
  Its load_env_file duplicated ray_runtime.py's credential-alias derivation,
  which stays and is pinned by seven tests.
• 78f162afe removes four categories of write-only output: the critic
  events.jsonl audit trail (7 call sites, no reader), framework_agent
decision.json / semantic_audit.json (not in the session package glob, no
  skill points at them), bypass_trace_analysis kernel_sequence.json (already
  inlined in the result), and the extra_sglang_args shim.
• fd81233c8 deletes 15 symbols with zero production callers, including the
  knowledge_plane typed read/write wrappers the codebase never adopted.
• 4b1ee80dd removes three reads of attributes SharedState doesn't define,
  each absorbed by a getattr default — most consequentially, the framework
  ranker prompt read best_throughput / baseline_throughput and therefore
  never told the model the current best throughput. It now shares
  resolve_grading_anchor_tput with the admission path.
• 2cb82ad7c collapses the ordinal kernel_opt_attempts index onto the
  stable kernel_opt_task_attempts ledger; the two diverged through three
  asymmetric cleanup paths and consumers fell back between them in inconsistent
  directions. A SharedState property keeps existing test doubles working.
• 212e6c8ab converges genuinely duplicated helpers: the lease-reap +
  task-reclaim + DB-retention sequence (copied verbatim between the maintenance
  tick and the cycle soft-restart), five Claude-SDK message-text extractors with
  slightly different gaps, and three identical _phase_at binary searches. Four
  other near-duplicates are left merged-in-comment-only with the reason recorded
  (differing error contracts, undocumented threshold differences, and one that
  must stay importable on remote nodes without the hyperloom package).
• fa264d8e6, 473aa8c8f, 5d49d0fdb, e509017d3 clear the remaining
  unread aliases, stale mirrors and follow-up breakages. e509017d3 is a review
  pass over the branch itself and fixes two defects the earlier mechanical stages
  introduced (an AttributeError on every GEMM-Roofline state sync, and a
  double-seeded plateau test).

Breaking changes

Change	Migration
Bare --resume removed	Use --resume-from <session-dir>; stale invocations exit 2
--robustness-server-url, --robustness-workload-uid, --robustness-enable-cluster-pod-metrics, --robustness-pod-metrics-categories removed	Drop the flags; they fed an undeployed server
$ROBUSTNESS_SERVER_URL, $ROBUSTNESS_ENABLE_CLUSTER_POD_METRICS no longer read	None
cluster_fault, pod_not_running, pod_no_metrics symptoms removed	None — they had no producer
Multi-node no longer implies --robustness-mock	Pass --robustness-mock explicitly for the old behaviour
parallel_e2e_runner.py removed	Covered by the KERNEL phase and ci-e2e.yml
session_breakdown.json loses final.cumulative_gain_pct_per_round_sum	Read cumulative_gain_pct_validated — it was equal in every session current code can produce
Bypass server now receives all of benchmark.envs	A static PATH in a shipped config now wins over the caller's inherited value, matching Magpie

Legacy state.json files load unchanged — from_dict drops unknown keys.

Reverted in-branch

dd163f2c9 recorded two blockers (an unwired MoE guard in compose_server_args,
and stop_ray_if_owned having no production caller so no path stops a Ray head
this process started). It was reverted in e39b39bbf — both findings stand but
belong in the issue tracker, not the changelog.

Review notes

The commits are ordered to be readable one at a time and each is
self-justifying; e509017d3 is a review of the rest of the branch and is worth
reading first for the list of what the earlier stages got wrong. The largest
single-file diffs (phases/kernel.py, loop/writeback.py,
phases/machine_state.py) are dominated by §1 — the convergence onto one
current_best writer — and are best reviewed against 227471e3d and
b4ff1b297 together.

[github-actions]

CI E2E report — ✅ Succeeded

item	value
result	✅ Succeeded
model	Qwen/Qwen3-0.6B (dense)
resources	1× GPU, TP=1
PR branch	feat/zgong/explore-opt-17
commit	48ed267e3eb0494707943b0c9935fb5906d72a52
session_id	4ef6d426-a39b-4902-8b98-d59e49e3948c
queue → dispatch	0s
run time	171m 3s
total	171m 3s

details