[Stepping] Correctness sweep across the optimization loop: measurement integrity, phase budgets, resume safety, and a large dead-code removal

CHANGELOG.md

@@ -7,6 +7,18 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
 
 ### Removed
 
+- **BREAKING — the robustness agent's remote cluster data path is gone**, along
+  with the flags that fed it: `--robustness-server-url`,
+  `--robustness-workload-uid`, `--robustness-enable-cluster-pod-metrics` /
+  `--no-...`, and `--robustness-pod-metrics-categories`. Callers still passing
+  any of them now fail in argparse. `$ROBUSTNESS_SERVER_URL` and
+  `$ROBUSTNESS_ENABLE_CLUSTER_POD_METRICS` are no longer read, and the startup
+  probe that tried `http://robustness-server:8000` and `http://localhost:8000`
+  on every tick is gone. No robustness-server is deployed and none of the five
+  workload-uid env keys was ever set, so the endpoints could only 404; the
+  `cluster_fault` and `pod_not_running` symptoms went with them, having had no
+  other producer.
+
 - **BREAKING — `kernel_optimization.py` no longer accepts `--test-command` or
   `--test-harness-path`.** The unittest-harness contract they fed had no
   reachable caller; an external invoker still passing either flag now fails in
@@ -56,8 +68,34 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/),
   read it; `KERNEL_OPT_BACKEND_ORDER` is the sole backend switch, and only an
   exact `forge` opts out of the default GEAK phase.
 
+- `agents/kernel/tools/parallel_e2e_runner.py` is gone. It was the
+  self-validation harness written alongside the original kernel-agent, back when
+  no KERNEL phase existed to prove the toolkit end to end; its own first step
+  (running the SGLang baseline) was removed in May, leaving a driver with no
+  caller whose `--backends` default was empty, so it raised on any plain
+  invocation. Its `load_env_file` duplicated the credential-alias derivation that
+  `tools/backends/ray_runtime.py` still performs under wider test coverage.
+
 ### Changed
 
+- **Multi-node runs now use the real robustness agent instead of the heartbeat
+  mock.** `--nodes >= 2` previously forced `--robustness-mock`, which produced
+  no symptoms at all — including `deadline_imminent`, the signal that drives the
+  `delegate(report)` wind-down. The downgrade guarded against LocalProbe false
+  positives, but `disable_local_probe` already defaults to True on multi-node
+  and swaps the probe for a silent stub, so the signals the agent reads straight
+  off the Coordinator prompt and inbox were being discarded for no reason. Those
+  now fire: the deadline and budget ladder, `gain_plateau`, `no_levers_found`,
+  crash escalation, `phase_budget_nearly_exhausted`,
+  `conversation_no_progress`, and the inbox-driven `agent_stall` /
+  `repeated_failure` / `repeated_policy_denied` family. Expect alerts on
+  multi-node where there were none; pass `--robustness-mock` for the old
+  behaviour.
+
+- `ReactorBundle.aclose()` now closes the RCA engine's provider client. It
+  previously closed only the robustness-server client, leaking the HTTP client
+  the LLM RCA engine owns.
+
 - **Remote Recipe knowledge now uses one current KB Store contract.** Remote
   mode reads one identity-addressed inference Recipe containing replay config,
   the ordered patch timeline, and nested kernel columns, then publishes one

docs/conceptual/optimization-loop.md

@@ -230,8 +230,9 @@ The old `backends` and `params` action names are compatibility aliases
 for archived reporting only. New sessions write the merged
 `explore_search` ledger.
 
-After each KEEP, the runtime revalidates the full stack end to end so
-the reported cumulative gain is not just a sum of per-round deltas.
+After each KEEP, the runtime revalidates the full stack end to end, so
+the reported `cumulative_gain_validated` always comes from a measurement
+taken with every accepted change applied.
 
 ## KERNEL_AGENT
 

docs/how-to/multi-node/hyperloom-remote-demo.md

@@ -114,8 +114,8 @@ ${NFS_SHARED_ROOT}/TraceLens-internal/     # TRACELENS_INTERNAL_ROOT (optional)
 ## What the agent produces
 
 A session under `$USER_DATA_PATH/<model_basename>/<UTC_timestamp>/` with launcher
-logs and a persisted `state.json` (holds `phase`, `cumulative_gain`,
-`crash_count`, `stop_reason`) for status and `--resume`. `$USER_DATA_PATH` comes
+logs and a persisted `state.json` (holds `phase`, `cumulative_gain_validated`,
+`crash_count`, `stop_reason`) for status and `--resume-from`. `$USER_DATA_PATH` comes
 from the environment (platform-injected).
 
 ---

docs/how-to/multi-node/hyperloom-remote-mn-qwen3-30b/SKILL.md

@@ -175,7 +175,8 @@ FORGE_PATH=${NFS_SHARED_ROOT}/KernelForge
 
 - Session lands in `$USER_DATA_PATH/<model_basename>/<UTC_ts>/`; `USER_DATA_PATH`
   is platform-injected and kept unchanged.
-- Crash recovery: `optimize --resume` on the **same** session dir (never a second
-  `optimize`; resume past a terminal `stop_reason` needs `--force-resume`).
+- Crash recovery: `optimize --resume-from "$SESSION_DIR"` on the **same** session
+  dir (never a second `optimize`; resume past a terminal `stop_reason` needs
+  `--force-resume`).
 - Releasing the cluster is the platform's job, not the optimizer's — it happens
   when the session ends.

docs/how-to/optimize-custom-workload.md

@@ -229,10 +229,10 @@ unset _dotenv_prev
 ```
 
 **Sessions.** `USER_DATA_PATH` sets the session root, and each `optimize`
-creates a new timestamped subdirectory under it. Use `--resume` to continue an
-existing session, optionally with `--resume-from <subdir>`; `--force-resume`
-pushes past the terminal-state guard. Without `--resume` you always get a fresh
-session, so an interrupted run is never picked up by accident.
+creates a new timestamped subdirectory under it. Use
+`--resume-from <subdir>` to continue an existing session; `--force-resume`
+pushes past the terminal-state guard. Without `--resume-from` you always get a
+fresh session, so an interrupted run is never picked up by accident.
 
 ## Monitor the run and read the output
 

docs/how-to/optimize.md

@@ -74,11 +74,12 @@ Paste this prompt into Cursor Chat to resume an existing session:
 Resume the existing Hyperloom optimization session.
 
 Requirements:
-1. Launch `python -m hyperloom.inference_optimizer.cli optimize --resume`; do not start a new session.
+1. Launch `python -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR"`; do not start a new session.
 2. Do not pass `--model`; read the model and workload from the saved manifest.
-3. Before launching, verify `manifest.json` and `state.json` exist.
-4. Report the log path, PID, health check, current phase, cumulative gain, and best config.
-5. Monitor the process every 300s until the optimization is complete or failed.
+3. Resolve `$SESSION_DIR` from the launch-info JSON or the `HYPERLOOM_LAUNCH` line, never from the newest timestamp dir.
+4. Before launching, verify `manifest.json` and `state.json` exist.
+5. Report the log path, PID, health check, current phase, cumulative gain, and best config.
+6. Monitor the process every 300s until the optimization is complete or failed.
 ```
 
 ## Output and artifacts

docs/install/slurm.md

@@ -189,7 +189,7 @@ cat <data-root>/<key>/$SID/state.json
 
 The artifact directory `<data-root>/<model_key>/<CLAW_SESSION_ID>/` contains:
 
-- `state.json`: live status (`baseline_tput`, `current_best`, `cumulative_gain`);
+- `state.json`: live status (`baseline_tput`, `current_best`, `cumulative_gain_validated`);
 - `manifest.json`: session manifest;
 - `ci_metrics.json`: baseline/optimized throughput plus `gain_pct`;
 - `optimizer_runs/`: `launch_<sid>.json` and logs;

docs/reference/environment-variables.md

@@ -61,9 +61,8 @@ The following variables configure filesystem paths for Hyperloom's runtime depen
 | `HYPERLOOM_ROOT`                          | No                   | `$HYPER`<br>`LOOM_R`<br>`UNTIME_`<br>`DIR/sou`<br>`rce-mirrors`                            | Legacy source-mirror root kept for compatibility. Current open-source dependency checkouts default to the repo-local cache root (`${HYPER`<br>`LOOM_CA`<br>`CHE_DIR:-`<br>`$REPO_ROOT`<br>`/.cache}`), not this path. |
 | `HYPERLOOM`<br>`_CACHE_`<br>`DIR`                          | No                   | `$REPO_ROOT`<br>`/.cache`                      | Writable, repo-local base for auto-cloned open-source deps (TraceLens, Magpie, etc.), cloned per revision as `<name>@<sha>`. Not under `$TMPDIR` so a reaper cannot wipe it mid-run. |
 | `MAGPIE_PATH`                              | No                   | Resolved from installed `Magpie` package unless explicitly set                               | Magpie package root for benchmark wrappers and patch inspection.                                                                                                                                            |
-| `INFERENCE_`<br>`OPTIMIZER`<br>`_MODEL_PATH_ROOTS` | No | Built-in model roots such as `/models` and `/shared_nfs` | `os.pathsep`-separated allowlist for absolute model paths restored from `state.json` during `--resume`. HuggingFace-style repo IDs remain allowed. Set this when production models live outside the built-in roots. |
+| `INFERENCE_`<br>`OPTIMIZER`<br>`_MODEL_PATH_ROOTS` | No | Built-in model roots such as `/models` and `/shared_nfs` | `os.pathsep`-separated allowlist for absolute model paths restored from `state.json` during a resume. HuggingFace-style repo IDs remain allowed. Set this when production models live outside the built-in roots. |
 | `SESSION_DIR`                             | No (robustness-agent)| Scan known paths                                                   | Path containing `storage/coordinator.db`; the robustness FindingSink writes under `{session_`<br>`dir}/ag`<br>`ents/ro`<br>`bustne`<br>`ss/fin`<br>`dings/`<br>`{sess`<br>`ion_id}.jsonl`.                                       |
-| `ROBUSTNESS_SERVER_URL`                   | No (robustness-agent)| Scan known DNS                                                     | M1 primary data source; empty disables the primary path and forces local-only probes.                                                                                                |
 | `WORKSPACE_PATH` *(legacy)*               | No                   | Unset                                                              | Legacy path variable. Still consumed in two narrow spots: the CLI `setdefault`s it to the repo root for the critic subprocess's static assets, and TraceLens uses it as a `USER_DATA_PATH` fallback. Prefer `USER_DATA_PATH`. See [Upgrade Hyperloom version](upgrade.md).                            |
 | `INFERENCE_`<br>`OPTIMI`<br>`ZER_SES`<br>`SION_DIR` *(deprecated)* | No            | Unset                                                              | **Retired** — replaced by `USER_DATA_PATH`. No longer read.                                                                                                                       |
 
@@ -95,8 +94,7 @@ Set with CLI flags, not env vars. Pre-set `ISL` / `OSL` / `CONC` / `PRECISION` /
   `--no-framework-agent`, `--no-framework-local-explore`, `--no-kernel`,
   `--no-explore`, `--no-eval`.
 - **Agent models:** `--claude-model`, `--codex-model`.
-- **Session / resume:** `--resume`, `--resume-from`, `--force-resume`,
-  `--reset-state`.
+- **Session / resume:** `--resume-from`, `--force-resume`, `--reset-state`.
 - **Quantization:** `--quantize`, `--quantize-scheme`.
 
 Run `inference_optimizer optimize --help` for the exhaustive flag list.
@@ -527,7 +525,7 @@ optional; defaults are safe for standard single-node deployments.
 | `INFERENCE_`<br>`OPTIMIZER_`<br>`AITER_JIT_DIR` | Aiter default | Per-attempt override set automatically to `<attempt_root>/aiter_jit` by each targeted build.  Override manually only when you need the global JIT cache to point at a pre-built location; leaving it unset lets each build use its own isolated directory. |
 | `PYTORCH_ROCM_ARCH` | Detected | Explicit GPU target architecture (e.g. `gfx942`, `gfx950`) injected into each compile.  Set automatically from the session `--gpu-type`; operator-override applies to bare-metal installs outside the session.  **Compile target only — it does not participate in architecture detection.**  It names the archs a wheel is *built* for, not the installed device, so provenance ignores it entirely and resolves `gfx_arch` from `HYPERLOOM_GFX_ARCH`, then `--gpu-type`, then `rocminfo`. |
 | `MAX_JOBS` | `8` | Parallelism cap for cmake/hipcc compile steps inside a targeted build.  Reduce on memory-constrained nodes (`MAX_JOBS=4` for a 64 GB compile node).  The default `8` is conservative enough for MI300X/MI355X nodes with 512 GB+. |
-| `HYPERLOOM_`<br>`FRAMEWORK_PYTHON` | Unset | Explicit interpreter that launches the server for a from-source build (the venv Python the artifact was compiled against).  Set automatically from `FrameworkRuntime.runtime_python_exe` via `apply_runtime_override` into the per-variant YAML `benchmark.envs`.  The bypass backend honors it by launching `python -m`; the Magpie backend re-exports it from the YAML `benchmark.envs` to the server env.  Operators normally do not set this by hand. |
+| `HYPERLOOM_`<br>`FRAMEWORK_PYTHON` | Unset | Explicit interpreter that launches the server for a from-source build (the venv Python the artifact was compiled against).  Set automatically from `FrameworkRuntime.runtime_python_exe` via `apply_runtime_override` into the per-variant YAML `benchmark.envs`.  Both backends export that mapping to the server env; the bypass backend additionally uses this value as the `python -m` interpreter.  Operators normally do not set this by hand. |
 | `HYPERLOOM_`<br>`VLLM_ROCM_`<br>`INDEX_URL` | Unset | ROCm pip index URL used as the default vLLM adapter wheel index; also seeds the index allowlist. |
 | `HYPERLOOM_`<br>`ENABLEMENT_`<br>`INDEX_ALLOWLIST` | Unset | Comma-separated allowlist of pip index URL prefixes; a candidate wheel index must match one of these prefixes or provisioning is refused (supply-chain safety). |
 | `HYPERLOOM_`<br>`ENABLEMENT_`<br>`ORIGIN_ALLOWLIST` | Unset | Comma-separated allowlist of git origin URL prefixes; a candidate repo origin must match one of these prefixes or provisioning is refused (supply-chain safety). |

docs/reference/operations.md

@@ -126,7 +126,7 @@ The following table describes the key lifecycle events for a Hyperloom session.
 | Session start   | API call / Job creation                            | Coordinator creates `$SESSION_DIR` and writes `manifest.json`, `state.json`. |
 | Heartbeat       | Every Coordinator tick (`--tick-interval-sec`, default `0` = no sleep) | Coordinator atomically rewrites `state.json` (temp `.state.json.*.tmp` + `os.replace`) inside `$SESSION_DIR`. |
 | Session end     | `target_reached` / `time_exhausted` / `global_converged` | Coordinator writes `session_breakdown.json`, exits 0. |
-| Crash recovery  | Pod OOM / preemption                               | Re-launch with `--resume` / `--resume-from`; reads `manifest.json` + `state.json`. |
+| Crash recovery  | Pod OOM / preemption                               | Re-launch with `--resume-from "$SESSION_DIR"`; reads `manifest.json` + `state.json`. |
 
 ---
 
@@ -210,9 +210,9 @@ ingest it whole on session end.
 
 1. Locate the affected session directory and verify the PV is intact:
    `ls "$SESSION_DIR/state.json"`.
-2. Relaunch with `--resume`:
+2. Relaunch with `--resume-from`:
    ```bash
-   python3 -m hyperloom.inference_optimizer.cli optimize --resume --resume-from "$SESSION_DIR"
+   python3 -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR"
    ```
 3. Coordinator reads `manifest.json` + `state.json`, re-enters the
    loop at the last completed action. The current in-flight action
@@ -222,7 +222,7 @@ ingest it whole on session end.
 
 To rebuild only the `session_breakdown` (and push it to Langfuse) for a run
 that exited abnormally — without re-running the optimization loop — use the
-dedicated subcommand instead of `--resume`:
+dedicated subcommand instead of `--resume-from`:
 
 ```bash
   python3 -m hyperloom.inference_optimizer.cli recover-session --session-dir "$SESSION_DIR" [--force] [--backfill-trace]
@@ -231,8 +231,8 @@ dedicated subcommand instead of `--resume`:
 `--force` re-runs even when the session already looks complete;
 `--backfill-trace` replays `reports/trace/llm_calls.jsonl` as Langfuse
 generations (use only when the live emitter never ran, or it duplicates
-generations). `--resume` = keep optimizing; `recover-session` = rebuild the
-breakdown artifact.
+generations). `--resume-from` = keep optimizing; `recover-session` = rebuild
+the breakdown artifact.
 
 ### Scenario B: PV lost or corrupted
 

docs/reference/session-breakdown.md

@@ -752,7 +752,6 @@ The following example shows a complete `session_breakdown.json` for a finished G
   "final": {
     "throughput_tok_s_per_gpu": 150.0,
     "cumulative_gain_pct_validated": 50.0,
-    "cumulative_gain_pct_per_round_sum": 50.0,
     "validated_at_stack_len": 4,
     "validated_ts": "2026-05-17T13:48:01Z",
     "stack_changed_after_validation": false,

docs/reference/troubleshooting.md

@@ -320,7 +320,7 @@ that path is missing or reaped.
 
 ## Resume fails: "manifest.json not found"
 
-**Symptom.** `python -m hyperloom.inference_optimizer.cli optimize --resume` exits with
+**Symptom.** `python -m hyperloom.inference_optimizer.cli optimize --resume-from` exits with
 `manifest.json not found under <dir>` or `state.json missing`.
 
 **Cause**: `USER_DATA_PATH` points at a different directory than the
@@ -336,10 +336,9 @@ original session, or the session never reached the point of writing
    echo "$USER_DATA_PATH"
    find "$USER_DATA_PATH" -name manifest.json
    ```
-2. If you used a custom path the first time, pass the actual session
-   directory explicitly:
+2. Pass the actual session directory:
    ```bash
-   python3 -m hyperloom.inference_optimizer.cli optimize --resume --resume-from "$SESSION_DIR"
+   python3 -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR"
    ```
 3. If `manifest.json` truly never existed, resume is not possible —
    restart with a fresh `--model …` launch.

docs/reference/upgrade.md

@@ -236,7 +236,7 @@ For any minor or patch upgrade:
    ```
 4. If you have ongoing sessions you want to resume across the upgrade,
    verify `manifest.json` and `state.json` are intact, then run
-   `python -m hyperloom.inference_optimizer.cli optimize --resume`.
+   `python -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR"`.
 
 Upgrades do not rewrite explicit `HYPERLOOM_LOCAL_KB_ROOT` paths or historical
 sessions. The one-time implicit Recipe-root migration described above is the

examples/hyperloom-custom-advanced/SKILL.md

@@ -405,7 +405,8 @@ and the stop reason. Never print API keys, tokens, or custom header values.
    on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`.
 5. Report the session ID, log path, PID, and initial health check result.
 6. Monitor the process every 300 seconds until work is done.
-7. To recover an unexpected crash, only run `optimize --resume` against the same
-   session dir. After the first launch, never start a new `optimize`; that
-   creates a new `<UTC_ts>` session and is forbidden.
+7. To recover an unexpected crash, only run
+   `optimize --resume-from "$SESSION_DIR"` against the same session dir. After
+   the first launch, never start a new `optimize`; that creates a new
+   `<UTC_ts>` session and is forbidden.
 8. If `stop_reason` in the current session `state.json` is final, stop and exit.

examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md

@@ -193,5 +193,5 @@ and the stop reason. Never print API keys, tokens, or custom header values.
 4. Pass all required optimize CLI flags in the `python -m hyperloom.inference_optimizer.cli optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`; CLI defaults can otherwise override the intended workload.
 5. Report the session ID, log path, PID, and initial health check result.
 6. Monitor the process every 300 seconds until work is done.
-7. To recover an unexpected crash, only run `optimize --resume` against the same session dir. After the first launch, never start a new `optimize`; that creates a new `<UTC_ts>` session and is forbidden.
+7. To recover an unexpected crash, only run `optimize --resume-from "$SESSION_DIR"` against the same session dir. After the first launch, never start a new `optimize`; that creates a new `<UTC_ts>` session and is forbidden.
 8. If `stop_reason` in the current session `state.json` is final, stop and exit.