diff --git a/CHANGELOG.md b/CHANGELOG.md index 107df296e0..08fe212265 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -13,6 +13,18 @@ for the user-facing summary. ### Removed +- **BREAKING — the robustness agent's remote cluster data path is gone**, along + with the flags that fed it: `--robustness-server-url`, + `--robustness-workload-uid`, `--robustness-enable-cluster-pod-metrics` / + `--no-...`, and `--robustness-pod-metrics-categories`. Callers still passing + any of them now fail in argparse. `$ROBUSTNESS_SERVER_URL` and + `$ROBUSTNESS_ENABLE_CLUSTER_POD_METRICS` are no longer read, and the startup + probe that tried `http://robustness-server:8000` and `http://localhost:8000` + on every tick is gone. No robustness-server is deployed and none of the five + workload-uid env keys was ever set, so the endpoints could only 404; the + `cluster_fault` and `pod_not_running` symptoms went with them, having had no + other producer. + - **BREAKING — `kernel_optimization.py` no longer accepts `--test-command` or `--test-harness-path`.** The unittest-harness contract they fed had no reachable caller; an external invoker still passing either flag now fails in @@ -62,8 +74,34 @@ for the user-facing summary. read it; `KERNEL_OPT_BACKEND_ORDER` is the sole backend switch, and only an exact `forge` opts out of the default GEAK phase. +- `agents/kernel/tools/parallel_e2e_runner.py` is gone. It was the + self-validation harness written alongside the original kernel-agent, back when + no KERNEL phase existed to prove the toolkit end to end; its own first step + (running the SGLang baseline) was removed in May, leaving a driver with no + caller whose `--backends` default was empty, so it raised on any plain + invocation. Its `load_env_file` duplicated the credential-alias derivation that + `tools/backends/ray_runtime.py` still performs under wider test coverage. + ### Changed +- **Multi-node runs now use the real robustness agent instead of the heartbeat + mock.** `--nodes >= 2` previously forced `--robustness-mock`, which produced + no symptoms at all — including `deadline_imminent`, the signal that drives the + `delegate(report)` wind-down. The downgrade guarded against LocalProbe false + positives, but `disable_local_probe` already defaults to True on multi-node + and swaps the probe for a silent stub, so the signals the agent reads straight + off the Coordinator prompt and inbox were being discarded for no reason. Those + now fire: the deadline and budget ladder, `gain_plateau`, `no_levers_found`, + crash escalation, `phase_budget_nearly_exhausted`, + `conversation_no_progress`, and the inbox-driven `agent_stall` / + `repeated_failure` / `repeated_policy_denied` family. Expect alerts on + multi-node where there were none; pass `--robustness-mock` for the old + behaviour. + +- `ReactorBundle.aclose()` now closes the RCA engine's provider client. It + previously closed only the robustness-server client, leaking the HTTP client + the LLM RCA engine owns. + - The recommended vLLM container image is now the official upstream `vllm/vllm-openai-rocm:v0.27.1` instead of `rocm/hyperloom:vllm-v0.27.1-rocm7.2.3`, because AMD deprecated `rocm/vllm` diff --git a/docs/conceptual/optimization-loop.md b/docs/conceptual/optimization-loop.md index 56c81142ee..befbf27bd5 100644 --- a/docs/conceptual/optimization-loop.md +++ b/docs/conceptual/optimization-loop.md @@ -230,8 +230,9 @@ The old `backends` and `params` action names are compatibility aliases for archived reporting only. New sessions write the merged `explore_search` ledger. -After each KEEP, the runtime revalidates the full stack end to end so -the reported cumulative gain is not just a sum of per-round deltas. +After each KEEP, the runtime revalidates the full stack end to end, so +the reported `cumulative_gain_validated` always comes from a measurement +taken with every accepted change applied. ## KERNEL_AGENT diff --git a/docs/how-to/multi-node/hyperloom-remote-demo.md b/docs/how-to/multi-node/hyperloom-remote-demo.md index 8e7a52b984..97596c8b13 100644 --- a/docs/how-to/multi-node/hyperloom-remote-demo.md +++ b/docs/how-to/multi-node/hyperloom-remote-demo.md @@ -114,8 +114,8 @@ ${NFS_SHARED_ROOT}/TraceLens-internal/ # TRACELENS_INTERNAL_ROOT (optional) ## What the agent produces A session under `$USER_DATA_PATH///` with launcher -logs and a persisted `state.json` (holds `phase`, `cumulative_gain`, -`crash_count`, `stop_reason`) for status and `--resume`. `$USER_DATA_PATH` comes +logs and a persisted `state.json` (holds `phase`, `cumulative_gain_validated`, +`crash_count`, `stop_reason`) for status and `--resume-from`. `$USER_DATA_PATH` comes from the environment (platform-injected). --- diff --git a/docs/how-to/multi-node/hyperloom-remote-mn-qwen3-30b/SKILL.md b/docs/how-to/multi-node/hyperloom-remote-mn-qwen3-30b/SKILL.md index f61f16bf97..aeb50343e9 100644 --- a/docs/how-to/multi-node/hyperloom-remote-mn-qwen3-30b/SKILL.md +++ b/docs/how-to/multi-node/hyperloom-remote-mn-qwen3-30b/SKILL.md @@ -175,7 +175,8 @@ FORGE_PATH=${NFS_SHARED_ROOT}/KernelForge - Session lands in `$USER_DATA_PATH///`; `USER_DATA_PATH` is platform-injected and kept unchanged. -- Crash recovery: `optimize --resume` on the **same** session dir (never a second - `optimize`; resume past a terminal `stop_reason` needs `--force-resume`). +- Crash recovery: `optimize --resume-from "$SESSION_DIR"` on the **same** session + dir (never a second `optimize`; resume past a terminal `stop_reason` needs + `--force-resume`). - Releasing the cluster is the platform's job, not the optimizer's — it happens when the session ends. diff --git a/docs/how-to/optimize-custom-workload.md b/docs/how-to/optimize-custom-workload.md index 7db0260998..14d0695ee0 100644 --- a/docs/how-to/optimize-custom-workload.md +++ b/docs/how-to/optimize-custom-workload.md @@ -229,10 +229,10 @@ unset _dotenv_prev ``` **Sessions.** `USER_DATA_PATH` sets the session root, and each `optimize` -creates a new timestamped subdirectory under it. Use `--resume` to continue an -existing session, optionally with `--resume-from `; `--force-resume` -pushes past the terminal-state guard. Without `--resume` you always get a fresh -session, so an interrupted run is never picked up by accident. +creates a new timestamped subdirectory under it. Use +`--resume-from ` to continue an existing session; `--force-resume` +pushes past the terminal-state guard. Without `--resume-from` you always get a +fresh session, so an interrupted run is never picked up by accident. ## Monitor the run and read the output diff --git a/docs/how-to/optimize.md b/docs/how-to/optimize.md index 0c5b30d1b5..ba4197d004 100644 --- a/docs/how-to/optimize.md +++ b/docs/how-to/optimize.md @@ -74,11 +74,12 @@ Paste this prompt into Cursor Chat to resume an existing session: Resume the existing Hyperloom optimization session. Requirements: -1. Launch `python -m hyperloom.inference_optimizer.cli optimize --resume`; do not start a new session. +1. Launch `python -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR"`; do not start a new session. 2. Do not pass `--model`; read the model and workload from the saved manifest. -3. Before launching, verify `manifest.json` and `state.json` exist. -4. Report the log path, PID, health check, current phase, cumulative gain, and best config. -5. Monitor the process every 300s until the optimization is complete or failed. +3. Resolve `$SESSION_DIR` from the launch-info JSON or the `HYPERLOOM_LAUNCH` line, never from the newest timestamp dir. +4. Before launching, verify `manifest.json` and `state.json` exist. +5. Report the log path, PID, health check, current phase, cumulative gain, and best config. +6. Monitor the process every 300s until the optimization is complete or failed. ``` ## Output and artifacts diff --git a/docs/install/slurm.md b/docs/install/slurm.md index 0f3e569282..80e7fd2a51 100644 --- a/docs/install/slurm.md +++ b/docs/install/slurm.md @@ -189,7 +189,7 @@ cat //$SID/state.json The artifact directory `///` contains: -- `state.json`: live status (`baseline_tput`, `current_best`, `cumulative_gain`); +- `state.json`: live status (`baseline_tput`, `current_best`, `cumulative_gain_validated`); - `manifest.json`: session manifest; - `ci_metrics.json`: baseline/optimized throughput plus `gain_pct`; - `optimizer_runs/`: `launch_.json` and logs; diff --git a/docs/reference/environment-variables.md b/docs/reference/environment-variables.md index 1d6872c499..fd01405fba 100644 --- a/docs/reference/environment-variables.md +++ b/docs/reference/environment-variables.md @@ -58,9 +58,8 @@ The following variables configure filesystem paths for Hyperloom's runtime depen | `HYPERLOOM_ROOT` | No | `$HYPER`
`LOOM_R`
`UNTIME_`
`DIR/sou`
`rce-mirrors` | Legacy source-mirror root kept for compatibility. Current open-source dependency checkouts default to the repo-local cache root (`${HYPER`
`LOOM_CA`
`CHE_DIR:-`
`$REPO_ROOT`
`/.cache}`), not this path. | | `HYPERLOOM`
`_CACHE_`
`DIR` | No | `$REPO_ROOT`
`/.cache` | Writable, repo-local base for auto-cloned open-source deps (TraceLens, Magpie, etc.), cloned per revision as `@`. Not under `$TMPDIR` so a reaper cannot wipe it mid-run. | | `MAGPIE_PATH` | No | Resolved from installed `Magpie` package unless explicitly set | Magpie package root for benchmark wrappers and patch inspection. | -| `INFERENCE_`
`OPTIMIZER`
`_MODEL_PATH_ROOTS` | No | Built-in model roots such as `/models` and `/shared_nfs` | `os.pathsep`-separated allowlist for absolute model paths restored from `state.json` during `--resume`. HuggingFace-style repo IDs remain allowed. Set this when production models live outside the built-in roots. | +| `INFERENCE_`
`OPTIMIZER`
`_MODEL_PATH_ROOTS` | No | Built-in model roots such as `/models` and `/shared_nfs` | `os.pathsep`-separated allowlist for absolute model paths restored from `state.json` during a resume. HuggingFace-style repo IDs remain allowed. Set this when production models live outside the built-in roots. | | `SESSION_DIR` | No (robustness-agent)| Scan known paths | Path containing `storage/coordinator.db`; the robustness FindingSink writes under `{session_`
`dir}/ag`
`ents/ro`
`bustne`
`ss/fin`
`dings/`
`{sess`
`ion_id}.jsonl`. | -| `ROBUSTNESS_SERVER_URL` | No (robustness-agent)| Scan known DNS | M1 primary data source; empty disables the primary path and forces local-only probes. | | `WORKSPACE_PATH` *(legacy)* | No | Unset | Legacy path variable. Still consumed in two narrow spots: the CLI `setdefault`s it to the repo root for the critic subprocess's static assets, and TraceLens uses it as a `USER_DATA_PATH` fallback. Prefer `USER_DATA_PATH`. See [Upgrade Hyperloom version](upgrade.md). | | `INFERENCE_`
`OPTIMI`
`ZER_SES`
`SION_DIR` *(deprecated)* | No | Unset | **Retired** — replaced by `USER_DATA_PATH`. No longer read. | @@ -92,8 +91,7 @@ Set with CLI flags, not env vars. Pre-set `ISL` / `OSL` / `CONC` / `PRECISION` / `--no-framework-agent`, `--no-framework-local-explore`, `--no-kernel`, `--no-explore`, `--no-eval`. - **Agent models:** `--claude-model`, `--codex-model`. -- **Session / resume:** `--resume`, `--resume-from`, `--force-resume`, - `--reset-state`. +- **Session / resume:** `--resume-from`, `--force-resume`, `--reset-state`. - **Quantization:** `--quantize`, `--quantize-scheme`. Run `inference_optimizer optimize --help` for the exhaustive flag list. @@ -522,7 +520,7 @@ optional; defaults are safe for standard single-node deployments. | `INFERENCE_`
`OPTIMIZER_`
`AITER_JIT_DIR` | Aiter default | Per-attempt override set automatically to `/aiter_jit` by each targeted build. Override manually only when you need the global JIT cache to point at a pre-built location; leaving it unset lets each build use its own isolated directory. | | `PYTORCH_ROCM_ARCH` | Detected | Explicit GPU target architecture (e.g. `gfx942`, `gfx950`) injected into each compile. Set automatically from the session `--gpu-type`; operator-override applies to bare-metal installs outside the session. **Compile target only — it does not participate in architecture detection.** It names the archs a wheel is *built* for, not the installed device, so provenance ignores it entirely and resolves `gfx_arch` from `HYPERLOOM_GFX_ARCH`, then `--gpu-type`, then `rocminfo`. | | `MAX_JOBS` | `8` | Parallelism cap for cmake/hipcc compile steps inside a targeted build. Reduce on memory-constrained nodes (`MAX_JOBS=4` for a 64 GB compile node). The default `8` is conservative enough for MI300X/MI355X nodes with 512 GB+. | -| `HYPERLOOM_`
`FRAMEWORK_PYTHON` | Unset | Explicit interpreter that launches the server for a from-source build (the venv Python the artifact was compiled against). Set automatically from `FrameworkRuntime.runtime_python_exe` via `apply_runtime_override` into the per-variant YAML `benchmark.envs`. The bypass backend honors it by launching `python -m`; the Magpie backend re-exports it from the YAML `benchmark.envs` to the server env. Operators normally do not set this by hand. | +| `HYPERLOOM_`
`FRAMEWORK_PYTHON` | Unset | Explicit interpreter that launches the server for a from-source build (the venv Python the artifact was compiled against). Set automatically from `FrameworkRuntime.runtime_python_exe` via `apply_runtime_override` into the per-variant YAML `benchmark.envs`. Both backends export that mapping to the server env; the bypass backend additionally uses this value as the `python -m` interpreter. Operators normally do not set this by hand. | | `HYPERLOOM_`
`VLLM_ROCM_`
`INDEX_URL` | Unset | ROCm pip index URL used as the default vLLM adapter wheel index; also seeds the index allowlist. | | `HYPERLOOM_`
`ENABLEMENT_`
`INDEX_ALLOWLIST` | Unset | Comma-separated allowlist of pip index URL prefixes; a candidate wheel index must match one of these prefixes or provisioning is refused (supply-chain safety). | | `HYPERLOOM_`
`ENABLEMENT_`
`ORIGIN_ALLOWLIST` | Unset | Comma-separated allowlist of git origin URL prefixes; a candidate repo origin must match one of these prefixes or provisioning is refused (supply-chain safety). | diff --git a/docs/reference/operations.md b/docs/reference/operations.md index c93c7e12c2..71d4d251f7 100644 --- a/docs/reference/operations.md +++ b/docs/reference/operations.md @@ -126,7 +126,7 @@ The following table describes the key lifecycle events for a Hyperloom session. | Session start | API call / Job creation | Coordinator creates `$SESSION_DIR` and writes `manifest.json`, `state.json`. | | Heartbeat | Every Coordinator tick (`--tick-interval-sec`, default `0` = no sleep) | Coordinator atomically rewrites `state.json` (temp `.state.json.*.tmp` + `os.replace`) inside `$SESSION_DIR`. | | Session end | `target_reached` / `time_exhausted` / `global_converged` | Coordinator writes `session_breakdown.json`, exits 0. | -| Crash recovery | Pod OOM / preemption | Re-launch with `--resume` / `--resume-from`; reads `manifest.json` + `state.json`. | +| Crash recovery | Pod OOM / preemption | Re-launch with `--resume-from "$SESSION_DIR"`; reads `manifest.json` + `state.json`. | --- @@ -210,9 +210,9 @@ ingest it whole on session end. 1. Locate the affected session directory and verify the PV is intact: `ls "$SESSION_DIR/state.json"`. -2. Relaunch with `--resume`: +2. Relaunch with `--resume-from`: ```bash - python3 -m hyperloom.inference_optimizer.cli optimize --resume --resume-from "$SESSION_DIR" + python3 -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR" ``` 3. Coordinator reads `manifest.json` + `state.json`, re-enters the loop at the last completed action. The current in-flight action @@ -222,7 +222,7 @@ ingest it whole on session end. To rebuild only the `session_breakdown` (and push it to Langfuse) for a run that exited abnormally — without re-running the optimization loop — use the -dedicated subcommand instead of `--resume`: +dedicated subcommand instead of `--resume-from`: ```bash python3 -m hyperloom.inference_optimizer.cli recover-session --session-dir "$SESSION_DIR" [--force] [--backfill-trace] @@ -231,8 +231,8 @@ dedicated subcommand instead of `--resume`: `--force` re-runs even when the session already looks complete; `--backfill-trace` replays `reports/trace/llm_calls.jsonl` as Langfuse generations (use only when the live emitter never ran, or it duplicates -generations). `--resume` = keep optimizing; `recover-session` = rebuild the -breakdown artifact. +generations). `--resume-from` = keep optimizing; `recover-session` = rebuild +the breakdown artifact. ### Scenario B: PV lost or corrupted diff --git a/docs/reference/session-breakdown.md b/docs/reference/session-breakdown.md index 219c5da3a1..241cf24da6 100644 --- a/docs/reference/session-breakdown.md +++ b/docs/reference/session-breakdown.md @@ -752,7 +752,6 @@ The following example shows a complete `session_breakdown.json` for a finished G "final": { "throughput_tok_s_per_gpu": 150.0, "cumulative_gain_pct_validated": 50.0, - "cumulative_gain_pct_per_round_sum": 50.0, "validated_at_stack_len": 4, "validated_ts": "2026-05-17T13:48:01Z", "stack_changed_after_validation": false, diff --git a/docs/reference/troubleshooting.md b/docs/reference/troubleshooting.md index cea263aa31..4c253ea72c 100644 --- a/docs/reference/troubleshooting.md +++ b/docs/reference/troubleshooting.md @@ -320,7 +320,7 @@ that path is missing or reaped. ## Resume fails: "manifest.json not found" -**Symptom.** `python -m hyperloom.inference_optimizer.cli optimize --resume` exits with +**Symptom.** `python -m hyperloom.inference_optimizer.cli optimize --resume-from` exits with `manifest.json not found under ` or `state.json missing`. **Cause**: `USER_DATA_PATH` points at a different directory than the @@ -336,10 +336,9 @@ original session, or the session never reached the point of writing echo "$USER_DATA_PATH" find "$USER_DATA_PATH" -name manifest.json ``` -2. If you used a custom path the first time, pass the actual session - directory explicitly: +2. Pass the actual session directory: ```bash - python3 -m hyperloom.inference_optimizer.cli optimize --resume --resume-from "$SESSION_DIR" + python3 -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR" ``` 3. If `manifest.json` truly never existed, resume is not possible — restart with a fresh `--model …` launch. diff --git a/docs/reference/upgrade.md b/docs/reference/upgrade.md index d36840db80..77a40edfab 100644 --- a/docs/reference/upgrade.md +++ b/docs/reference/upgrade.md @@ -236,7 +236,7 @@ For any minor or patch upgrade: ``` 4. If you have ongoing sessions you want to resume across the upgrade, verify `manifest.json` and `state.json` are intact, then run - `python -m hyperloom.inference_optimizer.cli optimize --resume`. + `python -m hyperloom.inference_optimizer.cli optimize --resume-from "$SESSION_DIR"`. Upgrades do not rewrite explicit `HYPERLOOM_LOCAL_KB_ROOT` paths or historical sessions. The one-time implicit Recipe-root migration described above is the diff --git a/examples/hyperloom-custom-advanced/SKILL.md b/examples/hyperloom-custom-advanced/SKILL.md index 98ad73a992..29ffb5a6e7 100644 --- a/examples/hyperloom-custom-advanced/SKILL.md +++ b/examples/hyperloom-custom-advanced/SKILL.md @@ -405,7 +405,8 @@ and the stop reason. Never print API keys, tokens, or custom header values. on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`. 5. Report the session ID, log path, PID, and initial health check result. 6. Monitor the process every 300 seconds until work is done. -7. To recover an unexpected crash, only run `optimize --resume` against the same - session dir. After the first launch, never start a new `optimize`; that - creates a new `` session and is forbidden. +7. To recover an unexpected crash, only run + `optimize --resume-from "$SESSION_DIR"` against the same session dir. After + the first launch, never start a new `optimize`; that creates a new + `` session and is forbidden. 8. If `stop_reason` in the current session `state.json` is final, stop and exit. diff --git a/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md b/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md index cca65534a6..1001a8d642 100644 --- a/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md +++ b/examples/hyperloom-qwen3-14b-fp8-12h/SKILL.md @@ -193,5 +193,5 @@ and the stop reason. Never print API keys, tokens, or custom header values. 4. Pass all required optimize CLI flags in the `python -m hyperloom.inference_optimizer.cli optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`; CLI defaults can otherwise override the intended workload. 5. Report the session ID, log path, PID, and initial health check result. 6. Monitor the process every 300 seconds until work is done. -7. To recover an unexpected crash, only run `optimize --resume` against the same session dir. After the first launch, never start a new `optimize`; that creates a new `` session and is forbidden. +7. To recover an unexpected crash, only run `optimize --resume-from "$SESSION_DIR"` against the same session dir. After the first launch, never start a new `optimize`; that creates a new `` session and is forbidden. 8. If `stop_reason` in the current session `state.json` is final, stop and exit. diff --git a/examples/hyperloom-qwen3-8b-3h/SKILL.md b/examples/hyperloom-qwen3-8b-3h/SKILL.md index bb53eff48e..4776e48ae1 100644 --- a/examples/hyperloom-qwen3-8b-3h/SKILL.md +++ b/examples/hyperloom-qwen3-8b-3h/SKILL.md @@ -80,7 +80,6 @@ Required optimize CLI flags: - `--max-minutes-explore-pct 0.39` - `--max-minutes-sweep-pct 0.01` - `--explore-force-exit-budget-pct 0.01` -- `--explore-force-exit-hours-remaining 0.05` - `--no-framework-agent` - `--no-kernel` - `--no-enable-conc-sweep` @@ -196,13 +195,12 @@ and the stop reason. Never print API keys, tokens, or custom header values. and critic subprocesses can import `hyperloom.agents` after changing cwd. 3. Run in background with `setsid nohup`. 4. Pass all required optimize CLI flags in the `python -m hyperloom.inference_optimizer.cli optimize` command. Do not rely on `.env` alone for `TP`, `CONC`, `ISL`, `OSL`, or `PRECISION`; CLI defaults can otherwise override the intended workload. -5. Include `--max-minutes-explore-pct 0.39`, - `--max-minutes-sweep-pct 0.01`, - `--explore-force-exit-budget-pct 0.01`, and - `--explore-force-exit-hours-remaining 0.05` in the optimize command. With - FRAMEWORK_AGENT and KERNEL_AGENT disabled, Hyperloom redistributes their - shares so most of the short run budget is reserved for EXPLORE while still - leaving SWEEP/CLOSE time to exit cleanly near the deadline. +5. Include `--max-minutes-explore-pct 0.39` and `--max-minutes-sweep-pct 0.01` + in the optimize command. With FRAMEWORK_AGENT and KERNEL_AGENT disabled, + Hyperloom redistributes their shares so most of the short run budget is + reserved for EXPLORE while still leaving SWEEP/CLOSE time to exit cleanly + near the deadline. Also include `--explore-force-exit-budget-pct 0.01`: with + no KERNEL phase to hand the reserve to, EXPLORE should spend its whole share. 6. Include `--no-framework-agent` in the optimize command so the FRAMEWORK_AGENT phase is skipped. 7. Include `--no-kernel` in the optimize command so the Kernel Agent phase is skipped. @@ -210,5 +208,5 @@ and the stop reason. Never print API keys, tokens, or custom header values. 9. Include `--no-enable-roofline` in the optimize command so PRELUDE uses the lighter profile path instead of roofline analysis. 10. Report the session ID, log path, PID, and initial health check result. 11. Monitor the process every 300 seconds until work is done. -12. To recover an unexpected crash, only run `optimize --resume` against the same session dir. After the first launch, never start a new `optimize`; that creates a new `` session and is forbidden. +12. To recover an unexpected crash, only run `optimize --resume-from "$SESSION_DIR"` against the same session dir. After the first launch, never start a new `optimize`; that creates a new `` session and is forbidden. 13. If `stop_reason` in the current session `state.json` is final, stop and exit. diff --git a/pyproject.toml b/pyproject.toml index 2d700d3b7c..662d7c81b6 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -271,7 +271,6 @@ omit = [ "src/hyperloom/agents/kernel/tools/kernel_optimization.py", "src/hyperloom/agents/kernel/tools/tracelens_analysis.py", "src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py", - "src/hyperloom/agents/kernel/tools/parallel_e2e_runner.py", "src/hyperloom/agents/kernel/tools/apply_kernel_patch.py", "src/hyperloom/agents/kernel/tools/geak_prompt_patcher.py", # Kernel-agent tool backends (subprocess submit). diff --git a/src/hyperloom/agents/critic/runtime/decision_reviewer.py b/src/hyperloom/agents/critic/runtime/decision_reviewer.py index 00629d6a19..2bb3d07fef 100644 --- a/src/hyperloom/agents/critic/runtime/decision_reviewer.py +++ b/src/hyperloom/agents/critic/runtime/decision_reviewer.py @@ -466,15 +466,6 @@ def init_session(self, raw_request: dict[str, Any]) -> dict[str, Any]: """ req = parse_request(raw_request) merge = self.session_memory.merge_context(req.session_id, req.context) - self.session_memory.append_event( - req.session_id, - { - "kind": "init_session", - "explicit_keys": merge.explicit_keys, - "from_memory_keys": merge.from_memory_keys, - "missing_keys": merge.missing_keys, - }, - ) return { "session_id": req.session_id, "merged_context": merge.merged, @@ -525,13 +516,6 @@ def close_session( "items": len(drafts), } ) - self.session_memory.append_event( - req.session_id, - { - "kind": "close_session", - "kb_writes": [w["result"]["status"] for w in outcome.kb_writes], - }, - ) return outcome # ------------------------------------------------------------------ @@ -576,14 +560,6 @@ def prepare_review(self, raw_request: dict[str, Any]) -> JudgeBundle: bundle.required_context = critical_missing bundle.kb_read_skipped_reason = "missing_critical_context" bundle.notes.append("model and/or framework unknown — KB priors not fetched") - self.session_memory.append_event( - req.session_id, - { - "kind": "prepare_review", - "missing_critical": critical_missing, - "kb_skipped": True, - }, - ) return bundle # Skip KB reads only if explicitly disabled or inappropriate kind. @@ -595,14 +571,6 @@ def prepare_review(self, raw_request: dict[str, Any]) -> JudgeBundle: if not self.kb_writer.read_enabled: bundle.kb_read_skipped_reason = "kb_read_disabled" bundle.notes.append("KB_READ_ENABLED=false — proceeding without priors") - self.session_memory.append_event( - req.session_id, - { - "kind": "prepare_review", - "kb_skipped": True, - "reason": "kb_read_disabled", - }, - ) return bundle # Breaker open from an earlier failure → skip another timeout. @@ -610,15 +578,6 @@ def prepare_review(self, raw_request: dict[str, Any]) -> JudgeBundle: bundle.kb_read_skipped_reason = "kb_unreachable" bundle.notes.append("KB service unreachable (circuit breaker open); proceeding without priors") bundle.review_constraints["kb_breaker"] = self.kb_writer.kb_breaker_state() - self.session_memory.append_event( - req.session_id, - { - "kind": "prepare_review", - "kb_skipped": True, - "reason": "kb_unreachable", - "breaker": self.kb_writer.kb_breaker_state(), - }, - ) return bundle scope = build_scope(req.context, session_context=merge.merged, require_critical=False) @@ -644,16 +603,6 @@ def prepare_review(self, raw_request: dict[str, Any]) -> JudgeBundle: topic_hits[p.msg_id] = priors.get("priors") or [] if priors.get("cache") == "kb_unreachable": any_kb_unreachable = True - self.session_memory.append_event( - req.session_id, - { - "kind": "kb_prior_lookup", - "msg_id": p.msg_id, - "topic": topic, - "cache": priors.get("cache"), - "count": len(topic_hits[p.msg_id]), - }, - ) priors_requests.append( { "msg_id": p.msg_id, @@ -676,15 +625,6 @@ def prepare_review(self, raw_request: dict[str, Any]) -> JudgeBundle: bundle.kb_priors_for_decision = priors.get("priors") or [] if priors.get("cache") == "kb_unreachable": any_kb_unreachable = True - self.session_memory.append_event( - req.session_id, - { - "kind": "kb_prior_lookup_decision", - "topic": topic, - "cache": priors.get("cache"), - "count": len(bundle.kb_priors_for_decision), - }, - ) priors_requests.append( { "msg_id": None, diff --git a/src/hyperloom/agents/critic/runtime/session_memory.py b/src/hyperloom/agents/critic/runtime/session_memory.py index b659f657d7..9af4ecc447 100644 --- a/src/hyperloom/agents/critic/runtime/session_memory.py +++ b/src/hyperloom/agents/critic/runtime/session_memory.py @@ -203,17 +203,6 @@ def _decisions_path(self, session_id: str) -> Path: """ return self.session_dir(session_id) / "decisions.jsonl" - def _events_path(self, session_id: str) -> Path: - """Return the path to the session's ``events.jsonl``. - - Args: - session_id (str): The opaque session identifier. - - Returns: - Path: Path to the append-only audit-trail log for the session. - """ - return self.session_dir(session_id) / "events.jsonl" - def _priors_cache_path(self, session_id: str) -> Path: """Return the path to the session's ``kb_priors_cache.json``. @@ -361,30 +350,6 @@ def append_decision(self, session_id: str, decision_review: dict[str, Any]) -> N } _common_append_jsonl(self._decisions_path(session_id), record, ensure_ascii=False) - # ------------------------------------------------------------------ - # Events (free-form audit trail) - # ------------------------------------------------------------------ - def append_event(self, session_id: str, event: dict[str, Any]) -> None: - """Append a free-form audit event to the session's events log. - - The event is timestamped (``ts``) and written as one JSONL line. - - Args: - session_id (str): The opaque session identifier. - event (dict[str, Any]): The event payload to persist. - - Raises: - SessionMemoryError: If ``event`` is not a dict. - """ - if not isinstance(event, dict): - raise SessionMemoryError("event must be a dict") - self._ensure_session_dir(session_id) - _common_append_jsonl( - self._events_path(session_id), - {"ts": now_iso(timespec="microseconds"), **event}, - ensure_ascii=False, - ) - # ------------------------------------------------------------------ # KB priors cache (per-scope+topic) # ------------------------------------------------------------------ diff --git a/src/hyperloom/agents/critic/runtime/tests/test_decision_reviewer.py b/src/hyperloom/agents/critic/runtime/tests/test_decision_reviewer.py index 607d8d172f..ef5f239aec 100644 --- a/src/hyperloom/agents/critic/runtime/tests/test_decision_reviewer.py +++ b/src/hyperloom/agents/critic/runtime/tests/test_decision_reviewer.py @@ -678,7 +678,7 @@ def test_decision_request_commit_emits_decision_review(reviewer): assert outcome.kb_writes -def test_init_session_records_event(reviewer): +def test_init_session_merges_context(reviewer): rev, kb, sm = reviewer out = rev.init_session( { @@ -689,12 +689,6 @@ def test_init_session_records_event(reviewer): } ) assert out["session_id"] == "sess_init" - events = [ - json.loads(line) - for line in (sm.session_dir("sess_init") / "events.jsonl").read_text("utf-8").splitlines() - if line.strip() - ] - assert events and events[0]["kind"] == "init_session" def test_close_session_writes_kb_drafts_when_provided(reviewer): diff --git a/src/hyperloom/agents/critic/runtime/tests/test_session_memory.py b/src/hyperloom/agents/critic/runtime/tests/test_session_memory.py index ac14f380d8..cad751b099 100644 --- a/src/hyperloom/agents/critic/runtime/tests/test_session_memory.py +++ b/src/hyperloom/agents/critic/runtime/tests/test_session_memory.py @@ -85,17 +85,6 @@ def test_append_decision_rejects_non_dict(tmp_session_root): sm.append_decision("sess_1", "not a dict") # type: ignore[arg-type] -def test_events_jsonl_roundtrip(tmp_session_root): - sm = SessionMemory(root=tmp_session_root) - sm.append_event("sess_1", {"kind": "kb_cache_miss"}) - sm.append_event("sess_1", {"kind": "kb_write_ok", "id": "kb_xxx"}) - events = [ - json.loads(line) - for line in (sm.session_dir("sess_1") / "events.jsonl").read_text("utf-8").splitlines() - if line.strip() - ] - assert [e["kind"] for e in events] == ["kb_cache_miss", "kb_write_ok"] - def test_priors_cache_hit_and_miss(tmp_session_root, monkeypatch): sm = SessionMemory(root=tmp_session_root) @@ -151,13 +140,3 @@ def test_atomic_write_does_not_leave_tmp_files(tmp_session_root): assert not list(sd.glob("*.tmp")) -def test_jsonl_records_are_well_formed_lines(tmp_session_root): - sm = SessionMemory(root=tmp_session_root) - sm.append_event("sess_1", {"kind": "x"}) - sm.append_event("sess_1", {"kind": "y"}) - raw = (sm.session_dir("sess_1") / "events.jsonl").read_text("utf-8") - lines = [ln for ln in raw.splitlines() if ln.strip()] - assert len(lines) == 2 - for ln in lines: - obj = json.loads(ln) - assert "ts" in obj and "kind" in obj diff --git a/src/hyperloom/agents/critic/runtime/tests/test_session_memory_units.py b/src/hyperloom/agents/critic/runtime/tests/test_session_memory_units.py index a3f2340beb..e5797bdfe8 100644 --- a/src/hyperloom/agents/critic/runtime/tests/test_session_memory_units.py +++ b/src/hyperloom/agents/critic/runtime/tests/test_session_memory_units.py @@ -69,13 +69,10 @@ def test_merge_context_fills_from_memory(sm): assert "workload" in result.explicit_keys -def test_append_decision_and_event_write_jsonl(sm): +def test_append_decision_writes_jsonl(sm): sm.append_decision("s1", {"verdict": "approve"}) - sm.append_event("s1", {"kind": "note", "text": "hi"}) decision = json.loads(sm._decisions_path("s1").read_text("utf-8").splitlines()[0]) - event = json.loads(sm._events_path("s1").read_text("utf-8").splitlines()[0]) assert decision["decision_review"]["verdict"] == "approve" - assert event["kind"] == "note" def test_append_decision_rejects_non_dict(sm): @@ -83,11 +80,6 @@ def test_append_decision_rejects_non_dict(sm): sm.append_decision("s1", "nope") # type: ignore[arg-type] -def test_append_event_rejects_non_dict(sm): - with pytest.raises(SessionMemoryError): - sm.append_event("s1", "nope") # type: ignore[arg-type] - - def test_get_cached_priors_absent_and_malformed(sm): assert sm.get_cached_priors("s1", "k") is None sm.put_cached_priors("s1", "good", [{"id": "x"}]) @@ -147,13 +139,3 @@ def test_read_json_corrupt_raises(sm): sm.load_context("s1") -def test_event_jsonl_records_are_parseable(sm): - sm._ensure_session_dir("s1") - path = sm._events_path("s1") - path.write_text('{"kind": "a"}\n\n \n', encoding="utf-8") - rows = [json.loads(line) for line in path.read_text("utf-8").splitlines() if line.strip()] - assert [e["kind"] for e in rows] == ["a"] - - path.write_text('{"kind": "a"}\n{bad json}\n', encoding="utf-8") - with pytest.raises(json.JSONDecodeError): - [json.loads(line) for line in path.read_text("utf-8").splitlines() if line.strip()] diff --git a/src/hyperloom/agents/framework/kb.py b/src/hyperloom/agents/framework/kb.py index 6a1dcccbd3..4f52c08947 100644 --- a/src/hyperloom/agents/framework/kb.py +++ b/src/hyperloom/agents/framework/kb.py @@ -634,29 +634,10 @@ async def _drive() -> None: def _iter_message_text(message) -> Iterable[str]: - """Best-effort text extraction from a claude_agent_sdk message. + """Yield the non-empty text fragments of a claude_agent_sdk message.""" + from hyperloom.common.claude_oneshot import message_text - Accepts a plain string, ``.text``, or a ``.content`` list of blocks each - with ``.text`` (SDK message shape varies across versions). - - Args: - message: An SDK message object or string. - - Yields: - Each non-empty text fragment found on the message. - """ - if isinstance(message, str): - yield message - return - text = getattr(message, "text", None) - if isinstance(text, str): - yield text - content = getattr(message, "content", None) - if isinstance(content, list): - for block in content: - block_text = getattr(block, "text", None) - if isinstance(block_text, str): - yield block_text + yield from (fragment for fragment in message_text(message) if fragment) def synthesize_findings( diff --git a/src/hyperloom/agents/kernel/tests/test_bypass_trace_analysis.py b/src/hyperloom/agents/kernel/tests/test_bypass_trace_analysis.py index a8dfac99b3..7981595032 100644 --- a/src/hyperloom/agents/kernel/tests/test_bypass_trace_analysis.py +++ b/src/hyperloom/agents/kernel/tests/test_bypass_trace_analysis.py @@ -677,19 +677,15 @@ def test_text_gen_default_full_trace_not_estimated(tmp_path, capsys, monkeypatch ] -def test_fusion_artifact_and_result(tmp_path, capsys, monkeypatch): - # Two consecutive Elementwise launches -> one fusable cluster, emitted in - # both the result summary and the kernel_sequence.json artifact. +def test_fusion_result_summary(tmp_path, capsys, monkeypatch): + """Two consecutive Elementwise launches -> one fusable cluster.""" trace = tmp_path / "f.trace.json" trace.write_bytes(json.dumps({"traceEvents": _FUSION_EVENTS}).encode("utf-8")) _, result, _ = _run(_base_argv(tmp_path, str(trace)), capsys) assert result["fusion"]["launch_count"] == 2 assert result["fusion"]["fusable_cluster_count"] == 1 - seq_path = result["artifact_paths"]["kernel_sequence"] - assert Path(seq_path).is_file() - seq = json.loads(Path(seq_path).read_text()) - assert seq["fusable_clusters"][0]["launch_count"] == 2 - assert "Elementwise" in seq["fusable_clusters"][0]["categories"] + assert result["fusion"]["fusable_time_us"] > 0.0 + assert "kernel_sequence" not in result["artifact_paths"] def test_csv_artifacts_written_and_paths_exposed(tmp_path, capsys, monkeypatch): diff --git a/src/hyperloom/agents/kernel/tests/test_kernel_agent_live.py b/src/hyperloom/agents/kernel/tests/test_kernel_agent_live.py index 85a8e80ed7..12ad4e9440 100644 --- a/src/hyperloom/agents/kernel/tests/test_kernel_agent_live.py +++ b/src/hyperloom/agents/kernel/tests/test_kernel_agent_live.py @@ -39,7 +39,7 @@ def load_dotenv(path: Path) -> dict[str, str]: key, value = line.split("=", 1) value = value.strip().strip('"').strip("'") env[key.strip()] = value - # Mirror parallel_e2e_runner.load_env_file: each side's aliases come from that + # Mirror ray_runtime.safe_runtime_env: each side's aliases come from that # side's own credentials, and the GEAK aliases are never derived. if "ANTHROPIC_AUTH_TOKEN" in env: env.setdefault("ANTHROPIC_API_KEY", env["ANTHROPIC_AUTH_TOKEN"]) diff --git a/src/hyperloom/agents/kernel/tests/test_llm_source_fallback.py b/src/hyperloom/agents/kernel/tests/test_llm_source_fallback.py index 4b58191e09..a61f540dcb 100644 --- a/src/hyperloom/agents/kernel/tests/test_llm_source_fallback.py +++ b/src/hyperloom/agents/kernel/tests/test_llm_source_fallback.py @@ -537,8 +537,10 @@ def _get_client(**kwargs): def test_message_text_accepts_sdk_text_shapes(): """Claude SDK string, text, and content-block forms must all be readable.""" - assert lsf._message_text("direct") == ["direct"] - assert lsf._message_text(types.SimpleNamespace(text="attribute")) == ["attribute"] + from hyperloom.common.claude_oneshot import message_text + + assert message_text("direct") == ["direct"] + assert message_text(types.SimpleNamespace(text="attribute")) == ["attribute"] message = types.SimpleNamespace( content=[ {"text": "dict"}, @@ -546,7 +548,7 @@ def test_message_text_accepts_sdk_text_shapes(): {"other": "ignored"}, ] ) - assert lsf._message_text(message) == ["dict", "object"] + assert message_text(message) == ["dict", "object"] def test_claude_provider_rejects_incomplete_sdk(monkeypatch): diff --git a/src/hyperloom/agents/kernel/tests/test_parallel_e2e_env.py b/src/hyperloom/agents/kernel/tests/test_parallel_e2e_env.py deleted file mode 100644 index 88ddbf478e..0000000000 --- a/src/hyperloom/agents/kernel/tests/test_parallel_e2e_env.py +++ /dev/null @@ -1,67 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Unit tests for ``parallel_e2e_runner.load_env_file`` key/URL derivation. - -Locks the per-side alias derivation: each side's aliases come from that side's -own credentials, and the GEAK aliases are never derived at all (GEAK runs on the -Anthropic side via GEAK_CLAUDE_MODEL + ANTHROPIC_*). -""" - -from __future__ import annotations - -import sys -from pathlib import Path - -sys.path.insert(0, str(Path(__file__).resolve().parent.parent / "tools")) - -import parallel_e2e_runner as per # noqa: E402 - - -def _write_env(tmp_path: Path, **vars: str) -> Path: - p = tmp_path / ".env" - p.write_text("\n".join(f"{k}={v}" for k, v in vars.items()) + "\n", encoding="utf-8") - return p - - -def test_openai_only_leaves_anthropic_aliases_unset(tmp_path): - """OpenAI side only: its own aliases are filled and the Anthropic side stays - unset, so the OpenAI key is never forwarded as an Anthropic credential.""" - env = per.load_env_file(_write_env(tmp_path, OPENAI_API_KEY="ak-gateway", OPENAI_BASE_URL="https://gw/v1")) - for alias in ("OPENAI_API_KEY", "LLM_API_KEY", "AMD_LLM_API_KEY"): - assert env[alias] == "ak-gateway", alias - for alias in ("OPENAI_BASE_URL", "LLM_API_BASE"): - assert env[alias] == "https://gw/v1", alias - for alias in ("ANTHROPIC_API_KEY", "ANTHROPIC_AUTH_TOKEN", "ANTHROPIC_BASE_URL"): - assert alias not in env, alias - # GEAK is Anthropic-only, so an OpenAI-side value is never handed to it. - for alias in ("GEAK_API_KEY", "GEAK_BASE_URL"): - assert alias not in env, alias - assert "_".join(("legacy backend", "API", "KEY")) not in env - assert "_".join(("legacy backend", "BASE", "URL")) not in env - - -def test_split_gateway_llm_aliases_take_openai_key(tmp_path): - """Split deploy: the generic OpenAI-protocol aliases derive from OpenAI, while - the GEAK aliases stay unset for either side to claim.""" - env = per.load_env_file( - _write_env( - tmp_path, - OPENAI_API_KEY="openai-test-key", - ANTHROPIC_API_KEY="anthropic-test-key", - OPENAI_BASE_URL="https://api.openai.com/v1", - ANTHROPIC_BASE_URL="https://api.anthropic.com", - ) - ) - for alias in ("LLM_API_KEY", "AMD_LLM_API_KEY"): - assert env[alias] == "openai-test-key", alias - assert env["ANTHROPIC_API_KEY"] == "anthropic-test-key" - assert env["LLM_API_BASE"] == "https://api.openai.com/v1" - for alias in ("GEAK_API_KEY", "GEAK_BASE_URL"): - assert alias not in env, alias - assert "_".join(("legacy backend", "API", "KEY")) not in env - assert "_".join(("legacy backend", "BASE", "URL")) not in env - - -def test_missing_file_returns_empty(tmp_path): - assert per.load_env_file(tmp_path / "nope.env") == {} diff --git a/src/hyperloom/agents/kernel/tests/test_payload_aliases_shim.py b/src/hyperloom/agents/kernel/tests/test_payload_aliases_shim.py deleted file mode 100644 index ab67992831..0000000000 --- a/src/hyperloom/agents/kernel/tests/test_payload_aliases_shim.py +++ /dev/null @@ -1,90 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Smoke tests for the kernel-agent payload-aliases compat shim.""" - -from __future__ import annotations - -import subprocess -import sys -import warnings -from pathlib import Path - -import pytest - -_TOOLS_DIR = Path(__file__).resolve().parent.parent / "tools" -if str(_TOOLS_DIR) not in sys.path: - sys.path.insert(0, str(_TOOLS_DIR)) - -from _payload_aliases import ( # type: ignore[import-not-found] # noqa: E402 - CANONICAL_KEY, - LEGACY_KEY, - read_extra_server_args, -) - - -def test_shim_constants_match_canonical_names(): - assert CANONICAL_KEY == "extra_server_args" - assert LEGACY_KEY == "extra_sglang_args" - - -def test_shim_canonical_key_returns_value_without_warning(): - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - out = read_extra_server_args({CANONICAL_KEY: "--x"}) - assert out == "--x" - assert not [w for w in caught if issubclass(w.category, DeprecationWarning)] - - -def test_shim_legacy_key_emits_warning(): - with pytest.warns(DeprecationWarning): - out = read_extra_server_args({LEGACY_KEY: "--legacy"}) - assert out == "--legacy" - - -def test_shim_default_returned_when_empty(): - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - assert read_extra_server_args({}) == "" - assert read_extra_server_args({}, default="z") == "z" - assert not caught - - -def test_shim_canonical_wins_over_legacy_when_both_present(): - with warnings.catch_warnings(record=True) as caught: - warnings.simplefilter("always") - out = read_extra_server_args({CANONICAL_KEY: "new", LEGACY_KEY: "old"}) - assert out == "new" - assert not [w for w in caught if issubclass(w.category, DeprecationWarning)] - - -def test_shim_has_no_hyperloom_import(): - """Static guard: the shim's source must not *import* ``hyperloom``. - - A re-export would defeat the standalone contract even though the module - technically still exists on disk. - """ - source = (_TOOLS_DIR / "_payload_aliases.py").read_text(encoding="utf-8") - assert "import hyperloom" not in source - - -def test_shim_importable_without_hyperloom_on_sys_path(): - """End-to-end contract check: run in a fresh subprocess with only - ``tools/`` on ``sys.path`` and no ``hyperloom`` package importable, - mirroring a real standalone remote-node invocation.""" - code = ( - "import sys; " - f"sys.path = [{str(_TOOLS_DIR)!r}] + [p for p in sys.path if p]; " - "import _payload_aliases as pa; " - "print(pa.read_extra_server_args({pa.CANONICAL_KEY: '--x'}))" - ) - proc = subprocess.run( - [sys.executable, "-c", code], - capture_output=True, - text=True, - timeout=30, - check=False, - ) - assert proc.returncode == 0, f"stdout={proc.stdout!r} stderr={proc.stderr!r}" - assert proc.stdout.strip() == "--x" - assert "hyperloom" not in proc.stderr diff --git a/src/hyperloom/agents/kernel/tools/_llm_source_fallback.py b/src/hyperloom/agents/kernel/tools/_llm_source_fallback.py index c3dda3f635..c0d3495986 100644 --- a/src/hyperloom/agents/kernel/tools/_llm_source_fallback.py +++ b/src/hyperloom/agents/kernel/tools/_llm_source_fallback.py @@ -382,23 +382,6 @@ def _complete_openai(prompt: str, model: str, timeout_sec: float) -> str: ).text -def _message_text(message: Any) -> list[str]: - """Extract text fragments from one Claude SDK message.""" - if isinstance(message, str): - return [message] - text = getattr(message, "text", None) - if isinstance(text, str): - return [text] - parts: list[str] = [] - content = getattr(message, "content", None) - if isinstance(content, list): - for block in content: - block_text = block.get("text") if isinstance(block, dict) else getattr(block, "text", None) - if isinstance(block_text, str): - parts.append(block_text) - return parts - - def _complete_claude_sdk(prompt: str, model: str, timeout_sec: float) -> str: """Run one tool-free completion through the native Claude Agent SDK.""" try: @@ -408,6 +391,7 @@ def _complete_claude_sdk(prompt: str, model: str, timeout_sec: float) -> str: if not (hasattr(sdk, "query") and hasattr(sdk, "ClaudeAgentOptions")): raise RuntimeError("claude_agent_sdk missing query / ClaudeAgentOptions") + from hyperloom.common.claude_oneshot import message_text # noqa: PLC0415 from hyperloom.common.llm_config import claude_sdk_env_options # noqa: PLC0415 kwargs: dict[str, Any] = dict(claude_sdk_env_options(model=model)) @@ -437,7 +421,7 @@ async def _drive() -> str: if isinstance(result, str) and result.strip(): final = result continue - chunks.extend(_message_text(message)) + chunks.extend(message_text(message)) return final.strip() or "".join(chunks).strip() try: diff --git a/src/hyperloom/agents/kernel/tools/_payload_aliases.py b/src/hyperloom/agents/kernel/tools/_payload_aliases.py deleted file mode 100644 index 37309f7a7d..0000000000 --- a/src/hyperloom/agents/kernel/tools/_payload_aliases.py +++ /dev/null @@ -1,78 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Stdlib-only payload-args helper for standalone kernel-agent tools. - -Kept independent so ``tools/`` scripts run standalone on remote nodes without a -``hyperloom`` import. Do not import ``hyperloom.common`` here. -""" - -from __future__ import annotations - -import warnings -from typing import Any - - -CANONICAL_KEY: str = "extra_server_args" -LEGACY_KEY: str = "extra_sglang_args" - - -_DEPRECATION_MESSAGE: str = ( - f"payload field {LEGACY_KEY!r} is a deprecation alias for " - f"{CANONICAL_KEY!r}. The legacy name carries the same value but " - f"will be removed in the next Hyperloom release — switch the " - f"writer site (or the operator script emitting this payload) to " - f"the canonical name." -) - - -def _coerce_str(value: Any) -> str: - """Coerce a payload value to a string: ``None`` -> ``""``, ``str`` - unchanged, list/tuple space-joined into shell tokens (each entry stripped, - blanks dropped), anything else through ``str()``. - """ - if value is None: - return "" - if isinstance(value, str): - return value - # Server flags emitted as a JSON list are space-joined into shell tokens. - if isinstance(value, (list, tuple)): - return " ".join(str(v).strip() for v in value if str(v).strip()) - return str(value) - - -def read_extra_server_args(payload: dict, *, default: str = "") -> str: - """Read ``extra_server_args`` from a payload dict, with a read-only - fallback to the legacy ``extra_sglang_args`` key. - - Resolution order (checks use ``in``, so an empty string value is distinct - from a missing key): - - 1. If ``payload[CANONICAL_KEY]`` is present (any value, including empty - string), return it coerced via :func:`_coerce_str`; no warning. - 2. Else if ``payload[LEGACY_KEY]`` is present, emit a single - ``DeprecationWarning`` (``stacklevel=3``) and return the coerced value. - 3. Else return ``default``. - - Args: - payload (dict): Dict-like payload (``Intent.payload`` / ``Task.params`` / - a JSON envelope body / a SharedState entry). - default (str): Returned when neither key is present. - - Returns: - str: The coerced canonical value, the coerced legacy value (with a - ``DeprecationWarning``), or ``default``. - """ - if CANONICAL_KEY in payload: - return _coerce_str(payload[CANONICAL_KEY]) - if LEGACY_KEY in payload: - warnings.warn(_DEPRECATION_MESSAGE, DeprecationWarning, stacklevel=3) - return _coerce_str(payload[LEGACY_KEY]) - return default - - -__all__ = [ - "CANONICAL_KEY", - "LEGACY_KEY", - "read_extra_server_args", -] diff --git a/src/hyperloom/agents/kernel/tools/bypass_trace_analysis.py b/src/hyperloom/agents/kernel/tools/bypass_trace_analysis.py index 7c06208e80..f12dbd1f27 100644 --- a/src/hyperloom/agents/kernel/tools/bypass_trace_analysis.py +++ b/src/hyperloom/agents/kernel/tools/bypass_trace_analysis.py @@ -741,7 +741,6 @@ def main(argv: list[str] | None = None) -> int: manifest_path = run_dir / "trace_input_manifest.json" roofline_name = args.roofline_output_name or "kernel_roofline.json" kernel_roofline_path = reports_dir / roofline_name - kernel_sequence_path = bypass_dir / "kernel_sequence.json" kernel_metrics_csv_path = bypass_dir / "kernel_metrics.csv" kernel_summary_csv_path = bypass_dir / "kernel_summary.csv" @@ -849,7 +848,6 @@ def main(argv: list[str] | None = None) -> int: # Kernel-fusion opportunities: launch adjacency -> fusable clusters. fusion = _report.build_fusion(analyze) - atomic_write_json(kernel_sequence_path, fusion, ensure_ascii=False, sort_keys=False, trailing_newline=False) # Diffusion / scriptable workload-level roofline: aggregate the per-kernel # analytical roofline into an end-to-end workload roofline + per-denoise-step @@ -911,7 +909,6 @@ def main(argv: list[str] | None = None) -> int: "trace_report_path": str(analysis_md_path), "kernel_roofline_path": str(kernel_roofline_path), "tracelens_summary_path": str(summary_path), - "kernel_sequence_path": str(kernel_sequence_path), "kernel_metrics_csv_path": str(kernel_metrics_csv_path), "kernel_summary_csv_path": str(kernel_summary_csv_path), "fusion": { @@ -929,7 +926,6 @@ def main(argv: list[str] | None = None) -> int: "kernel_candidates": str(candidates_path), "kernel_roofline": str(kernel_roofline_path), "tracelens_summary": str(summary_path), - "kernel_sequence": str(kernel_sequence_path), "kernel_metrics_csv": str(kernel_metrics_csv_path), "kernel_summary_csv": str(kernel_summary_csv_path), "trace_input_manifest": str(manifest_path), diff --git a/src/hyperloom/agents/kernel/tools/kernel_optimization.py b/src/hyperloom/agents/kernel/tools/kernel_optimization.py index 2e6a242d8d..336c0b7b9d 100755 --- a/src/hyperloom/agents/kernel/tools/kernel_optimization.py +++ b/src/hyperloom/agents/kernel/tools/kernel_optimization.py @@ -1402,14 +1402,9 @@ def build_kernel_metadata(candidate: dict[str, Any], args: argparse.Namespace) - runtime_flags.update(candidate["runtime_flags"]) runtime_flags.setdefault("is_multigpu", bool(candidate.get("is_multigpu"))) runtime_flags.setdefault("num_gpus_recommended", candidate.get("num_gpus_recommended")) - # Standalone shim: keeps kernel-agent scripts independent from ``hyperloom``. - from _payload_aliases import ( # type: ignore[import-not-found] - read_extra_server_args as _read_eserver, - ) - extra_server_args = ( getattr(args, "extra_server_args", "") - or _read_eserver(candidate) + or candidate.get("extra_server_args", "") or candidate.get("candidate_extra_server_args", "") ) parsed_sglang_args = parse_extra_server_args(str(extra_server_args)) diff --git a/src/hyperloom/agents/kernel/tools/kernel_source_index.py b/src/hyperloom/agents/kernel/tools/kernel_source_index.py index feab81cadc..382d8a7ff7 100644 --- a/src/hyperloom/agents/kernel/tools/kernel_source_index.py +++ b/src/hyperloom/agents/kernel/tools/kernel_source_index.py @@ -242,11 +242,6 @@ def _save_cache(index: SourceIndex) -> None: _PROCESS_INDEX: SourceIndex | None = None -def reset_index_cache() -> None: - """Drop the in-process index singleton (for tests / a forced rebuild).""" - global _PROCESS_INDEX - _PROCESS_INDEX = None - def load_or_build(frameworks: dict[str, source_env.FrameworkRoot] | None = None) -> SourceIndex: """Return a cached index for the current versions, or build + cache one. diff --git a/src/hyperloom/agents/kernel/tools/parallel_e2e_runner.py b/src/hyperloom/agents/kernel/tools/parallel_e2e_runner.py deleted file mode 100644 index f5395d9d8a..0000000000 --- a/src/hyperloom/agents/kernel/tools/parallel_e2e_runner.py +++ /dev/null @@ -1,538 +0,0 @@ -#!/usr/bin/env python3 -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""End-to-end Kernel-agent runner for real model/profile/backend testing. - -Takes a pre-generated trace (``--trace-path``), picks a hot kernel, launches -backend optimization attempts in parallel (backend x replicas), and summarizes. -Does not fabricate patch effectiveness; records absence of patchable source / -benchmark file as the outcome. -""" - -from __future__ import annotations - -import argparse -import json -import os -import subprocess -import sys -import time -from concurrent.futures import ThreadPoolExecutor, as_completed -from pathlib import Path -from typing import Any - - -ROOT = Path(__file__).resolve().parents[1] -TRACE_TOOL = ROOT / "tools" / "tracelens_analysis.py" -OPT_TOOL = ROOT / "tools" / "kernel_optimization.py" - -# Local sibling import for the collective-name fallback (tools/ on sys.path). -sys.path.insert(0, str(ROOT / "tools")) -from _collective_names import kernel_name_implies_multigpu # noqa: E402 -from _io_utils import extract_last_json, utc_now # noqa: E402 -from _paths import workspace_root # noqa: E402 - -sys.path.pop(0) - - -def write_json(path: Path, data: dict[str, Any]) -> None: - """Write pretty-printed sorted JSON, creating parents.""" - path.parent.mkdir(parents=True, exist_ok=True) - path.write_text(json.dumps(data, indent=2, sort_keys=True) + "\n", encoding="utf-8") - - -def load_env_file(path: Path) -> dict[str, str]: - """Parse a ``KEY=VALUE`` env file and derive provider-key aliases.""" - env: dict[str, str] = {} - if not str(path) or not path.is_file(): - return env - for line in path.read_text(encoding="utf-8", errors="replace").splitlines(): - line = line.strip() - if not line or line.startswith("#") or "=" not in line: - continue - key, value = line.split("=", 1) - env[key.strip()] = value.strip().strip('"').strip("'") - # Each side's aliases come from that side's own credentials. GEAK_API_KEY / - # GEAK_BASE_URL are never derived: GEAK runs on the Anthropic side via - # GEAK_CLAUDE_MODEL + ANTHROPIC_*, so an OpenAI-side value could not start it. - openai_key = env.get("OPENAI_API_KEY") or env.get("AMD_API_KEY") - if openai_key: - env.setdefault("LLM_API_KEY", openai_key) - env.setdefault("AMD_LLM_API_KEY", openai_key) - anthropic_key = env.get("ANTHROPIC_API_KEY") or env.get("ANTHROPIC_AUTH_TOKEN") - if anthropic_key: - env.setdefault("ANTHROPIC_API_KEY", anthropic_key) - env.setdefault("ANTHROPIC_AUTH_TOKEN", anthropic_key) - openai_url = env.get("OPENAI_BASE_URL") - if openai_url: - env.setdefault("LLM_API_BASE", openai_url) - return env - - -def run_json(cmd: list[str], *, env: dict[str, str], timeout_s: int, log_path: Path) -> dict[str, Any]: - """Run a subprocess, tee output to a log, and parse trailing JSON.""" - log_path.parent.mkdir(parents=True, exist_ok=True) - with log_path.open("a", encoding="utf-8") as log: - log.write("$ " + " ".join(cmd) + "\n") - proc = subprocess.run( - cmd, - text=True, - stdout=subprocess.PIPE, - stderr=subprocess.STDOUT, - env=env, - timeout=timeout_s, - ) - log.write(proc.stdout or "") - log.write(f"\n[exit_code] {proc.returncode}\n") - if proc.returncode != 0: - raise RuntimeError(f"command failed: {' '.join(cmd)}; see {log_path}") - parsed = extract_last_json(proc.stdout or "") - if parsed is None: - raise ValueError("no trailing JSON object in stdout") - return parsed - - -def _ensure_ray_via_helper(num_gpus: int, log_path: Path) -> bool: - """Use the kernel-agent self-contained ray_runtime helper.""" - sys.path.insert(0, str(ROOT / "tools" / "backends")) - from ray_runtime import ensure_ray_cluster # type: ignore - - return ensure_ray_cluster(num_gpus=num_gpus, log_path=log_path) - - -def _stop_ray_via_helper(started: bool, log_path: Path) -> None: - """Stop Ray via the ray_runtime helper if this runner started it. - - Args: - started (bool): The return value from - :func:`_ensure_ray_via_helper`. - log_path (Path): File for Ray lifecycle output. - """ - sys.path.insert(0, str(ROOT / "tools" / "backends")) - from ray_runtime import stop_ray_if_owned # type: ignore - - stop_ray_if_owned(started, log_path=log_path) - - -def choose_candidate(candidates: list[dict[str, Any]], kernel_name: str = "", kernel_id: str = "") -> dict[str, Any]: - """Select a hot-kernel candidate from the analyzed list. - - Resolution order: by ``kernel_id`` if given, else by ``kernel_name``, - else the first candidate with an existing patchable ``source_file``, - else the first candidate overall. - - Args: - candidates (list[dict[str, Any]]): Hot-kernel candidate dicts. - kernel_name (str): Optional exact kernel name to match. - kernel_id (str): Optional kernel id to match; takes precedence - over ``kernel_name``. - - Returns: - dict[str, Any]: The selected candidate dict. - - Raises: - RuntimeError: If a requested id/name is not found, or the - candidate list is empty. - """ - if kernel_id: - for c in candidates: - if c.get("kernel_id") == kernel_id: - return c - raise RuntimeError(f"kernel_id not found in candidates: {kernel_id}") - if kernel_name: - for c in candidates: - if c.get("name") == kernel_name: - return c - raise RuntimeError(f"kernel name not found in candidates: {kernel_name}") - patchable = [c for c in candidates if c.get("source_file") and Path(str(c["source_file"])).exists()] - if patchable: - return patchable[0] - if not candidates: - raise RuntimeError("no hot kernels found") - return candidates[0] - - -def run_one_attempt( - *, - backend: str, - replica: int, - gpu_id: int, - args: argparse.Namespace, - run_dir: Path, - env: dict[str, str], - kernel_id: str, - source_file: str, - benchmark_file: str, - num_gpus: int = 1, -) -> dict[str, Any]: - """Run a single backend/replica kernel-optimization attempt. - - Args: - backend: Backend name to run (e.g. ``forge``). - replica: Replica index within the backend's parallel fan-out. - gpu_id: Logical GPU id assigned to this attempt (informational; Ray - sets the visible-device env vars in workers). - args: Parsed CLI arguments for the run. - run_dir: Per-run output directory. - env: Base environment to extend for the child process. - kernel_id: Identifier of the kernel being optimized. - source_file: Path to the kernel source file. - benchmark_file: Path to the benchmark file used for patch validation. - num_gpus: Number of GPUs allotted to this attempt. - - Returns: - A result dict describing the attempt's outcome and artifact paths. - """ - # Do NOT set HIP/ROCR/CUDA_VISIBLE_DEVICES here; Ray assigns them in workers. - local_env = { - **env, - # Forward workspace-path so nested subprocesses share the artefact root. - "USER_DATA_PATH": str(args.workspace_path), - "KERNEL_AGENT_NUM_GPUS": str(num_gpus), - } - log_path = run_dir / "logs" / "parallel" / f"{backend}_replica{replica}.log" - cmd = [ - sys.executable, - str(OPT_TOOL), - "--kernel-id", - kernel_id, - "--session-id", - args.session_id, - "--backends", - backend, - "--budget-minutes", - str(args.backend_budget_min), - "--num-gpus", - str(num_gpus), - ] - if source_file: - cmd.extend(["--source-file", source_file]) - if benchmark_file: - cmd.extend(["--benchmark-file", benchmark_file]) - started = time.time() - _effective_budget_min = args.backend_budget_min - try: - result = run_json(cmd, env=local_env, timeout_s=int(_effective_budget_min * 60) + 360, log_path=log_path) - status = "ok" - except Exception as exc: - result = {"error": f"{type(exc).__name__}: {exc}"} - status = "failed" - return { - "backend": backend, - "replica": replica, - "gpu_id": gpu_id, - "num_gpus": num_gpus, - "status": status, - "elapsed_s": round(time.time() - started, 2), - "log_path": str(log_path), - "result": result, - } - - -def write_summary(run_dir: Path, summary: dict[str, Any]) -> None: - """Write the run summary as JSON and Markdown.""" - write_json(run_dir / "parallel_e2e_summary.json", summary) - lines = [ - "# Kernel-agent Parallel E2E Summary", - "", - f"- Session: `{summary['session_id']}`", - f"- Model: `{summary['model_path']}`", - f"- Trace: `{summary.get('trace_path', '')}`", - f"- Selected kernel: `{summary.get('selected_kernel', {}).get('name', '')}`", - "", - "## Backend Attempts", - "", - ] - for item in summary.get("parallel_results", []): - result = item.get("result", {}) - attempts = result.get("attempts") or [] - attempt_status = attempts[0].get("status") if attempts else item.get("status") - decision = result.get("proposal", {}).get("decision", "n/a") - lines.append( - f"- {item['backend']} replica {item['replica']} GPU {item['gpu_id']}: " - f"{attempt_status}, decision={decision}, elapsed={item['elapsed_s']}s" - ) - lines.extend( - [ - "", - "## Patch/Retest", - "", - summary.get("patch_retest_status", "not attempted"), - "", - ] - ) - (run_dir / "parallel_e2e_summary.md").write_text("\n".join(lines), encoding="utf-8") - - -def main() -> int: - """Drive the full parallel end-to-end run.""" - parser = argparse.ArgumentParser(description="Run Kernel-agent real parallel E2E") - parser.add_argument("--model-path", default="/models/Qwen3-30B-A3B") - parser.add_argument( - "--workspace-path", - default=workspace_root(), - help="Root the tool writes under; defaults to $USER_DATA_PATH.", - ) - parser.add_argument("--session-id", default=f"qwen3-30b-{int(time.time())}") - parser.add_argument("--env-file", default="") - parser.add_argument("--tp", type=int, default=8) - parser.add_argument("--conc", type=int, default=4) - parser.add_argument("--isl", type=int, default=256) - parser.add_argument("--osl", type=int, default=128) - parser.add_argument( - "--backend-budget-min", - type=float, - default=60, - help="Wall-clock budget per backend attempt in minutes " - "(default 60). Agents are told to early-exit as " - "soon as they hit >=1.50x with passing correctness; " - "otherwise they iterate up to ~85%% of this budget " - "and SIGTERM at 100%%.", - ) - parser.add_argument("--replicas-per-backend", type=int, default=2) - parser.add_argument( - "--backends", - default="", - help=( - "Comma list of agentic backends. Default is empty because current " - "GEAK owns the KERNEL phase; per-kernel forge requires exact " - "KERNEL_OPT_BACKEND_ORDER=forge." - ), - ) - parser.add_argument( - "--num-gpus-override", - type=int, - default=0, - help="If >0, override candidate.num_gpus_recommended for " - "every backend task. Use 2 to test the multi-GPU " - "communication-kernel path explicitly.", - ) - parser.add_argument( - "--total-gpus", - type=int, - default=8, - help="Total GPUs available on this host; used to cap concurrency (default 8 for MI355X box).", - ) - parser.add_argument( - "--trace-path", - required=True, - help=( - "Path to a pre-generated trace (``.json`` / ``.json.gz``) or a " - "torch_trace dir. Use ``python -m hyperloom.inference_optimizer.cli optimize`` to " - "produce baseline+profile traces." - ), - ) - parser.add_argument( - "--kernel-name", - default="", - help="Pick this exact kernel name from the trace (default: first patchable hot kernel).", - ) - parser.add_argument( - "--kernel-id", default="", help="Pick by kernel_id (k001/k002/...); takes precedence over --kernel-name." - ) - parser.add_argument( - "--reuse-candidates-from", - default="", - help="Reuse a previous run's kernel_candidates.json instead of re-running the trace analysis.", - ) - args = parser.parse_args() - - workspace = Path(args.workspace_path) - run_dir = workspace / "kernel-agent" / "runs" / args.session_id - run_dir.mkdir(parents=True, exist_ok=True) - env = { - **os.environ, - **load_env_file(Path(args.env_file)), - # Forward workspace-path so children share the artefact root. - "USER_DATA_PATH": str(workspace), - } - - summary: dict[str, Any] = { - "session_id": args.session_id, - "model_path": args.model_path, - "created_at": utc_now(), - } - try: - trace_path = args.trace_path - baseline = {"trace_path": trace_path} - if not trace_path or not Path(trace_path).exists(): - raise RuntimeError( - f"--trace-path missing or does not exist: {trace_path}. " - "Produce a trace with ``python -m hyperloom.inference_optimizer.cli optimize`` first." - ) - summary["baseline"] = baseline - summary["trace_path"] = trace_path - - if args.reuse_candidates_from: - src = Path(args.reuse_candidates_from) - if not src.exists(): - raise RuntimeError(f"--reuse-candidates-from path missing: {src}") - data = json.loads(src.read_text()) - candidates = ( - data if isinstance(data, list) else (data.get("hot_kernels") or data.get("kernel_candidates") or []) - ) - # Mirror to the default candidates_path so kernel_optimization finds it. - (run_dir / "kernel_candidates.json").write_text(json.dumps(candidates, indent=2)) - analysis = {"trace_report_path": str(src), "reused": True} - else: - analysis = run_json( - [ - sys.executable, - str(TRACE_TOOL), - "--trace-input", - trace_path, - "--session-id", - args.session_id, - "--model-name", - Path(args.model_path).name, - "--framework", - "sglang", - "--budget-minutes", - "60", - ], - env=env, - timeout_s=3600, - log_path=run_dir / "logs" / "tracelens_analysis_driver.log", - ) - candidates = analysis.get("hot_kernels", []) - selected = choose_candidate(candidates, kernel_name=args.kernel_name, kernel_id=args.kernel_id) - summary["analysis"] = { - "trace_report_path": analysis.get("trace_report_path"), - "num_hot_kernels": len(candidates), - } - summary["selected_kernel"] = selected - - source_file = str(selected.get("source_file") or "") - # Prefer `bench`-style scripts then `test_*`. - bench_files = list(selected.get("benchmark_files") or []) - # is_multigpu := TraceLens flag OR kernel name matches a known collective. - selected_name = str(selected.get("name") or "") - name_says_collective = kernel_name_implies_multigpu(selected_name) - is_multigpu = bool(selected.get("is_multigpu")) or name_says_collective - if name_says_collective and not bool(selected.get("is_multigpu")): - summary["multigpu_inferred_from_name"] = ( - f"is_multigpu inferred from kernel name {selected_name!r} (TraceLens did not flag is_multigpu=True)" - ) - benchmark_file = "" - if bench_files: - preferred = [b for b in bench_files if "bench" in Path(b).name.lower()] - if not preferred and not is_multigpu: - preferred = [b for b in bench_files if Path(b).name.startswith("test_")] - benchmark_file = (preferred or bench_files)[0] - if not source_file: - summary["source_resolution"] = "no source_file in trace/TraceLens output; optimization will run prompt-only" - if not benchmark_file: - summary["benchmark_resolution"] = "no benchmark resolved; GEAK may be slower or fail" - - # GPU budgeting: collectives need >=2 GPUs; compute kernels run on 1. - if args.num_gpus_override > 0: - per_task_gpus = args.num_gpus_override - else: - per_task_gpus = int(selected.get("num_gpus_recommended") or 1) - if is_multigpu and per_task_gpus < 2: - # Collective but TraceLens reported <2; force-bump to args.tp (2 floor). - per_task_gpus = max(2, int(getattr(args, "tp", 0) or 2)) - summary.setdefault( - "per_task_gpus_inferred", - f"raised to {per_task_gpus} from collective name " - "pattern; TraceLens reported num_gpus_recommended<2", - ) - backends = [b.strip() for b in args.backends.split(",") if b.strip()] - backends_dropped: list[str] = [] - max_concurrent = max(1, args.total_gpus // max(1, per_task_gpus)) - total_jobs = len(backends) * args.replicas_per_backend - summary["gpu_plan"] = { - "per_task_gpus": per_task_gpus, - "total_gpus": args.total_gpus, - "max_concurrent_tasks": max_concurrent, - "total_jobs": total_jobs, - "backends_dropped": backends_dropped, - } - if not backends: - raise RuntimeError( - "All backends were dropped (likely all incompatible with this " - f"kernel's per_task_gpus={per_task_gpus}). Selected backends: " - f"{args.backends}. Dropped: {backends_dropped}" - ) - jobs = [] - ray_log = run_dir / "logs" / "ray.log" - ray_started_by_runner = _ensure_ray_via_helper(args.tp, ray_log) - summary["ray_started_by_runner"] = ray_started_by_runner - # ThreadPool only issues Ray submissions; Ray serialises GPU contention. - with ThreadPoolExecutor(max_workers=min(total_jobs, max_concurrent)) as pool: - for backend in backends: - for replica in range(args.replicas_per_backend): - jobs.append( - pool.submit( - run_one_attempt, - backend=backend, - replica=replica, - gpu_id=-1, - args=args, - run_dir=run_dir, - env=env, - kernel_id=selected["kernel_id"], - source_file=source_file, - benchmark_file=benchmark_file, - num_gpus=per_task_gpus, - ) - ) - parallel_results = [job.result() for job in as_completed(jobs)] - _stop_ray_via_helper(ray_started_by_runner, ray_log) - parallel_results.sort(key=lambda x: (x["backend"], x["replica"])) - summary["parallel_results"] = parallel_results - if not source_file: - summary["patch_retest_status"] = "not attempted: no patchable source resolved from real trace" - elif not benchmark_file: - summary["patch_retest_status"] = ( - "not attempted: source resolved but no benchmark was resolved for safe patch validation" - ) - else: - summary["patch_retest_status"] = ( - "not attempted automatically: backend outputs require review before applying to runtime source" - ) - summary["completed_at"] = utc_now() - write_summary(run_dir, summary) - print( - json.dumps( - { - "status": "succeeded", - "run_dir": str(run_dir), - "summary_json": str(run_dir / "parallel_e2e_summary.json"), - "summary_md": str(run_dir / "parallel_e2e_summary.md"), - "selected_kernel": selected, - "patch_retest_status": summary["patch_retest_status"], - }, - indent=2, - sort_keys=True, - ) - ) - return 0 - except Exception as exc: - try: - _stop_ray_via_helper(bool(summary.get("ray_started_by_runner")), run_dir / "logs" / "ray.log") - except Exception: - # Best-effort Ray teardown; ignore failures during cleanup. - pass - summary["status"] = "failed" - summary["error"] = f"{type(exc).__name__}: {exc}" - summary["completed_at"] = utc_now() - write_summary(run_dir, summary) - print( - json.dumps( - { - "status": "failed", - "run_dir": str(run_dir), - "summary_json": str(run_dir / "parallel_e2e_summary.json"), - "error": summary["error"], - }, - indent=2, - sort_keys=True, - ) - ) - return 1 - - -if __name__ == "__main__": - raise SystemExit(main()) diff --git a/src/hyperloom/agents/kernel/tools/tracelens_analysis.py b/src/hyperloom/agents/kernel/tools/tracelens_analysis.py index 503eea60fb..7a04fb04a8 100755 --- a/src/hyperloom/agents/kernel/tools/tracelens_analysis.py +++ b/src/hyperloom/agents/kernel/tools/tracelens_analysis.py @@ -131,8 +131,8 @@ log = logging.getLogger(__name__) -# Artifact name kept local so the standalone-script path does not need the -# shared contract module just to know where the file goes. +# Duplicated from kernel_source_contract.SOURCE_RESOLUTION_FILENAME: the +# standalone-script path cannot import hyperloom.common. Keep the two in sync. _SOURCE_RESOLUTION_NAME = "kernel_source_resolution.json" diff --git a/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py b/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py index 7052dbe8f2..4195d53fd8 100644 --- a/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py +++ b/src/hyperloom/agents/kernel/tools/tracelens_skill_runner.py @@ -437,26 +437,9 @@ def _should_use_codex_runner() -> bool: def _iter_message_text(message: Any) -> Iterable[str]: - """Yield text fragments from an SDK message. + from hyperloom.common.claude_oneshot import message_text # noqa: PLC0415 - Handles both content blocks exposing a ``.text`` attribute or ``"text"`` - dict key, plus a top-level ``.result`` string. - - Args: - message (Any): An SDK message object. - - Yields: - str: Each non-empty text fragment found on the message. - """ - for block in list(getattr(message, "content", None) or []): - text = getattr(block, "text", None) - if isinstance(text, str) and text: - yield text - elif isinstance(block, dict) and isinstance(block.get("text"), str): - yield block["text"] - result_text = getattr(message, "result", None) - if isinstance(result_text, str) and result_text: - yield result_text + yield from (t for t in message_text(message) if t) async def _run_tracelens_skill_codex( diff --git a/src/hyperloom/agents/quantization/driver/runner.py b/src/hyperloom/agents/quantization/driver/runner.py index 113b1d6c91..bc50fe8337 100644 --- a/src/hyperloom/agents/quantization/driver/runner.py +++ b/src/hyperloom/agents/quantization/driver/runner.py @@ -67,26 +67,10 @@ def _import_sdk() -> tuple[Any, Any]: def _iter_message_text(message: Any) -> Iterable[str]: - """Yield text fragments from a Claude Agent SDK message. + """Yield the non-empty text fragments of a Claude Agent SDK message.""" + from hyperloom.common.claude_oneshot import message_text # noqa: PLC0415 - Handles the varying SDK message shapes: ``.content`` blocks exposing - ``.text`` (object or dict) and a top-level ``.result`` string. - - Args: - message: An SDK message object. - - Yields: - Each non-empty text fragment found on the message. - """ - for block in list(getattr(message, "content", None) or []): - text = getattr(block, "text", None) - if isinstance(text, str) and text: - yield text - elif isinstance(block, dict) and isinstance(block.get("text"), str): - yield block["text"] - result_text = getattr(message, "result", None) - if isinstance(result_text, str) and result_text: - yield result_text + yield from (fragment for fragment in message_text(message) if fragment) def resolve_skill_path(package_root: Path | None = None) -> Path: diff --git a/src/hyperloom/agents/robustness/SKILL.md b/src/hyperloom/agents/robustness/SKILL.md index b018c35eee..0e51360974 100644 --- a/src/hyperloom/agents/robustness/SKILL.md +++ b/src/hyperloom/agents/robustness/SKILL.md @@ -19,8 +19,7 @@ on-disk findings. # from the repo root — the repo is a single distribution python3 -m venv .venv && .venv/bin/pip install -e ".[test]" -# Reactor mode. Auto-discovers session_dir and probes -# robustness-server before falling back to local probes. +# Reactor mode. Auto-discovers session_dir and runs the local probes. .venv/bin/robustness-agent ``` @@ -50,9 +49,7 @@ src/hyperloom/agents/robustness/ │ └── symptom.py # Symptom / SymptomSeverity dataclasses ├── sources/ │ ├── base.py # Source / SourceData / DegradeRouter -│ ├── server_client.py # robustness-server REST + Source adapter -│ ├── cluster_decoder.py # cluster pods/GPU/fault payload decoding -│ └── local_probe.py # local fallback (coordinator.db, ps, df, parsed rocm-smi, http probes, log error patterns) +│ └── local_probe.py # the collector (coordinator.db, ps, df, parsed rocm-smi, http probes, log error patterns) ├── factory.py # Config -> ReactorBundle (build_reactor_components) ├── config.py # discovery + tunables ├── state_store.py # per-detector state persisted across ticks @@ -85,9 +82,8 @@ python -m hyperloom.agents.robustness.runtime.cli tick \ "raw_prompt": "=== Shared session state ===\n...", "context": {"tick_index": 0, "now_unix": 1700000000.0}, "options": {"session_dir": "/tmp/sess-1", - "robustness_server_url": "http://...", "llm_rca_enabled": false, - "metrics_window_s": 300} + "disable_local_probe": false} } ``` @@ -119,7 +115,6 @@ host -> subprocess -> envelope -> upstream PolicyGate path. | Variable | Required | Default | Description | |----------|----------|---------|-------------| | `SESSION_DIR` | no | scan known paths | Path containing `storage/coordinator.db`; the FindingSink writes under `{session_dir}/agents/robustness/findings/{session_id}.jsonl`. | -| `ROBUSTNESS_SERVER_URL` | no | scan known DNS | M1 primary data source; empty disables the primary path and forces local-only mode. | | `OPENAI_BASE_URL` | no | — | LLM endpoint for RCA (used as `llm_base_url`). | | `OPENAI_API_KEY` | no | — | API key for the LLM proxy (used as `llm_api_key`). | | `ROBUSTNESS_LLM_MODEL` | no | — | RCA model name; takes precedence over `LLM_MODEL`. | @@ -127,38 +122,81 @@ host -> subprocess -> envelope -> upstream PolicyGate path. | `ROBUSTNESS_LLM_RCA_DISABLED` | no | unset | Set to `1` to forcibly disable the LlmRcaEngine even when credentials are present. | `Config.discover()` reads the variables above plus the deployment-shape ones -(`ROBUSTNESS_DISABLE_LOCAL_PROBE`, `ROBUSTNESS_ENABLE_CLUSTER_POD_METRICS`, -`ROBUSTNESS_NODES`) and nothing else. Every threshold — stall timeouts, disk and -shm percentages, GPU temperatures — is a field on `Config` with a default in -`config.py`, changed in code or by whoever constructs the `Config`, not from the -environment. - -## Symptom -> intent mapping (M1 / M1.5) - -| Symptom | Severity | Intents emitted | Source | -|---------|----------|-----------------|--------| -| `agent_stall` (≥ stall_timeout_s) | medium | `alert(medium)` | M1 | -| `agent_stall` (≥ severity_high_after_s) | high | `alert(high)` | M1 | -| `agent_quiet_work_progressing` (own dispatched work reported within `stall_timeout_s`) | low | `send_message(observation)` | M1.5 | -| `crash_count_rising` (≥ 2) | medium | `alert(medium)` | M1 | -| `crash_count_high` (≥ 5) | high | `alert(high)` | M1 | -| `crash_count_emergency` (≥ 10) | high | `alert(high)` | M1 | -| `repeated_policy_denied` (≥ 3) | medium | `alert(medium)` | M1 | -| `repeated_failure` (≥ 2 same family) | medium / high (≥ prune threshold) | `alert(medium)`; HIGH tier also emits `prune_branch(family)` | M1 | -| `pod_not_running` (Failed) | high | `alert(high)` | M1 | -| `pod_not_running` (other non-Running) | medium | `alert(medium)` | M1 | -| `pod_no_metrics` (≥ no_metrics_warn_s) | low | `send_message(observation)` | M1 | -| `local_server_unreachable` (any target down) | medium / high (all down) | `alert(medium)` / `alert(high)` | M1.5 | -| `local_server_unreachable`, no server process and no benchmark client of this session | — | suppressed (an idle stretch, not an outage) | M1.5 | -| `log_error_pattern` (CUDA OOM / NCCL / segfault) | high | `alert(high)` | M1.5 | -| `log_error_pattern` (RuntimeError / generic) | medium | `alert(medium)` | M1.5 | -| `gpu_thermal_high` (≥ warn_c) | medium | `alert(medium)` | M1.5 | -| `gpu_thermal_high` (≥ crit_c) | high | `alert(high)` | M1.5 | -| `stale_lease` | high | `alert(high)` + `kill_task(task_id)` | M1 | -| `gpu_memory_leaked` | high | `alert(high)` + `delegate(recover, force_gpu_cleanup=True)` | M1 | -| `deadline_warning` / `deadline_imminent` / `deadline_hard_cutoff` / `recover_unsuccessful` | high | `alert(high)` + `delegate(report)` | M1 | -| `same_payload_loop` / `kernel_opt_no_progress` / `geak_budget_starvation` / `amdahl_kernel_ceiling_low` | high | `alert(high)` + `prune_branch(family)` | M1 | -| (no symptoms) | — | `send_message(heartbeat)` | M1 | +(`ROBUSTNESS_DISABLE_LOCAL_PROBE`, `ROBUSTNESS_NODES`) and nothing else. Every +threshold — stall timeouts, disk and shm percentages, GPU temperatures — is a +field on `Config` with a default in `config.py`, changed in code or by whoever +constructs the `Config`, not from the environment. + +## Symptom -> intent mapping + +Complete inventory. The `Rule` column is the `SignalSpec.name` from +`_SIGNAL_REGISTRY` in `signals/classifier.py`, which is also the module the +rule lives in (`signals/.py`; `ray_pending` and `kernel_pipeline` share +`kernel_pipeline.py`, and the three preflight rules share `preflight.py`). + +Severity drives the ladder tier: low emits `send_message(observation)`, medium +emits `alert(medium)`, high emits `alert(high)` plus any remediation intent +listed below. + +| Symptom | Severity | Intents emitted | Rule | +|---------|----------|-----------------|------| +| `agent_stall` (≥ stall_timeout_s) | medium | `alert(medium)` | `stall` | +| `agent_stall` (≥ severity_high_after_s) | high | `alert(high)` | `stall` | +| `agent_quiet_work_progressing` (own dispatched work reported within `stall_timeout_s`) | low | `send_message(observation)` | `stall` | +| `crash_count_rising` (≥ 2) | medium | `alert(medium)` | `crash` | +| `crash_count_high` (≥ 5) | high | `alert(high)` | `crash` | +| `crash_count_emergency` (≥ 10) | high | `alert(high)` | `crash` | +| `repeated_policy_denied` (≥ 3) | medium | `alert(medium)` | `event` | +| `repeated_failure` (≥ 2 same family) | medium / high (≥ prune threshold) | `alert(...)`; HIGH tier also emits `prune_branch(family)` | `event` | +| `idempotency_replay` | medium | `alert(medium)` | `event` | +| `recover_unsuccessful` | high | `alert(high)` + `delegate(report)` | `event` | +| `local_server_unreachable` (any target down) | medium / high (all down) | `alert(medium)` / `alert(high)` | `local_health` | +| `local_server_unreachable`, no server process and no benchmark client of this session | — | suppressed (an idle stretch, not an outage) | `local_health` | +| `log_error_pattern` (CUDA OOM / NCCL / segfault) | high | `alert(high)` | `local_health` | +| `log_error_pattern` (RuntimeError / generic) | medium | `alert(medium)` | `local_health` | +| `gpu_thermal_high` (≥ warn_c / ≥ crit_c) | medium / high | `alert(medium)` / `alert(high)` | `local_health` | +| `disk_pressure` (≥ warn_pct / ≥ crit_pct) | medium / high | `alert(medium)` / `alert(high)` | `local_health` | +| `shm_pressure` (≥ warn_pct / ≥ crit_pct) | medium / high | `alert(medium)` / `alert(high)` | `local_health` | +| `fd_pressure` (≥ warn_pct / ≥ crit_pct) | medium / high | `alert(medium)` / `alert(high)` | `local_health` | +| `ray_head_dead` | high | `alert(high)` | `local_health` | +| `gpu_memory_leaked` | high | `alert(high)` + `delegate(recover, force_gpu_cleanup=True)` | `gpu_leak` | +| `deadline_warning` | medium / high | `alert(...)`; HIGH tier also emits `delegate(report)` | `budget` | +| `deadline_imminent` | high | `alert(high)` + `delegate(report)` | `budget` | +| `deadline_hard_cutoff` | high | `alert(high)` + `delegate(report)` | `budget` | +| `budget_burn_no_gain` | medium | `alert(medium)` | `budget` | +| `budget_strategy_drift` | medium | `alert(medium)` | `budget` | +| `phase_budget_nearly_exhausted` | medium | `alert(medium)` | `phase_budget` | +| `conversation_no_progress` | high | `alert(high)` | `conversation_progress` | +| `aiter_jit_regressed` | high | `alert(high)` | `aiter_jit` | +| `aiter_jit_build_stuck` | medium | `alert(medium)` | `aiter_jit` | +| `gain_plateau` | medium | `alert(medium)` | `progress` | +| `no_levers_found` | medium | `alert(medium)` | `progress` | +| `same_payload_loop` | high | `alert(high)` + `prune_branch(family)` | `repeated_payload` | +| `empty_patch_kept` | high | `alert(high)` | `decision_audit` | +| `decision_threshold_violated` | medium | `alert(medium)` | `decision_audit` | +| `kernel_dispatch_bypassed` | high | `alert(high)` | `decision_audit` | +| `kernel_negative_delta_kept` | high | `alert(high)` | `decision_audit` | +| `ci_metrics_baseline_zero` | high | `alert(high)` | `decision_audit` | +| `ci_metrics_schema_drift` | medium | `alert(medium)` | `decision_audit` | +| `model_gpu_infeasible` | high | `alert(high)` | `model_gpu_fit` | +| `amdahl_kernel_ceiling_low` | high | `alert(high)` + `prune_branch(kernel_opt)` | `amdahl_ceiling` | +| `cold_start_budget_exhausted` | high | `alert(high)` | `cold_start` | +| `critic_kb_outage` | high | `alert(high)` | `critic_health` | +| `critic_unavailable_streak` | high | `alert(high)` | `critic_health` | +| `critic_prune_stuck` | medium | `alert(medium)` | `critic_health` | +| `critic_runtime_stuck` | high | `alert(high)` | `critic_health` | +| `ray_pending_starvation` | high | `alert(high)` | `ray_pending` | +| `geak_budget_starvation` | high | `alert(high)` + `prune_branch(kernel_opt)` | `kernel_pipeline` | +| `kernel_opt_no_progress` | high | `alert(high)` + `prune_branch(kernel_opt)` | `kernel_pipeline` | +| `state_json_corrupt` | high | `alert(high)` | `state_integrity` | +| `coordinator_wal_bloat` (≥ warn / ≥ critical bytes) | medium / high | `alert(medium)` / `alert(high)` | `state_integrity` | +| `stale_lease` | high | `alert(high)` + `kill_task(task_id)` | `state_integrity` | +| `inbox_bloat` (≥ warn / ≥ critical bytes) | low / medium | `send_message(observation)` / `alert(medium)` | `state_integrity` | +| `coordinator_zombie` | high | `alert(high)` | `state_integrity` | +| `gateway_auth_outage` | high | `alert(high)` | `external_deps` | +| `wekafs_degraded` (unreachable, or ≥ warn / ≥ critical latency) | medium / high | `alert(medium)` / `alert(high)` | `external_deps` | +| `tracelens_cli_missing` | high (once per session) | `alert(high)` | `external_deps` | +| (no symptoms) | — | `send_message(heartbeat)` | — | Every other HIGH symptom is strategic: the recommendation rides the alert's `detail.suggestion` field and the ladder never auto-emits @@ -170,35 +208,40 @@ allowlist (`accuracy_gate` / `recover` / `report` / `server_lifecycle`). Cooldown: identical `(symptom_name, subject)` keys are silenced for `config.cooldown_ticks` ticks (default 5) to avoid inbox flooding. -## Data sources (M1 / M1.5) - -* **Primary:** `robustness-server` - * `/api/v1/sessions/{id}/pods` - * `/api/v1/sessions/{id}/events` - * `/api/v1/sessions/{id}/summary` - * `/api/v1/cluster/faults` (on by default) - * `/api/v1/cluster/workloads/{id}/hierarchy` - * `/api/v1/cluster/pods/{ns}/{name}/metrics` — gated by - `Config.enable_cluster_pod_metrics` (default `False`, env-settable) -* **Fallback:** local probes +## Data sources + +Two of the rule families need no source at all: everything driven by the +rendered Coordinator prompt (`budget`, `phase_budget`, +`conversation_progress`, `progress`, `crash`) and everything driven by the +inbox (`event`, `repeated_payload`, and part of `stall` / `critic_health`). +Those keep working when the probe is off, which is what multi-node relies on. + +The rest read `SourceData`, collected by: + +* **LocalProbe** — the only collector * `coordinator.db` (read-only) for Coordinator events * `shutil.disk_usage`, `ps`, parsed `rocm-smi --csv` / `nvidia-smi` * `Config.health_probe_targets[]` — local HTTP `/health` probes for - inference servers running on the same host (M1.5) + inference servers running on the same host * tail of a configured log file + error-pattern extraction (`CUDA out of memory`, `NCCL error`, `Segmentation fault`, etc.) - -LocalProbe stays small-scope by design: it only collects what the -agent itself can see. GPU time-series, workload inference health, and -node-level fault detection stay with primus-robust + robustness-server -ownership. - -`DegradeRouter` switches to the fallback after -`source_fail_threshold` (default 3) consecutive primary failures and -re-probes the primary every `source_recheck_interval_s` (default 30s). + * `state.json` / WAL / lease integrity, decision-audit and critic-workdir + scans, and the external-dependency probes (gateway `/models`, source + mounts, TraceLens CLI) +* **Quiet stub** — substituted when `Config.disable_local_probe` is set + (the multi-node default). Returns an empty snapshot without raising, so + probe-derived rules stay silent instead of false-firing on a single pod. + +LocalProbe stays small-scope by design: it only collects what the agent +itself can see, so on multi-node it sees one pod and is therefore disabled. + +`DegradeRouter` keeps the collector behind a silent fallback: after +`source_fail_threshold` (default 3) consecutive failures it serves an empty +snapshot instead, and re-probes every `source_recheck_interval_s` (default +30s). A tick therefore degrades to "no data" rather than failing outright. State transitions emit one WARN log; no spam in steady state. -## LLM RCA (M1.5) +## LLM RCA When `llm_base_url` and `llm_api_key` are both set (and `ROBUSTNESS_LLM_RCA_DISABLED` is not `1`), the factory wires @@ -230,11 +273,10 @@ Each tick that emits a non-heartbeat intent writes one Fields: `tick_index`, `timestamp_unix`, `symptom_name`, `severity`, `summary`, `intents` (envelope dicts), `evidence`, `rca_text`. -These records are the hand-off point for a future findings publisher -that POSTs them to the robustness-server for dashboards / alerting; +These records are the hand-off point for a future findings publisher; today they remain local-only. -## Session-end postmortem (L1 + L2) +## Session-end postmortem When the Coordinator sets `state.json::stop_reason` (run wind-down) the reactor fires :class:`hyperloom.agents.robustness.role.postmortem.PostmortemFinalizer` @@ -252,7 +294,7 @@ finalizer post-hoc via `hyperloom.agents.robustness.role.postmortem.finalize_session(session_dir, session_id=...)` (noop when the marker exists). -## Critic feedback loop (L4) +## Critic feedback loop The Critic agent's `prepare-review` phase reads the most recent N HIGH- severity findings from `findings/.jsonl` and injects them @@ -276,6 +318,6 @@ Knobs (env): `CRITIC_ROBUSTNESS_PRIORS_LIMIT` (default 5), feeding the same reactor. Not shipped: the only transport today is the subprocess one above, and `ReactorContext` is only ever built from a rendered Coordinator prompt. -* **Findings publisher** — POST the on-disk findings to - robustness-server for cross-session reporting and advisory pull-back. - Nothing ships today; findings stay local-only. +* **Findings publisher** — POST the on-disk findings to a collector for + cross-session reporting and advisory pull-back. Nothing ships today; + findings stay local-only. diff --git a/src/hyperloom/agents/robustness/config.py b/src/hyperloom/agents/robustness/config.py index a09db3b05d..f330878050 100644 --- a/src/hyperloom/agents/robustness/config.py +++ b/src/hyperloom/agents/robustness/config.py @@ -3,13 +3,11 @@ """Configuration for the Robustness Agent. -Env-first with auto-detection fallbacks: session_dir (``SESSION_DIR``), the -robustness-server endpoint (``ROBUSTNESS_SERVER_URL``) and the LLM endpoint / -credentials (``OPENAI_BASE_URL`` / ``OPENAI_API_KEY``, plus Anthropic and -DeepSeek variants) fall back to probing well-known paths and endpoints when -unset. ``ROBUSTNESS_DISABLE_LOCAL_PROBE``, -``ROBUSTNESS_ENABLE_CLUSTER_POD_METRICS`` and ``ROBUSTNESS_NODES`` are env-only -with fixed defaults. +Env-first with auto-detection fallbacks: session_dir (``SESSION_DIR``) and the +LLM endpoint / credentials (``OPENAI_BASE_URL`` / ``OPENAI_API_KEY``, plus +Anthropic and DeepSeek variants) fall back to probing well-known paths and +endpoints when unset. ``ROBUSTNESS_DISABLE_LOCAL_PROBE`` and +``ROBUSTNESS_NODES`` are env-only with fixed defaults. """ from __future__ import annotations @@ -21,8 +19,6 @@ from pathlib import Path from typing import Optional -import httpx - from hyperloom.common.env import env_bool, env_int from hyperloom.common.llm_config import ( CLAUDE_OAUTH_TOKEN_ENV, @@ -34,14 +30,6 @@ log = logging.getLogger(__name__) -# Primary data source: an optional explicit endpoint (ROBUSTNESS_SERVER_URL), -# then generic in-cluster / local-dev fallbacks. No internal cluster DNS is -# hardcoded; set ROBUSTNESS_SERVER_URL for a specific deployment. -ROBUSTNESS_SERVER_CANDIDATES: list[str] = [u for u in (os.environ.get("ROBUSTNESS_SERVER_URL", "").strip(),) if u] + [ - "http://robustness-server:8000", - "http://localhost:8000", -] - SESSION_DIR_CANDIDATES: list[Path] = [ Path("/workspace/session"), Path(tempfile.gettempdir()) / "robustness-session", @@ -61,8 +49,6 @@ class Config: Attributes: session_dir (Path): Directory containing the session's storage (including ``coordinator.db``). - robustness_server_url (str): Primary M1 data source endpoint; - empty means skip the server and use only the local probe. llm_model (str): Model name used for LLM-driven root-cause analysis. llm_base_url (str): LLM API base URL discovered from the sandbox. Empty @@ -72,20 +58,15 @@ class Config: without ever handing it to this process. llm_rca_enabled (Optional[bool]): Tri-state RCA activation flag; ``None`` auto-enables when credentials are present. - metrics_window_s (int): Rolling window, in seconds, over which - metrics-based signals are evaluated. Note: Many additional threshold, interval, and per-signal fields exist on this dataclass; see the inline comments grouped by signal - family (A–L) for their meaning. + family for their meaning. """ session_dir: Path = field(default_factory=lambda: Path(tempfile.gettempdir()) / "robustness-session") - # Primary data source; empty means "skip server, only use local probe". - robustness_server_url: str = "" - @property def coordinator_db_path(self) -> Path: """Filesystem path to the session's Coordinator SQLite database. @@ -124,8 +105,6 @@ def coordinator_db_path(self) -> Path: # -- reactor knobs -- cooldown_ticks: int = 5 - metrics_window_s: int = 300 - server_request_timeout_s: float = 5.0 source_fail_threshold: int = 3 source_recheck_interval_s: float = 30.0 standalone_tick_interval_s: float = 10.0 @@ -136,13 +115,8 @@ def coordinator_db_path(self) -> Path: # -- multi-node knobs -- # Required in multi-node runs: per-pod ps/HTTP/rocm-smi probes false-fire - # local_server_unreachable / ray_head_dead on Ray workers without a server. + # local_server_unreachable / ray_head_dead on Ray workers. disable_local_probe: bool = False - # RobustnessServerSource fan-out so local_health sees per-pod GPU snapshots. - enable_cluster_pod_metrics: bool = False - pod_metrics_categories: tuple[str, ...] = ("gpu",) - # Resolves the full multi-node RayJob pod set via the hierarchy endpoint. - workload_uid: str = "" # Informational only (mirrors --nodes); policy driven by the flags above. nodes: int = 1 @@ -302,53 +276,39 @@ def coordinator_db_path(self) -> Path: ) @classmethod - async def discover(cls) -> "Config": + def discover(cls) -> "Config": """Auto-detect all configuration from the runtime environment. - Discovers the session directory, probes the robustness-server - endpoint, and reads LLM credentials from the sandbox environment. + Scans for the session directory and reads the LLM credentials and + the env-only knobs from the sandbox environment. Returns: Config: A new instance populated with the discovered values. """ session_dir = _discover_session_dir() - server_url = await _probe_robustness_server() llm_base_url, llm_api_key, llm_provider = _discover_llm_credentials() - workload_uid = _discover_workload_uid() disable_local_probe = env_bool("ROBUSTNESS_DISABLE_LOCAL_PROBE", False) - enable_cluster_pod_metrics = env_bool( - "ROBUSTNESS_ENABLE_CLUSTER_POD_METRICS", - False, - ) nodes = env_int("ROBUSTNESS_NODES", 1) config = cls( session_dir=session_dir, - robustness_server_url=server_url, llm_model=_discover_llm_model(llm_provider), llm_base_url=llm_base_url, llm_api_key=llm_api_key, llm_provider=llm_provider, - workload_uid=workload_uid, disable_local_probe=disable_local_probe, - enable_cluster_pod_metrics=enable_cluster_pod_metrics, nodes=nodes, ) log.info( - "Config discovered: session_dir=%s server=%s llm=%s " - "nodes=%d workload_uid=%s disable_local_probe=%s " - "enable_cluster_pod_metrics=%s", + "Config discovered: session_dir=%s llm=%s nodes=%d disable_local_probe=%s", config.session_dir, - config.robustness_server_url or "(local-only)", # A subscription-token host resolves no base_url at all, so the URL # alone would report "(not available)" for an RCA engine that is # about to start issuing calls. "(configured)" if (config.llm_base_url or config.llm_provider == "anthropic") else "(not available)", config.nodes, - config.workload_uid or "(unset)", config.disable_local_probe, - config.enable_cluster_pod_metrics, ) return config @@ -387,33 +347,6 @@ def _discover_session_dir() -> Path: return fallback -async def _probe_robustness_server() -> str: - """Probe known robustness-server endpoints + ROBUSTNESS_SERVER_URL env. - - Returns: - str: The first candidate URL whose ``/healthz`` endpoint returns - 200, or an empty string if none are reachable. - """ - candidates: list[str] = [] - env_url = os.environ.get("ROBUSTNESS_SERVER_URL", "").strip() - if env_url: - candidates.append(env_url.rstrip("/")) - candidates.extend(ROBUSTNESS_SERVER_CANDIDATES) - - for url in candidates: - try: - async with httpx.AsyncClient(timeout=httpx.Timeout(3.0)) as client: - resp = await client.get(f"{url}/healthz") - if resp.status_code == 200: - log.info("Robustness-server reachable at %s", url) - return url - except Exception: - continue - - log.info("Robustness-server not reachable, will use local-only fallback") - return "" - - def _provider_env() -> dict[str, str]: """Return the process environment with retired provider variables normalized. @@ -471,26 +404,3 @@ def _discover_llm_model(provider: str) -> str: return env.get("ANTHROPIC_MODEL", "").strip() or env.get("CLAUDE_MODEL", "").strip() or "claude-opus-5" -_WORKLOAD_UID_ENV_KEYS: tuple[str, ...] = ( - "ROBUSTNESS_WORKLOAD_UID", - "CLAW_WORKLOAD_UID", - "WORKLOAD_UID", - "KUBE_WORKLOAD_UID", - "RAY_JOB_ID", -) - - -def _discover_workload_uid() -> str: - """Resolve the multi-node workload uid (first non-empty env key above). - - Lets a RayJob sandbox opt into hierarchy-based pod discovery; single-node - runs leave every key unset and fall back to ``list_session_pods``. - - Returns: - The first non-empty workload-uid env value, or ``""`` if none set. - """ - for key in _WORKLOAD_UID_ENV_KEYS: - value = (os.environ.get(key) or "").strip() - if value: - return value - return "" diff --git a/src/hyperloom/agents/robustness/decision/action_ladder.py b/src/hyperloom/agents/robustness/decision/action_ladder.py index 3e025f8535..f5cc61711d 100644 --- a/src/hyperloom/agents/robustness/decision/action_ladder.py +++ b/src/hyperloom/agents/robustness/decision/action_ladder.py @@ -70,8 +70,7 @@ def _prune_family_for(sym: Symptom) -> str: class Finding: """Persistent record describing one ladder firing. - Stored on disk by the FindingSink for later inspection / reporting - to robustness-server. + Stored on disk by the FindingSink for later inspection. """ tick_index: int diff --git a/src/hyperloom/agents/robustness/decision/rca_engine.py b/src/hyperloom/agents/robustness/decision/rca_engine.py index ed2fa0ca03..e9f5e148e2 100644 --- a/src/hyperloom/agents/robustness/decision/rca_engine.py +++ b/src/hyperloom/agents/robustness/decision/rca_engine.py @@ -53,6 +53,9 @@ async def summarize(self, symptom: Symptom) -> str: str: Root-cause summary text, or an empty string when none. """ + async def aclose(self) -> None: + """Release any provider client the engine owns.""" + @dataclass class NoopRcaEngine: @@ -75,6 +78,9 @@ def drain_usage(self) -> dict[str, Any] | None: """No LLM is ever contacted, so there is never any usage to drain.""" return None + async def aclose(self) -> None: + """No client is ever created, so there is nothing to close.""" + @dataclass class RcaThrottleConfig: diff --git a/src/hyperloom/agents/robustness/factory.py b/src/hyperloom/agents/robustness/factory.py index d8d726fd4d..fc06370b18 100644 --- a/src/hyperloom/agents/robustness/factory.py +++ b/src/hyperloom/agents/robustness/factory.py @@ -4,11 +4,10 @@ """High-level builders that turn a :class:`Config` into a running reactor. :func:`build_reactor_components` returns a :class:`ReactorBundle` (reactor + -ReactorComponents + FindingSink + RobustnessServerClient) so callers can -manage lifecycles via ``await bundle.aclose()``. All hosts drive -``bundle.reactor.tick(ctx)`` per tick; the blessed transport is the -subprocess CLI (mirrors critic-agent), no in-process Backend adapter. The -factory never blocks on remote services — ``Config.discover`` already probed. +ReactorComponents + FindingSink) so callers can manage lifecycles via +``await bundle.aclose()``. All hosts drive ``bundle.reactor.tick(ctx)`` per +tick; the blessed transport is the subprocess CLI (mirrors critic-agent), no +in-process Backend adapter. """ from __future__ import annotations @@ -47,7 +46,6 @@ from .signals.decision_audit import DecisionAuditConfig from .signals.event import EventConfig from .signals.gpu_leak import GpuLeakConfig -from .signals.health import HealthConfig from .signals.critic_health import CriticHealthConfig from .signals.external_deps import ExternalDepsConfig from .signals.kernel_pipeline import KernelPipelineConfig @@ -63,12 +61,8 @@ from .signals.repeated_payload import RepeatedPayloadConfig from .signals.state_integrity import StateIntegrityConfig from .signals.stall import StallConfig -from .sources.base import DegradeRouter, Source, SourceData, SourceUnavailable +from .sources.base import DegradeRouter, Source, SourceData from .sources.local_probe import LocalProbeConfig, LocalProbeSource -from .sources.server_client import ( - RobustnessServerClient, - RobustnessServerSource, -) log = logging.getLogger(__name__) @@ -80,17 +74,15 @@ class ReactorBundle: reactor: Reactor components: ReactorComponents - server_client: RobustnessServerClient | None sink: FindingSink async def aclose(self) -> None: - """Release lifecycle resources held by the bundle. + """Close the RCA engine's provider client. - Closes the underlying robustness-server HTTP client when one was - created; a no-op in local-only mode. + The engine only acts on a client it created, so an injected one is + left to its owner. """ - if self.server_client is not None: - await self.server_client.aclose() + await self.components.rca.aclose() def _build_local_probe_config(config: Config) -> LocalProbeConfig: @@ -163,12 +155,12 @@ def build_reactor_components( ) -> ReactorBundle: """Construct everything the reactor needs. - Wires the primary/fallback sources, degrade router, detectors, - state store, finding sink, and RCA engine into a single bundle. + Wires the source, degrade router, detectors, state store, finding sink, + and RCA engine into a single bundle. Args: config (Config): Discovered configuration — typically the result - of ``await Config.discover()``. + of ``Config.discover()``. rca (RcaEngine | None): Optional RCA engine override. Defaults to an auto-selected engine (Noop unless LLM RCA is enabled). session_id (str | None): Override for the FindingSink filename. @@ -176,39 +168,24 @@ def build_reactor_components( sandbox writes to a stable file. Returns: - ReactorBundle: The assembled reactor plus the lifecycle handles - (components, server client, and sink) the caller must manage. + ReactorBundle: The assembled reactor plus the components and sink + the caller must manage. """ - # Primary source: robustness-server (omitted in local-only mode). - server_client: RobustnessServerClient | None = None + # Multi-node guard: the probe only sees its own pod, so it is disabled there. primary: Source - if config.robustness_server_url: - server_client = RobustnessServerClient( - config.robustness_server_url, - timeout_s=config.server_request_timeout_s, - ) - primary = RobustnessServerSource( - server_client, - metrics_window_s=config.metrics_window_s, - enable_cluster_pod_metrics=config.enable_cluster_pod_metrics, - pod_metrics_categories=tuple(config.pod_metrics_categories), - workload_uid=config.workload_uid, - ) - else: - primary = _NoServerSource( - "robustness-server", - "config.robustness_server_url is empty", - ) - - # Multi-node guard: ``disable_local_probe`` swaps LocalProbe for a quiet stub. - fallback: Source if config.disable_local_probe: - fallback = _QuietFallback( + primary = _QuietSource( name="local-probe", - reason="config.disable_local_probe is True (multi-node policy)", + reason="local-probe disabled: config.disable_local_probe is True", ) else: - fallback = LocalProbeSource(_build_local_probe_config(config)) + primary = LocalProbeSource(_build_local_probe_config(config)) + + # LocalProbe raises when every sub-probe is empty; degrade to silence. + fallback: Source = _QuietSource( + name="quiet-fallback", + reason="local probe produced no data", + ) router = DegradeRouter( primary, @@ -222,9 +199,8 @@ def build_reactor_components( DetectorStateStore(session_dir=config.session_dir) if config.state_store_enabled else None ) - # Config->SignalConfig map keyed by ``SignalSpec.config_attr``; slots the - # registry defaults (e.g. ``cluster_fault``) are omitted here and filled by - # the classifier from the registry ``config_factory``. + # Config->SignalConfig map keyed by ``SignalSpec.config_attr``; omitted + # slots are filled by the classifier from the registry ``config_factory``. signal_configs: dict[str, Any] = { "stall": StallConfig( stall_timeout_s=config.agent_stall_timeout_s, @@ -234,7 +210,6 @@ def build_reactor_components( "event": EventConfig( idempotency_replay_threshold=config.idempotency_replay_threshold, ), - "health": HealthConfig(), "local_health": LocalHealthConfig( gpu_temp_warn_c=config.gpu_temp_warn_c, disk_used_warn_pct=config.disk_used_warn_pct, @@ -367,7 +342,6 @@ def build_reactor_components( return ReactorBundle( reactor=Reactor(components), components=components, - server_client=server_client, sink=sink, ) @@ -466,57 +440,29 @@ def _parse_severity(value: str) -> SymptomSeverity: @dataclass -class _NoServerSource: - """Permanent stub used when no robustness-server URL is configured. - - DegradeRouter degrades it after ``fail_threshold`` ticks, after which - the LocalProbe takes over without further probes. - """ - - name: str - reason: str - - async def fetch(self, ctx: Any) -> SourceData: - """Always fail, signalling the source is unavailable. - - Args: - ctx (Any): The fetch context (ignored). - - Returns: - SourceData: Never returns normally. - - Raises: - SourceUnavailable: Always, with the configured reason. - """ - raise SourceUnavailable(self.reason) - - -@dataclass -class _QuietFallback: - """Silent fallback used when ``disable_local_probe`` is on. +class _QuietSource: + """Source that collects nothing, carrying its reason for saying so. - Returns empty :class:`SourceData` (not raising) so DegradeRouter never - enters FAILED; the signal layer treats empty fields as "no data" and - stays quiet — the multi-node policy of no LocalProbe symptoms. The - transition is still visible via ``primary_state`` / ``degraded_reason``. + Returns empty :class:`SourceData` rather than raising, so probe-derived + rules see "no data" and stay quiet instead of failing the tick. The reason + stays visible via ``degraded_reason``. """ name: str reason: str async def fetch(self, ctx: Any) -> SourceData: # noqa: ARG002 - protocol - """Return empty source data describing the disabled local probe. + """Return empty source data annotated with this source's reason. Args: ctx: Fetch context supplied by the source protocol; unused because - this fallback never collects data. + this source never collects data. Returns: - A :class:`SourceData` with no signals, annotated with a - ``degraded_reason`` explaining that the local probe is disabled. + A :class:`SourceData` with no signals and a ``degraded_reason``. """ return SourceData( - degraded_reason=f"local-probe disabled: {self.reason}", + degraded_reason=self.reason, # Nothing looked: an empty process list here is ignorance, not # evidence that no server is running. local_processes_known=False, diff --git a/src/hyperloom/agents/robustness/main.py b/src/hyperloom/agents/robustness/main.py index 6d65b222df..b435ee5752 100644 --- a/src/hyperloom/agents/robustness/main.py +++ b/src/hyperloom/agents/robustness/main.py @@ -70,9 +70,8 @@ def _shutdown(sig: signal.Signals) -> None: pass log.info( - "Reactor mode running tick=%.1fs server=%s session_dir=%s", + "Reactor mode running tick=%.1fs session_dir=%s", config.standalone_tick_interval_s, - config.robustness_server_url or "(local-only)", config.session_dir, ) @@ -80,7 +79,7 @@ def _shutdown(sig: signal.Signals) -> None: while not stop.is_set(): ctx = ReactorContext( tick_index=0, - shared_state=SharedStateSnapshot(session_id=config.session_dir.name), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=time.time(), ) @@ -122,7 +121,7 @@ async def _async_main(argv: list[str] | None = None) -> None: :func:`_parse_args`. Defaults to ``None``. """ _parse_args(argv) - config = await Config.discover() + config = Config.discover() await _run_reactor_mode(config) diff --git a/src/hyperloom/agents/robustness/prompts/rca.md b/src/hyperloom/agents/robustness/prompts/rca.md index 067277c54d..309c07f6a3 100644 --- a/src/hyperloom/agents/robustness/prompts/rca.md +++ b/src/hyperloom/agents/robustness/prompts/rca.md @@ -26,7 +26,7 @@ Symptoms come from these signal families; knowing the family helps you interpret | H — Time / budget | `budget_strategy_drift`, `budget_burn_no_gain`, `deadline_imminent`, `deadline_warning`, `deadline_hard_cutoff` | | I — State integrity | `state_json_corrupt`, `coordinator_wal_bloat`, `stale_lease`, `inbox_bloat`, `coordinator_zombie` | | J — External deps | `gateway_auth_outage`, `wekafs_degraded`, `tracelens_cli_missing` | -| baseline / stall | `agent_stall`, `agent_quiet_work_progressing`, `crash_count_rising`, `crash_count_high`, `crash_count_emergency`, `repeated_policy_denied`, `repeated_failure`, `recover_unsuccessful`, `cluster_fault` | +| baseline / stall | `agent_stall`, `agent_quiet_work_progressing`, `crash_count_rising`, `crash_count_high`, `crash_count_emergency`, `repeated_policy_denied`, `repeated_failure`, `recover_unsuccessful` | ## Output contract diff --git a/src/hyperloom/agents/robustness/role/envelope.py b/src/hyperloom/agents/robustness/role/envelope.py index 4931f5b01d..577e46cc08 100644 --- a/src/hyperloom/agents/robustness/role/envelope.py +++ b/src/hyperloom/agents/robustness/role/envelope.py @@ -99,7 +99,6 @@ class IntentType(str, Enum): "stop_reason", "stop_ts", "last_tick_exception", - "cumulative_gain", "cumulative_gain_validated", "cumulative_gain_validated_ts", "cumulative_gain_validated_stack_len", @@ -120,7 +119,6 @@ class IntentType(str, Enum): "schema_version", # Recipe KB integration. "recipe_kb_session_id", - "recipe_kb_session_summary", "warm_start_recipe", "warm_start_pitfalls", "warm_start_lessons", @@ -202,7 +200,6 @@ class IntentType(str, Enum): "last_trace_analyze", "last_kernel_opt", "last_kernel_opt_dispatch_skip", - "kernel_opt_attempts", "kernel_opt_task_attempts", "pending_kernel_integrations", "last_collective", diff --git a/src/hyperloom/agents/robustness/role/postmortem.py b/src/hyperloom/agents/robustness/role/postmortem.py index b23a9cbf7f..37099e6ef3 100644 --- a/src/hyperloom/agents/robustness/role/postmortem.py +++ b/src/hyperloom/agents/robustness/role/postmortem.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Session-end postmortem + decision trace finalizer (L1 + L2). +"""Session-end postmortem + decision trace finalizer. Fired once per session on the first non-empty ``stop_reason``; idempotent via a ``.robustness_finalized`` marker under ``/reports/``. @@ -34,7 +34,7 @@ _POSTMORTEM_FILENAME: str = "robustness_postmortem.md" _DECISION_TRACE_FILENAME: str = "decision_trace.json" -# Action families scanned under ``runs/`` for L2. +# Action families scanned under ``runs/`` for the decision trace. _DECISION_TRACE_ACTION_DIRS: tuple[str, ...] = ( "baseline", "profile", @@ -69,7 +69,7 @@ class PostmortemFinalizerConfig: class PostmortemFinalizer: - """Once-per-session aggregator for L1 (flashpoint) + L2 (decision trace). + """Once-per-session aggregator for the flashpoint report + decision trace. Lifecycle: callers (the reactor) invoke :meth:`finalize` once when ``stop_reason`` is first observed non-empty. Re-running has no @@ -125,7 +125,7 @@ def is_finalized(self) -> bool: return self.marker_path.is_file() def finalize(self, *, stop_reason: str) -> bool: - """Run the L1+L2 pipeline. Returns True if we wrote new files. + """Run the finalizer pipeline. Returns True if we wrote new files. Best-effort: any IO error is logged and swallowed. The reactor must never crash because the postmortem failed to write. diff --git a/src/hyperloom/agents/robustness/role/prompt_inputs.py b/src/hyperloom/agents/robustness/role/prompt_inputs.py index 7b4acf2f71..d4f1ba44c7 100644 --- a/src/hyperloom/agents/robustness/role/prompt_inputs.py +++ b/src/hyperloom/agents/robustness/role/prompt_inputs.py @@ -11,7 +11,7 @@ ... === Shared session state === - session_id=... + tick=... ... === Time budget === @@ -48,10 +48,18 @@ # Anchored to the two-space row prefix; ``\S+`` topic guards against payloads -# whose dict repr contains a literal ``topic=``. +# whose dict repr contains a literal ``topic=``. ``msg_id`` is absent on +# messages carrying none, and the fields after ``topic`` are per-topic +# (``_format_inbox_event``), so the remainder is captured as a tail. _INBOX_LINE_RE = re.compile( - r"^\s+seq=(?P\d+)\s+msg_id=(?P\S+)\s+from=(?P\S+)\s+" - r"topic=(?P\S+)\s+payload=(?P.+)$" + r"^\s+seq=(?P\d+)\s+(?:msg_id=(?P\S+)\s+)?from=(?P\S+)\s+" + r"topic=(?P\S+)\s*(?P.*)$" +) + +# One ``key=`` pair of a tail. Quoted values are matched whole so +# a ``k=v`` inside a rendered ``error=``/``notes=`` string cannot split it. +_INBOX_FIELD_RE = re.compile( + r"(?P\w+)=(?P'(?:[^'\\]|\\.)*'|\"(?:[^\"\\]|\\.)*\"|\S+)" ) _SHARED_HEADER = "=== Shared session state ===" @@ -78,9 +86,7 @@ # SharedState lines we care about. _SCALAR_KEYS = { - "session_id", "baseline_tput", - "cumulative_gain", "cumulative_gain_validated", "crash_count", "current_action", @@ -175,17 +181,15 @@ class InboxItem: class SharedStateSnapshot: """Subset of the Coordinator SharedState the robustness reactor reads. - Only the fields the M1 reactor consumes are parsed; every other + Only the fields the reactor consumes are parsed; every other rendered line is ignored. All fields default to a neutral value so a parse miss degrades to "no signal" rather than raising. Attributes: - session_id (str): Current session id, or ``""`` when unset. model_name (str): Target model name, or ``""`` when unset. model_class (str): Target model class, or ``""`` when unset. baseline_tput (float): Baseline throughput reported by the Coordinator. - cumulative_gain (float): Cumulative (unvalidated) gain percentage. cumulative_gain_validated (float): Cumulative validated gain percentage. crash_count (int): Number of crashes recorded this session. @@ -204,17 +208,15 @@ class SharedStateSnapshot: means no budget configured. closing_phase (bool): True when the Coordinator signals the closing phase. - kernel_opt_attempts_count (int): Count of unique kernel ids with at + kernel_opt_attempts_count (int): Count of unique kernel task identities with at least one recorded kernel_opt attempt. has_keep_pending_integrate (bool): True when a multi-KEEP integrate queue still has work pending. """ - session_id: str = "" model_name: str = "" model_class: str = "" baseline_tput: float = 0.0 - cumulative_gain: float = 0.0 cumulative_gain_validated: float = 0.0 crash_count: int = 0 current_action: str = "" @@ -235,9 +237,8 @@ class SharedStateSnapshot: class ReactorContext: """Per-tick input for :class:`Reactor`. - Built by :func:`from_coordinator_prompt` (SINGLE_PROC) or - :func:`from_inbox_jsonl` (MULTI_CLI; M3). The reactor does not need - to know which transport produced the context. + Built by :func:`from_coordinator_prompt` from the rendered Coordinator + prompt, which is the only transport. Attributes: tick_index (int): In-process tick counter. @@ -470,9 +471,7 @@ def _coerce_cumulative_gain_validated(head: str) -> float: #: attr name differs from its rendered key. Explore-family keys are handled #: separately because they set a shared flag idempotently rather than a 1:1 attr. _SCALAR_FIELD_TABLE: dict[str, tuple[str, Callable[[str], Any]]] = { - "session_id": ("session_id", lambda head: "" if head == "(unset)" else head), "baseline_tput": ("baseline_tput", lambda head: to_float(head, default=0.0)), - "cumulative_gain": ("cumulative_gain", lambda head: to_float(head.rstrip("%"), default=0.0)), "cumulative_gain_validated": ("cumulative_gain_validated", _coerce_cumulative_gain_validated), "crash_count": ("crash_count", lambda head: to_int(head, default=0)), "current_action": ("current_action", lambda head: "" if head == "(idle)" else head), @@ -649,14 +648,12 @@ def _parse_inbox(body: str) -> tuple[list[InboxItem], list[str]]: except ValueError: warnings.append(f"non-integer seq in {raw!r}") continue - payload_text = match.group("payload") - payload, payload_warn = _decode_payload(payload_text) - if payload_warn: - warnings.append(payload_warn) + payload, tail_warnings = _decode_tail(match.group("tail")) + warnings.extend(tail_warnings) items.append( InboxItem( seq=seq, - msg_id=match.group("msg_id"), + msg_id=match.group("msg_id") or "", from_agent=match.group("from_agent"), topic=match.group("topic"), payload=payload, @@ -665,6 +662,38 @@ def _parse_inbox(body: str) -> tuple[list[InboxItem], list[str]]: return items, warnings +def _decode_tail(tail: str) -> tuple[dict[str, Any], list[str]]: + """Decode the per-topic field tail of an inbox line into a payload dict. + + ``payload=`` carries the whole dict and wins; the summary fields rendered + before it are folded in underneath, so a topic that emits no ``payload=`` + (``delegated_result``) still yields ``kind`` / ``state`` / ``error``. + + Args: + tail (str): Everything after ``topic=`` on the line. + + Returns: + tuple[dict[str, Any], list[str]]: The payload dict and any decode + warnings. + """ + head, sep, payload_text = tail.strip().partition("payload=") + fields: dict[str, Any] = {} + warnings: list[str] = [] + for match in _INBOX_FIELD_RE.finditer(head): + key, raw = match.group("key"), match.group("value") + try: + fields[key] = ast.literal_eval(raw) + except (SyntaxError, ValueError): + fields[key] = raw + warnings.append(f"field {key} not a python literal: {raw!r}") + if not sep: + return fields, warnings + payload, payload_warn = _decode_payload(payload_text) + if payload_warn: + warnings.append(payload_warn) + return {**fields, **payload}, warnings + + def _decode_payload(text: str) -> tuple[dict[str, Any], str | None]: """Decode a rendered payload literal into a dict. diff --git a/src/hyperloom/agents/robustness/role/reactor.py b/src/hyperloom/agents/robustness/role/reactor.py index 41d9cb2720..50bf8037f1 100644 --- a/src/hyperloom/agents/robustness/role/reactor.py +++ b/src/hyperloom/agents/robustness/role/reactor.py @@ -3,7 +3,7 @@ """Reactor: the heart of the robustness role. -A single :meth:`Reactor.tick` runs the M1 pipeline: :class:`DegradeRouter` -> +A single :meth:`Reactor.tick` runs the pipeline: :class:`DegradeRouter` -> :class:`Classifier` -> :class:`ActionLadder` -> :class:`PolicyAware` filter -> :class:`FindingSink` persist -> return validated intents. The Reactor holds tick state but no business logic (that lives in classifier + ladder) so transports diff --git a/src/hyperloom/agents/robustness/runtime/cli.py b/src/hyperloom/agents/robustness/runtime/cli.py index ac949add99..70f5ddc063 100644 --- a/src/hyperloom/agents/robustness/runtime/cli.py +++ b/src/hyperloom/agents/robustness/runtime/cli.py @@ -18,9 +18,8 @@ "raw_prompt": "=== Shared session state ===\\n...", "context": {"tick_index": 0, "now_unix": 1700000000.0}, "options": {"session_dir": "/tmp/sess-1", - "robustness_server_url": "http://...", "llm_rca_enabled": false, - "metrics_window_s": 300} + "disable_local_probe": false} } ``raw_prompt`` is parsed by :func:`from_coordinator_prompt`; ``context`` @@ -62,11 +61,7 @@ from ..factory import build_reactor_components from ..role.envelope import build_envelope_dict from ..role.postmortem import finalize_session -from ..role.prompt_inputs import ( - ReactorContext, - SharedStateSnapshot, - from_coordinator_prompt, -) +from ..role.prompt_inputs import from_coordinator_prompt log = logging.getLogger("robustness_agent.runtime.cli") @@ -137,31 +132,13 @@ async def _run_tick(request: dict[str, Any]) -> dict[str, Any]: context = dict(request.get("context") or {}) options = dict(request.get("options") or {}) - config = await Config.discover() + config = Config.discover() if "session_dir" in options: config.session_dir = Path(str(options["session_dir"])) - if "robustness_server_url" in options: - config.robustness_server_url = str(options["robustness_server_url"] or "") if "llm_rca_enabled" in options: config.llm_rca_enabled = bool(options["llm_rca_enabled"]) - if "metrics_window_s" in options: - config.metrics_window_s = int(options["metrics_window_s"]) if "disable_local_probe" in options: config.disable_local_probe = bool(options["disable_local_probe"]) - if "enable_cluster_pod_metrics" in options: - config.enable_cluster_pod_metrics = bool(options["enable_cluster_pod_metrics"]) - if "pod_metrics_categories" in options: - raw_cats = options["pod_metrics_categories"] - if isinstance(raw_cats, str): - cats = tuple(part.strip() for part in raw_cats.split(",") if part.strip()) - elif isinstance(raw_cats, (list, tuple)): - cats = tuple(str(c).strip() for c in raw_cats if str(c).strip()) - else: - cats = () - if cats: - config.pod_metrics_categories = cats - if "workload_uid" in options: - config.workload_uid = str(options["workload_uid"] or "") if "nodes" in options: try: config.nodes = max(1, int(options["nodes"])) @@ -177,7 +154,7 @@ async def _run_tick(request: dict[str, Any]) -> dict[str, Any]: # Disable the ``external_deps`` probe (TraceLens CLI / WekaFS mount) on inert CI hosts. if "external_deps_enabled" in options: config.external_deps_enabled = bool(options["external_deps_enabled"]) - # B3 ``no_levers_found`` floor knobs override the default window. + # ``no_levers_found`` floor knobs override the default window. if "progress_no_levers_min_minutes" in options: config.progress_no_levers_min_minutes = float(options["progress_no_levers_min_minutes"]) if "progress_no_levers_min_ticks" in options: @@ -199,25 +176,6 @@ async def _run_tick(request: dict[str, Any]) -> dict[str, Any]: tick_index=tick_index, now_unix=now_unix, ) - if not reactor_ctx.shared_state.session_id: - reactor_ctx = ReactorContext( - tick_index=reactor_ctx.tick_index, - shared_state=SharedStateSnapshot( - session_id=session_id, - model_name=reactor_ctx.shared_state.model_name, - model_class=reactor_ctx.shared_state.model_class, - baseline_tput=reactor_ctx.shared_state.baseline_tput, - cumulative_gain=reactor_ctx.shared_state.cumulative_gain, - crash_count=reactor_ctx.shared_state.crash_count, - current_action=reactor_ctx.shared_state.current_action, - ), - inbox=list(reactor_ctx.inbox), - now_unix=reactor_ctx.now_unix, - parse_warnings=list(reactor_ctx.parse_warnings), - phase=reactor_ctx.phase, - phase_budget=list(reactor_ctx.phase_budget), - conversation_progress=reactor_ctx.conversation_progress, - ) bundle = build_reactor_components(config, session_id=session_id) try: @@ -261,7 +219,7 @@ def _cmd_tick(args: argparse.Namespace) -> None: def _cmd_finalize(args: argparse.Namespace) -> None: - """Run the L1+L2 postmortem finalizer as a one-shot operator tool. + """Run the postmortem finalizer as a one-shot operator tool. Use when the reactor never observed ``stop_reason`` going non-empty (e.g. Coordinator killed by SIGKILL before the wind-down @@ -332,7 +290,7 @@ def _build_parser() -> argparse.ArgumentParser: finalize = sub.add_parser( "finalize", help=( - "Run the L1+L2 postmortem finalizer post-hoc " + "Run the postmortem finalizer post-hoc " "(for sessions whose Coordinator died before stop_reason " "was written)." ), diff --git a/src/hyperloom/agents/robustness/signals/__init__.py b/src/hyperloom/agents/robustness/signals/__init__.py index dc4b087890..27b8db7967 100644 --- a/src/hyperloom/agents/robustness/signals/__init__.py +++ b/src/hyperloom/agents/robustness/signals/__init__.py @@ -14,7 +14,6 @@ ) from .budget import BudgetConfig, evaluate_budget_signals from .classifier import Classifier, SignalSpec -from .cluster_fault import evaluate_cluster_fault_signals from .conversation_progress import ( ConversationProgressConfig, evaluate_conversation_progress_signals, @@ -38,7 +37,6 @@ GpuLeakConfig, GpuLeakDetector, ) -from .health import evaluate_health_signals from .kernel_pipeline import ( KernelPipelineConfig, RayPendingDetector, @@ -97,7 +95,6 @@ "SymptomSeverity", "TraceLensCliFiredOnce", "evaluate_budget_signals", - "evaluate_cluster_fault_signals", "evaluate_cold_start_signals", "evaluate_conversation_progress_signals", "evaluate_crash_signals", @@ -105,7 +102,6 @@ "evaluate_decision_audit_signals", "evaluate_event_signals", "evaluate_external_deps_signals", - "evaluate_health_signals", "evaluate_kernel_pipeline_signals", "evaluate_local_health_signals", "evaluate_phase_budget_signals", diff --git a/src/hyperloom/agents/robustness/signals/classifier.py b/src/hyperloom/agents/robustness/signals/classifier.py index e0f8aeae27..f9b8dbf73c 100644 --- a/src/hyperloom/agents/robustness/signals/classifier.py +++ b/src/hyperloom/agents/robustness/signals/classifier.py @@ -22,7 +22,6 @@ from ..state_store import DetectorStateStore from .aiter_jit import AiterJitConfig, AiterJitDetector from .budget import BudgetConfig, evaluate_budget_signals -from .cluster_fault import ClusterFaultConfig, evaluate_cluster_fault_signals from .conversation_progress import ConversationProgressConfig, evaluate_conversation_progress_signals from .crash import CrashConfig, evaluate_crash_signals from .critic_health import ( @@ -40,7 +39,6 @@ evaluate_external_deps_signals, ) from .gpu_leak import GpuLeakConfig, GpuLeakDetector -from .health import HealthConfig, evaluate_health_signals from .kernel_pipeline import ( KernelPipelineConfig, RayPendingDetector, @@ -104,7 +102,7 @@ class SignalSpec: def _external_deps_extra_kwargs(classifier: "Classifier") -> dict[str, Any]: """Inject the shared TraceLens CLI latch into ``evaluate_external_deps_signals``. - The latch is a configless one-shot helper owned by the classifier (so J3 + The latch is a configless one-shot helper owned by the classifier (so the rule fires at most once per session); it is a sibling of the external_deps row rather than its own signal. @@ -125,7 +123,6 @@ def _external_deps_extra_kwargs(classifier: "Classifier") -> dict[str, Any]: SignalSpec("stall", "stall", StallConfig, evaluator=evaluate_stall_signals), SignalSpec("crash", "crash", CrashConfig, evaluator=evaluate_crash_signals), SignalSpec("event", "event", EventConfig, evaluator=evaluate_event_signals), - SignalSpec("health", "health", HealthConfig, evaluator=evaluate_health_signals), SignalSpec( "local_health", "local_health", @@ -139,12 +136,6 @@ def _external_deps_extra_kwargs(classifier: "Classifier") -> dict[str, Any]: detector_cls=GpuLeakDetector, state_view_key="gpu_leak", ), - SignalSpec( - "cluster_fault", - "cluster_fault", - ClusterFaultConfig, - evaluator=evaluate_cluster_fault_signals, - ), SignalSpec( "budget", "budget", @@ -218,7 +209,7 @@ def _external_deps_extra_kwargs(classifier: "Classifier") -> dict[str, Any]: CriticHealthConfig, evaluator=evaluate_critic_health_signals, ), - # F1 ray-pending is stateful; F2/F5 live in the module helper — both + # Ray-pending is stateful; the other two live in the module helper — both # driven off one KernelPipelineConfig slot. SignalSpec( "ray_pending", @@ -239,7 +230,7 @@ def _external_deps_extra_kwargs(classifier: "Classifier") -> dict[str, Any]: StateIntegrityConfig, evaluator=evaluate_state_integrity_signals, ), - # TraceLens CLI latch is owned by the classifier and injected here so J3 + # TraceLens CLI latch is owned by the classifier and injected here so it # fires at most once per session. SignalSpec( "external_deps", diff --git a/src/hyperloom/agents/robustness/signals/cluster_fault.py b/src/hyperloom/agents/robustness/signals/cluster_fault.py deleted file mode 100644 index 239782a130..0000000000 --- a/src/hyperloom/agents/robustness/signals/cluster_fault.py +++ /dev/null @@ -1,184 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Symptoms derived from cluster-fault snapshots (M2). - -Consumes :data:`SourceData.cluster_faults` rows shaped like the upstream -robust-api ``FaultSummary``:: - - { - "name": "g53-gpu_ecc", - "monitor_id": "gpu_ecc", - "node_name": "g53", - "phase": "Isolating", # Isolating / Succeeded / Failed - "auto_repair": false, - "action": "isolate", - "created_at": "...", - "affected_workload_count": 3, - "affected_gpu_count": 8, - } - -Severity: ``Failed`` -> HIGH; ``Isolating`` -> MEDIUM, escalated to HIGH -on a wide blast radius (>= ``high_workload_threshold`` workloads or ->= ``high_gpu_threshold`` GPUs); ``Succeeded`` -> silent. -""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Any - -from hyperloom.common.coerce import to_int - -from ..role.prompt_inputs import ReactorContext -from ..sources.base import SourceData -from .symptom import Symptom, SymptomSeverity - - -@dataclass -class ClusterFaultConfig: - """Tunables for the cluster_fault rule.""" - - # Actionable phases; "Succeeded" is excluded. - actionable_phases: frozenset[str] = frozenset({"Isolating", "Failed"}) - high_workload_threshold: int = 4 - high_gpu_threshold: int = 8 - - -def evaluate_cluster_fault_signals( - ctx: ReactorContext, - data: SourceData, - *, - config: ClusterFaultConfig | None = None, -) -> list[Symptom]: - """Convert actionable cluster-fault rows into ``cluster_fault`` symptoms. - - Args: - ctx (ReactorContext): Reactor context for the current tick. - data (SourceData): Collected source data including ``cluster_faults``. - config (ClusterFaultConfig | None): Tunables; defaults to - :class:`ClusterFaultConfig` when ``None``. - - Returns: - list[Symptom]: One ``cluster_fault`` symptom per actionable fault, - possibly empty. - """ - cfg = config or ClusterFaultConfig() - if not data.cluster_faults: - return [] - out: list[Symptom] = [] - for entry in data.cluster_faults: - if not isinstance(entry, dict): - continue - sym = _fault_to_symptom(entry, cfg) - if sym is not None: - out.append(sym) - return out - - -def _fault_to_symptom( - entry: dict[str, Any], - cfg: ClusterFaultConfig, -) -> Symptom | None: - """Build a ``cluster_fault`` symptom from a single fault row. - - Args: - entry (dict[str, Any]): A single cluster-fault row. - cfg (ClusterFaultConfig): Tunables (actionable phases + thresholds). - - Returns: - Symptom | None: The corresponding symptom, or ``None`` when the fault's - phase is not actionable. - """ - phase = str(entry.get("phase") or "") - if phase not in cfg.actionable_phases: - return None - - name = str(entry.get("name") or "") - monitor_id = str(entry.get("monitor_id") or "") - node = str(entry.get("node_name") or "") - affected_workloads = to_int(entry.get("affected_workload_count"), default=0) - affected_gpus = to_int(entry.get("affected_gpu_count"), default=0) - auto_repair = bool(entry.get("auto_repair")) - - severity = _severity_for( - phase=phase, - affected_workloads=affected_workloads, - affected_gpus=affected_gpus, - cfg=cfg, - ) - - return Symptom( - name="cluster_fault", - severity=severity, - summary=( - f"cluster fault {name or monitor_id!r} on node {node or 'unknown'} " - f"phase={phase} workloads={affected_workloads} gpus={affected_gpus}" - ), - evidence={ - "fault_name": name, - "monitor_id": monitor_id, - "node": node, - "phase": phase, - "auto_repair": auto_repair, - "action": entry.get("action"), - "affected_workload_count": affected_workloads, - "affected_gpu_count": affected_gpus, - "created_at": entry.get("created_at"), - }, - # Fault name in subject so the ladder cools down per-fault. - subject={"node": node, "fault": name or monitor_id}, - source="server", - suggestion=_suggestion(phase, severity, auto_repair), - ) - - -def _severity_for( - *, - phase: str, - affected_workloads: int, - affected_gpus: int, - cfg: ClusterFaultConfig, -) -> SymptomSeverity: - """Compute the severity for a fault from its phase and blast radius. - - Args: - phase (str): The fault phase (e.g. ``"Isolating"`` / ``"Failed"``). - affected_workloads (int): Number of impacted workloads. - affected_gpus (int): Number of impacted GPUs. - cfg (ClusterFaultConfig): Tunables (provides escalation thresholds). - - Returns: - SymptomSeverity: HIGH for failed faults or wide blast radius, otherwise - MEDIUM. - """ - if phase == "Failed": - return SymptomSeverity.HIGH - if affected_workloads >= cfg.high_workload_threshold: - return SymptomSeverity.HIGH - if affected_gpus >= cfg.high_gpu_threshold: - return SymptomSeverity.HIGH - return SymptomSeverity.MEDIUM - - -def _suggestion(phase: str, severity: SymptomSeverity, auto_repair: bool) -> str: - """Pick an operator-facing suggestion for a fault symptom. - - Args: - phase (str): The fault phase. - severity (SymptomSeverity): The computed severity. - auto_repair (bool): Whether auto-repair is active for the fault. - - Returns: - str: A short remediation hint tailored to the phase/severity. - """ - if phase == "Failed": - return "auto-repair failed; delegate(server_lifecycle) or escalate strategy to drain affected workloads" - if severity is SymptomSeverity.HIGH: - return "blast radius is wide; escalate strategy or pause new dispatches to the affected node" - if auto_repair: - return "auto-repair in progress; observe and re-evaluate next tick" - return "monitor the fault; alert orchestration if it persists" - - -__all__ = ["ClusterFaultConfig", "evaluate_cluster_fault_signals"] diff --git a/src/hyperloom/agents/robustness/signals/critic_health.py b/src/hyperloom/agents/robustness/signals/critic_health.py index 9bb4f3f1b4..d3757a00c0 100644 --- a/src/hyperloom/agents/robustness/signals/critic_health.py +++ b/src/hyperloom/agents/robustness/signals/critic_health.py @@ -1,21 +1,18 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Critic-health signals (E1 / E2 / E4 / E5). +"""Critic-health signals. Critic is the reviewer and no one reviews the reviewer; these detectors catch the session silently losing its decision gate: -* **E1 ``critic_kb_outage``** — ``judge_bundle.json`` marks +* **``critic_kb_outage``** — ``judge_bundle.json`` marks ``kb_read_skipped_reason="kb_unreachable"`` for ``min_outage_judges`` consecutive turns. -* **E2 ``critic_unavailable_streak``** — ``review_verdict`` events with +* **``critic_unavailable_streak``** — ``review_verdict`` events with ``source="critic_unavailable"`` for ``min_unavailable_verdicts`` consecutive verdicts. -* **E4 ``critic_prune_stuck``** — ``critic-workdir/`` count past ``max_workdir_count`` (pruner broken). -* **E5 ``critic_runtime_stuck``** — runtime-cli timeout pattern in server logs +* **``critic_prune_stuck``** — ``critic-workdir/`` count past ``max_workdir_count`` (pruner broken). +* **``critic_runtime_stuck``** — runtime-cli timeout pattern in server logs (reuses :data:`local_log_errors`), collapsed into one critic-attributed symptom. - -E3 ("critic full-approve drift") is out of scope: it needs critic-agent runtime -invariants Robustness cannot see from coordinator_events. """ from __future__ import annotations @@ -35,15 +32,15 @@ class CriticHealthConfig: Threshold defaults are permissive; escalate only when an outage persists. """ - # E1 — KB unreachable across N+ consecutive recent turns. + # KB unreachable across N+ consecutive recent turns. min_outage_judges: int = 3 - # E2 — critic_unavailable verdicts across N+ consecutive recent items. + # critic_unavailable verdicts across N+ consecutive recent items. min_unavailable_verdicts: int = 3 - # E4 — ``critic-workdir/`` count above this is a leak (2x the backend keep count). + # ``critic-workdir/`` count above this is a leak (2x the backend keep count). max_workdir_count: int = 100 - # E5 — log-pattern marker that the runtime-cli timed out. + # Log-pattern marker that the runtime-cli timed out. runtime_stuck_pattern_marker: str = "runtime.cli" - # E5 needs log lines with the marker AND a ``timed out`` substring to fire. + # Needs log lines with the marker AND a ``timed out`` substring to fire. min_runtime_stuck_hits: int = 1 @@ -53,7 +50,7 @@ def evaluate_critic_health_signals( *, config: CriticHealthConfig | None = None, ) -> list[Symptom]: - """Run the E1/E2/E4/E5 critic-health rules and aggregate their symptoms. + """Run the critic-health rules and aggregate their symptoms. Args: ctx (ReactorContext): Reactor context for the current tick. @@ -76,7 +73,7 @@ def evaluate_critic_health_signals( # --------------------------------------------------------------------------- -# E1 — KB outage streak +# KB outage streak # --------------------------------------------------------------------------- @@ -84,7 +81,7 @@ def _kb_outage_symptoms( data: SourceData, cfg: CriticHealthConfig, ) -> list[Symptom]: - """E1: fire ``critic_kb_outage`` for a streak of KB-unreachable judges. + """Fire ``critic_kb_outage`` for a streak of KB-unreachable judges. Args: data (SourceData): Collected source data including @@ -143,7 +140,7 @@ def _kb_outage_symptoms( # --------------------------------------------------------------------------- -# E2 — critic_unavailable verdict streak +# critic_unavailable verdict streak # --------------------------------------------------------------------------- @@ -221,7 +218,7 @@ def _unavailable_streak_symptoms( # --------------------------------------------------------------------------- -# E4 — workdir prune stuck +# workdir prune stuck # --------------------------------------------------------------------------- @@ -229,7 +226,7 @@ def _prune_stuck_symptoms( data: SourceData, cfg: CriticHealthConfig, ) -> list[Symptom]: - """E4: fire ``critic_prune_stuck`` when the workdir count leaks past cap. + """Fire ``critic_prune_stuck`` when the workdir count leaks past cap. Args: data (SourceData): Collected source data including @@ -271,7 +268,7 @@ def _prune_stuck_symptoms( # --------------------------------------------------------------------------- -# E5 — runtime-cli timeout +# runtime-cli timeout # --------------------------------------------------------------------------- diff --git a/src/hyperloom/agents/robustness/signals/event.py b/src/hyperloom/agents/robustness/signals/event.py index 1b2d014710..854f322824 100644 --- a/src/hyperloom/agents/robustness/signals/event.py +++ b/src/hyperloom/agents/robustness/signals/event.py @@ -7,7 +7,7 @@ (misconfigured / systemically-rejected agent → MEDIUM alert), ``delegated_result`` failures clustering on one action family (stuck branch → ``prune_branch``), ``recover_unsuccessful`` recovery outcomes, -and B4 ``idempotency_replay`` (repeated idempotency keys). Reads both +and ``idempotency_replay`` (repeated idempotency keys). Reads both ``ctx.inbox`` and ``data.coordinator_events``. """ @@ -33,7 +33,7 @@ class EventConfig: delegated_failure_prune_threshold: int = 4 # Lookback over inbox + coordinator_events for the most recent recover result. recover_lookback_events: int = 50 - # B4 ``idempotency_replay``: fire when >= threshold distinct idempotency_keys + # ``idempotency_replay``: fire when >= threshold distinct idempotency_keys # share the same action+payload hash within one tick. idempotency_replay_threshold: int = 2 diff --git a/src/hyperloom/agents/robustness/signals/external_deps.py b/src/hyperloom/agents/robustness/signals/external_deps.py index 8600afac96..368298b88c 100644 --- a/src/hyperloom/agents/robustness/signals/external_deps.py +++ b/src/hyperloom/agents/robustness/signals/external_deps.py @@ -1,20 +1,17 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""External-dependency signals (J1 / J2 / J3). +"""External-dependency signals. Failure modes originating outside Hyperloom but manifesting as opaque hangs / 401 storms: -* **J1 ``gateway_auth_outage``** — gateway ``/models`` returns 401/403 +* **``gateway_auth_outage``** — gateway ``/models`` returns 401/403 (key lost/revoked); every claude/codex CLI will fail at the gateway. -* **J2 ``wekafs_degraded``** — ``stat`` on a source mount errored or +* **``wekafs_degraded``** — ``stat`` on a source mount errored or exceeded the latency budget; ``trace_analyze`` / external CLI hang silently. -* **J3 ``tracelens_cli_missing``** — neither TraceLens perf-report CLI +* **``tracelens_cli_missing``** — neither TraceLens perf-report CLI is on ``PATH``. Boot-time-only, so the detector latches after first fire. - -J4 ``cluster_gpu_quota_anomaly`` is covered by F1 -``ray_pending_starvation`` in ``signals/kernel_pipeline.py``. """ from __future__ import annotations @@ -34,7 +31,7 @@ class ExternalDepsConfig: """Tunables for :func:`evaluate_external_deps_signals`.""" - # J2 mount stat latency budget; ``ok=False`` fires HIGH regardless of latency. + # Mount stat latency budget; ``ok=False`` fires HIGH regardless of latency. mount_latency_warn_ms: float = 5000.0 mount_latency_critical_ms: float = 15000.0 @@ -59,7 +56,7 @@ def __init__( @property def value(self) -> bool: - """Whether the J3 symptom has already fired this session. + """Whether the ``tracelens_cli_missing`` symptom has already fired this session. Returns: bool: ``True`` once the latch has tripped, otherwise ``False``. @@ -85,7 +82,7 @@ def evaluate_external_deps_signals( config: ExternalDepsConfig | None = None, tracelens_latch: TraceLensCliFiredOnce | None = None, ) -> list[Symptom]: - """Run the J1/J2/J3 external-dependency rules and aggregate symptoms. + """Run the external-dependency rules and aggregate symptoms. Args: ctx (ReactorContext): Reactor context for the current tick. @@ -94,7 +91,7 @@ def evaluate_external_deps_signals( config (ExternalDepsConfig | None): Tunables; defaults to :class:`ExternalDepsConfig` when ``None``. tracelens_latch (TraceLensCliFiredOnce | None): One-shot latch for the - J3 rule; when ``None`` the J3 check is skipped. + TraceLens rule; when ``None`` that check is skipped. Returns: list[Symptom]: All external-dependency symptoms found this tick, @@ -118,14 +115,14 @@ def evaluate_external_deps_signals( # --------------------------------------------------------------------------- -# J1 — Upstream gateway 401 / forbidden +# Upstream gateway 401 / forbidden # --------------------------------------------------------------------------- def _gateway_symptoms( gateway: dict[str, Any], ) -> list[Symptom]: - """J1: fire ``gateway_auth_outage`` when the LLM gateway returns 401/403. + """Fire ``gateway_auth_outage`` when the LLM gateway returns 401/403. Args: gateway (dict[str, Any]): Gateway probe result (status/status_code/url). @@ -166,7 +163,7 @@ def _gateway_symptoms( # --------------------------------------------------------------------------- -# J2 — WekaFS / external mount degraded +# WekaFS / external mount degraded # --------------------------------------------------------------------------- @@ -174,7 +171,7 @@ def _mount_symptoms( mounts: list[Any], cfg: ExternalDepsConfig, ) -> list[Symptom]: - """J2: fire ``wekafs_degraded`` for unreachable or slow external mounts. + """Fire ``wekafs_degraded`` for unreachable or slow external mounts. Unreachable mounts (``ok`` falsey) fire HIGH; reachable-but-slow mounts fire HIGH/MEDIUM based on the configured latency thresholds. @@ -257,7 +254,7 @@ def _mount_symptoms( # --------------------------------------------------------------------------- -# J3 — TraceLens CLI missing +# TraceLens CLI missing # --------------------------------------------------------------------------- @@ -265,7 +262,7 @@ def _tracelens_symptoms( cli_info: dict[str, Any], latch: TraceLensCliFiredOnce, ) -> list[Symptom]: - """J3: fire ``tracelens_cli_missing`` once when no TraceLens CLI is on PATH. + """Fire ``tracelens_cli_missing`` once when no TraceLens CLI is on PATH. Latches via ``latch`` so the symptom is emitted at most once per session. diff --git a/src/hyperloom/agents/robustness/signals/health.py b/src/hyperloom/agents/robustness/signals/health.py deleted file mode 100644 index 7ac4515952..0000000000 --- a/src/hyperloom/agents/robustness/signals/health.py +++ /dev/null @@ -1,167 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Pod-health signal driven by robustness-server's session snapshot. - -Emits alerts when a ``session_pods`` row has a non-empty ``phase`` other than ``Running``, -or when ``session_summary.pods`` shows empty ``available_metrics`` for a pod older than -``no_metrics_warn_s`` (alive but no telemetry → LOW). -""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Any - -from ..role.prompt_inputs import ReactorContext -from ..sources.base import SourceData -from .symptom import Symptom, SymptomSeverity - - -@dataclass -class HealthConfig: - """Tunables for :func:`evaluate_health_signals`. - - Attributes: - no_metrics_warn_s (float): Minimum pod age in seconds before an empty - ``available_metrics`` list triggers a ``pod_no_metrics`` symptom. - """ - - no_metrics_warn_s: float = 600.0 - - -_POD_RUNNING_PHASES: frozenset[str] = frozenset( - { - "Running", - "Succeeded", - "Pending", # transient; we do not flag Pending here - "", # missing phase - } -) - - -def evaluate_health_signals( - ctx: ReactorContext, - data: SourceData, - *, - config: HealthConfig | None = None, -) -> list[Symptom]: - """Emit pod-health symptoms from the server session snapshot. - - Flags pods in a non-running phase (``pod_not_running``) and hands pods that - have produced no metric series for longer than the configured age - (``pod_no_metrics``). - - Args: - ctx (ReactorContext): Reactor context (provides the current unix time). - data (SourceData): Collected source data including ``session_pods`` and - ``session_summary``. - config (HealthConfig | None): Tunables; defaults to :class:`HealthConfig` - when ``None``. - - Returns: - list[Symptom]: All pod-health symptoms found this tick, possibly empty. - """ - cfg = config or HealthConfig() - out: list[Symptom] = [] - - for assignment in data.session_pods: - if not isinstance(assignment, dict): - continue - pod = _pod_dict(assignment) - phase = _phase(assignment, pod) - if phase and phase not in _POD_RUNNING_PHASES: - ns = str(pod.get("namespace") or "") - name = str(pod.get("name") or "") - role = str(assignment.get("role") or "") - severity = SymptomSeverity.HIGH if phase == "Failed" else SymptomSeverity.MEDIUM - out.append( - Symptom( - name="pod_not_running", - severity=severity, - summary=f"pod {ns}/{name} ({role}) phase={phase}", - evidence={ - "namespace": ns, - "name": name, - "role": role, - "phase": phase, - "assignment_id": assignment.get("assignment_id"), - }, - subject={"namespace": ns, "name": name, "phase": phase}, - source="server", - suggestion=( - "kill_task on the related task and escalate strategy" - if phase == "Failed" - else "escalate_strategy_change to monitor recovery" - ), - ) - ) - - summary = data.session_summary - if isinstance(summary, dict): - for entry in summary.get("pods") or []: - if not isinstance(entry, dict): - continue - available = entry.get("available_metrics") - if available not in (None, []): - continue - t_start = entry.get("t_start") - if not isinstance(t_start, (int, float)): - continue - age = ctx.now_unix - float(t_start) - if age < cfg.no_metrics_warn_s: - continue - pod = _pod_dict(entry) - ns = str(pod.get("namespace") or "") - name = str(pod.get("name") or "") - role = str(entry.get("role") or "") - out.append( - Symptom( - name="pod_no_metrics", - severity=SymptomSeverity.LOW, - summary=(f"pod {ns}/{name} ({role}) has no metric series for {int(age)}s"), - evidence={ - "namespace": ns, - "name": name, - "role": role, - "age_seconds": int(age), - }, - subject={"namespace": ns, "name": name, "kind": "no_metrics"}, - source="server", - suggestion="verify exporter / cluster_proxy data path", - ) - ) - - return out - - -def _pod_dict(entry: dict[str, Any]) -> dict[str, Any]: - """Extract the nested ``pod`` dict from an assignment/summary entry. - - Args: - entry (dict[str, Any]): A session-pod assignment or summary row. - - Returns: - dict[str, Any]: The nested ``pod`` dict, or an empty dict when absent. - """ - pod = entry.get("pod") - if isinstance(pod, dict): - return pod - return {} - - -def _phase(entry: dict[str, Any], pod: dict[str, Any]) -> str: - """Resolve a pod's phase from the entry or its nested pod dict. - - Args: - entry (dict[str, Any]): The assignment/summary row. - pod (dict[str, Any]): The nested pod dict (see :func:`_pod_dict`). - - Returns: - str: The trimmed phase string, or an empty string when unset. - """ - phase = entry.get("phase") or pod.get("phase") - return str(phase or "").strip() - - -__all__ = ["HealthConfig", "evaluate_health_signals"] diff --git a/src/hyperloom/agents/robustness/signals/kernel_pipeline.py b/src/hyperloom/agents/robustness/signals/kernel_pipeline.py index 4e68e7b125..683a9e65c0 100644 --- a/src/hyperloom/agents/robustness/signals/kernel_pipeline.py +++ b/src/hyperloom/agents/robustness/signals/kernel_pipeline.py @@ -1,13 +1,13 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Kernel-pipeline / external-backend health signals (F1-F5). +"""Kernel-pipeline / external-backend health signals. -* **F1 ``ray_pending_starvation``** — non-zero ``ray status`` Pending across +* **``ray_pending_starvation``** — non-zero ``ray status`` Pending across ``min_pending_ticks`` consecutive ticks (cluster quota ledger wedged). -* **F2 ``geak_budget_starvation``** — same kernel_id's GEAK attempt SIGTERM'd across +* **``geak_budget_starvation``** — same kernel_id's GEAK attempt SIGTERM'd across ``min_geak_sigterm_attempts`` rows; budget too short for ``select_patch``. -* **F5 ``kernel_opt_no_progress``** — ``min_kernels_with_no_progress`` kernel_ids where +* **``kernel_opt_no_progress``** — ``min_kernels_with_no_progress`` kernel_ids where no backend attempt reached a >=1.2x microbench speedup and no integrate row recorded a KEEP decision (>=2 distinct backends tried); prune kernel_opt toward params/sweep. """ @@ -39,18 +39,18 @@ class KernelPipelineConfig: """Tunables for :func:`evaluate_kernel_pipeline_signals`.""" - # F1 — pending count above this for N consecutive ticks → fire. + # Pending count above this for N consecutive ticks → fire. pending_count_threshold: int = 1 min_pending_ticks: int = 3 - # F2 — same kernel_id has GEAK backend SIGTERM'd this many times. + # Same kernel_id has GEAK backend SIGTERM'd this many times. min_geak_sigterm_attempts: int = 2 - # F5 — kernel_ids with no >=1.2x microbench speedup and no KEEP integrate + # kernel_ids with no >=1.2x microbench speedup and no KEEP integrate # decision across the recent oob_attempts window. min_kernels_with_no_progress: int = 3 # --------------------------------------------------------------------------- -# F1 — Ray pending starvation (stateful — counts consecutive ticks) +# Ray pending starvation (stateful — counts consecutive ticks) # --------------------------------------------------------------------------- @@ -100,7 +100,7 @@ def evaluate( ctx: ReactorContext, data: SourceData, ) -> list[Symptom]: - """Advance the pending streak and fire F1 once it crosses threshold. + """Advance the pending streak and fire once it crosses threshold. Resets the streak when Ray data is missing, the head is unhealthy, or the pending count is at/below the configured threshold. @@ -161,7 +161,7 @@ def evaluate( # --------------------------------------------------------------------------- -# F2 — GEAK budget starvation +# GEAK budget starvation # --------------------------------------------------------------------------- @@ -169,7 +169,7 @@ def _geak_budget_symptoms( data: SourceData, cfg: KernelPipelineConfig, ) -> list[Symptom]: - """F2: fire ``geak_budget_starvation`` for kernels whose GEAK runs SIGTERM. + """Fire ``geak_budget_starvation`` for kernels whose GEAK runs SIGTERM. Args: data (SourceData): Collected source data including the decision-audit @@ -232,7 +232,7 @@ def _geak_budget_symptoms( # --------------------------------------------------------------------------- -# F5 — Kernel-opt no-progress +# Kernel-opt no-progress # --------------------------------------------------------------------------- @@ -346,7 +346,7 @@ def _kernel_opt_no_progress_symptoms( # --------------------------------------------------------------------------- -# Public entry point — module-level helper (stateful F1 lives in the class) +# Public entry point — module-level helper (the stateful rule lives in the class) # --------------------------------------------------------------------------- @@ -356,9 +356,10 @@ def evaluate_kernel_pipeline_signals( *, config: KernelPipelineConfig | None = None, ) -> list[Symptom]: - """Evaluate the stateless kernel-pipeline signals (F2 / F5). + """Evaluate the stateless kernel-pipeline signals. - F1 is stateful and lives in :class:`RayPendingDetector`. + ``ray_pending_starvation`` is stateful and lives in + :class:`RayPendingDetector`. Args: ctx: Reactor context for the current tick. diff --git a/src/hyperloom/agents/robustness/signals/local_health.py b/src/hyperloom/agents/robustness/signals/local_health.py index a4e805c44b..e8cc661123 100644 --- a/src/hyperloom/agents/robustness/signals/local_health.py +++ b/src/hyperloom/agents/robustness/signals/local_health.py @@ -3,8 +3,8 @@ """Symptoms derived from LocalProbe-only data. -Fire only when DegradeRouter hands control to :class:`LocalProbeSource` (robustness-server -unreachable); silent otherwise since the SourceData fields are empty. Covers +Fire only when :class:`LocalProbeSource` is active; silent when the probe is +disabled, since the SourceData fields are then empty. Covers ``local_server_unreachable`` (HIGH if all targets fail), ``log_error_pattern`` (OOM/NCCL → HIGH), ``gpu_thermal_high``, plus disk/shm/ray-head/fd pressure rules. """ diff --git a/src/hyperloom/agents/robustness/signals/progress.py b/src/hyperloom/agents/robustness/signals/progress.py index 515fd73494..5e77b6d876 100644 --- a/src/hyperloom/agents/robustness/signals/progress.py +++ b/src/hyperloom/agents/robustness/signals/progress.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Progress-stagnation detectors (B2 / B3). +"""Progress-stagnation detectors. Two stateful rules: ``gain_plateau`` (validated gain flat within ``epsilon_pct`` across ``window_ticks`` while still proposing actions → @@ -102,7 +102,7 @@ def evaluate( ctx: ReactorContext, data: SourceData, ) -> list[Symptom]: - """Update the gain history and evaluate the B2/B3 stagnation rules. + """Update the gain history and evaluate the stagnation rules. Appends one gain sample per new Coordinator tick, then runs the ``gain_plateau`` and ``no_levers_found`` checks. Short-circuits when the @@ -142,7 +142,7 @@ def evaluate( return out def _gain_plateau_symptom(self, snap: SharedStateSnapshot) -> Symptom | None: - """B2: build a ``gain_plateau`` symptom when validated gain has flatlined. + """Build a ``gain_plateau`` symptom when validated gain has flatlined. Args: snap (SharedStateSnapshot): Current shared-state snapshot. @@ -193,7 +193,7 @@ def _gain_plateau_symptom(self, snap: SharedStateSnapshot) -> Symptom | None: ) def _no_levers_symptom(self, snap: SharedStateSnapshot) -> Symptom | None: - """B3: build a ``no_levers_found`` symptom for an empty, gainless run. + """Build a ``no_levers_found`` symptom for an empty, gainless run. Fires only after the explore phase has started and the configured elapsed-minute and tick floors are met, while no kernel_opt work is in diff --git a/src/hyperloom/agents/robustness/signals/repeated_payload.py b/src/hyperloom/agents/robustness/signals/repeated_payload.py index fdce3742b1..0f84a528a4 100644 --- a/src/hyperloom/agents/robustness/signals/repeated_payload.py +++ b/src/hyperloom/agents/robustness/signals/repeated_payload.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Detect same-fingerprint action retries (B1 / same_payload_loop). +"""Detect same-fingerprint action retries (``same_payload_loop``). Hashes the action-defining subset of each ``delegated_result`` payload (from coordinator_events + inbox) and fires ``same_payload_loop`` when a diff --git a/src/hyperloom/agents/robustness/sources/__init__.py b/src/hyperloom/agents/robustness/sources/__init__.py index 9aa05b2b56..94559a9f87 100644 --- a/src/hyperloom/agents/robustness/sources/__init__.py +++ b/src/hyperloom/agents/robustness/sources/__init__.py @@ -3,9 +3,10 @@ """Data sources used by the reactor. -Layered as ``robustness-server`` (preferred) -> ``local probe`` (fallback) -through :class:`DegradeRouter`. The contract is narrow so each source is -a drop-in replacement. +The local probe is the only collector; :class:`DegradeRouter` keeps it +behind a silent fallback so a failing probe degrades to "no data" instead +of failing the tick. The contract is narrow so each source is a drop-in +replacement. """ from .base import ( @@ -16,15 +17,12 @@ SourceUnavailable, ) from .local_probe import LocalProbeConfig, LocalProbeSource -from .server_client import RobustnessServerClient, RobustnessServerSource __all__ = [ "DegradeRouter", "HealthState", "LocalProbeConfig", "LocalProbeSource", - "RobustnessServerClient", - "RobustnessServerSource", "Source", "SourceData", "SourceUnavailable", diff --git a/src/hyperloom/agents/robustness/sources/base.py b/src/hyperloom/agents/robustness/sources/base.py index a5642771d6..e297771a99 100644 --- a/src/hyperloom/agents/robustness/sources/base.py +++ b/src/hyperloom/agents/robustness/sources/base.py @@ -61,11 +61,6 @@ class SourceData: produced each tick; ``degraded_reason`` is set on fallback. """ - session_pods: list[dict[str, Any]] = field(default_factory=list) - session_metrics: dict[str, Any] = field(default_factory=dict) - session_events: list[dict[str, Any]] = field(default_factory=list) - session_summary: dict[str, Any] = field(default_factory=dict) - cluster_faults: list[dict[str, Any]] = field(default_factory=list) local_gpu: dict[str, Any] = field(default_factory=dict) local_processes: list[dict[str, Any]] = field(default_factory=list) # ``False`` when the process probe could not answer (``ps`` missing, timed @@ -88,7 +83,7 @@ class SourceData: # ``local_kernel_breakdown`` ``{tier_pcts, total_kernels, total_gpu_pct, mtime}``. local_manifest: dict[str, Any] = field(default_factory=dict) local_kernel_breakdown: dict[str, Any] = field(default_factory=dict) - # Critic health: ``recent_judges`` + ``workdir_count`` (subdirs under critic-workdir/, E4). + # Critic health: ``recent_judges`` + ``workdir_count`` (subdirs under critic-workdir/). local_critic_health: dict[str, Any] = field(default_factory=dict) # State-integrity slots: ``state_json``, ``wal`` {wal_bytes, db_bytes, db_path}, # ``leases`` (pid liveness), ``agents`` {: {inbox_bytes, outbox_bytes}}, diff --git a/src/hyperloom/agents/robustness/sources/cluster_decoder.py b/src/hyperloom/agents/robustness/sources/cluster_decoder.py deleted file mode 100644 index 90d973c5bd..0000000000 --- a/src/hyperloom/agents/robustness/sources/cluster_decoder.py +++ /dev/null @@ -1,223 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Decode robust-api raw responses into LocalProbe-equivalent schemas. - -robust-api emits Prometheus-style time-series via ``pod-metrics/batch``; -this module flattens them to one row per device (latest value per -metric) so ``signals/local_health.py`` is agnostic to which source -filled :data:`SourceData.local_gpu`. GPU metrics only for now. -""" - -from __future__ import annotations - -from typing import Any, Mapping - -from hyperloom.common.coerce import to_int - - -# metric_name -> SourceData.local_gpu field, covering rocm/DCGM/generic exporters. -_GPU_METRIC_FIELD: Mapping[str, str] = { - # rocm-exporter - "rocm_temperature_celsius": "temperature_c", - "rocm_temperature_edge_celsius": "temperature_c", - "rocm_temperature_junction_celsius": "temperature_junction_c", - "rocm_temperature_memory_celsius": "temperature_memory_c", - "rocm_gpu_utilization": "util_gpu_pct", - "rocm_memory_utilization": "util_mem_pct", - "rocm_power_average_watts": "power_watts", - # NVIDIA DCGM exporter - "DCGM_FI_DEV_GPU_TEMP": "temperature_c", - "DCGM_FI_DEV_MEMORY_TEMP": "temperature_memory_c", - "DCGM_FI_DEV_GPU_UTIL": "util_gpu_pct", - "DCGM_FI_DEV_MEM_COPY_UTIL": "util_mem_pct", - "DCGM_FI_DEV_POWER_USAGE": "power_watts", - # Generic / re-labelled - "gpu_temperature_celsius": "temperature_c", - "gpu_temperature_c": "temperature_c", - "gpu_util_percent": "util_gpu_pct", - "gpu_memory_util_percent": "util_mem_pct", -} - - -# Series labels used to deduce gpu_id; first match wins. -_GPU_ID_LABELS: tuple[str, ...] = ( - "gpu", - "device", - "device_id", - "minor", - "minor_number", - "DCGM_FI_DRIVER_DEVICE_ID", - "card", -) - - -def decode_gpu_snapshot( - response: Mapping[str, Any] | None, -) -> dict[str, Any]: - """Decode pod-metrics/batch response into LocalProbe local_gpu shape. - - Each gpu row carries the same fields the LocalProbe rocm-smi parser - produces (``gpu_id``, ``temperature_c``, ``util_gpu_pct`` etc.) plus - ``pod_namespace`` / ``pod_name`` so signals can pinpoint where the - heat is coming from. Rows are keyed by ``(namespace, name, gpu_id)`` - so two pods sharing a GPU id on one node do not collide. - - Args: - response (Mapping[str, Any] | None): The raw - ``pod-metrics/batch`` response, expected to nest the per-pod - results under ``data.pods``. Any other shape yields ``{}``. - - Returns: - dict[str, Any]: ``{"gpus": [...], "tool": "robust-api"}`` with - one row per decoded device, or ``{}`` when no GPU metric is - found. - """ - - if not isinstance(response, Mapping): - return {} - data = response.get("data") - if not isinstance(data, Mapping): - return {} - pod_results = data.get("pods") - if not isinstance(pod_results, list): - return {} - - by_id: dict[tuple[str, str, str], dict[str, Any]] = {} - - for pod in pod_results: - if not isinstance(pod, Mapping): - continue - ns = str(pod.get("namespace") or "") - name = str(pod.get("name") or "") - results = pod.get("results") - if not isinstance(results, list): - continue - for result in results: - if not isinstance(result, Mapping): - continue - metric_name = str(result.get("name") or "") - field = _GPU_METRIC_FIELD.get(metric_name) - if field is None: - continue - for series in result.get("series") or []: - if not isinstance(series, Mapping): - continue - gpu_id = _extract_gpu_id(series.get("labels")) - latest = _latest_value(series.get("values")) - if latest is None: - continue - key = (ns, name, gpu_id) - snap = by_id.setdefault( - key, - { - "gpu_id": to_int(gpu_id, default=gpu_id), - "pod_namespace": ns, - "pod_name": name, - }, - ) - snap[field] = latest - - if not by_id: - return {} - return { - "gpus": [by_id[k] for k in sorted(by_id)], - "tool": "robust-api", - } - - -def merge_gpu_snapshots( - snapshots: list[Mapping[str, Any]], -) -> dict[str, Any]: - """Combine multiple per-pod snapshots into a single ``local_gpu``. - - Rows from every snapshot are concatenated and sorted by - (pod_namespace, pod_name, gpu_id); no per-key merge or field-clash - resolution happens, so callers must pass at most one snapshot per pod - (see ``server_client._unique_pod_refs``). - - Args: - snapshots: Per-pod GPU snapshot mappings. - - Returns: - A merged ``local_gpu`` mapping. - """ - - rows: list[dict[str, Any]] = [] - for snap in snapshots: - if not isinstance(snap, Mapping): - continue - gpus = snap.get("gpus") - if not isinstance(gpus, list): - continue - for row in gpus: - if isinstance(row, Mapping): - rows.append(dict(row)) - if not rows: - return {} - rows.sort( - key=lambda r: ( - str(r.get("pod_namespace") or ""), - str(r.get("pod_name") or ""), - str(r.get("gpu_id") or ""), - ) - ) - return {"gpus": rows, "tool": "robust-api"} - - -def _extract_gpu_id(labels: Any) -> str: - """Deduce a GPU id string from a Prometheus series' labels. - - Walks :data:`_GPU_ID_LABELS` in priority order so the differing - exporter conventions (rocm / DCGM / generic) resolve to one id. - - Args: - labels (Any): The ``labels`` mapping from a metric series. - Non-mapping values yield an empty string. - - Returns: - str: The first matching label's value as a string, or ``""`` - when no known label is present. - """ - if not isinstance(labels, Mapping): - return "" - for key in _GPU_ID_LABELS: - if key in labels: - return str(labels[key]) - return "" - - -def _latest_value(values: Any) -> float | None: - """Return the most recent numeric value from a metric series. - - Scans the ``values`` list for the entry with the highest - ``timestamp`` whose ``value`` coerces to ``float``. - - Args: - values (Any): The series ``values`` list, each entry expected - to be a mapping with ``timestamp`` and ``value`` keys. - - Returns: - float | None: The value at the latest timestamp, or ``None`` - when the list is empty or carries no usable entry. - """ - if not isinstance(values, list) or not values: - return None - best_ts = -1 - best_val: float | None = None - for entry in values: - if not isinstance(entry, Mapping): - continue - ts = entry.get("timestamp") - if not isinstance(ts, (int, float)): - continue - if ts > best_ts: - try: - best_val = float(entry.get("value")) - except (TypeError, ValueError): - continue - best_ts = ts - return best_val - - -__all__ = ["decode_gpu_snapshot", "merge_gpu_snapshots"] diff --git a/src/hyperloom/agents/robustness/sources/local_probe.py b/src/hyperloom/agents/robustness/sources/local_probe.py index 7de0b421e5..b32d267644 100644 --- a/src/hyperloom/agents/robustness/sources/local_probe.py +++ b/src/hyperloom/agents/robustness/sources/local_probe.py @@ -9,7 +9,7 @@ without raising; :class:`LocalProbeSource` raises :class:`SourceUnavailable` only when *every* sub-probe yields nothing, so :class:`DegradeRouter` does not flap. Cluster-wide metrics and -node-level fault detection stay with robustness-server. +node-level fault detection are out of scope. """ from __future__ import annotations @@ -1724,7 +1724,7 @@ def _sample_critic_workdir( session_dir: Path | None, max_judges: int, ) -> dict[str, Any]: - """Scan ``critic-workdir//judge_bundle.json`` for E1+E4 signals. + """Scan ``critic-workdir//judge_bundle.json`` for critic-health signals. Args: session_dir (Path | None): Session root containing @@ -2096,7 +2096,7 @@ async def _probe_external_deps( mount_timeout_s: float, http_timeout_s: float, ) -> dict[str, Any]: - """Async wrapper that runs J1+J2+J3 probes once per tick. + """Async wrapper that runs the external-dependency probes once per tick. Args: gateway_probe_url_override (str): Explicit gateway probe URL; @@ -2138,7 +2138,7 @@ async def _probe_gateway_health( A 401 here (with the same token + custom headers that critic + kernel-agent use) means the upstream gateway has revoked / lost the - key. Matching the main LLM auth resolver avoids false J1 alerts on + key. Matching the main LLM auth resolver avoids false gateway alerts on gateways that require ``Ocp-Apim-Subscription-Key`` for ``/models``. Args: diff --git a/src/hyperloom/agents/robustness/sources/server_client.py b/src/hyperloom/agents/robustness/sources/server_client.py deleted file mode 100644 index 9aabe2f606..0000000000 --- a/src/hyperloom/agents/robustness/sources/server_client.py +++ /dev/null @@ -1,660 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""robustness-server client + Source adapter. - -The client wraps a small subset of the robustness-server REST API (the -``/api/v1/sessions/{id}/{pods,events,summary}`` and ``/api/v1/cluster/*`` -endpoints). Networking errors (timeout / connect refused / 5xx) -are translated to :class:`SourceUnavailable` so :class:`DegradeRouter` -counts failures and degrades to the local fallback. 404 and other 4xx -responses yield ``None`` without decoding the body, since they usually -mean "no data for this session" rather than an upstream outage. -""" - -from __future__ import annotations - -from dataclasses import dataclass -from typing import Any - -import httpx - -from .base import SourceData, SourceUnavailable -from .cluster_decoder import decode_gpu_snapshot, merge_gpu_snapshots - - -@dataclass -class _MetricsWindow: - """Explicit ``start`` / ``end`` Unix-second window for ``/metrics`` and ``/summary``.""" - - start_unix: int - end_unix: int - - -class RobustnessServerClient: - """HTTP client for the subset of robustness-server we use.""" - - def __init__( - self, - base_url: str, - *, - timeout_s: float = 5.0, - client: httpx.AsyncClient | None = None, - ) -> None: - """Build a client for the robustness-server REST subset. - - Args: - base_url (str): Base URL of the robustness-server; trailing - slash is stripped. Must be non-empty. - timeout_s (float): Per-request timeout in seconds, applied - only when this client owns its ``httpx`` client. - client (httpx.AsyncClient | None): Optional pre-built async - client to reuse; when ``None`` a new one is created and - owned by this instance. - - Raises: - ValueError: If ``base_url`` is empty. - """ - if not base_url: - raise ValueError("base_url must be non-empty") - self._base_url = base_url.rstrip("/") - self._timeout = httpx.Timeout(timeout_s) - self._owns_client = client is None - self._client = client or httpx.AsyncClient( - base_url=self._base_url, - timeout=self._timeout, - ) - - @property - def base_url(self) -> str: - """Normalised base URL (without a trailing slash). - - Returns: - str: The base URL the client issues requests against. - """ - return self._base_url - - async def aclose(self) -> None: - """Close the underlying ``httpx`` client when this owns it. - - No-op when an external client was injected, since its lifecycle - belongs to the caller. - """ - if self._owns_client: - await self._client.aclose() - - # -- low-level GET --------------------------------------------------- - - async def _get_json(self, path: str, *, params: dict[str, Any] | None = None) -> Any: - """Issue a GET and decode JSON, mapping transport errors. - - Timeouts, transport errors, 5xx responses and invalid JSON are - translated to :class:`SourceUnavailable` so the DegradeRouter - can count them. 404 / other 4xx responses return ``None`` since - they usually mean "no data for this session" rather than outage. - - Args: - path (str): Request path appended to the base URL. - params (dict[str, Any] | None): Optional query parameters. - - Returns: - Any: The decoded JSON body, or ``None`` for 4xx responses. - - Raises: - SourceUnavailable: On timeout, transport error, 5xx status, - or undecodable JSON. - """ - try: - resp = await self._client.get(path, params=params) - except httpx.TimeoutException as exc: - raise SourceUnavailable(f"GET {path}: timeout") from exc - except httpx.RequestError as exc: - raise SourceUnavailable(f"GET {path}: {type(exc).__name__}: {exc}") from exc - if resp.status_code >= 500: - raise SourceUnavailable(f"GET {path}: upstream {resp.status_code}") - if resp.status_code == 404: - return None - if resp.status_code >= 400: - return None - try: - return resp.json() - except ValueError as exc: - raise SourceUnavailable(f"GET {path}: invalid json") from exc - - # -- public methods -------------------------------------------------- - - async def list_session_pods( - self, - session_id: str, - *, - start_unix: int | None = None, - end_unix: int | None = None, - ) -> list[dict[str, Any]]: - """List a session's pods via ``GET .../{id}/pods``. - - Args: - session_id (str): Identifier of the session. - start_unix (int | None): Optional window start in Unix - seconds; converted to ISO-8601 for the ``start`` query. - end_unix (int | None): Optional window end in Unix seconds; - converted to ISO-8601 for the ``end`` query. - - Returns: - list[dict[str, Any]]: The pod rows, or ``[]`` when the - response is not a list. - """ - params: dict[str, Any] = {} - if start_unix is not None: - params["start"] = _to_iso(start_unix) - if end_unix is not None: - params["end"] = _to_iso(end_unix) - body = await self._get_json( - f"/api/v1/sessions/{session_id}/pods", - params=params or None, - ) - if isinstance(body, list): - return body - return [] - - async def list_session_events( - self, - session_id: str, - *, - start_unix: int | None = None, - end_unix: int | None = None, - limit: int = 200, - ) -> list[dict[str, Any]]: - """List a session's events via ``GET .../{id}/events``. - - Args: - session_id (str): Identifier of the session. - start_unix (int | None): Optional window start in Unix - seconds; converted to ISO-8601 for the ``start`` query. - end_unix (int | None): Optional window end in Unix seconds; - converted to ISO-8601 for the ``end`` query. - limit (int): Maximum number of events to request. - - Returns: - list[dict[str, Any]]: The ``events`` array from the - response, or ``[]`` when absent. - """ - params: dict[str, Any] = {"limit": limit} - if start_unix is not None: - params["start"] = _to_iso(start_unix) - if end_unix is not None: - params["end"] = _to_iso(end_unix) - body = await self._get_json( - f"/api/v1/sessions/{session_id}/events", - params=params, - ) - if isinstance(body, dict): - events = body.get("events") - if isinstance(events, list): - return events - return [] - - async def get_session_summary( - self, - session_id: str, - window: _MetricsWindow, - ) -> dict[str, Any]: - """Fetch a session summary via ``GET .../{id}/summary``. - - Args: - session_id (str): Identifier of the session. - window (_MetricsWindow): Explicit ``start`` / ``end`` Unix - second bounds for the summary. - - Returns: - dict[str, Any]: The summary object, or ``{}`` when the - response is not a dict. - """ - body = await self._get_json( - f"/api/v1/sessions/{session_id}/summary", - params={"start": str(window.start_unix), "end": str(window.end_unix)}, - ) - return body if isinstance(body, dict) else {} - - # -- cluster-physical proxies --------------------------------------- - - async def get_cluster_pod_metrics( - self, - namespace: str, - name: str, - window: _MetricsWindow, - *, - categories: list[str] | None = None, - step: str | None = None, - ) -> dict[str, Any]: - """GET ``/api/v1/cluster/pods/{ns}/{name}/metrics``. - - Single-pod metrics; the response nests results under ``data.pods`` - (see :func:`cluster_decoder.decode_gpu_snapshot`). - - Args: - namespace (str): Kubernetes namespace of the pod. - name (str): Pod name. - window (_MetricsWindow): Explicit ``start`` / ``end`` Unix - second bounds for the query. - categories (list[str] | None): Optional metric categories; - joined comma-separated into the ``categories`` query. - step (str | None): Optional sampling step passed through. - - Returns: - dict[str, Any]: The metrics object, or ``{}`` when the - response is not a dict. - """ - - params: dict[str, Any] = { - "start": str(window.start_unix), - "end": str(window.end_unix), - } - if categories: - params["categories"] = ",".join(categories) - if step: - params["step"] = step - body = await self._get_json( - f"/api/v1/cluster/pods/{namespace}/{name}/metrics", - params=params, - ) - return body if isinstance(body, dict) else {} - - async def get_cluster_workload_hierarchy( - self, - workload_id: str, - ) -> dict[str, Any]: - """GET ``/api/v1/cluster/workloads/{id}/hierarchy``. - - Args: - workload_id (str): Identifier of the workload whose pod / - container hierarchy is requested. - - Returns: - dict[str, Any]: The hierarchy object, or ``{}`` when the - response is not a dict. - """ - - body = await self._get_json( - f"/api/v1/cluster/workloads/{workload_id}/hierarchy", - ) - return body if isinstance(body, dict) else {} - - async def list_cluster_faults( - self, - *, - since: str | None = None, - node: str | None = None, - phase: str | None = None, - page_size: int | None = None, - ) -> list[dict[str, Any]]: - """GET ``/api/v1/cluster/faults``. - - Flattens dict / list responses so callers don't branch on - shape; the array is already paginated upstream. - - Args: - since (str | None): Optional lower time bound passed through - as the ``since`` query. - node (str | None): Optional node filter. - phase (str | None): Optional fault-phase filter. - page_size (int | None): Optional page size; stringified into - the ``page_size`` query. - - Returns: - list[dict[str, Any]]: The ``faults`` array, or ``[]`` when - absent. - """ - - params: dict[str, Any] = {} - if since: - params["since"] = since - if node: - params["node"] = node - if phase: - params["phase"] = phase - if page_size is not None: - params["page_size"] = str(page_size) - body = await self._get_json( - "/api/v1/cluster/faults", - params=params or None, - ) - if isinstance(body, dict): - faults = body.get("faults") - if isinstance(faults, list): - return faults - if isinstance(body, list): - return body - return [] - - -def _to_iso(unix_seconds: int) -> str: - """Format Unix seconds as ISO-8601 UTC for ``start`` / ``end`` query args. - - Args: - unix_seconds (int): The timestamp in Unix seconds. - - Returns: - str: The ISO-8601 UTC string for the given instant. - """ - from datetime import datetime, timezone - - return datetime.fromtimestamp(int(unix_seconds), tz=timezone.utc).isoformat() - - -# --------------------------------------------------------------------------- -# Source adapter -# --------------------------------------------------------------------------- - - -class RobustnessServerSource: - """Adapter wrapping :class:`RobustnessServerClient` as a :class:`Source`. - - Per tick fetches session-scoped data (``pods`` + ``events`` + - ``summary``) for ``ctx.shared_state.session_id`` plus cluster - ``cluster_faults``. Cluster fetches tolerate 404 / 4xx (treated as no - data); a 5xx or transport failure raises :class:`SourceUnavailable` and - fails the tick so the DegradeRouter degrades to the local probe. - """ - - name = "robustness-server" - - def __init__( - self, - client: RobustnessServerClient, - *, - metrics_window_s: int = 300, - events_limit: int = 200, - faults_lookback_s: int = 300, - faults_page_size: int = 50, - enable_cluster_faults: bool = True, - enable_cluster_pod_metrics: bool = False, - pod_metrics_categories: tuple[str, ...] = ("gpu",), - max_pods_per_tick: int = 16, - workload_uid: str = "", - ) -> None: - """Configure the source adapter's per-tick fetch behaviour. - - Args: - client (RobustnessServerClient): The HTTP client used for - all fetches. - metrics_window_s (int): Look-back window for session - metrics / summary; clamped to at least 60 seconds. - events_limit (int): Max events fetched per tick; clamped to - at least 1. - faults_lookback_s (int): Look-back for cluster faults; - clamped to at least 0. - faults_page_size (int): Faults page size; clamped to the - 1..500 range. - enable_cluster_faults (bool): Whether to fetch cluster - faults each tick. - enable_cluster_pod_metrics (bool): Whether to fan out - per-pod cluster GPU metrics; off by default due to cost. - pod_metrics_categories (tuple[str, ...]): Metric categories - requested when pod metrics are enabled. - max_pods_per_tick (int): Upper bound on pods queried per - tick; clamped to at least 1. - """ - self._client = client - self._metrics_window_s = max(60, int(metrics_window_s)) - self._events_limit = max(1, int(events_limit)) - self._faults_lookback_s = max(0, int(faults_lookback_s)) - self._faults_page_size = max(1, min(500, int(faults_page_size))) - self._enable_cluster_faults = bool(enable_cluster_faults) - # Off by default: fans out one HTTP call per pod per tick. - self._enable_cluster_pod_metrics = bool(enable_cluster_pod_metrics) - self._pod_metrics_categories = tuple(pod_metrics_categories) - self._max_pods_per_tick = max(1, int(max_pods_per_tick)) - # ``workload_uid`` opts into hierarchy-based pod discovery; empty keeps - # the ``list_session_pods`` path. - self._workload_uid = (workload_uid or "").strip() - - async def fetch(self, ctx: Any) -> SourceData: - """Collect one tick of session- and cluster-scoped data. - - Fetches pods, events and (when the window is set) a summary for - the context's session, plus best-effort cluster faults and - optional per-pod GPU metrics. Transport failures on the cluster - endpoints re-raise so the DegradeRouter can degrade. - - Args: - ctx (Any): The reactor context; must expose a session id via - ``ctx.shared_state.session_id`` and may carry - ``ctx.now_unix``. - - Returns: - SourceData: The assembled snapshot for the tick. - - Raises: - SourceUnavailable: When no session id is present, or a - cluster fetch hits a transport / 5xx failure. - """ - session_id = _extract_session_id(ctx) - if not session_id: - raise SourceUnavailable("no session_id in reactor context") - - now_unix = int(getattr(ctx, "now_unix", 0)) or 0 - window = _MetricsWindow( - start_unix=now_unix - self._metrics_window_s if now_unix else 0, - end_unix=now_unix or 0, - ) - - session_pods = await self._client.list_session_pods( - session_id, - start_unix=window.start_unix or None, - end_unix=window.end_unix or None, - ) - events = await self._client.list_session_events( - session_id, - start_unix=window.start_unix or None, - end_unix=window.end_unix or None, - limit=self._events_limit, - ) - summary: dict[str, Any] = {} - if window.start_unix and window.end_unix: - summary = await self._client.get_session_summary(session_id, window) - - # When workload_uid is set, merge cluster-hierarchy pods with the session view. - hierarchy_pods: list[dict[str, Any]] = [] - if self._workload_uid: - try: - hierarchy = await self._client.get_cluster_workload_hierarchy( - self._workload_uid, - ) - except SourceUnavailable: - raise - hierarchy_pods = _extract_hierarchy_pods(hierarchy) - merged_pods = _merge_pods(session_pods, hierarchy_pods) - - cluster_faults: list[dict[str, Any]] = [] - if self._enable_cluster_faults: - since = str(now_unix - self._faults_lookback_s) if now_unix and self._faults_lookback_s else None - try: - cluster_faults = await self._client.list_cluster_faults( - since=since, - page_size=self._faults_page_size, - ) - except SourceUnavailable: - # Transport-level failure: re-raise so DegradeRouter counts it. - raise - - local_gpu: dict[str, Any] = {} - if self._enable_cluster_pod_metrics and merged_pods and window.start_unix and window.end_unix: - local_gpu = await self._fetch_cluster_pod_metrics(merged_pods, window) - - return SourceData( - session_pods=merged_pods, - session_events=events, - session_summary=summary, - cluster_faults=cluster_faults, - local_gpu=local_gpu, - sources_used=[self.name], - ) - - async def _fetch_cluster_pod_metrics( - self, - pods: list[dict[str, Any]], - window: _MetricsWindow, - ) -> dict[str, Any]: - """Fan out cluster pod metrics across the session's pods. - - Decodes each per-pod response into the LocalProbe ``local_gpu`` - schema and merges them into one snapshot. - - Args: - pods: Session pod rows to fetch metrics for. - window: Metrics time window to request. - - Returns: - A merged ``local_gpu`` snapshot mapping (empty when no pods). - - Raises: - SourceUnavailable: On a 5xx / transport failure for any pod, so - the DegradeRouter degrades. - """ - - refs = _unique_pod_refs(pods) - if not refs: - return {} - if len(refs) > self._max_pods_per_tick: - refs = refs[: self._max_pods_per_tick] - - snapshots: list[dict[str, Any]] = [] - for ns, name in refs: - try: - metrics = await self._client.get_cluster_pod_metrics( - ns, - name, - window, - categories=list(self._pod_metrics_categories), - ) - except SourceUnavailable: - raise - decoded = decode_gpu_snapshot(metrics) - if decoded: - snapshots.append(decoded) - return merge_gpu_snapshots(snapshots) - - -def _extract_session_id(ctx: Any) -> str: - """Pull the session id out of a reactor context. - - Args: - ctx (Any): The reactor context, expected to expose - ``shared_state.session_id``. - - Returns: - str: The session id, or ``""`` when it is missing / falsy. - """ - shared = getattr(ctx, "shared_state", None) - return getattr(shared, "session_id", "") or "" - - -def _unique_pod_refs(pods: list[dict[str, Any]]) -> list[tuple[str, str]]: - """Return the distinct ``(namespace, name)`` tuples from session pods. - - Rows carry the pod under ``pod.namespace`` / ``pod.name``; a pod may - recur across open/close cycles so we collapse to the unique set to - avoid duplicating the cluster-metrics fan-out. - - Args: - pods: Session pod rows. - - Returns: - The unique ``(namespace, name)`` tuples in first-seen order. - """ - - seen: set[tuple[str, str]] = set() - out: list[tuple[str, str]] = [] - for entry in pods or []: - if not isinstance(entry, dict): - continue - pod = entry.get("pod") if isinstance(entry.get("pod"), dict) else entry - ns = str(pod.get("namespace") or "") - name = str(pod.get("name") or "") - if not ns or not name: - continue - key = (ns, name) - if key in seen: - continue - seen.add(key) - out.append(key) - return out - - -def _extract_hierarchy_pods( - hierarchy: dict[str, Any] | None, -) -> list[dict[str, Any]]: - """Pluck pod rows from a workload hierarchy response. - - Accepts ``pods`` (documented), ``children`` / ``items`` (mirrors), - and a single ``pod`` (degraded response) so an upstream schema nudge - does not silently disable multi-node fan-out. - - Args: - hierarchy: The workload hierarchy response, if any. - - Returns: - The extracted pod row dicts (empty when none found). - """ - - if not isinstance(hierarchy, dict): - return [] - for key in ("pods", "children", "items"): - rows = hierarchy.get(key) - if isinstance(rows, list): - return [r for r in rows if isinstance(r, dict)] - pod = hierarchy.get("pod") - if isinstance(pod, dict): - return [pod] - return [] - - -def _merge_pods( - session_pods: list[dict[str, Any]], - extra_pods: list[dict[str, Any]], -) -> list[dict[str, Any]]: - """Combine session_pods with hierarchy-derived pods, deduping by ref. - - Hierarchy rows are wrapped in the session-pod envelope - (``{"pod": {...}}``) so downstream consumers see a uniform shape. - Session entries win on conflicts (richer phase / role metadata). - - Args: - session_pods: Pods reported by the session source. - extra_pods: Pods derived from the workload hierarchy. - - Returns: - The merged, de-duplicated pod list. - """ - - out: list[dict[str, Any]] = list(session_pods or []) - seen: set[tuple[str, str]] = set() - for entry in out: - if not isinstance(entry, dict): - continue - pod = entry.get("pod") if isinstance(entry.get("pod"), dict) else entry - ns = str(pod.get("namespace") or "") - name = str(pod.get("name") or "") - if ns and name: - seen.add((ns, name)) - - for raw in extra_pods or []: - pod = raw.get("pod") if isinstance(raw.get("pod"), dict) else raw - if not isinstance(pod, dict): - continue - ns = str(pod.get("namespace") or "") - name = str(pod.get("name") or "") - if not ns or not name: - continue - key = (ns, name) - if key in seen: - continue - seen.add(key) - out.append({"pod": {"namespace": ns, "name": name}, "source": "hierarchy"}) - return out - - -__all__ = [ - "RobustnessServerClient", - "RobustnessServerSource", -] diff --git a/src/hyperloom/agents/robustness/tests/test_decision_action_ladder.py b/src/hyperloom/agents/robustness/tests/test_decision_action_ladder.py index 54d4dc0d9c..0c04833b10 100644 --- a/src/hyperloom/agents/robustness/tests/test_decision_action_ladder.py +++ b/src/hyperloom/agents/robustness/tests/test_decision_action_ladder.py @@ -43,14 +43,14 @@ def _sym( async def test_low_severity_yields_observation_send_message(): ladder = ActionLadder() out = await ladder.decide( - [_sym("pod_no_metrics", SymptomSeverity.LOW)], + [_sym("inbox_bloat", SymptomSeverity.LOW)], tick_index=0, now_unix=1.0, ) assert len(out.intents) == 1 assert out.intents[0].type is IntentType.SEND_MESSAGE assert out.intents[0].payload["topic"] == "observation" - assert out.findings and out.findings[0].symptom_name == "pod_no_metrics" + assert out.findings and out.findings[0].symptom_name == "inbox_bloat" async def test_medium_severity_yields_alert_only(): @@ -82,17 +82,17 @@ async def test_high_crash_emits_alert_only(): assert alert.payload["detail"]["suggestion"] == "revert" -async def test_high_cluster_fault_emits_alert_only(): - """Cluster faults are diagnostic — alert + suggestion only.""" +async def test_high_diagnostic_symptom_emits_alert_only(): + """Diagnostic symptoms are alert + suggestion only.""" ladder = ActionLadder() out = await ladder.decide( [ _sym( - "cluster_fault", + "log_error_pattern", SymptomSeverity.HIGH, - summary="cluster fault on g53", - subject={"node": "g53", "fault": "g53-gpu_ecc"}, - suggestion="drain g53", + summary="CUDA out of memory in server.log", + subject={"pattern": "CUDA out of memory"}, + suggestion="lower the batch size", ) ], tick_index=0, @@ -103,18 +103,18 @@ async def test_high_cluster_fault_emits_alert_only(): assert IntentType.ESCALATE_STRATEGY_CHANGE not in types alert = next(i for i in out.intents if i.type is IntentType.ALERT) assert alert.payload["severity"] == "high" - assert alert.payload["detail"]["suggestion"] == "drain g53" + assert alert.payload["detail"]["suggestion"] == "lower the batch size" -async def test_medium_cluster_fault_emits_alert_only(): +async def test_medium_diagnostic_symptom_emits_alert_only(): ladder = ActionLadder() out = await ladder.decide( [ _sym( - "cluster_fault", + "log_error_pattern", SymptomSeverity.MEDIUM, - summary="cluster fault on g53", - subject={"node": "g53", "fault": "g53-gpu_ecc"}, + summary="RuntimeError in server.log", + subject={"pattern": "RuntimeError"}, ) ], tick_index=0, @@ -378,7 +378,7 @@ async def test_coordinator_zombie_alert_only(): async def test_gateway_auth_outage_alert_only(): - """J1: HIGH alert — the operator has to rotate the gateway key.""" + """HIGH alert — the operator has to rotate the gateway key.""" ladder = ActionLadder() out = await ladder.decide( [ @@ -398,7 +398,7 @@ async def test_gateway_auth_outage_alert_only(): async def test_wekafs_degraded_alert_only(): - """J2: HIGH alert — operator decides wait vs remount.""" + """HIGH alert — operator decides wait vs remount.""" ladder = ActionLadder() out = await ladder.decide( [ @@ -419,7 +419,7 @@ async def test_wekafs_degraded_alert_only(): async def test_tracelens_cli_missing_alert_only(): - """J3: HIGH alert — the operator has to re-run install.sh.""" + """HIGH alert — the operator has to re-run install.sh.""" ladder = ActionLadder() out = await ladder.decide( [_sym("tracelens_cli_missing", SymptomSeverity.HIGH, evidence={"cli_names": ["a", "b"]}, subject={})], @@ -470,7 +470,7 @@ async def test_critic_runtime_stuck_alert_only(): async def test_ray_pending_starvation_alert_only(): - """F1: kernel pipeline is wedged — alert + suggestion only. + """Kernel pipeline is wedged — alert + suggestion only. The auto prune_branch was dropped.""" ladder = ActionLadder() out = await ladder.decide( @@ -505,7 +505,7 @@ async def test_geak_budget_starvation_emits_alert_plus_prune_kernel_opt(): async def test_kernel_opt_no_progress_emits_alert_plus_prune_kernel_opt(): - """F5: pipeline structurally cannot optimise — prune kernel_opt toward params/sweep.""" + """Pipeline structurally cannot optimise — prune kernel_opt toward params/sweep.""" ladder = ActionLadder() out = await ladder.decide( [_sym("kernel_opt_no_progress", SymptomSeverity.HIGH, evidence={"kernel_count": 3}, subject={})], @@ -520,7 +520,7 @@ async def test_kernel_opt_no_progress_emits_alert_plus_prune_kernel_opt(): async def test_critic_prune_stuck_falls_to_medium_alert(): - """E4 — MEDIUM severity → alert only, no destructive action.""" + """MEDIUM severity → alert only, no destructive action.""" ladder = ActionLadder() out = await ladder.decide( [_sym("critic_prune_stuck", SymptomSeverity.MEDIUM, subject={})], diff --git a/src/hyperloom/agents/robustness/tests/test_factory.py b/src/hyperloom/agents/robustness/tests/test_factory.py index 8626b51350..fa18349da8 100644 --- a/src/hyperloom/agents/robustness/tests/test_factory.py +++ b/src/hyperloom/agents/robustness/tests/test_factory.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Smoke tests for the M1 factory.""" +"""Smoke tests for the factory.""" from __future__ import annotations @@ -22,19 +22,17 @@ @pytest.mark.asyncio async def test_build_reactor_components_local_only_mode_runs_a_tick(tmp_path: Path): config = Config(session_dir=tmp_path) - config.robustness_server_url = "" bundle = build_reactor_components(config) try: ctx = ReactorContext( tick_index=0, - shared_state=SharedStateSnapshot(session_id="sess-1", crash_count=2), + shared_state=SharedStateSnapshot(crash_count=2), inbox=[], now_unix=1000.0, ) intents = await bundle.reactor.tick(ctx) assert intents assert any(i.type is IntentType.ALERT for i in intents) - assert bundle.server_client is None finally: await bundle.aclose() @@ -42,13 +40,13 @@ async def test_build_reactor_components_local_only_mode_runs_a_tick(tmp_path: Pa @pytest.mark.asyncio async def test_factory_config_map_covers_all_registry_entries(tmp_path: Path): """The factory-built classifier must resolve a config for every registry - slot: entries the factory omits (e.g. cluster_fault) fall back to the - registry default, so nothing is left unconfigured.""" + slot: entries the factory omits fall back to the registry default, so + nothing is left unconfigured.""" from hyperloom.agents.robustness.signals.classifier import _SIGNAL_REGISTRY expected_slots = {spec.config_attr for spec in _SIGNAL_REGISTRY if spec.config_attr} - config = Config(session_dir=tmp_path, robustness_server_url="") + config = Config(session_dir=tmp_path) bundle = build_reactor_components(config) try: resolved = bundle.components.classifier.signal_configs @@ -66,7 +64,7 @@ async def test_the_stall_escalation_threshold_reaches_the_signal_from_the_config In code: like every other threshold on ``Config`` it is set by whoever constructs it, not by an environment variable ``discover`` reads. """ - config = Config(session_dir=tmp_path, robustness_server_url="", agent_stall_high_after_s=7200.0) + config = Config(session_dir=tmp_path, agent_stall_high_after_s=7200.0) bundle = build_reactor_components(config) try: assert bundle.components.classifier.signal_configs["stall"].severity_high_after_s == 7200.0 @@ -74,21 +72,6 @@ async def test_the_stall_escalation_threshold_reaches_the_signal_from_the_config await bundle.aclose() -@pytest.mark.asyncio -async def test_build_reactor_components_uses_server_url_when_set(tmp_path: Path): - config = Config( - session_dir=tmp_path, - robustness_server_url="http://example.invalid:8000", - ) - bundle = build_reactor_components(config) - try: - # Assert the bundle wires the server_client when a URL is configured. - assert bundle.server_client is not None - assert bundle.server_client.base_url == "http://example.invalid:8000" - finally: - await bundle.aclose() - - # LLM RCA wiring @@ -96,7 +79,7 @@ async def test_build_reactor_components_uses_server_url_when_set(tmp_path: Path) async def test_factory_uses_noop_engine_when_credentials_missing(tmp_path: Path): from hyperloom.agents.robustness.decision.rca_engine import NoopRcaEngine - config = Config(session_dir=tmp_path, robustness_server_url="") + config = Config(session_dir=tmp_path) bundle = build_reactor_components(config) try: assert isinstance(bundle.components.rca, NoopRcaEngine) @@ -123,9 +106,7 @@ async def test_config_discover_normalizes_retired_deepseek_env(monkeypatch, tmp_ monkeypatch.delenv("CLAUDE_MODEL", raising=False) monkeypatch.delenv("ROBUSTNESS_LLM_MODEL", raising=False) monkeypatch.delenv("LLM_MODEL", raising=False) - monkeypatch.setattr("hyperloom.agents.robustness.config._probe_robustness_server", lambda: _async_value("")) - - config = await Config.discover() + config = Config.discover() # The OpenAI side is filled too, and it is checked first. assert config.llm_provider == "openai" @@ -145,9 +126,7 @@ async def test_config_discover_uses_dual_protocol_gateway_anthropic_side(monkeyp monkeypatch.delenv("_".join(("OPENAI", "API", "KEY")), raising=False) monkeypatch.delenv("OPENAI_BASE_URL", raising=False) monkeypatch.delenv("_".join(("SAFE", "API", "KEY")), raising=False) - monkeypatch.setattr("hyperloom.agents.robustness.config._probe_robustness_server", lambda: _async_value("")) - - config = await Config.discover() + config = Config.discover() assert config.llm_provider == "anthropic" assert config.llm_base_url == "https://api.deepseek.com/anthropic" @@ -171,9 +150,7 @@ async def test_config_discover_selects_anthropic_for_a_subscription_token(monkey monkeypatch.delenv("OPENAI_BASE_URL", raising=False) monkeypatch.delenv("_".join(("DEEPSEEK", "API", "KEY")), raising=False) monkeypatch.delenv("_".join(("SAFE", "API", "KEY")), raising=False) - monkeypatch.setattr("hyperloom.agents.robustness.config._probe_robustness_server", lambda: _async_value("")) - - config = await Config.discover() + config = Config.discover() assert config.llm_provider == "anthropic" assert config.llm_api_key == "" @@ -188,9 +165,7 @@ async def test_config_discover_anthropic_model_follows_claude_model(monkeypatch, monkeypatch.delenv("_".join(("OPENAI", "API", "KEY")), raising=False) monkeypatch.delenv("OPENAI_BASE_URL", raising=False) monkeypatch.delenv("_".join(("DEEPSEEK", "API", "KEY")), raising=False) - monkeypatch.setattr("hyperloom.agents.robustness.config._probe_robustness_server", lambda: _async_value("")) - - config = await Config.discover() + config = Config.discover() assert config.llm_provider == "anthropic" assert config.llm_model == "claude-opus-4-6" @@ -204,9 +179,7 @@ async def test_config_discover_openai_model_follows_codex_model(monkeypatch, tmp monkeypatch.delenv("_".join(("ANTHROPIC", "API", "KEY")), raising=False) monkeypatch.delenv("_".join(("ANTHROPIC", "AUTH", "TOKEN")), raising=False) monkeypatch.delenv("_".join(("DEEPSEEK", "API", "KEY")), raising=False) - monkeypatch.setattr("hyperloom.agents.robustness.config._probe_robustness_server", lambda: _async_value("")) - - config = await Config.discover() + config = Config.discover() assert config.llm_provider == "openai" assert config.llm_model == "gpt-5.5" @@ -221,9 +194,7 @@ async def test_config_discover_does_not_treat_gateway_key_as_official_openai(mon monkeypatch.delenv("_".join(("ANTHROPIC", "API", "KEY")), raising=False) monkeypatch.delenv("_".join(("ANTHROPIC", "AUTH", "TOKEN")), raising=False) monkeypatch.delenv("_".join(("DEEPSEEK", "API", "KEY")), raising=False) - monkeypatch.setattr("hyperloom.agents.robustness.config._probe_robustness_server", lambda: _async_value("")) - - config = await Config.discover() + config = Config.discover() assert config.llm_base_url == "" assert config.llm_api_key == "" @@ -240,7 +211,6 @@ async def test_factory_uses_llm_engine_when_credentials_present(tmp_path: Path): config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url="http://chat-server.invalid/v1", llm_api_key="secret", ) @@ -261,7 +231,6 @@ async def test_factory_uses_anthropic_engine_for_provider(tmp_path: Path, monkey monkeypatch.setattr("hyperloom.common.llm_config.anthropic_transport_ready", lambda *_a, **_kw: True) config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url="https://api.deepseek.com/anthropic", llm_api_key="secret", llm_provider="anthropic", @@ -296,7 +265,6 @@ async def test_factory_uses_anthropic_engine_for_a_subscription_token_host(tmp_p config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url=base_url, llm_api_key=api_key, llm_provider=provider, @@ -322,7 +290,6 @@ async def test_factory_falls_back_to_noop_when_the_anthropic_transport_is_unusab monkeypatch.setattr("hyperloom.common.llm_config.anthropic_transport_ready", lambda *_a, **_kw: False) config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url="", llm_api_key="", llm_provider="anthropic", @@ -341,7 +308,6 @@ async def test_factory_still_noops_when_the_openai_side_has_no_key(tmp_path: Pat config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url="http://chat-server.invalid/v1", llm_api_key="", llm_provider="openai", @@ -359,7 +325,6 @@ async def test_factory_respects_explicit_disable(tmp_path: Path): config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url="http://chat-server.invalid/v1", llm_api_key="secret", llm_rca_enabled=False, @@ -378,7 +343,6 @@ async def test_factory_respects_env_disable(monkeypatch, tmp_path: Path): monkeypatch.setenv("ROBUSTNESS_LLM_RCA_DISABLED", "1") config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url="http://chat-server.invalid/v1", llm_api_key="secret", ) @@ -396,7 +360,6 @@ async def test_factory_propagates_severity_min_config(tmp_path: Path): config = Config( session_dir=tmp_path, - robustness_server_url="", llm_base_url="http://chat-server.invalid/v1", llm_api_key="secret", llm_rca_severity_min="medium", @@ -416,19 +379,19 @@ async def test_factory_propagates_severity_min_config(tmp_path: Path): @pytest.mark.asyncio -async def test_factory_uses_quiet_fallback_when_local_probe_disabled(tmp_path: Path): +async def test_factory_uses_quiet_source_when_local_probe_disabled(tmp_path: Path): """``disable_local_probe`` swaps the LocalProbe for a quiet stub that never yields high-severity local symptoms.""" - from hyperloom.agents.robustness.factory import _QuietFallback + from hyperloom.agents.robustness.factory import _QuietSource from hyperloom.agents.robustness.sources.local_probe import LocalProbeSource config = Config(session_dir=tmp_path, disable_local_probe=True) bundle = build_reactor_components(config) try: router = bundle.components.router - fallback = router._fallback # type: ignore[attr-defined] - assert isinstance(fallback, _QuietFallback) - assert not isinstance(fallback, LocalProbeSource) - data = await fallback.fetch(None) + primary = router._primary # type: ignore[attr-defined] + assert isinstance(primary, _QuietSource) + assert not isinstance(primary, LocalProbeSource) + data = await primary.fetch(None) assert data.local_processes == [] assert data.local_server_health == [] assert data.degraded_reason and "local-probe disabled" in data.degraded_reason @@ -453,7 +416,7 @@ async def test_the_quiet_fallback_does_not_pass_its_empty_process_list_off_as_ev config = Config(session_dir=tmp_path, disable_local_probe=True) bundle = build_reactor_components(config) try: - data = await bundle.components.router._fallback.fetch(None) # type: ignore[attr-defined] + data = await bundle.components.router._primary.fetch(None) # type: ignore[attr-defined] assert data.local_processes_known is False probed = replace( data, @@ -463,7 +426,7 @@ async def test_the_quiet_fallback_does_not_pass_its_empty_process_list_off_as_ev ) ctx = ReactorContext( tick_index=0, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) @@ -475,15 +438,15 @@ async def test_the_quiet_fallback_does_not_pass_its_empty_process_list_off_as_ev @pytest.mark.asyncio -async def test_factory_default_keeps_local_probe_fallback(tmp_path: Path): +async def test_factory_default_keeps_local_probe_primary(tmp_path: Path): from hyperloom.agents.robustness.sources.local_probe import LocalProbeSource config = Config(session_dir=tmp_path) bundle = build_reactor_components(config) try: router = bundle.components.router - fallback = router._fallback # type: ignore[attr-defined] - assert isinstance(fallback, LocalProbeSource) + primary = router._primary # type: ignore[attr-defined] + assert isinstance(primary, LocalProbeSource) finally: await bundle.aclose() @@ -496,9 +459,9 @@ async def test_factory_default_auto_probes_inference_server(tmp_path: Path): config = Config(session_dir=tmp_path) bundle = build_reactor_components(config) try: - fallback = bundle.components.router._fallback # type: ignore[attr-defined] - assert isinstance(fallback, LocalProbeSource) - targets = fallback._config.health_probe_targets # type: ignore[attr-defined] + primary = bundle.components.router._primary # type: ignore[attr-defined] + assert isinstance(primary, LocalProbeSource) + targets = primary._config.health_probe_targets # type: ignore[attr-defined] assert config.inference_server_health_url in targets finally: await bundle.aclose() @@ -513,9 +476,9 @@ async def test_factory_scriptable_skips_inference_server_probe(tmp_path: Path): config = Config(session_dir=tmp_path, auto_probe_inference_server=False) bundle = build_reactor_components(config) try: - fallback = bundle.components.router._fallback # type: ignore[attr-defined] - assert isinstance(fallback, LocalProbeSource) - targets = fallback._config.health_probe_targets # type: ignore[attr-defined] + primary = bundle.components.router._primary # type: ignore[attr-defined] + assert isinstance(primary, LocalProbeSource) + targets = primary._config.health_probe_targets # type: ignore[attr-defined] assert config.inference_server_health_url not in targets finally: await bundle.aclose() @@ -537,7 +500,7 @@ async def test_a_framework_added_to_the_config_knob_is_recognised_as_a_server(tm config.server_process_patterns.append("tinyserve.entrypoint") bundle = build_reactor_components(config) try: - probe_cfg = bundle.components.router._fallback._config # type: ignore[attr-defined] + probe_cfg = bundle.components.router._primary._config # type: ignore[attr-defined] monkeypatch.setattr( local_probe.subprocess, "run", @@ -557,25 +520,3 @@ async def test_a_framework_added_to_the_config_knob_is_recognised_as_a_server(tm ] finally: await bundle.aclose() - - -@pytest.mark.asyncio -async def test_factory_forwards_multi_node_options_to_server_source(tmp_path: Path): - """``enable_cluster_pod_metrics`` / ``workload_uid`` reach the server source.""" - - config = Config( - session_dir=tmp_path, - robustness_server_url="http://example.invalid:8000", - enable_cluster_pod_metrics=True, - pod_metrics_categories=("gpu", "memory"), - workload_uid="wl-42", - ) - bundle = build_reactor_components(config) - try: - router = bundle.components.router - primary = router._primary # type: ignore[attr-defined] - assert primary._enable_cluster_pod_metrics is True # type: ignore[attr-defined] - assert primary._pod_metrics_categories == ("gpu", "memory") # type: ignore[attr-defined] - assert primary._workload_uid == "wl-42" # type: ignore[attr-defined] - finally: - await bundle.aclose() diff --git a/src/hyperloom/agents/robustness/tests/test_inference_optimizer_integration.py b/src/hyperloom/agents/robustness/tests/test_inference_optimizer_integration.py index 362e1f43d7..d1681627a7 100644 --- a/src/hyperloom/agents/robustness/tests/test_inference_optimizer_integration.py +++ b/src/hyperloom/agents/robustness/tests/test_inference_optimizer_integration.py @@ -76,7 +76,7 @@ async def _drive_reactor_with_prompt(config, prompt: str): async def test_backend_intents_pass_upstream_policy_gate(tmp_path): from hyperloom.agents.robustness.config import Config - config = Config(session_dir=tmp_path, robustness_server_url="") + config = Config(session_dir=tmp_path) intents, bundle = await _drive_reactor_with_prompt( config, "=== Shared session state ===\n" @@ -102,7 +102,7 @@ async def test_backend_intents_pass_upstream_policy_gate(tmp_path): async def test_backend_high_severity_path_passes_gate(tmp_path): from hyperloom.agents.robustness.config import Config - config = Config(session_dir=tmp_path, robustness_server_url="") + config = Config(session_dir=tmp_path) intents, bundle = await _drive_reactor_with_prompt( config, "=== Shared session state ===\n" @@ -131,7 +131,6 @@ async def test_heartbeat_passes_gate(tmp_path): # Disable auto-probe so an inert test host doesn't fire alerts that mask the heartbeat. config = Config( session_dir=tmp_path, - robustness_server_url="", auto_probe_inference_server=False, # Inert hosts have no Ray head; disable the probe so it doesn't fire alongside the heartbeat. ray_probe_enabled=False, @@ -167,7 +166,7 @@ async def test_gpu_memory_leaked_round_trips_through_upstream_policy_gate(tmp_pa ) from hyperloom.agents.robustness.sources.base import SourceData - config = Config(session_dir=tmp_path, robustness_server_url="") + config = Config(session_dir=tmp_path) bundle = build_reactor_components(config) try: classifier = bundle.components.classifier @@ -190,7 +189,7 @@ async def test_gpu_memory_leaked_round_trips_through_upstream_policy_gate(tmp_pa ) ctx_t0 = ReactorContext( tick_index=0, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) @@ -200,7 +199,7 @@ async def test_gpu_memory_leaked_round_trips_through_upstream_policy_gate(tmp_pa ctx_t1 = ReactorContext( tick_index=1, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=2.0, ) @@ -240,7 +239,7 @@ async def test_gpu_memory_leaked_silent_when_live_owner_present(tmp_path): ) from hyperloom.agents.robustness.sources.base import SourceData - config = Config(session_dir=tmp_path, robustness_server_url="") + config = Config(session_dir=tmp_path) bundle = build_reactor_components(config) try: full_with_owner = SourceData( @@ -266,7 +265,7 @@ async def test_gpu_memory_leaked_silent_when_live_owner_present(tmp_path): for tick in range(4): ctx = ReactorContext( tick_index=tick, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=float(tick + 1), ) @@ -316,7 +315,7 @@ async def test_repeated_failure_emits_prune_branch_passing_gate(tmp_path): conn.commit() conn.close() - config = Config(session_dir=tmp_path, robustness_server_url="") + config = Config(session_dir=tmp_path) intents, bundle = await _drive_reactor_with_prompt( config, "=== Shared session state ===\n" diff --git a/src/hyperloom/agents/robustness/tests/test_persistence_integration.py b/src/hyperloom/agents/robustness/tests/test_persistence_integration.py index 9ab813815a..1b576eeb38 100644 --- a/src/hyperloom/agents/robustness/tests/test_persistence_integration.py +++ b/src/hyperloom/agents/robustness/tests/test_persistence_integration.py @@ -60,7 +60,7 @@ def _fresh_classifier( "progress": ProgressConfig( gain_window_ticks=3, gain_epsilon_pct=0.1, - no_levers_min_minutes=10_000.0, # disable B3 in this test + no_levers_min_minutes=10_000.0, # disable no_levers_found in this test ), "aiter_jit": AiterJitConfig(), }, @@ -77,7 +77,6 @@ def _ctx_with_tick( return ReactorContext( tick_index=tick, shared_state=SharedStateSnapshot( - session_id="sess-int", tick=tick, cumulative_gain_validated=cumulative_gain_validated, optimization_stack_size=optimization_stack_size, diff --git a/src/hyperloom/agents/robustness/tests/test_reactor.py b/src/hyperloom/agents/robustness/tests/test_reactor.py index 5974c4bbdc..1649e01e9b 100644 --- a/src/hyperloom/agents/robustness/tests/test_reactor.py +++ b/src/hyperloom/agents/robustness/tests/test_reactor.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""End-to-end reactor tests plus L1/L2 finalizer integration. The subprocess-transport JSON-IO contract is exercised in test_runtime_cli.py.""" +"""End-to-end reactor tests plus postmortem finalizer integration. The subprocess-transport JSON-IO contract is exercised in test_runtime_cli.py.""" from __future__ import annotations @@ -105,14 +105,12 @@ def _build_reactor_with_finalizer( def _ctx( crash_count: int = 0, *, - session_id: str = "sess-1", now_unix: float = 1.0, stop_reason: str = "", ) -> ReactorContext: return ReactorContext( tick_index=0, shared_state=SharedStateSnapshot( - session_id=session_id, crash_count=crash_count, stop_reason=stop_reason, ), @@ -159,16 +157,18 @@ async def test_reactor_emits_alert_for_crash_count_and_persists_finding(tmp_path @pytest.mark.asyncio async def test_reactor_falls_back_to_secondary_when_primary_fails(tmp_path: Path): - primary = _FakeSource("server", SourceUnavailable("down")) + primary = _FakeSource("local-probe", SourceUnavailable("down")) fallback = _FakeSource( - "local", + "quiet-fallback", SourceData( - session_pods=[{"pod": {"namespace": "ns", "name": "p"}, "phase": "Failed"}], - sources_used=["local"], + local_log_errors=[{"pattern": "CUDA out of memory", "line": "boom"}], + sources_used=["quiet-fallback"], ), ) reactor, _ = _build_reactor(primary=primary, fallback=fallback, tmp_path=tmp_path) intents = await reactor.tick(_ctx()) + assert primary.calls == 1 + assert fallback.calls == 1 assert any(i.type is IntentType.ALERT for i in intents) assert any(i.payload.get("severity") == "high" for i in intents if i.type is IntentType.ALERT) diff --git a/src/hyperloom/agents/robustness/tests/test_role_prompt_inputs.py b/src/hyperloom/agents/robustness/tests/test_role_prompt_inputs.py index c632ddf00c..1b9fd2d9af 100644 --- a/src/hyperloom/agents/robustness/tests/test_role_prompt_inputs.py +++ b/src/hyperloom/agents/robustness/tests/test_role_prompt_inputs.py @@ -29,7 +29,6 @@ def _prompt(shared: str, inbox: str, *, time_budget: str | None = None) -> str: def test_empty_prompt_returns_empty_context(): ctx = from_coordinator_prompt("") assert isinstance(ctx, ReactorContext) - assert ctx.shared_state.session_id == "" assert ctx.inbox == [] assert ctx.parse_warnings == ["empty prompt"] @@ -41,7 +40,7 @@ def test_no_new_messages_yields_empty_inbox(): session_id=sess-1 model=qwen3-8b class=qwen3 baseline_tput=10.5 baseline_acc=0.8 - cumulative_gain=12.5% + cumulative_gain_validated=12.5% crash_count=0 current_action=(idle) """ @@ -56,11 +55,10 @@ def test_no_new_messages_yields_empty_inbox(): ctx = from_coordinator_prompt(prompt, tick_index=3, now_unix=100.0) assert ctx.tick_index == 3 assert ctx.now_unix == 100.0 - assert ctx.shared_state.session_id == "sess-1" assert ctx.shared_state.model_name == "qwen3-8b" assert ctx.shared_state.model_class == "qwen3" assert ctx.shared_state.baseline_tput == 10.5 - assert ctx.shared_state.cumulative_gain == 12.5 + assert ctx.shared_state.cumulative_gain_validated == 12.5 assert ctx.shared_state.crash_count == 0 assert ctx.shared_state.current_action == "" assert ctx.inbox == [] @@ -88,7 +86,6 @@ def test_inbox_parses_multiple_messages_with_python_repr_payload(): ), ) ctx = from_coordinator_prompt(prompt) - assert ctx.shared_state.session_id == "sess-2" assert ctx.shared_state.model_name == "" assert ctx.shared_state.crash_count == 2 assert ctx.shared_state.current_action == "baseline" @@ -102,6 +99,35 @@ def test_inbox_parses_multiple_messages_with_python_repr_payload(): assert ctx.parse_warnings == [] +def test_inbox_parses_per_topic_summary_fields_without_a_payload(): + """``delegated_result`` renders ``kind=/state=/error=`` and no ``payload=``; + the repeated-failure signal reads those keys, so they must survive.""" + prompt = _prompt( + "session_id=sess-2b\ncrash_count=0\n", + textwrap.dedent( + """\ + === Inbox for robustness (newest last) === + seq=1 msg_id=abc from=coordinator topic=delegated_result kind='baseline' state='succeeded' gain=4.875 kept=True + seq=2 from=coordinator topic=delegated_result kind='explore' state='failed' error='exited -8: run_1stage = False, k=v inside' + seq=3 msg_id=ghi from=coordinator topic=observation kind='retry' payload={'kind': 'retry', 'task_id': 't-1'} + """ + ), + ) + ctx = from_coordinator_prompt(prompt) + first, second, third = ctx.inbox + assert first.payload == {"kind": "baseline", "state": "succeeded", "gain": 4.875, "kept": True} + # A quoted value owns its inner ``k=v``; splitting there would corrupt the + # error text and invent a bogus ``k`` key. + assert second.msg_id == "" + assert second.payload == { + "kind": "explore", + "state": "failed", + "error": "exited -8: run_1stage = False, k=v inside", + } + assert third.payload == {"kind": "retry", "task_id": "t-1"} + assert ctx.parse_warnings == [] + + def test_unparsable_inbox_line_is_warned_not_raised(): prompt = _prompt( "session_id=sess-3\ncrash_count=0\n", @@ -141,15 +167,14 @@ def test_payload_with_non_dict_repr_is_preserved_as_raw(): def test_kb_section_is_ignored_for_robustness_role(): prompt = ( "=== Shared session state ===\n" - "session_id=sess-kb\n" - "crash_count=0\n" + "crash_count=4\n" "=== Knowledge base hints ===\n" "kb-hint-do-not-parse\n" "=== Inbox for robustness ===\n" "(no new messages)\n" ) ctx = from_coordinator_prompt(prompt) - assert ctx.shared_state.session_id == "sess-kb" + assert ctx.shared_state.crash_count == 4 assert ctx.inbox == [] diff --git a/src/hyperloom/agents/robustness/tests/test_runtime_cli.py b/src/hyperloom/agents/robustness/tests/test_runtime_cli.py index b642f90783..dc7bfc8813 100644 --- a/src/hyperloom/agents/robustness/tests/test_runtime_cli.py +++ b/src/hyperloom/agents/robustness/tests/test_runtime_cli.py @@ -88,20 +88,48 @@ async def test_run_tick_emits_alert_on_high_crash_count(tmp_path: Path): @pytest.mark.asyncio -async def test_run_tick_propagates_session_id_when_prompt_lacks_it(tmp_path: Path): +async def test_run_tick_hands_the_parsed_snapshot_to_the_reactor(tmp_path: Path, monkeypatch): + """Every parsed shared-state field must survive the trip into the reactor.""" + captured: dict[str, object] = {} + + async def _capture_tick(_self, ctx): + captured["ctx"] = ctx + return [] + + from hyperloom.agents.robustness.role.reactor import Reactor + + monkeypatch.setattr(Reactor, "tick", _capture_tick, raising=True) + from hyperloom.agents.robustness.runtime.cli import _coerce_request, _run_tick - prompt = "=== Shared session state ===\ncrash_count=0\n=== Inbox for robustness ===\n(no new messages)\n" + prompt = ( + "=== Time budget ===\n" + "elapsed=116.0min remaining=4.0min budget=120min closing_phase=False\n" + "=== Shared session state ===\n" + "tick=42\n" + "stop_reason=time_exhausted\n" + "crash_count=3\n" + "=== Inbox for robustness ===\n" + "(no new messages)\n" + ) request = _coerce_request( { "kind": "coordinator_inbox", - "session_id": "sess-fallback", + "session_id": "sess-snapshot", "raw_prompt": prompt, "options": {"session_dir": str(tmp_path)}, } ) emit = await _run_tick(request) - assert emit["session_id"] == "sess-fallback" + + assert emit["session_id"] == "sess-snapshot" + snap = captured["ctx"].shared_state # type: ignore[union-attr] + assert snap.tick == 42 + assert snap.stop_reason == "time_exhausted" + assert snap.crash_count == 3 + assert snap.budget_minutes == 120.0 + assert snap.remaining_minutes == 4.0 + assert snap.elapsed_minutes == 116.0 def test_coerce_request_rejects_bad_kind(): @@ -220,24 +248,15 @@ def test_subprocess_tick_help_smoke(): # --------------------------------------------------------------------------- -# M2 multi-node options plumbing +# Multi-node options plumbing # --------------------------------------------------------------------------- @pytest.mark.asyncio async def test_run_tick_applies_multi_node_options(tmp_path: Path, monkeypatch): """``request.options`` overrides land on the per-tick :class:`Config`.""" - # Drop workload-uid env so the only non-default Config value comes from options. - for key in ( - "ROBUSTNESS_WORKLOAD_UID", - "CLAW_WORKLOAD_UID", - "WORKLOAD_UID", - "KUBE_WORKLOAD_UID", - "RAY_JOB_ID", - "ROBUSTNESS_DISABLE_LOCAL_PROBE", - "ROBUSTNESS_ENABLE_CLUSTER_POD_METRICS", - "ROBUSTNESS_NODES", - ): + # Drop the env so the only non-default Config value comes from options. + for key in ("ROBUSTNESS_DISABLE_LOCAL_PROBE", "ROBUSTNESS_NODES"): monkeypatch.delenv(key, raising=False) captured: dict[str, object] = {} @@ -266,11 +285,7 @@ async def _zero_tick(_self, _ctx): **_REQUEST_HEARTBEAT, "options": { "session_dir": str(tmp_path), - "robustness_server_url": "", "disable_local_probe": True, - "enable_cluster_pod_metrics": True, - "pod_metrics_categories": "gpu,memory", - "workload_uid": "wl-123", "nodes": 4, }, } @@ -279,9 +294,6 @@ async def _zero_tick(_self, _ctx): config = captured["config"] assert config.disable_local_probe is True - assert config.enable_cluster_pod_metrics is True - assert config.pod_metrics_categories == ("gpu", "memory") - assert config.workload_uid == "wl-123" assert config.nodes == 4 @@ -302,6 +314,9 @@ class _StubRca: def drain_usage(self): return usage + async def aclose(self): + return None + bundle.components.rca = _StubRca() return bundle @@ -318,7 +333,7 @@ async def _zero_tick(_self, _ctx): request = _coerce_request( { **_REQUEST_HEARTBEAT, - "options": {"session_dir": str(tmp_path), "robustness_server_url": ""}, + "options": {"session_dir": str(tmp_path)}, } ) emit = await _run_tick(request) @@ -348,7 +363,7 @@ async def _zero_tick(_self, _ctx): request = _coerce_request( { **_REQUEST_HEARTBEAT, - "options": {"session_dir": str(tmp_path), "robustness_server_url": ""}, + "options": {"session_dir": str(tmp_path)}, } ) emit = await _run_tick(request) diff --git a/src/hyperloom/agents/robustness/tests/test_signals.py b/src/hyperloom/agents/robustness/tests/test_signals.py index a5b7561abe..c4b2ed746e 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals.py +++ b/src/hyperloom/agents/robustness/tests/test_signals.py @@ -18,12 +18,10 @@ SymptomSeverity, evaluate_crash_signals, evaluate_event_signals, - evaluate_health_signals, evaluate_stall_signals, ) from hyperloom.agents.robustness.signals.crash import CrashConfig from hyperloom.agents.robustness.signals.event import EventConfig -from hyperloom.agents.robustness.signals.health import HealthConfig from hyperloom.agents.robustness.signals.stall import StallConfig from hyperloom.agents.robustness.sources.base import SourceData @@ -38,7 +36,6 @@ def _ctx( return ReactorContext( tick_index=1, shared_state=SharedStateSnapshot( - session_id="sess-1", crash_count=crash_count, current_action=current_action, ), @@ -423,81 +420,6 @@ def test_event_handles_combined_inbox_and_coordinator_events(): assert any(s.name == "repeated_policy_denied" for s in out) -# Health - - -def test_health_flags_failed_pod_as_high_severity(): - data = SourceData( - session_pods=[ - { - "pod": {"namespace": "ns", "name": "brain-0"}, - "role": "brain", - "phase": "Failed", - "assignment_id": "a1", - }, - ], - ) - out = evaluate_health_signals(_ctx(), data) - assert out and out[0].severity is SymptomSeverity.HIGH - assert out[0].name == "pod_not_running" - - -def test_health_flags_unknown_phase_as_medium(): - data = SourceData( - session_pods=[ - { - "pod": {"namespace": "ns", "name": "hands-0"}, - "role": "hands", - "phase": "CrashLoopBackOff", - }, - ], - ) - out = evaluate_health_signals(_ctx(), data) - assert out and out[0].severity is SymptomSeverity.MEDIUM - - -def test_health_silent_when_pods_running(): - data = SourceData( - session_pods=[{"pod": {"namespace": "ns", "name": "p"}, "phase": "Running"}], - ) - assert evaluate_health_signals(_ctx(), data) == [] - - -def test_health_warns_no_metrics_after_threshold(): - now = 10_000.0 - data = SourceData( - session_summary={ - "pods": [ - { - "pod": {"namespace": "ns", "name": "p1"}, - "role": "hands", - "t_start": now - 1200.0, - "available_metrics": [], - } - ] - } - ) - out = evaluate_health_signals(_ctx(now_unix=now), data, config=HealthConfig(no_metrics_warn_s=600)) - assert any(s.name == "pod_no_metrics" for s in out) - - -def test_health_does_not_flag_recently_started_pods_with_no_metrics(): - now = 10_000.0 - data = SourceData( - session_summary={ - "pods": [ - { - "pod": {"namespace": "ns", "name": "p2"}, - "role": "hands", - "t_start": now - 60.0, - "available_metrics": [], - } - ] - } - ) - assert evaluate_health_signals(_ctx(now_unix=now), data, config=HealthConfig(no_metrics_warn_s=600)) == [] - - # Classifier @@ -544,16 +466,12 @@ def test_classifier_runs_all_default_rules(): for i in range(3) ] ctx = _ctx(crash_count=2, inbox=inbox) - data = SourceData( - session_pods=[{"pod": {"namespace": "ns", "name": "p"}, "phase": "Failed"}], - coordinator_events=[], - ) + data = SourceData(coordinator_events=[]) classifier = Classifier(configs={"crash": CrashConfig(medium_threshold=2)}) out = classifier.classify(data, ctx) names = {s.name for s in out} assert "crash_count_rising" in names assert "repeated_policy_denied" in names - assert "pod_not_running" in names # --------------------------------------------------------------------------- @@ -571,10 +489,8 @@ def test_signal_registry_order_is_pinned(): "stall", "crash", "event", - "health", "local_health", "gpu_leak", - "cluster_fault", "budget", "phase_budget", "conversation_progress", diff --git a/src/hyperloom/agents/robustness/tests/test_signals_aiter_jit.py b/src/hyperloom/agents/robustness/tests/test_signals_aiter_jit.py index 8169cf228a..99bfc29293 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_aiter_jit.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_aiter_jit.py @@ -20,7 +20,7 @@ def _ctx(tick: int = 0) -> ReactorContext: return ReactorContext( tick_index=tick, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) diff --git a/src/hyperloom/agents/robustness/tests/test_signals_cluster_fault.py b/src/hyperloom/agents/robustness/tests/test_signals_cluster_fault.py deleted file mode 100644 index ed4e645ab2..0000000000 --- a/src/hyperloom/agents/robustness/tests/test_signals_cluster_fault.py +++ /dev/null @@ -1,148 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Unit tests for ``signals/cluster_fault.py``.""" - -from __future__ import annotations - - -from hyperloom.agents.robustness.role.prompt_inputs import ( - ReactorContext, - SharedStateSnapshot, -) -from hyperloom.agents.robustness.signals import SymptomSeverity -from hyperloom.agents.robustness.signals.cluster_fault import ( - ClusterFaultConfig, - evaluate_cluster_fault_signals, -) -from hyperloom.agents.robustness.sources.base import SourceData - - -def _ctx() -> ReactorContext: - return ReactorContext( - tick_index=1, - shared_state=SharedStateSnapshot(session_id="sess-1"), - inbox=[], - now_unix=1_700_000_000.0, - ) - - -def _fault( - *, - phase: str = "Isolating", - name: str = "g53-gpu_ecc", - node: str = "g53", - monitor_id: str = "gpu_ecc", - affected_workloads: int = 1, - affected_gpus: int = 1, - auto_repair: bool = True, -) -> dict: - return { - "name": name, - "monitor_id": monitor_id, - "node_name": node, - "phase": phase, - "auto_repair": auto_repair, - "affected_workload_count": affected_workloads, - "affected_gpu_count": affected_gpus, - "action": "isolate", - "created_at": "2026-05-09T12:00:00Z", - } - - -def test_no_faults_yields_no_symptoms(): - out = evaluate_cluster_fault_signals(_ctx(), SourceData(cluster_faults=[])) - assert out == [] - - -def test_succeeded_phase_is_silent(): - """Auto-repair completed -> no agent action expected.""" - - data = SourceData(cluster_faults=[_fault(phase="Succeeded")]) - out = evaluate_cluster_fault_signals(_ctx(), data) - assert out == [] - - -def test_isolating_low_blast_radius_is_medium(): - data = SourceData(cluster_faults=[_fault(phase="Isolating", affected_workloads=1, affected_gpus=2)]) - out = evaluate_cluster_fault_signals(_ctx(), data) - assert len(out) == 1 - assert out[0].name == "cluster_fault" - assert out[0].severity is SymptomSeverity.MEDIUM - assert out[0].source == "server" - assert out[0].subject == {"node": "g53", "fault": "g53-gpu_ecc"} - assert out[0].evidence["phase"] == "Isolating" - assert out[0].evidence["affected_workload_count"] == 1 - - -def test_failed_phase_is_high_regardless_of_blast_radius(): - data = SourceData(cluster_faults=[_fault(phase="Failed", affected_workloads=0, affected_gpus=0)]) - out = evaluate_cluster_fault_signals(_ctx(), data) - assert len(out) == 1 - assert out[0].severity is SymptomSeverity.HIGH - assert "auto-repair failed" in out[0].suggestion.lower() - - -def test_isolating_promotes_to_high_on_workload_threshold(): - cfg = ClusterFaultConfig(high_workload_threshold=4) - data = SourceData(cluster_faults=[_fault(phase="Isolating", affected_workloads=4, affected_gpus=1)]) - out = evaluate_cluster_fault_signals(_ctx(), data, config=cfg) - assert out[0].severity is SymptomSeverity.HIGH - - -def test_isolating_promotes_to_high_on_gpu_threshold(): - cfg = ClusterFaultConfig(high_gpu_threshold=8) - data = SourceData(cluster_faults=[_fault(phase="Isolating", affected_workloads=1, affected_gpus=8)]) - out = evaluate_cluster_fault_signals(_ctx(), data, config=cfg) - assert out[0].severity is SymptomSeverity.HIGH - - -def test_unknown_phase_is_ignored(): - """Future phases the upstream might add should not throw.""" - - data = SourceData(cluster_faults=[_fault(phase="DraftAdded")]) - out = evaluate_cluster_fault_signals(_ctx(), data) - assert out == [] - - -def test_non_dict_entries_are_skipped(): - data = SourceData(cluster_faults=["not a dict", None, _fault()]) # type: ignore[list-item] - out = evaluate_cluster_fault_signals(_ctx(), data) - assert len(out) == 1 - assert out[0].evidence["fault_name"] == "g53-gpu_ecc" - - -def test_string_counts_are_coerced(): - """robust-api emits ints, but be defensive against str-typed envs.""" - - fault = _fault(phase="Isolating", affected_gpus=1) - fault["affected_workload_count"] = "5" # promotes to high via string coercion - data = SourceData(cluster_faults=[fault]) - out = evaluate_cluster_fault_signals(_ctx(), data, config=ClusterFaultConfig(high_workload_threshold=4)) - assert len(out) == 1 - assert out[0].severity is SymptomSeverity.HIGH - - -def test_multiple_faults_each_yield_one_symptom(): - data = SourceData( - cluster_faults=[ - _fault(name="g53-gpu_ecc", node="g53", phase="Isolating"), - _fault(name="g54-net_drop", node="g54", phase="Failed"), - ] - ) - out = evaluate_cluster_fault_signals(_ctx(), data) - assert len(out) == 2 - keys = {s.subject["fault"] for s in out} - assert keys == {"g53-gpu_ecc", "g54-net_drop"} - - -def test_classifier_includes_cluster_fault_rule(): - """Make sure the rule is wired into the central classifier.""" - - from hyperloom.agents.robustness.signals import Classifier - - clf = Classifier() - data = SourceData(cluster_faults=[_fault(phase="Failed", affected_workloads=0, affected_gpus=0)]) - out = clf.classify(data, _ctx()) - names = [s.name for s in out] - assert "cluster_fault" in names diff --git a/src/hyperloom/agents/robustness/tests/test_signals_critic_health.py b/src/hyperloom/agents/robustness/tests/test_signals_critic_health.py index 816f403817..46a1ddfbd6 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_critic_health.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_critic_health.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""E1 / E2 / E4 / E5 critic-health signal tests.""" +"""Critic-health signal tests.""" from __future__ import annotations @@ -21,7 +21,7 @@ def _ctx(*, inbox=None) -> ReactorContext: return ReactorContext( tick_index=1, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=list(inbox or []), now_unix=1.0, ) diff --git a/src/hyperloom/agents/robustness/tests/test_signals_decision_audit.py b/src/hyperloom/agents/robustness/tests/test_signals_decision_audit.py index 780214cf69..e8786bc412 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_decision_audit.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_decision_audit.py @@ -20,7 +20,7 @@ def _ctx() -> ReactorContext: return ReactorContext( tick_index=1, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) diff --git a/src/hyperloom/agents/robustness/tests/test_signals_external_deps.py b/src/hyperloom/agents/robustness/tests/test_signals_external_deps.py index 8234a34715..e0dbbb2e86 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_external_deps.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_external_deps.py @@ -232,7 +232,7 @@ def test_j3_silent_when_any_cli_present(): def test_j3_silent_without_latch_passed(): - """When the caller doesn't supply a latch, J3 is skipped entirely.""" + """When the caller doesn't supply a latch, the TraceLens rule is skipped entirely.""" data = SourceData( local_external_deps={ "tracelens_cli": {"any_present": False, "found": {"a": False}}, diff --git a/src/hyperloom/agents/robustness/tests/test_signals_gpu_leak.py b/src/hyperloom/agents/robustness/tests/test_signals_gpu_leak.py index 5a250bd6f4..d6537b6525 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_gpu_leak.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_gpu_leak.py @@ -24,7 +24,7 @@ def _ctx(tick: int = 0) -> ReactorContext: return ReactorContext( tick_index=tick, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) diff --git a/src/hyperloom/agents/robustness/tests/test_signals_kernel_pipeline.py b/src/hyperloom/agents/robustness/tests/test_signals_kernel_pipeline.py index 0e89a44af7..a2b803f303 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_kernel_pipeline.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_kernel_pipeline.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""F1-F5 kernel-pipeline health signal tests.""" +"""Kernel-pipeline health signal tests.""" from __future__ import annotations diff --git a/src/hyperloom/agents/robustness/tests/test_signals_local_health.py b/src/hyperloom/agents/robustness/tests/test_signals_local_health.py index ca88b34a4e..c7121cc1cc 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_local_health.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_local_health.py @@ -27,7 +27,7 @@ def _ctx() -> ReactorContext: return ReactorContext( tick_index=0, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) diff --git a/src/hyperloom/agents/robustness/tests/test_signals_preflight.py b/src/hyperloom/agents/robustness/tests/test_signals_preflight.py index 3e070310ec..4b57bc18c2 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_preflight.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_preflight.py @@ -38,7 +38,6 @@ def _ctx( return ReactorContext( tick_index=tick, shared_state=SharedStateSnapshot( - session_id="sess-1", budget_minutes=budget_minutes, remaining_minutes=remaining_minutes, closing_phase=closing_phase, diff --git a/src/hyperloom/agents/robustness/tests/test_signals_progress.py b/src/hyperloom/agents/robustness/tests/test_signals_progress.py index 25e3b709a7..d6e4d19324 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_progress.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_progress.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Unit tests for ``gain_plateau`` and ``no_levers_found`` signals (B2 / B3).""" +"""Unit tests for the ``gain_plateau`` and ``no_levers_found`` signals.""" from __future__ import annotations @@ -28,7 +28,6 @@ def _ctx( explore_started: bool = False, ) -> ReactorContext: snap = SharedStateSnapshot( - session_id="sess-1", tick=tick, macro_cycle=macro_cycle, cumulative_gain_validated=cumulative_gain_validated, diff --git a/src/hyperloom/agents/robustness/tests/test_signals_repeated_payload.py b/src/hyperloom/agents/robustness/tests/test_signals_repeated_payload.py index 5e91c35172..5d3509e6ce 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_repeated_payload.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_repeated_payload.py @@ -1,7 +1,7 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT -"""Unit tests for the ``same_payload_loop`` signal (B1).""" +"""Unit tests for the ``same_payload_loop`` signal.""" from __future__ import annotations @@ -21,7 +21,7 @@ def _ctx() -> ReactorContext: return ReactorContext( tick_index=1, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) @@ -205,7 +205,7 @@ def test_inbox_and_coordinator_events_combined(): ] ctx = ReactorContext( tick_index=1, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=inbox, now_unix=1.0, ) diff --git a/src/hyperloom/agents/robustness/tests/test_signals_state_integrity.py b/src/hyperloom/agents/robustness/tests/test_signals_state_integrity.py index 54b707fd9e..5a9bf31c88 100644 --- a/src/hyperloom/agents/robustness/tests/test_signals_state_integrity.py +++ b/src/hyperloom/agents/robustness/tests/test_signals_state_integrity.py @@ -20,7 +20,7 @@ def _ctx(*, now_unix: float = 2_000_000.0) -> ReactorContext: return ReactorContext( tick_index=1, - shared_state=SharedStateSnapshot(session_id="sess-1"), + shared_state=SharedStateSnapshot(), inbox=[], now_unix=now_unix, ) diff --git a/src/hyperloom/agents/robustness/tests/test_sources_base.py b/src/hyperloom/agents/robustness/tests/test_sources_base.py index 0a82d1ea8b..d0f1d7bf64 100644 --- a/src/hyperloom/agents/robustness/tests/test_sources_base.py +++ b/src/hyperloom/agents/robustness/tests/test_sources_base.py @@ -53,7 +53,7 @@ async def fetch(self, ctx: object) -> SourceData: def _data(label: str) -> SourceData: - return SourceData(session_summary={"from": label}) + return SourceData(local_gpu={"from": label}) @pytest.mark.asyncio @@ -65,7 +65,7 @@ async def test_router_happy_path_uses_primary_only(): for _ in range(3): snap = await router.collect(ctx=None) assert snap.sources_used == ["server"] - assert snap.session_summary == {"from": "server"} + assert snap.local_gpu == {"from": "server"} assert snap.degraded_reason is None assert primary.calls == 3 assert fallback.calls == 0 diff --git a/src/hyperloom/agents/robustness/tests/test_sources_cluster_decoder.py b/src/hyperloom/agents/robustness/tests/test_sources_cluster_decoder.py deleted file mode 100644 index 3e926fc7ff..0000000000 --- a/src/hyperloom/agents/robustness/tests/test_sources_cluster_decoder.py +++ /dev/null @@ -1,304 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Unit tests for ``sources/cluster_decoder.py`` (M2.5).""" - -from __future__ import annotations - -from hyperloom.agents.robustness.sources.cluster_decoder import ( - decode_gpu_snapshot, - merge_gpu_snapshots, -) - - -def _series(labels, values): - return {"labels": labels, "values": values} - - -def _result(name, *, category="gpu", unit="C", series=()): - return { - "name": name, - "category": category, - "unit": unit, - "series": list(series), - } - - -def _pod(ns, name, *, results=()): - return {"namespace": ns, "name": name, "results": list(results)} - - -def _response(*, pods=()): - return {"data": {"pods": list(pods)}} - - -def test_decode_returns_empty_on_garbage(): - assert decode_gpu_snapshot(None) == {} - assert decode_gpu_snapshot({}) == {} - assert decode_gpu_snapshot({"data": {}}) == {} - assert decode_gpu_snapshot({"data": {"pods": "wat"}}) == {} - - -def test_decode_picks_latest_value_per_metric(): - """Multiple samples per series -> only the latest one wins.""" - - response = _response( - pods=[ - _pod( - "ns1", - "podA", - results=[ - _result( - "rocm_temperature_celsius", - series=[ - _series( - {"gpu": "0"}, - [ - {"timestamp": 1, "value": 70.0}, - {"timestamp": 5, "value": 95.0}, - {"timestamp": 3, "value": 80.0}, - ], - ) - ], - ) - ], - ) - ] - ) - out = decode_gpu_snapshot(response) - assert out["tool"] == "robust-api" - assert len(out["gpus"]) == 1 - snap = out["gpus"][0] - assert snap["gpu_id"] == 0 - assert snap["temperature_c"] == 95.0 - assert snap["pod_namespace"] == "ns1" - assert snap["pod_name"] == "podA" - - -def test_decode_merges_metrics_per_gpu(): - """Same gpu_id, multiple metric kinds -> single snapshot row.""" - - response = _response( - pods=[ - _pod( - "ns1", - "podA", - results=[ - _result( - "rocm_temperature_celsius", - series=[ - _series( - {"gpu": "0"}, - [{"timestamp": 10, "value": 92.0}], - ) - ], - ), - _result( - "rocm_gpu_utilization", - series=[ - _series( - {"gpu": "0"}, - [{"timestamp": 10, "value": 87.5}], - ) - ], - ), - _result( - "rocm_power_average_watts", - series=[ - _series( - {"gpu": "0"}, - [{"timestamp": 10, "value": 250.0}], - ) - ], - ), - ], - ) - ] - ) - out = decode_gpu_snapshot(response) - assert len(out["gpus"]) == 1 - snap = out["gpus"][0] - assert snap["temperature_c"] == 92.0 - assert snap["util_gpu_pct"] == 87.5 - assert snap["power_watts"] == 250.0 - - -def test_decode_handles_dcgm_metric_names(): - response = _response( - pods=[ - _pod( - "ns1", - "gpu-pod", - results=[ - _result( - "DCGM_FI_DEV_GPU_TEMP", - series=[ - _series( - {"DCGM_FI_DRIVER_DEVICE_ID": "3"}, - [{"timestamp": 50, "value": 88.0}], - ) - ], - ) - ], - ) - ] - ) - out = decode_gpu_snapshot(response) - assert out["gpus"][0]["gpu_id"] == 3 - assert out["gpus"][0]["temperature_c"] == 88.0 - - -def test_decode_keeps_string_id_when_label_is_not_numeric(): - response = _response( - pods=[ - _pod( - "ns1", - "p1", - results=[ - _result( - "rocm_temperature_celsius", - series=[ - _series( - {"device": "amdgpu0"}, - [{"timestamp": 1, "value": 60.0}], - ) - ], - ) - ], - ) - ] - ) - out = decode_gpu_snapshot(response) - assert out["gpus"][0]["gpu_id"] == "amdgpu0" - - -def test_decode_distinguishes_pods_with_overlapping_gpu_ids(): - response = _response( - pods=[ - _pod( - "ns1", - "podA", - results=[ - _result( - "rocm_temperature_celsius", - series=[ - _series( - {"gpu": "0"}, - [{"timestamp": 1, "value": 80.0}], - ) - ], - ) - ], - ), - _pod( - "ns1", - "podB", - results=[ - _result( - "rocm_temperature_celsius", - series=[ - _series( - {"gpu": "0"}, - [{"timestamp": 1, "value": 95.0}], - ) - ], - ) - ], - ), - ] - ) - out = decode_gpu_snapshot(response) - assert len(out["gpus"]) == 2 - by_pod = {r["pod_name"]: r["temperature_c"] for r in out["gpus"]} - assert by_pod == {"podA": 80.0, "podB": 95.0} - - -def test_decode_skips_unknown_metrics(): - response = _response( - pods=[ - _pod( - "ns1", - "podA", - results=[ - _result( - "vendor_specific_unknown_metric", - series=[_series({"gpu": "0"}, [{"timestamp": 1, "value": 1.0}])], - ) - ], - ) - ] - ) - assert decode_gpu_snapshot(response) == {} - - -def test_decode_skips_series_without_values(): - response = _response( - pods=[ - _pod( - "ns1", - "podA", - results=[ - _result( - "rocm_temperature_celsius", - series=[_series({"gpu": "0"}, [])], - ) - ], - ) - ] - ) - assert decode_gpu_snapshot(response) == {} - - -def test_merge_combines_multiple_pod_snapshots_in_stable_order(): - snap1 = decode_gpu_snapshot( - _response( - pods=[ - _pod( - "ns2", - "podZ", - results=[ - _result( - "rocm_temperature_celsius", - series=[ - _series( - {"gpu": "0"}, - [{"timestamp": 1, "value": 80.0}], - ) - ], - ) - ], - ) - ] - ) - ) - snap2 = decode_gpu_snapshot( - _response( - pods=[ - _pod( - "ns1", - "podA", - results=[ - _result( - "rocm_temperature_celsius", - series=[ - _series( - {"gpu": "0"}, - [{"timestamp": 1, "value": 90.0}], - ) - ], - ) - ], - ) - ] - ) - ) - merged = merge_gpu_snapshots([snap1, snap2]) - assert merged["tool"] == "robust-api" - keys = [(g["pod_namespace"], g["pod_name"], g["gpu_id"]) for g in merged["gpus"]] - # Merged output is sorted by (namespace, pod, gpu_id) regardless of decode order. - assert keys == [("ns1", "podA", 0), ("ns2", "podZ", 0)] - - -def test_merge_returns_empty_when_no_snapshots_have_gpus(): - assert merge_gpu_snapshots([]) == {} - assert merge_gpu_snapshots([{}, {"tool": "x"}, {"gpus": []}]) == {} diff --git a/src/hyperloom/agents/robustness/tests/test_sources_server_client.py b/src/hyperloom/agents/robustness/tests/test_sources_server_client.py deleted file mode 100644 index ef02c900c2..0000000000 --- a/src/hyperloom/agents/robustness/tests/test_sources_server_client.py +++ /dev/null @@ -1,750 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Unit tests for the robustness-server client + Source adapter.""" - -from __future__ import annotations - - -import httpx -import pytest - -from hyperloom.agents.robustness.role.prompt_inputs import ( - ReactorContext, - SharedStateSnapshot, -) -from hyperloom.agents.robustness.sources.base import SourceUnavailable -from hyperloom.agents.robustness.sources.server_client import ( - RobustnessServerClient, - RobustnessServerSource, -) - - -def _ctx( - *, - session_id: str = "sess-1", - now_unix: float = 1_700_000_000.0, -) -> ReactorContext: - return ReactorContext( - tick_index=1, - shared_state=SharedStateSnapshot(session_id=session_id), - inbox=[], - now_unix=now_unix, - ) - - -def _client(handler) -> RobustnessServerClient: - transport = httpx.MockTransport(handler) - http = httpx.AsyncClient(base_url="http://server.test", transport=transport, timeout=5.0) - return RobustnessServerClient("http://server.test", client=http) - - -@pytest.mark.asyncio -async def test_list_session_events_unwraps_envelope(): - def handler(request: httpx.Request) -> httpx.Response: - return httpx.Response(200, json={"events": [{"id": 1}, {"id": 2}]}) - - client = _client(handler) - try: - events = await client.list_session_events("sess-1") - finally: - await client.aclose() - assert events == [{"id": 1}, {"id": 2}] - - -@pytest.mark.asyncio -async def test_5xx_raises_source_unavailable(): - def handler(request: httpx.Request) -> httpx.Response: - return httpx.Response(503, json={"detail": "down"}) - - client = _client(handler) - try: - with pytest.raises(SourceUnavailable): - await client.list_session_pods("sess-1") - finally: - await client.aclose() - - -@pytest.mark.asyncio -async def test_connect_error_raises_source_unavailable(): - def handler(request: httpx.Request) -> httpx.Response: - raise httpx.ConnectError("boom", request=request) - - client = _client(handler) - try: - with pytest.raises(SourceUnavailable): - await client.list_session_pods("sess-1") - finally: - await client.aclose() - - -@pytest.mark.asyncio -async def test_source_returns_pods_events_summary(): - requests_seen: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - requests_seen.append(str(request.url)) - if "/pods" in request.url.path: - return httpx.Response(200, json=[{"pod": {"name": "brain-0"}}]) - if "/events" in request.url.path: - return httpx.Response(200, json={"events": [{"kind": "ping"}]}) - if "/summary" in request.url.path: - return httpx.Response(200, json={"pods": [], "session": {}}) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - data = await source.fetch(_ctx()) - finally: - await client.aclose() - assert data.session_pods == [{"pod": {"name": "brain-0"}}] - assert data.session_events == [{"kind": "ping"}] - assert data.session_summary == {"pods": [], "session": {}} - assert data.sources_used == ["robustness-server"] - assert any("/pods" in u for u in requests_seen) - assert any("/events" in u for u in requests_seen) - assert any("/summary" in u for u in requests_seen) - - -@pytest.mark.asyncio -async def test_source_raises_when_session_id_missing(): - def handler(request: httpx.Request) -> httpx.Response: - return httpx.Response(200, json=[]) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - with pytest.raises(SourceUnavailable): - await source.fetch(_ctx(session_id="")) - finally: - await client.aclose() - - -@pytest.mark.asyncio -async def test_source_propagates_5xx_as_source_unavailable(): - def handler(request: httpx.Request) -> httpx.Response: - return httpx.Response(503, json={"detail": "boom"}) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - with pytest.raises(SourceUnavailable): - await source.fetch(_ctx()) - finally: - await client.aclose() - - -@pytest.mark.asyncio -async def test_source_skips_summary_when_now_unix_is_zero(): - requested_paths: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - requested_paths.append(request.url.path) - if "/pods" in request.url.path: - return httpx.Response(200, json=[]) - if "/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - data = await source.fetch(_ctx(now_unix=0.0)) - finally: - await client.aclose() - assert "/api/v1/sessions/sess-1/summary" not in requested_paths - assert data.session_summary == {} - - -@pytest.mark.asyncio -async def test_get_cluster_pod_metrics_forwards_window_and_categories(): - seen: dict[str, str] = {} - - def handler(request: httpx.Request) -> httpx.Response: - seen["path"] = request.url.path - seen["query"] = str(request.url.query) - return httpx.Response(200, json={"data": {"pods": []}}) - - client = _client(handler) - from hyperloom.agents.robustness.sources.server_client import _MetricsWindow - - try: - body = await client.get_cluster_pod_metrics( - "ns1", - "podA", - _MetricsWindow(start_unix=100, end_unix=200), - categories=["gpu"], - step="15s", - ) - finally: - await client.aclose() - assert seen["path"] == "/api/v1/cluster/pods/ns1/podA/metrics" - assert "start=100" in seen["query"] and "end=200" in seen["query"] - assert "step=15s" in seen["query"] - assert "categories=gpu" in seen["query"] - assert body == {"data": {"pods": []}} - - -@pytest.mark.asyncio -async def test_get_cluster_workload_hierarchy_returns_dict(): - def handler(request: httpx.Request) -> httpx.Response: - return httpx.Response( - 200, - json={ - "workload_id": "wl-1", - "pods": [{"namespace": "ns1", "name": "podA"}], - }, - ) - - client = _client(handler) - try: - body = await client.get_cluster_workload_hierarchy("wl-1") - finally: - await client.aclose() - assert body == { - "workload_id": "wl-1", - "pods": [{"namespace": "ns1", "name": "podA"}], - } - - -@pytest.mark.asyncio -async def test_list_cluster_faults_forwards_filters_and_unwraps(): - seen: dict[str, str] = {} - - def handler(request: httpx.Request) -> httpx.Response: - seen["path"] = request.url.path - seen["query"] = str(request.url.query) - return httpx.Response( - 200, - json={ - "faults": [{"name": "g1-ecc", "phase": "Isolating"}], - "total_count": 1, - }, - ) - - client = _client(handler) - try: - faults = await client.list_cluster_faults(since="1700000000", node="g1", phase="Isolating", page_size=100) - finally: - await client.aclose() - assert seen["path"] == "/api/v1/cluster/faults" - assert "since=1700000000" in seen["query"] - assert "node=g1" in seen["query"] - assert "phase=Isolating" in seen["query"] - assert "page_size=100" in seen["query"] - assert faults == [{"name": "g1-ecc", "phase": "Isolating"}] - - -@pytest.mark.asyncio -async def test_list_cluster_faults_handles_bare_array_response(): - """Be tolerant of older robust-api builds that return a list.""" - - def handler(request: httpx.Request) -> httpx.Response: - return httpx.Response(200, json=[{"name": "g1-ecc"}, {"name": "g2-ecc"}]) - - client = _client(handler) - try: - faults = await client.list_cluster_faults() - finally: - await client.aclose() - assert faults == [{"name": "g1-ecc"}, {"name": "g2-ecc"}] - - -@pytest.mark.asyncio -async def test_source_fetch_populates_cluster_faults(): - """The Source adapter calls /cluster/faults and surfaces them.""" - - paths_hit: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - paths_hit.append(request.url.path) - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response(200, json=[]) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response( - 200, - json={ - "faults": [ - { - "name": "g53-gpu_ecc", - "monitor_id": "gpu_ecc", - "node_name": "g53", - "phase": "Isolating", - "auto_repair": True, - "affected_workload_count": 2, - "affected_gpu_count": 4, - } - ], - "total_count": 1, - }, - ) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - data = await source.fetch(_ctx()) - finally: - await client.aclose() - - assert "/api/v1/cluster/faults" in paths_hit - assert len(data.cluster_faults) == 1 - assert data.cluster_faults[0]["phase"] == "Isolating" - assert data.sources_used == ["robustness-server"] - - -@pytest.mark.asyncio -async def test_source_propagates_cluster_faults_5xx_as_source_unavailable(): - """A 5xx on /cluster/faults must trigger DegradeRouter, not be swallowed.""" - - def handler(request: httpx.Request) -> httpx.Response: - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response(200, json=[]) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(503, json={"detail": "robust-api down"}) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - with pytest.raises(SourceUnavailable): - await source.fetch(_ctx()) - finally: - await client.aclose() - - -@pytest.mark.asyncio -async def test_source_can_disable_cluster_faults(): - paths_hit: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - paths_hit.append(request.url.path) - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response(200, json=[]) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client, enable_cluster_faults=False) - data = await source.fetch(_ctx()) - finally: - await client.aclose() - - assert "/api/v1/cluster/faults" not in paths_hit - assert data.cluster_faults == [] - - -def _gpu_metric_response(value: float, *, gpu_id: str = "0", ts: int = 100): - """Build a robust-api-shaped pod-metrics response for one GPU.""" - - return { - "data": { - "pods": [ - { - "namespace": "ns1", - "name": "podA", - "results": [ - { - "name": "rocm_temperature_celsius", - "category": "gpu", - "unit": "C", - "series": [ - { - "labels": {"gpu": gpu_id}, - "values": [{"timestamp": ts, "value": value}], - } - ], - } - ], - } - ] - } - } - - -@pytest.mark.asyncio -async def test_source_disables_cluster_pod_metrics_by_default(): - """The fan-out costs one HTTP call per pod; default off.""" - - paths_hit: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - paths_hit.append(request.url.path) - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": "podA"}}], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - data = await source.fetch(_ctx()) - finally: - await client.aclose() - - assert not any("/cluster/pods/" in p for p in paths_hit) - assert data.local_gpu == {} - - -@pytest.mark.asyncio -async def test_source_fans_out_pod_metrics_when_enabled(): - """With enable_cluster_pod_metrics=True, fetch hits /cluster/pods/{ns}/{name}/metrics.""" - - paths_hit: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - paths_hit.append(request.url.path) - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": "podA"}}], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - if request.url.path == "/api/v1/cluster/pods/ns1/podA/metrics": - return httpx.Response(200, json=_gpu_metric_response(95.0)) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client, enable_cluster_pod_metrics=True) - data = await source.fetch(_ctx()) - finally: - await client.aclose() - - assert "/api/v1/cluster/pods/ns1/podA/metrics" in paths_hit - assert data.local_gpu["tool"] == "robust-api" - assert len(data.local_gpu["gpus"]) == 1 - assert data.local_gpu["gpus"][0]["temperature_c"] == 95.0 - assert data.local_gpu["gpus"][0]["pod_name"] == "podA" - - -@pytest.mark.asyncio -async def test_source_pod_metrics_5xx_propagates_for_degrade(): - """Transport / 5xx on cluster metrics still triggers DegradeRouter.""" - - def handler(request: httpx.Request) -> httpx.Response: - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": "podA"}}], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - if request.url.path == "/api/v1/cluster/pods/ns1/podA/metrics": - return httpx.Response(503, text="busy") - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client, enable_cluster_pod_metrics=True) - with pytest.raises(SourceUnavailable): - await source.fetch(_ctx()) - finally: - await client.aclose() - - -@pytest.mark.asyncio -async def test_source_pod_metrics_dedups_repeated_pod_refs(): - """Same pod appearing twice in session_pods should fan out once.""" - - seen_metrics_paths: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[ - {"pod": {"namespace": "ns1", "name": "podA"}}, - {"pod": {"namespace": "ns1", "name": "podA"}}, - {"pod": {"namespace": "ns1", "name": "podB"}}, - ], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - if "/api/v1/cluster/pods/" in request.url.path: - seen_metrics_paths.append(request.url.path) - return httpx.Response(200, json=_gpu_metric_response(80.0)) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client, enable_cluster_pod_metrics=True) - await source.fetch(_ctx()) - finally: - await client.aclose() - - assert sorted(seen_metrics_paths) == [ - "/api/v1/cluster/pods/ns1/podA/metrics", - "/api/v1/cluster/pods/ns1/podB/metrics", - ] - - -@pytest.mark.asyncio -async def test_source_pod_metrics_caps_fan_out_per_tick(): - """Sessions with too many pods must not blow the per-tick budget.""" - - metrics_calls = 0 - - def handler(request: httpx.Request) -> httpx.Response: - nonlocal metrics_calls - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": f"pod-{i:02d}"}} for i in range(10)], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - if "/api/v1/cluster/pods/" in request.url.path: - metrics_calls += 1 - return httpx.Response(200, json=_gpu_metric_response(80.0)) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource( - client, - enable_cluster_pod_metrics=True, - max_pods_per_tick=3, - ) - await source.fetch(_ctx()) - finally: - await client.aclose() - - assert metrics_calls == 3 - - -@pytest.mark.asyncio -async def test_server_pod_metrics_drive_local_health_gpu_signal(): - """End-to-end: server-decoded GPU >= warn threshold fires gpu_thermal_high.""" - - def handler(request: httpx.Request) -> httpx.Response: - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": "podA"}}], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - if request.url.path == "/api/v1/cluster/pods/ns1/podA/metrics": - # 95 C: warn (>= 90) but below crit (100) -> medium. - return httpx.Response(200, json=_gpu_metric_response(95.0)) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client, enable_cluster_pod_metrics=True) - data = await source.fetch(_ctx()) - finally: - await client.aclose() - - from hyperloom.agents.robustness.signals import ( - Classifier, - SymptomSeverity, - ) - - classifier = Classifier() - symptoms = classifier.classify(data, _ctx()) - thermal = [s for s in symptoms if s.name == "gpu_thermal_high"] - assert len(thermal) == 1 - assert thermal[0].severity is SymptomSeverity.MEDIUM - assert thermal[0].evidence["temperature_c"] == 95.0 - - -@pytest.mark.asyncio -async def test_source_workload_uid_merges_hierarchy_pods_into_session_pods(): - """Hierarchy-derived workers are added even when session_pods skipped them.""" - - paths_hit: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - paths_hit.append(request.url.path) - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": "head-pod"}}], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/workloads/wl-1/hierarchy": - return httpx.Response( - 200, - json={ - "workload_id": "wl-1", - "pods": [ - {"namespace": "ns1", "name": "head-pod"}, - {"namespace": "ns1", "name": "worker-0"}, - {"namespace": "ns1", "name": "worker-1"}, - ], - }, - ) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client, workload_uid="wl-1") - data = await source.fetch(_ctx()) - finally: - await client.aclose() - - assert "/api/v1/cluster/workloads/wl-1/hierarchy" in paths_hit - pod_refs = {(entry["pod"]["namespace"], entry["pod"]["name"]) for entry in data.session_pods} - assert pod_refs == { - ("ns1", "head-pod"), - ("ns1", "worker-0"), - ("ns1", "worker-1"), - } - - -@pytest.mark.asyncio -async def test_source_workload_uid_drives_multi_node_pod_metric_fan_out(): - """Cluster pod-metrics fans out across hierarchy-only pods, not just session_pods.""" - - metric_paths: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - if "/sessions/sess-1/pods" in request.url.path: - # Session only knows the head pod; workers exist only in the cluster hierarchy view. - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": "head"}}], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/workloads/wl-1/hierarchy": - return httpx.Response( - 200, - json={ - "workload_id": "wl-1", - "pods": [ - {"namespace": "ns1", "name": "head"}, - {"namespace": "ns1", "name": "worker-0"}, - ], - }, - ) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - if "/api/v1/cluster/pods/" in request.url.path: - metric_paths.append(request.url.path) - return httpx.Response(200, json=_gpu_metric_response(80.0)) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource( - client, - workload_uid="wl-1", - enable_cluster_pod_metrics=True, - ) - await source.fetch(_ctx()) - finally: - await client.aclose() - - assert sorted(metric_paths) == [ - "/api/v1/cluster/pods/ns1/head/metrics", - "/api/v1/cluster/pods/ns1/worker-0/metrics", - ] - - -@pytest.mark.asyncio -async def test_source_workload_uid_hierarchy_5xx_triggers_degrade(): - """A 5xx on the hierarchy endpoint must propagate as SourceUnavailable.""" - - def handler(request: httpx.Request) -> httpx.Response: - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response(200, json=[]) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/workloads/wl-1/hierarchy": - return httpx.Response(503, text="busy") - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client, workload_uid="wl-1") - with pytest.raises(SourceUnavailable): - await source.fetch(_ctx()) - finally: - await client.aclose() - - -@pytest.mark.asyncio -async def test_source_no_workload_uid_skips_hierarchy_call(): - """Single-node path keeps the existing list_session_pods behaviour.""" - - paths_hit: list[str] = [] - - def handler(request: httpx.Request) -> httpx.Response: - paths_hit.append(request.url.path) - if "/sessions/sess-1/pods" in request.url.path: - return httpx.Response( - 200, - json=[{"pod": {"namespace": "ns1", "name": "head"}}], - ) - if "/sessions/sess-1/events" in request.url.path: - return httpx.Response(200, json={"events": []}) - if "/sessions/sess-1/summary" in request.url.path: - return httpx.Response(200, json={}) - if request.url.path == "/api/v1/cluster/faults": - return httpx.Response(200, json={"faults": []}) - return httpx.Response(404) - - client = _client(handler) - try: - source = RobustnessServerSource(client) - data = await source.fetch(_ctx()) - finally: - await client.aclose() - - assert not any("workloads" in p for p in paths_hit) - assert len(data.session_pods) == 1 - assert data.session_pods[0]["pod"]["name"] == "head" diff --git a/src/hyperloom/common/claude_oneshot.py b/src/hyperloom/common/claude_oneshot.py index 3a3e0f2e2f..8d93deb49c 100644 --- a/src/hyperloom/common/claude_oneshot.py +++ b/src/hyperloom/common/claude_oneshot.py @@ -130,6 +130,10 @@ def ensure_available() -> None: def message_text(message: Any) -> list[str]: """Extract text fragments from one Claude SDK message. + Covers every shape the SDK emits across versions: a bare string, a ``.text`` + attribute, ``.content`` blocks exposing ``.text`` (object or dict), and a + ``ResultMessage.result`` summary string. + Args: message: A message yielded by ``claude_agent_sdk.query``. @@ -149,6 +153,9 @@ def message_text(message: Any) -> list[str]: block_text = block.get("text") if isinstance(block, dict) else getattr(block, "text", None) if isinstance(block_text, str): parts.append(block_text) + result_text = getattr(message, "result", None) + if isinstance(result_text, str) and result_text: + parts.append(result_text) return parts diff --git a/src/hyperloom/common/env_safety.py b/src/hyperloom/common/env_safety.py index c768922731..beea2721e8 100644 --- a/src/hyperloom/common/env_safety.py +++ b/src/hyperloom/common/env_safety.py @@ -10,6 +10,7 @@ from __future__ import annotations +import os import re from collections.abc import Mapping @@ -265,6 +266,10 @@ ) ) +# Env names a per-variant override may never set. Workload pins stay allowed: +# the sweep and shape-capture grids set them from code. +BLOCKED_VARIANT_ENV_NAMES: frozenset[str] = BLOCKED_UNTRUSTED_ENV_NAMES | BENCHMARK_SECRET_ENV_NAMES + # Credential-shaped name fragments, so an unlisted secret cannot be persisted # into a session YAML by name alone. _SECRET_NAME_FRAGMENTS: tuple[str, ...] = ("APIKEY", "API_KEY", "TOKEN", "SECRET", "PASSWORD", "CREDENTIAL") @@ -290,6 +295,12 @@ def is_allowed_external_env_key(key: object) -> bool: return not is_secret_shaped_env_name(upper) +def is_allowed_variant_env_key(key: object) -> bool: + """True when a per-variant env override is safe to hand a benchmark subprocess.""" + upper = str(key or "").strip().upper() + return valid_env_key(upper) and upper not in BLOCKED_VARIANT_ENV_NAMES + + def is_python_package_root(path: object) -> bool: """True when ``path`` is a ``site-packages``/``dist-packages`` dir. @@ -364,13 +375,15 @@ def scrub_benchmark_process_env(env: dict[str, str]) -> dict[str, str]: return env -def filter_benchmark_env_mapping(envs: Mapping[str, object] | None) -> dict[str, str]: - """Return benchmark env overrides without control-plane credentials.""" - return { - str(name): str(value) - for name, value in (envs or {}).items() - if str(name).strip().upper() not in BENCHMARK_SECRET_ENV_NAMES - } +def build_benchmark_env(*layers: Mapping[str, object] | None) -> dict[str, str]: + """Build a benchmark subprocess env: parent env under each layer, later winning. + + Keys are upper-cased and values stringified for the YAML's plain scalars. + """ + env = os.environ.copy() + for layer in layers: + env.update({str(key).upper(): str(value) for key, value in (layer or {}).items()}) + return scrub_benchmark_process_env(env) def redact_secret_values(text: str) -> str: @@ -386,12 +399,14 @@ def redact_secret_values(text: str) -> str: "BLOCKED_CHILD_ENV_NAMES", "BLOCKED_EXTERNAL_ENV_NAMES", "BLOCKED_UNTRUSTED_ENV_NAMES", + "BLOCKED_VARIANT_ENV_NAMES", "GPU_MASK_ENV_NAMES", - "filter_benchmark_env_mapping", + "build_benchmark_env", "filter_untrusted_env_mapping", "is_allowed_dotenv_key", "is_allowed_external_env_key", "is_allowed_kernel_agent_env_key", + "is_allowed_variant_env_key", "is_python_package_root", "is_secret_shaped_env_name", "redact_secret_values", diff --git a/src/hyperloom/common/kernel_shape_contract.py b/src/hyperloom/common/kernel_shape_contract.py index 43e941996c..31a275b208 100644 --- a/src/hyperloom/common/kernel_shape_contract.py +++ b/src/hyperloom/common/kernel_shape_contract.py @@ -11,7 +11,3 @@ # Alias used by the kernel-opt predispatch validator. ALLOWED_SHAPE_PROVENANCE = DISPATCHABLE_SHAPE_PROVENANCE - -def is_dispatchable_shape_provenance(provenance: str) -> bool: - """Return whether ``provenance`` carries dispatch-grade operand dims.""" - return provenance in DISPATCHABLE_SHAPE_PROVENANCE diff --git a/src/hyperloom/common/kernel_source_contract.py b/src/hyperloom/common/kernel_source_contract.py index 4cff89af63..3062627082 100644 --- a/src/hyperloom/common/kernel_source_contract.py +++ b/src/hyperloom/common/kernel_source_contract.py @@ -23,17 +23,16 @@ from __future__ import annotations -import json import math import os import re -from pathlib import Path from typing import Any #: Bump the major on any field removal or meaning change; consumers gate on it. SOURCE_RESOLUTION_SCHEMA_VERSION = "1.0.0" -#: Canonical artifact name, relative to the analysis run directory. +#: Canonical artifact name, relative to the analysis run directory. Mirrored by +#: ``tracelens_analysis._SOURCE_RESOLUTION_NAME`` for the standalone path. SOURCE_RESOLUTION_FILENAME = "kernel_source_resolution.json" #: How a location was decided, best evidence first. ``llm_review`` outranks the @@ -272,10 +271,3 @@ def path_is_acceptable(path: str, roots: tuple[str, ...]) -> bool: """Whether a rewriting tier may write ``path`` as a resolved location.""" return bool(canonical_source_path(path, roots)) - -def read_document(path: Path | str) -> dict[str, Any] | None: - """Load the artifact, or ``None`` when it is absent or unreadable.""" - try: - return json.loads(Path(path).read_text(encoding="utf-8")) - except (OSError, ValueError): - return None diff --git a/src/hyperloom/common/subprocess_bridge.py b/src/hyperloom/common/subprocess_bridge.py index 8f4597d24e..f15202bcc1 100644 --- a/src/hyperloom/common/subprocess_bridge.py +++ b/src/hyperloom/common/subprocess_bridge.py @@ -32,6 +32,9 @@ class RuntimeAdapterError(RuntimeError): def read_json(path: str | Path) -> Any: """Read a UTF-8 JSON file, returning ``None`` for a blank file. + Raises rather than degrading so a bridge caller exits non-zero; use + ``hyperloom.common.jsonio.read_json`` for the tolerant variant. + Args: path: Path to the JSON file to read. diff --git a/src/hyperloom/common/tests/test_kernel_shape_contract.py b/src/hyperloom/common/tests/test_kernel_shape_contract.py index 1a6862e3ce..2c9f144cd7 100644 --- a/src/hyperloom/common/tests/test_kernel_shape_contract.py +++ b/src/hyperloom/common/tests/test_kernel_shape_contract.py @@ -1,10 +1,11 @@ # SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. # SPDX-License-Identifier: MIT +"""The shape-provenance set the dispatch gate tests membership against.""" + from hyperloom.common.kernel_shape_contract import ( ALLOWED_SHAPE_PROVENANCE, DISPATCHABLE_SHAPE_PROVENANCE, - is_dispatchable_shape_provenance, ) @@ -13,8 +14,8 @@ def test_dispatchable_and_allowed_match(): def test_capture_backfill_is_dispatchable(): - assert is_dispatchable_shape_provenance("capture_backfill") + assert "capture_backfill" in DISPATCHABLE_SHAPE_PROVENANCE def test_geometry_provenance_not_dispatchable(): - assert not is_dispatchable_shape_provenance("launch_grid") + assert "launch_grid" not in DISPATCHABLE_SHAPE_PROVENANCE diff --git a/src/hyperloom/inference_optimizer/README.md b/src/hyperloom/inference_optimizer/README.md index 5ee4853c3d..908d07d27a 100644 --- a/src/hyperloom/inference_optimizer/README.md +++ b/src/hyperloom/inference_optimizer/README.md @@ -46,10 +46,13 @@ python3 -m hyperloom.inference_optimizer.cli optimize \ --max-hours 2.0 ``` -Resume an interrupted session: +Resume an interrupted session (the session dir is always named explicitly — +take it from the launch-info JSON or the `HYPERLOOM_LAUNCH` line the CLI +printed at launch): ```bash -python3 -m hyperloom.inference_optimizer.cli optimize --resume +python3 -m hyperloom.inference_optimizer.cli optimize \ + --resume-from "$SESSION_DIR" ``` See `python -m hyperloom.inference_optimizer.cli optimize --help` for the full flag set and diff --git a/src/hyperloom/inference_optimizer/SKILL.md b/src/hyperloom/inference_optimizer/SKILL.md index 034cfa82a0..916d8f1e54 100644 --- a/src/hyperloom/inference_optimizer/SKILL.md +++ b/src/hyperloom/inference_optimizer/SKILL.md @@ -56,7 +56,7 @@ $USER_DATA_PATH/ # workspace_root — set by operator / │ # Open-source deps are installed by install.sh. ├── logs/ # workspace-shared launcher stdout └── / # e.g. DeepSeek-R1-0528, deepseek-ai-DeepSeek-V3 - └── / # session_dir — manifest.json, state.json, runs/, … + └── -/ # session_dir — manifest.json, state.json, runs/, … ├── manifest.json ├── state.json ├── storage/coordinator.db @@ -73,11 +73,11 @@ $USER_DATA_PATH/ # workspace_root — set by operator / run-scoped path *before* the optimizer starts, e.g. ``/hyperloom/users//deepseek-ai-DeepSeek-V3-20260522_034024/``. That outer directory is **platform isolation** (one Claw job). The -optimizer then creates ``//`` inside it. Full +optimizer then creates ``/-/`` inside it. Full session path example:: /hyperloom/users//deepseek-ai-DeepSeek-V3-20260522_034024/ ← USER_DATA_PATH (Claw) - deepseek-ai-DeepSeek-V3/20260522T035359Z/ ← session_dir (optimizer) + deepseek-ai-DeepSeek-V3/20260522T035359Z-9f3c1a04/ ← session_dir (optimizer) ### Path resolution (do not guess) @@ -150,7 +150,7 @@ that runs **before** `python -m hyperloom.inference_optimizer.cli optimize` is e ### IR-1 — GPU MUST be unoccupied before every launch Before every `python -m hyperloom.inference_optimizer.cli optimize` invocation (fresh start OR -`--resume`), verify that every visible GPU on this pod has **zero +`--resume-from`), verify that every visible GPU on this pod has **zero foreign serving PIDs and ≲ 500 MiB VRAM in use**. A leftover `sglang.launch_server` / `vllm.entrypoints` / `Magpie` from a previous run silently degrades the next `baseline` by 5–30 % (shares VRAM + @@ -174,11 +174,10 @@ Ray head → `kernel_opt` tasks hang; missing `kernel-agent.env.sh` → first kernel-opt gateway call returns `401`. `install.sh --check-only` is a *diagnostic*, never a substitute. -**Resume carve-out.** `... optimize --resume` may skip install only when +**Resume carve-out.** `... optimize --resume-from` may skip install only when ALL hold: (1) `install.sh` exited 0 earlier in the *same shell*; (2) -`kernel-agent.env.sh` is still sourced; (3) the session being resumed is -known (explicit `--resume-from`, `$INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR`, -or launch-info JSON) and its `manifest.json` exists under that session dir. +`kernel-agent.env.sh` is still sourced; (3) `manifest.json` exists under the +session dir passed to `--resume-from`. Any failure → treat as fresh launch and re-run `install.sh`. > The in-loop equivalent is `_preflight()` steps 1–12 (drift repair, not @@ -235,10 +234,11 @@ brief: (any port that is not the production serving port 8888), profile, autotune, and run real benchmark loops. The one invariant is that they must not touch the production serving process, its cards, or port 8888. -- **IR-6 HARD force-exit**: EXPLORE exits the moment wall-clock remaining - < `--explore-force-exit-hours-remaining` (default 3.0 h) OR phase - budget < `--explore-force-exit-budget-pct` (default 20%). Non-negotiable - — leaves buffer for KERNEL_AGENT → SWEEP → CLOSE + report. +- **IR-6 HARD force-exit**: EXPLORE exits the moment the unspent fraction of + its own phase budget drops to `--explore-force-exit-budget-pct` (default + 20%). Non-negotiable. The buffer for KERNEL_AGENT → SWEEP → CLOSE + report is + already inside that fraction, since charge-back rebuilds the allotment from + the time left at phase entry; there is no session-remaining arm. - **Plateau advisory**: EXPLORE / KERNEL_AGENT / FRAMEWORK plateau signals are computed every tick and rendered as advisory in the orchestration prompt. They do NOT drive phase advance — the LLM may emit @@ -490,8 +490,8 @@ and the operator's stated value is lost: | Budget | `--max-hours` | Pass the prompt's time budget. Default `2.0`. | | Max model len | `--max-model-len` | Optional; auto-derived from ISL+OSL+headroom when omitted. | | External reference GPU | `--compare-against-gpu` | Coordinator *always* hard-gates `target_analysis` to run first so `$SESSION_DIR/target_analysis/target_baseline.json` exists before `baseline` runs. When this flag is set the JSON carries the InferenceX reference (`reason="ok"`); when unset the JSON carries a structured `reason="no_target_gpu_configured"` marker. The report renders the "External baseline" section from this JSON in both cases (heading switches to "(not requested)" for the marker variant) | -| Quantization prelude | `--quantize` | Optional. Natural-language quantization request. Runs the quantization-agent once before the loop and rewrites `--model` to the quantized model. See Step 2b. Ignored on `--resume`. | -| Env pins | `--extra-env NAME=VALUE` | Repeatable; forward **every** one verbatim as its own flag (do not drop any or fold into the `Environment:` block). The CLI persists them in `state.json` and serializes them into `$INFERENCE_OPTIMIZER_EXTRA_ENV`; a dropped pin is lost silently — e.g. a missing `SGLANG_USE_AITER=0` leaves the explore aiter-MoE filter blind. A `--resume` re-exports the persisted set, so re-pass them only to change the set. | +| Quantization prelude | `--quantize` | Optional. Natural-language quantization request. Runs the quantization-agent once before the loop and rewrites `--model` to the quantized model. See Step 2b. Never runs on a resume. | +| Env pins | `--extra-env NAME=VALUE` | Repeatable; forward **every** one verbatim as its own flag (do not drop any or fold into the `Environment:` block). The CLI persists them in `state.json` and serializes them into `$INFERENCE_OPTIMIZER_EXTRA_ENV`; a dropped pin is lost silently — e.g. a missing `SGLANG_USE_AITER=0` leaves the explore aiter-MoE filter blind. A `--resume-from` re-exports the persisted set, so re-pass them only to change the set. | ### Step 2b — Optional quantization prelude (`--quantize`) @@ -526,7 +526,7 @@ python3 -m hyperloom.inference_optimizer.cli optimize \ benchmark configs, display names, and the optimization report carry the stale operator-supplied precision label (e.g. `fp8`/`bf16`) and **mislabel** an actually-quantized model. Never leave a conflicting precision when quantizing. -- Behavior: one-shot, **skipped on `--resume`**. On a failed/unusable +- Behavior: one-shot, **never runs on a resume**. On a failed/unusable quantization the run **hard-stops (`SystemExit(3)`)** — it never silently optimizes the un-quantized source after an explicit `--quantize`. The one exception is a **pre-flight scheme/GPU mismatch** via @@ -882,9 +882,9 @@ below describe the runtime behavior operators observe. - **Interpreter switch.** A from-source build records the venv interpreter it was compiled against (`runtime_python_exe`) and emits it as the `HYPERLOOM_FRAMEWORK_PYTHON` env into the per-variant YAML benchmark envs - (`benchmark.envs`). The bypass backend launches the server via `python -m` - with that interpreter; the Magpie backend re-exports it from the YAML - benchmark envs. This guarantees the server loads the exact build. + (`benchmark.envs`). Both backends export that mapping to the server env, and + bypass additionally launches the server via `python -m` with that + interpreter. This guarantees the server loads the exact build. - **`build_budget_sec`.** Per-build-action wall-clock timeout knob; `0` selects the per-component default. @@ -1026,16 +1026,19 @@ exist + no early `stop_reason`. ## Resume Existing Session -`--resume` auto-picks the latest `$USER_DATA_PATH///` -(without `--resume-from`) or an explicit path via `--resume-from`. +`--resume-from "$SESSION_DIR"` is the only way to resume; there is no +flag that lets the CLI choose a session for you. Get `$SESSION_DIR` from the +authoritative sources listed under the **Session rule** above — never by +walking `$USER_DATA_PATH///` for the newest dir. `$USER_DATA_PATH` must stay at the **workspace root** so `runtime/kernel-agent.env.sh` resolves. The CLI refuses to start if -`manifest.json` or `state.json` is missing in the picked session dir. +`manifest.json` or `state.json` is missing in that session dir. Reuse the Launch template above with these diffs: drop `--model`, add -`--resume`, set `RUN_TAG="resume-$(date +%Y%m%d_%H%M%S)"`. Resume preserves -baseline, current best, params-search state, event history, and kernel-agent -artifacts; the CLI clears stale `stop_reason` and `crash_count` before retrying. +`--resume-from "$SESSION_DIR"`, set `RUN_TAG="resume-$(date +%Y%m%d_%H%M%S)"`. +Resume preserves baseline, current best, params-search state, event history, +and kernel-agent artifacts; the CLI clears stale `stop_reason` and +`crash_count` before retrying. **Most of the launch shape does not need re-passing.** `state.json` is the authority for it, so a bare `--resume` keeps `--server-args`, every @@ -1068,8 +1071,8 @@ For runs > 5 min, start a monitor in its own `setsid nohup` process. It polls `state.json` every 5 min, exits without resuming when the session is terminal (any `stop_reason` in `STOP_REASON_VOCAB`, `phase=CLOSE`, or `reports/final.md` present — including failure sentinels like -`baseline_failed`), and resumes via `--resume` only when the optimizer dies -without those markers (unexpected crash). +`baseline_failed`), and resumes via `--resume-from` only when the optimizer +dies without those markers (unexpected crash). ```bash export RUN_DIR="${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs" @@ -1105,7 +1108,7 @@ export SESSION="${INFERENCE_OPTIMIZER_SESSION_DIR:-$(python3 -c 'import json,sys python3 "$REPO_ROOT/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py" "$SESSION" ``` -It prints `stop_reason`, `baseline_tput`, `cumulative_gain`, `current_best`, +It prints `stop_reason`, `baseline_tput`, `cumulative_gain_validated`, `current_best`, `last_kernel_opt`, `last_trace_analyze`, `last_sweep`, `explore_last_round`, `phase`, plus the recent lifecycle events. @@ -1263,14 +1266,14 @@ Bypass with `--critic-mock` for offline / smoke runs. See - `correctness_passed=false`: do not integrate; the kernel-agent report must contain explicit correctness evidence. - `stop_reason=no_more_leverage`: stop and report; only resume if the user changes workload / search space / model / strategy. - `stop_reason=policy_loop`: a legacy stop_reason kept in the vocabulary for resuming old sessions; nothing in the runtime sets it. Repeated `policy_denied` for the same (action, rule) pair is advisory only — there is no auto-prune at streak ≥5 and no `policy_loop` stop at streak ≥10. Inspect `SharedState.policy_denial_history` via the `why_denied` tool or the `=== Recent policy denials ===` block, then change something substantive (a new `params.grid` variant, a different `benchmark_script`, or a sibling action family). Do not hand-edit `state.json`. -- `stop_reason=time_exhausted`: resume same session (`--resume`); do not start fresh. +- `stop_reason=time_exhausted`: resume same session (`--resume-from`); do not start fresh. ## Report Back To User Report concise status: - session id (from `manifest.json`) and log path -- `cumulative_gain` and `current_best` +- `cumulative_gain_validated` and `current_best` - explore accepted/rejected summary - last kernel optimized, correctness, micro speedup, E2E gain, decision - whether the process is still running or stopped and why diff --git a/src/hyperloom/inference_optimizer/actions/integrate_patch.md b/src/hyperloom/inference_optimizer/actions/integrate_patch.md index 97d6bccd43..81af043615 100644 --- a/src/hyperloom/inference_optimizer/actions/integrate_patch.md +++ b/src/hyperloom/inference_optimizer/actions/integrate_patch.md @@ -40,7 +40,7 @@ patch *files*; this action produces *outcomes*. | `specialist_task_id` | string | yes | Task id of the specialist whose worktree carries the patches. | | `patches` | list[str]| no | Explicit patch path list (relative to specialist workspace or absolute under `SESSION_DIR`). Defaults to `specialist_done.patches_written`. | | `config_changes` | object | no | `env_var -> value` map layered onto the server-launch env before restart. Reverted with the patches on gate failure. | -| `keep_threshold_pct` | float | no | KEEP threshold over baseline_tput, default 1.0. | +| `keep_threshold_pct` | float | no | KEEP threshold over baseline_tput; defaults to the session's decaying per-cycle bar (read from `SharedState`). | | `accuracy_baseline` | float | no | Baseline accuracy score (0-1). Backfilled from `SharedState.baseline_accuracy` when omitted; `<= 0` skips the gate. | ## EMIT format @@ -75,10 +75,10 @@ delegate{ the shared `_accuracy_gate.parse_eval_results(...)` + `_accuracy_gate.accuracy_passed(...)` helpers. 8. Decide: - - KEEP — bench tput ≥ baseline * (1 + keep_threshold_pct/100) AND + - KEEP — bench tput ≥ grading anchor * (1 + keep_threshold_pct/100) AND accuracy did not drop more than 0.05 absolute. Append the patch + config_changes to - `SharedState.optimization_stack`, update `current_best`, increment - `cumulative_gain`. + `SharedState.optimization_stack`, update `current_best`, restamp + `cumulative_gain_validated`. - REVERT — any gate failure. `git checkout` the framework source roots, drop `config_changes`, restart with the previous config, record evidence in `last_action_failures`. diff --git a/src/hyperloom/inference_optimizer/assets/configs/profile_vllm.yaml b/src/hyperloom/inference_optimizer/assets/configs/profile_vllm.yaml index 72089d392b..c9f0ee2702 100644 --- a/src/hyperloom/inference_optimizer/assets/configs/profile_vllm.yaml +++ b/src/hyperloom/inference_optimizer/assets/configs/profile_vllm.yaml @@ -14,7 +14,7 @@ # vllm_*.sh when PROFILE=1. # These are required for TraceLens fusion / roofline analysis. # -# Issue #194 §4: `capture_torch_profiler` and `detailed_trace_annotation` +# Issue #194 §4: `capture_torch_profiler_dir` and `detailed_trace_annotation` # ship upstream from vLLM 0.26; below that they come from the TraceLens # patch set at # TraceLens-internal/examples/custom_workflows/inference_analysis/vllm_patches/. @@ -22,11 +22,13 @@ # Hyperloom's runtime patcher (HYPERLOOM_ENABLE_PATCH=1, default) tries # to apply the matching `config_vllm_v.patch` against the # in-container vLLM install at the start of every profile. Capture -# sidecars land under /capture_traces. If patching -# succeeds, materialize_config_with_envs auto-appends those two flags -# to EXTRA_VLLM_ARGS; if it fails (version unsupported, fs read-only, -# already-patched fork, etc.) we fall back to today's safe behaviour: -# only delay_iterations / max_iterations + ignore_frontend get injected. +# sidecars land under /capture_traces, driven by the +# `capture_torch_profiler_dir` path Magpie's TraceLens route supplies. If +# patching succeeds, materialize_config_with_envs auto-appends +# detailed_trace_annotation to EXTRA_VLLM_ARGS; if it fails (version +# unsupported, fs read-only, already-patched fork, etc.) we fall back to +# today's safe behaviour: only delay_iterations / max_iterations + +# ignore_frontend get injected. # Set HYPERLOOM_ENABLE_PATCH=0 to disable runtime patching entirely. benchmark: diff --git a/src/hyperloom/inference_optimizer/breakdown/SKILL.md b/src/hyperloom/inference_optimizer/breakdown/SKILL.md index c9a560282d..ac50d84c72 100644 --- a/src/hyperloom/inference_optimizer/breakdown/SKILL.md +++ b/src/hyperloom/inference_optimizer/breakdown/SKILL.md @@ -18,7 +18,7 @@ globs: A single JSON file: **`/session_breakdown.json`**. -- Schema: `hyperloom.session_breakdown.v3.0` when recorder fragments are present, else `…v2` (collector-only fallback). Same additive wire shape; gate consumers on the major version, not exact-string equality. See `breakdown/schema.py` (`SCHEMA_VERSION` / `SCHEMA_VERSION_V3`). +- Schema: `hyperloom.session_breakdown.v5.0` (hardcoded; see `SCHEMA_VERSION` in `breakdown/schema.py`). - Producer: `src/hyperloom/inference_optimizer/breakdown/exporter.py` - Filename: `BREAKDOWN_FILENAME` (= `session_breakdown.json`) @@ -131,7 +131,7 @@ this reference is partial — `breakdown/exporter.py` is authoritative. | `session` | `manifest.json` + `state.{session_id, stop_reason, stop_ts, max_minutes, tick, start_ts, resumed_ts}` | | `workload` | `manifest.{framework, model_*, gpu_type, tp, workload, objective}` + `state.{model_class, framework, gpu_type}` | | `baseline` | `state.{baseline_tput, baseline_accuracy, last_baseline.workspace, baseline_attempts}` + `/benchmark_*/benchmark_report.json` | -| `final` | `state.{current_best, cumulative_gain, cumulative_gain_validated_*, optimization_stack}` | +| `final` | `state.{current_best, cumulative_gain_validated, cumulative_gain_validated_*, optimization_stack}` | | `phase_timeline` | `state.{_attempts, kernel_opt_attempts.history, kernel_integrate_attempts.attempts}` sorted by `ts` | | `capability_summary` | Reduces invocations + per-action attempts + search ledgers into 8 rows: geak / forge / explore / sweep / specialist plus the backends / params / validate_stack compatibility rows | | `optimizations` | The recorder's own streams only — `operations` / `adoptions` / `measurements` / `artifacts`, as the producers wrote them. Never rebuilt from `state.json`; when the records are absent the section reports `available: false` instead. | diff --git a/src/hyperloom/inference_optimizer/breakdown/collectors/_common.py b/src/hyperloom/inference_optimizer/breakdown/collectors/_common.py index edfb2e82f4..b038f256f2 100644 --- a/src/hyperloom/inference_optimizer/breakdown/collectors/_common.py +++ b/src/hyperloom/inference_optimizer/breakdown/collectors/_common.py @@ -321,6 +321,32 @@ def _parse_iso_unix(ts: Any) -> float | None: return None +def phase_at( + ts_unix: float, + phase_boundaries: list[tuple[float, str]], + *, + fallback: str = "", +) -> str: + """Return the phase active at ``ts_unix``. + + Args: + ts_unix: The timestamp to classify. + phase_boundaries: ``(unix_ts, phase_name)`` transition points sorted + ascending; the last one at or before ``ts_unix`` wins. + fallback: Phase name returned when every boundary is later. + + Returns: + The phase name active at ``ts_unix``, or ``fallback``. + """ + current = fallback + for boundary, phase in phase_boundaries: + if boundary <= ts_unix: + current = phase + else: + break + return current + + def _load_optimization_journal( session_dir: Path | None, warnings: list[str], diff --git a/src/hyperloom/inference_optimizer/breakdown/collectors/attribution.py b/src/hyperloom/inference_optimizer/breakdown/collectors/attribution.py index faf2df2c57..780fc29b41 100644 --- a/src/hyperloom/inference_optimizer/breakdown/collectors/attribution.py +++ b/src/hyperloom/inference_optimizer/breakdown/collectors/attribution.py @@ -18,6 +18,7 @@ from ._common import ( _to_float, + phase_at, ) @@ -142,17 +143,10 @@ def _entry_ts(entry: dict[str, Any]) -> float | None: def _phase_at(ts_unix: float | None, timeline: list[tuple[float, str]]) -> str: - """Return the phase active at ``ts_unix``, if one can be inferred.""" - + """Return the phase active at ``ts_unix``, or ``""`` when it is unknown.""" if ts_unix is None: return "" - current = "" - for ts, phase in timeline: - if ts <= ts_unix: - current = phase - else: - break - return current + return phase_at(ts_unix, timeline) def _entry_family(entry: dict[str, Any]) -> str: diff --git a/src/hyperloom/inference_optimizer/breakdown/collectors/decision.py b/src/hyperloom/inference_optimizer/breakdown/collectors/decision.py index a33c57c440..fcace7f019 100644 --- a/src/hyperloom/inference_optimizer/breakdown/collectors/decision.py +++ b/src/hyperloom/inference_optimizer/breakdown/collectors/decision.py @@ -26,6 +26,7 @@ _load_optimization_journal, _parse_iso_unix, _to_float, + phase_at, ) @@ -287,7 +288,7 @@ def _build_phase_windows( def _phase_at(ts: Any, windows: list[tuple[float, str]]) -> str: - """Return the phase active at ``ts`` per ``windows`` (latest <= ts). + """Return the phase active at ISO-or-numeric ``ts`` per ``windows``. Args: ts (Any): An ISO-8601 timestamp (or numeric Unix value). @@ -301,13 +302,7 @@ def _phase_at(ts: Any, windows: list[tuple[float, str]]) -> str: unix = _parse_iso_unix(ts) if unix is None or not windows: return "" - phase = "" - for entered, name in windows: - if entered <= unix: - phase = name - else: - break - return phase + return phase_at(unix, windows) # Components whose unjoined LLM spend is legitimately not tied to a single diff --git a/src/hyperloom/inference_optimizer/breakdown/collectors/roofline.py b/src/hyperloom/inference_optimizer/breakdown/collectors/roofline.py index 2952534a5f..cbe845bf5a 100644 --- a/src/hyperloom/inference_optimizer/breakdown/collectors/roofline.py +++ b/src/hyperloom/inference_optimizer/breakdown/collectors/roofline.py @@ -161,7 +161,7 @@ def collect_roofline_progress( # Headline numbers. current_best_tput = trajectory[-1]["tput"] if trajectory else 0.0 - cumulative_gain_pct = _to_float(state.get("cumulative_gain")) or 0.0 + cumulative_gain_pct = _to_float(state.get("cumulative_gain_validated")) or 0.0 pct_of_ceiling = ( round(current_best_tput / ceiling_tok * 100.0, 4) if ceiling_available and current_best_tput > 0 else None ) diff --git a/src/hyperloom/inference_optimizer/breakdown/collectors/sessions.py b/src/hyperloom/inference_optimizer/breakdown/collectors/sessions.py index d2b3b0ce42..758316d2b6 100644 --- a/src/hyperloom/inference_optimizer/breakdown/collectors/sessions.py +++ b/src/hyperloom/inference_optimizer/breakdown/collectors/sessions.py @@ -671,10 +671,9 @@ def _should_use_close_stop_reason(stop_reason: str, close_stop_reason: str) -> b def _collect_recovery(state: dict[str, Any]) -> dict[str, Any]: """Project SharedState's crash / interruption / resume signals. - Folds crash / steward-continuation / degraded-mode / pending-revalidation - signals into the ``session.recovery`` block so a resumed run is not read as - a clean monotonic one. Pure / best-effort: unparseable fields are skipped, - never raised. + Folds crash / degraded-mode / pending-revalidation signals into the + ``session.recovery`` block so a resumed run is not read as a clean monotonic + one. Pure / best-effort: unparseable fields are skipped, never raised. Args: state (dict[str, Any]): Parsed ``state.json`` (SharedState-shaped). @@ -705,30 +704,15 @@ def _collect_recovery(state: dict[str, Any]) -> dict[str, Any]: "message": (str(lte.get("message") or "")[:500] or None), } - infra = state.get("steward_infra_failures_by_round") - infra_by_round: dict[str, int] = {} - infra_total = 0 - if isinstance(infra, dict): - for k, v in infra.items(): - iv = _to_int(v) - if iv is None: - continue - infra_by_round[str(k)] = iv - infra_total += iv - - steward_continuation = bool(state.get("steward_continuation_used")) resume_pending = bool(state.get("resume_pending_revalidation")) degraded = bool(state.get("degraded_mode")) - recovered = bool(crash_count > 0 or crash_ts_iso or steward_continuation or resume_pending or last_exc) + recovered = bool(crash_count > 0 or crash_ts_iso or resume_pending or last_exc) return { "recovered": recovered, "crash_count": crash_count, "crash_timestamps": crash_ts_iso, "degraded_mode": degraded, - "steward_continuation_used": steward_continuation, "resume_pending_revalidation": resume_pending, - "steward_infra_failures_total": infra_total, - "steward_infra_failures_by_round": infra_by_round, "last_tick_exception": last_exc, } @@ -1225,9 +1209,9 @@ def collect_final( warnings (list[str]): Shared warnings list (mutated in place). Returns: - dict[str, Any]: The final section (throughput, validated/per-round - cumulative gain, stack-length bookkeeping, action path, ttft / e2el, - invocation, and closing-phase markers). + dict[str, Any]: The final section (throughput, validated cumulative + gain, stack-length bookkeeping, action path, ttft / e2el, invocation, + and closing-phase markers). """ cb = state.get("current_best") or {} stack = state.get("optimization_stack") or [] @@ -1304,10 +1288,6 @@ def collect_final( # Which field holds the primary result (e2el_mean_ms vs throughput). "primary_metric": framework_registry.primary_metric_name(state.get("framework")), "cumulative_gain_pct_validated": _to_float(state.get("cumulative_gain_validated")) or 0.0, - "cumulative_gain_pct_per_round_sum": _to_float(state.get("cumulative_gain")) or 0.0, - # Provenance/basis of the recorded gain (same-harness validated vs - # cross-harness PROVISIONAL). Empty on native/legacy sessions. - "cumulative_gain_provenance": str(state.get("cumulative_gain_provenance") or ""), "revalidation_pending": bool(state.get("resume_pending_revalidation") or False), # A GEAK e2e candidate whose self-reported win is not yet confirmed by a # main-flow rebench; surfaced as an audit-only note and EXCLUDED from the diff --git a/src/hyperloom/inference_optimizer/breakdown/collectors/telemetry.py b/src/hyperloom/inference_optimizer/breakdown/collectors/telemetry.py index 729748eb1e..f5fbb0088c 100644 --- a/src/hyperloom/inference_optimizer/breakdown/collectors/telemetry.py +++ b/src/hyperloom/inference_optimizer/breakdown/collectors/telemetry.py @@ -821,7 +821,6 @@ def collect_specialist_runs( "proposals_kept": int(raw.get("proposals_kept") or 0), "proposals_rejected": int(raw.get("proposals_rejected") or 0), "proposals_skipped": int(raw.get("proposals_skipped") or 0), - "kb_edge_ids": list(raw.get("kb_edge_ids") or []), "confidence_avg": _to_float( raw.get("confidence_avg") if raw.get("confidence_avg") is not None else raw.get("confidence") ), diff --git a/src/hyperloom/inference_optimizer/breakdown/exporter.py b/src/hyperloom/inference_optimizer/breakdown/exporter.py index 9a885ed774..220efb2129 100644 --- a/src/hyperloom/inference_optimizer/breakdown/exporter.py +++ b/src/hyperloom/inference_optimizer/breakdown/exporter.py @@ -325,19 +325,7 @@ def _pick(section: str, collector_value: Any) -> Any: baseline = _pick( "baseline", _safe_collect("baseline", lambda: collectors.collect_baseline(sd, state, warnings), warnings) ) - final_collector = _safe_collect("final", lambda: collectors.collect_final(sd, state, warnings), warnings) - final_frag = assembled.get("final") - # Merge: fragment live-scalars win, but collector structural fields (invocation, - # action_path, source_layers) are preserved when absent from the fragment. - if isinstance(final_frag, dict) and final_frag: - merged_final = dict(final_collector or {}) - merged_final.update(final_frag) - for _structural in ("invocation", "action_path", "source_layers"): - if _structural in (final_collector or {}): - merged_final[_structural] = (final_collector or {})[_structural] - final = merged_final - else: - final = final_collector + final = _safe_collect("final", lambda: collectors.collect_final(sd, state, warnings), warnings) # Enablement attempt-runtime observability; {} → dashboard hides the block. enablement = _pick( "enablement", @@ -601,16 +589,12 @@ def _pick(section: str, collector_value: Any) -> Any: "phase_timeline": phase_timeline, # v1 readers use flat ``phase_timeline``, v2 prefer ``phase_segments``. "phase_segments": phase_segments, - # v1-reader alias mirroring the flat per-action timeline. - "action_timeline": phase_timeline, "capability_summary": capability_summary, "kernel_lifecycle": kernel_lifecycle, # Collective lane audit trail; survives a campaign the E2E gate rejected, # which never reaches ``optimizations``. "collective": collective, "param_search": explore_search, - # v2-native name for the merged ledger; mirrors ``param_search``. - "explore_search": explore_search, "sweep": sweep, "critic_robustness": critic_robustness, "telemetry": telemetry, @@ -977,7 +961,7 @@ def _fmt_attempt(d: dict[str, Any] | None, label: str) -> str: f"- stop_reason : `{state.stop_reason or '-'}`", f"- baseline : `{baseline_metric_s}`", f"- current_best : `{cb_action}` @ `{cb_metric_s}`", - f"- cumul_gain : `{state.cumulative_gain:.2f}%` (validated `{state.cumulative_gain_validated:.2f}%`)", + f"- cumul_gain : `{state.cumulative_gain_validated:.2f}%` (validated)", f"- stack_entries : `{len(state.optimization_stack or [])}`", f"- sweep summary : {sw_line}", "", @@ -1102,7 +1086,6 @@ def write_minimal_final_json( "baseline_tput": state.baseline_tput, "baseline_accuracy": state.baseline_accuracy, "current_best": state.current_best, - "cumulative_gain": state.cumulative_gain, "cumulative_gain_validated": state.cumulative_gain_validated, "optimization_stack_len": len(state.optimization_stack or []), "crash_count": state.crash_count, diff --git a/src/hyperloom/inference_optimizer/breakdown/recorder/instrument.py b/src/hyperloom/inference_optimizer/breakdown/recorder/instrument.py index 9de74242fb..c180361872 100644 --- a/src/hyperloom/inference_optimizer/breakdown/recorder/instrument.py +++ b/src/hyperloom/inference_optimizer/breakdown/recorder/instrument.py @@ -991,10 +991,7 @@ def snapshot_state_sections( for name, fn in ( ("session", _snapshot_session), - ("workload", _snapshot_workload), - ("final", _snapshot_final), ("explore_search", _snapshot_explore_search), - ("sweep", _snapshot_sweep), ("optimization_stack", _snapshot_optimization_stack), ("roofline", _snapshot_roofline), ): @@ -1071,7 +1068,6 @@ def _snapshot_v4_run(rec, st: Any) -> None: "baseline_throughput": to_float(getattr(st, "baseline_tput", None)), "baseline_accuracy": to_float(getattr(st, "baseline_accuracy", None)), "current_best": current_best, - "cumulative_gain_pct": to_float(getattr(st, "cumulative_gain", None)), "cumulative_gain_validated_pct": to_float( getattr(st, "cumulative_gain_validated", None) ), @@ -1119,82 +1115,6 @@ def _snapshot_session(rec, st: Any) -> None: ) -def _snapshot_workload(rec, st: Any) -> None: - """Snapshot the ``workload`` singleton from ``st``. - - A no-op when neither a framework nor a model is set. - - Args: - rec: the recorder used to write the singleton. - st (Any): the live ``SharedState`` to snapshot. - """ - framework = str(getattr(st, "framework", "") or "") - model = str(getattr(st, "model_name", "") or getattr(st, "model_path", "") or "") - if not framework and not model: - return - rec.record_singleton( - "workload", - { - "framework": framework, - "model_name": str(getattr(st, "model_name", "") or ""), - "model_path": str(getattr(st, "model_path", "") or ""), - "model_class": str(getattr(st, "model_class", "") or ""), - "gpu_type": str(getattr(st, "gpu_type", "") or ""), - "tp": int(getattr(st, "tp", 0) or 0), - "ep": int(getattr(st, "ep", 0) or 0), - "precision": str(getattr(st, "precision", "") or ""), - "conc": int(getattr(st, "conc", 0) or 0), - "isl": int(getattr(st, "isl", 0) or 0), - "osl": int(getattr(st, "osl", 0) or 0), - "max_model_len": int(getattr(st, "max_model_len", 0) or 0), - }, - ) - - -def _snapshot_final(rec, st: Any) -> None: - """Snapshot the ``final`` singleton (current best + cumulative gains) from ``st``. - - A no-op when there is neither a current best nor an optimization stack. - - Args: - rec: the recorder used to write the singleton. - st (Any): the live ``SharedState`` to snapshot. - """ - cb = getattr(st, "current_best", None) or {} - stack = getattr(st, "optimization_stack", None) or [] - if not cb and not stack: - return - from ... import framework_registry - - framework = str(getattr(st, "framework", "") or "") - tput = to_float(cb.get("tput")) - # Latency is the primary result for scriptable/diffusion (xDiT) image models - # (throughput_tok_s_per_gpu is only ``1 / latency`` there and misleading as a - # headline). Emit e2el/ttft alongside the throughput-unit + primary-metric - # markers so consumers pick the right result field per framework. e2el falls - # back to the tput-derived per-image latency when no measured value exists. - e2el = to_float(cb.get("e2el_mean_ms")) - if e2el is None and framework_registry.is_scriptable(framework) and tput is not None and tput > 0: - derived = framework_registry.primary_metric_value(framework, tput) - e2el = round(float(derived), 4) if derived is not None and derived > 0 else None - rec.record_singleton( - "final", - { - "current_best_action": str(cb.get("action") or ""), - "throughput_tok_s_per_gpu": tput, - "throughput_unit": framework_registry.throughput_unit(framework), - "primary_metric": framework_registry.primary_metric_name(framework), - "e2el_mean_ms": e2el, - "ttft_mean_ms": to_float(cb.get("ttft_mean_ms")), - "cumulative_gain_pct_validated": to_float(getattr(st, "cumulative_gain_validated", 0.0)) or 0.0, - "cumulative_gain_pct_per_round_sum": to_float(getattr(st, "cumulative_gain", 0.0)) or 0.0, - "validated_ts": str(getattr(st, "cumulative_gain_validated_ts", "") or ""), - "stack_len": len(stack), - "extra_server_args": str(cb.get("extra_server_args") or ""), - "extra_envs": dict(cb.get("extra_envs") or {}), - }, - ) - def _snapshot_explore_search(rec, st: Any) -> None: """Snapshot the ``explore_search`` singleton from ``st`` (no-op when empty). @@ -1217,19 +1137,6 @@ def _snapshot_explore_search(rec, st: Any) -> None: rec.record_singleton("explore_search", search) -def _snapshot_sweep(rec, st: Any) -> None: - """Snapshot the ``sweep`` singleton from ``st.last_sweep`` (no-op when empty). - - Args: - rec: the recorder used to write the singleton. - st (Any): the live ``SharedState`` to snapshot. - """ - last_sweep = dict(getattr(st, "last_sweep", None) or {}) - if not last_sweep: - return - rec.record_singleton("sweep", last_sweep) - - def _snapshot_optimization_stack(rec, st: Any) -> None: """Snapshot each ``optimization_stack`` entry from ``st`` as a keyed item. diff --git a/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py b/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py index 794ff1fce0..694051839b 100644 --- a/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py +++ b/src/hyperloom/inference_optimizer/breakdown/reporters/_renderers/final.py @@ -38,21 +38,17 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: final_tput = f.get("throughput_tok_s_per_gpu") base_tput = b.get("throughput_tok_s_per_gpu") gain_v = f.get("cumulative_gain_pct_validated") - gain_round = f.get("cumulative_gain_pct_per_round_sum") val_stack_len = f.get("validated_at_stack_len") val_ts = f.get("validated_ts") stack_changed = bool(f.get("stack_changed_after_validation")) extra_args = f.get("extra_server_args") or "" action_path = f.get("action_path") or [] - gain_provenance = str(f.get("cumulative_gain_provenance") or "") revalidation_pending = bool(f.get("revalidation_pending")) # Self-reported GEAK candidate excluded from the headline; surfaced as an audit-only note. geak_pending = f.get("geak_pending") if isinstance(f.get("geak_pending"), dict) else {} pending_awaiting = geak_pending.get("status") == "awaiting_rebench" - # Gain is provisional when provenance says so, or a revalidation is pending with no positive validated number. - is_provisional = ("provisional" in gain_provenance) or ( - revalidation_pending and not (isinstance(gain_v, (int, float)) and gain_v > 0) - ) + # Gain is provisional when a cross-harness revalidation is pending with no confirmed validated number. + is_provisional = revalidation_pending and not (isinstance(gain_v, (int, float)) and gain_v > 0) # Headline is unvalidated when a GEAK candidate is pending with no positive validated gain. headline_unvalidated = pending_awaiting and not (isinstance(gain_v, (int, float)) and gain_v > 0) @@ -74,33 +70,23 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: note = " (negative = faster)" if framework_registry.is_scriptable(fw) else "" facts.append(f"Delta vs baseline: {final_v - base_v:+.2f} {_unit}{note}.") if is_provisional: - if gain_round is not None: - facts.append( - f"Provisional cumulative gain: {fmt_pct(gain_round, plus=True)} " - "— PENDING same-harness revalidation, NOT yet validated." - ) + facts.append("Cumulative gain is PENDING same-harness revalidation; no validated number exists yet.") warnings.append( - "Reported gain is PROVISIONAL and cross-harness " - f"(provenance={gain_provenance or 'unknown'}): the numerator was " - "measured by the delegated optimizer's harness and the denominator " - "is the orchestrator baseline. A same-harness full-stack rebench is " - "pending; the validated gain will replace this number once it lands." + "The recorded gain basis is PROVISIONAL and cross-harness: measured by the " + "delegated optimizer's harness against the orchestrator baseline, so " + "no gain is reported here. A same-harness full-stack rebench is " + "pending and will supply the validated number." ) - else: - if gain_v is not None and not headline_unvalidated: - facts.append(f"Validated cumulative gain: {fmt_pct(gain_v, plus=True)}.") - decisions.append( - Decision( - kind="kept" if (gain_v or 0) > 0 else "attempted", - subject="final", - metric_pct=float(gain_v), - rationale=f"validated at stack_len={val_stack_len} ts={val_ts}", - ) - ) - if gain_round is not None and not headline_unvalidated: - facts.append( - f"Per-round summed gain: {fmt_pct(gain_round)} (non-additive, do not present as the user-visible number)." + elif gain_v is not None and not headline_unvalidated: + facts.append(f"Validated cumulative gain: {fmt_pct(gain_v, plus=True)}.") + decisions.append( + Decision( + kind="kept" if (gain_v or 0) > 0 else "attempted", + subject="final", + metric_pct=float(gain_v), + rationale=f"validated at stack_len={val_stack_len} ts={val_ts}", ) + ) if geak_pending and geak_pending.get("status") == "awaiting_rebench": self_gain = geak_pending.get("self_reported_gain_pct") self_gain_str = fmt_pct(self_gain, plus=True) if isinstance(self_gain, (int, float)) else "unknown" @@ -139,8 +125,6 @@ def render(breakdown: dict[str, Any]) -> RenderedSection: ("final_throughput_tok_s_per_gpu", final_tput), ("throughput_unit", f.get("throughput_unit") or None), ("cumulative_gain_pct_validated", gain_v), - ("cumulative_gain_pct_per_round_sum", gain_round), - ("cumulative_gain_provenance", gain_provenance or None), ("revalidation_pending", revalidation_pending or None), ("geak_pending", geak_pending or None), ("validated_at_stack_len", val_stack_len), diff --git a/src/hyperloom/inference_optimizer/breakdown/schema.py b/src/hyperloom/inference_optimizer/breakdown/schema.py index 367371e433..d209384854 100644 --- a/src/hyperloom/inference_optimizer/breakdown/schema.py +++ b/src/hyperloom/inference_optimizer/breakdown/schema.py @@ -46,14 +46,8 @@ class Recovery(TypedDict, total=False): crash_timestamps (list[str]): ISO UTC timestamps of recent crashes (bounded tail). degraded_mode (bool): Whether the run entered degraded operation. - steward_continuation_used (bool): The steward continued the run after an - interruption / budget event. resume_pending_revalidation (bool): Accepted stack awaits post-resume revalidation (validated gain not yet re-trusted). - steward_infra_failures_total (int): Sum of steward-observed infra - failures across rounds. - steward_infra_failures_by_round (dict[str, int]): Per-round infra - failure counts. last_tick_exception (dict[str, Any] | None): Compact summary of the last Coordinator tick exception (tick / stage / type / message), traceback omitted. @@ -63,10 +57,7 @@ class Recovery(TypedDict, total=False): crash_count: int crash_timestamps: list[str] degraded_mode: bool - steward_continuation_used: bool resume_pending_revalidation: bool - steward_infra_failures_total: int - steward_infra_failures_by_round: dict[str, int] last_tick_exception: dict[str, Any] | None @@ -331,7 +322,6 @@ class Final(TypedDict, total=False): Attributes: throughput_tok_s_per_gpu (float | None): Final throughput (tok/s/GPU), or None. cumulative_gain_pct_validated (float): Validated cumulative gain percent. - cumulative_gain_pct_per_round_sum (float): Sum of per-round gain percents. validated_at_stack_len (int): Stack depth at which validation occurred. validated_ts (str): ISO UTC timestamp of the validation. stack_changed_after_validation (bool): Whether the stack changed post-validation. @@ -351,7 +341,6 @@ class Final(TypedDict, total=False): throughput_tok_s_per_gpu: float | None throughput_unit: str # "tok/s" (serving) or "img/s" (scriptable xDiT) cumulative_gain_pct_validated: float - cumulative_gain_pct_per_round_sum: float validated_at_stack_len: int validated_ts: str stack_changed_after_validation: bool @@ -1539,8 +1528,6 @@ class SpecialistRound(TypedDict, total=False): proposals_kept: int proposals_rejected: int proposals_skipped: int - # Retired field, kept (always empty) for backward compatibility with existing readers. - kb_edge_ids: list[str] confidence_avg: float | None domain_breakdown: dict[str, SpecialistDomainBreakdown] transcripts: list[SpecialistTranscriptRef] @@ -2889,15 +2876,13 @@ class SessionBreakdown(TypedDict, total=False): Empty {} on non-transformers models or pre-field sessions. baseline (Baseline): Pre-optimization reference performance. final (Final): Final validated optimization state. - phase_timeline (list[PhaseEvent]): Flat per-action timeline (v1-reader compat). + phase_timeline (list[PhaseEvent]): Flat per-action timeline. phase_segments (list[PhaseSegment]): Phase-boundary view. - action_timeline (list[PhaseEvent]): v2 canonical flat per-action timeline. capability_summary (CapabilitySummary): Per-capability roll-up. kernel_lifecycle (KernelLifecycle): Kernels grouped by lifecycle stage. collective (Collective): Collective-lane campaigns and their E2E verdicts; empty {} when the lane never ran. - param_search (ParamSearch): v1-reader compat alias for ``explore_search``. - explore_search (ParamSearch): Merged explore-search ledger. + param_search (ParamSearch): Merged explore-search ledger. sweep (Sweep): Concurrency/shape sweep results. critic_robustness (CriticRobustness): Critic reviews and robustness signals. telemetry (Telemetry): Telemetry artifacts and aggregated metrics. diff --git a/src/hyperloom/inference_optimizer/cli/__init__.py b/src/hyperloom/inference_optimizer/cli/__init__.py index 7e2642f2be..b7dd008649 100644 --- a/src/hyperloom/inference_optimizer/cli/__init__.py +++ b/src/hyperloom/inference_optimizer/cli/__init__.py @@ -37,7 +37,6 @@ _build_proposal_scorer, _official_anthropic_only, _official_openai_only, - _robustness_server_configured, resolve_robustness_options, ) from .model_gate import ( @@ -100,6 +99,7 @@ ) from ..session.lock import SessionAlreadyRunning, SessionLock from ..session.paths import ( + ENV_USER_DATA_PATH, asset_system_prompts_dir, make_session_dir, ) @@ -444,7 +444,7 @@ def _emit_launch_info( def _acquire_session_lock_or_exit(session_dir: Path) -> SessionLock: """Take the single-optimizer session lock or exit ``SESSION_BUSY_EXIT_CODE``. - Guards both fresh ``optimize`` and ``--resume`` against a second optimizer + Guards both fresh ``optimize`` and ``--resume-from`` against a second optimizer attaching to the same ``session_dir``. When a live optimizer already owns the session this refuses to run before any ``state.json`` / lease mutation. @@ -1065,13 +1065,9 @@ def _resolve_critic_choice(args: argparse.Namespace) -> str: def _resolve_robustness_choice(args: argparse.Namespace) -> str: """Resolve the active robustness backend choice (arg → DEFAULT_ROBUSTNESS_BACKEND); hard-fails on invalid. - Multi-node policy: on ``nodes>=2`` the agent's LocalProbe targets sandbox-local resources that live in - separate pods (HIGH false positives). Keep ``agent`` only when a robustness-server is configured; else - auto-downgrade to ``mock`` (explicit --robustness-agent gets a WARN). - Args: args (argparse.Namespace): The parsed CLI namespace (reads - ``robustness_backend`` / ``nodes`` and server config). + ``robustness_backend``). Returns: str: The resolved robustness backend (one of @@ -1080,31 +1076,13 @@ def _resolve_robustness_choice(args: argparse.Namespace) -> str: Raises: SystemExit: With code 2 when the chosen backend is invalid. """ - chosen, explicit = _resolve_choice( + chosen, _ = _resolve_choice( "robustness_backend", DEFAULT_ROBUSTNESS_BACKEND, _VALID_ROBUSTNESS_BACKENDS, "--robustness-mock / --robustness-agent or INFERENCE_OPTIMIZER_DEFAULT_ROBUSTNESS_BACKEND", args=args, ) - nodes = int(getattr(args, "nodes", 1) or 1) - if nodes >= 2 and chosen == "agent" and not _robustness_server_configured(args): - if explicit: - print( - f"WARN: --robustness-agent selected but nodes={nodes} and " - f"no robustness-server configured — the agent's LocalProbe " - f"family targets sandbox-local resources (ray, inference " - f"server, GPU, ...) that all live in separate pods on " - f"multi-node and surface as HIGH false positives. " - f"Auto-downgrading to --robustness-mock; configure " - f"--robustness-server-url / ROBUSTNESS_SERVER_URL to keep " - f"the agent backend, or pass --robustness-mock explicitly " - f"to suppress this warning. See " - f"src/hyperloom/inference_optimizer/multi_node/SKILL.md " - f"(Robustness limitation in multi-node mode).", - file=sys.stderr, - ) - chosen = "mock" return chosen @@ -1457,7 +1435,7 @@ async def _run_optimize(args: argparse.Namespace) -> int: # (``None`` -> 1) because the persisted SharedState is loaded later; the # resume branch re-exports the real values after ``_resolve_workload_knobs`` # so downstream (incl. preflight) never sees the placeholder default. - if not args.resume and not args.resume_from: + if not args.resume_from: _export_workload_envs_for_optimize( args, nodes_resolved=nodes_resolved, @@ -1533,53 +1511,32 @@ async def _run_optimize(args: argparse.Namespace) -> int: codex_follows_claude=codex_follows_claude, ) - # `--resume-from ` implies `--resume` (operator convenience). - if args.resume_from and not args.resume: - args.resume = True - - if args.resume: - # Resume mode: USER_DATA_PATH stays at workspace level; pick the - # per-session subdir via --resume-from or auto-pick the latest. Pin - # INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR for consistent resolution. + if args.resume_from: + # USER_DATA_PATH stays at the workspace root; --resume-from names a subdir under it. from ..session.paths import ( ENV_CURRENT_SESSION_DIR, - find_latest_per_session_dir, workspace_root, ) ws = workspace_root() - if args.resume_from: - session_dir = Path(args.resume_from).expanduser().resolve() - try: - session_dir.relative_to(ws.resolve()) - except ValueError: - print( - f"ERROR: --resume-from {session_dir!r} is not under " - f"$USER_DATA_PATH={ws}. Move USER_DATA_PATH to the " - f"workspace root (the parent of the per-session subdirs) " - f"and pass the per-session subdir via --resume-from.", - file=sys.stderr, - ) - sys.exit(2) - if not session_dir.is_dir(): - print( - f"ERROR: --resume-from {session_dir!r} does not exist.", - file=sys.stderr, - ) - sys.exit(2) - else: - picked = find_latest_per_session_dir() - if picked is not None: - session_dir = picked - print(" --resume: auto-picked latest per-session subdir") - else: - # No per-session subdir found — workspace_root itself is the - # session_dir (e.g. resuming a pre-existing single-dir session). - session_dir = ws - print( - f" --resume: no per-session subdir found under " - f"{ws}///; falling back to {ws}" - ) + session_dir = Path(args.resume_from).expanduser().resolve() + try: + session_dir.relative_to(ws.resolve()) + except ValueError: + print( + f"ERROR: --resume-from {session_dir!r} is not under " + f"$USER_DATA_PATH={ws}. Move USER_DATA_PATH to the " + f"workspace root (the parent of the per-session subdirs) " + f"and pass the per-session subdir via --resume-from.", + file=sys.stderr, + ) + sys.exit(2) + if not session_dir.is_dir(): + print( + f"ERROR: --resume-from {session_dir!r} does not exist.", + file=sys.stderr, + ) + sys.exit(2) # Pin before Coordinator/SharedState load so paths/subprocesses inherit the resolved location. os.environ[ENV_CURRENT_SESSION_DIR] = str(session_dir) # Ensure per-session skeleton exists (idempotent mkdir -p). @@ -1595,11 +1552,11 @@ async def _run_optimize(args: argparse.Namespace) -> int: try: manifest = load_manifest(session_dir) except FileNotFoundError as exc: - print(f"ERROR: --resume failed: {exc}", file=sys.stderr) + print(f"ERROR: --resume-from failed: {exc}", file=sys.stderr) sys.exit(2) if not (session_dir / "state.json").exists(): print( - f"ERROR: --resume failed: {session_dir}/state.json missing " + f"ERROR: --resume-from failed: {session_dir}/state.json missing " f"(manifest exists but Coordinator never wrote SharedState)", file=sys.stderr, ) @@ -1609,7 +1566,7 @@ async def _run_optimize(args: argparse.Namespace) -> int: print(f"Resuming session: {session_dir}") print(f" manifest.session_id : {manifest.get('session_id')}") print(f" prior baseline_tput : {state.baseline_tput:.1f}") - print(f" prior cumul_gain : {state.cumulative_gain:.2f}%") + print(f" prior cumul_gain : {state.cumulative_gain_validated:.2f}%") print( f" prior current_best : " f"{(state.current_best or {}).get('action')}/" @@ -1766,7 +1723,7 @@ async def _run_optimize(args: argparse.Namespace) -> int: gated_terminal = {"target_reached"} if prior_stop in gated_terminal and not force_resume: print( - f"\nERROR: --resume blocked by terminal stop_reason=" + f"\nERROR: --resume-from blocked by terminal stop_reason=" f"{prior_stop!r}.\n" f"\n" f" SKILL.md (Run-time signals): {prior_stop!r} is a " @@ -1836,8 +1793,8 @@ async def _run_optimize(args: argparse.Namespace) -> int: if not args.model: print( "ERROR: model is required. Pass --model or set " - "MODEL_PATH env (or use --resume to continue an existing " - "session at the canonical session_dir).", + "MODEL_PATH env (or use --resume-from to " + "continue an existing session).", file=sys.stderr, ) sys.exit(2) @@ -1961,7 +1918,7 @@ async def _run_optimize(args: argparse.Namespace) -> int: f"FRAMEWORK_VERSION={_fw_version_for_env or ''}" ) - # session_dir defaults to ///. + # session_dir defaults to //-/. # Use the resolved identity so a quantized run is named after the source # model (e.g. "-quantized") instead of the generic export-dir # basename "quantized". @@ -2171,7 +2128,7 @@ async def _run_optimize(args: argparse.Namespace) -> int: critic_protocol=args.critic_protocol, ) # Expose active session_dir to in-process executors via the canonical pin - # env var; reinforced here for --resume paths. Do NOT overwrite + # env var; reinforced here for resume paths. Do NOT overwrite # USER_DATA_PATH — it must remain the workspace root for concurrent sessions # and install.sh on shared filesystems (WekaFS). os.environ["INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR"] = str(session_dir) @@ -2203,7 +2160,7 @@ async def _run_optimize(args: argparse.Namespace) -> int: # Advisory multi-model specialist-proposal scorer, disabled by default # (enable via --proposal-scoring). When active it scores each # proposal_set and surfaces results to Orchestration without gating. - # Not persisted across --resume. ``session_dir`` lets it append per-model + # Not persisted across a resume. ``session_dir`` lets it append per-model # token usage to the full-trace ledger (component=proposal_scorer). proposal_scorer=_build_proposal_scorer(args, session_dir), # Warm-recipe replay controls. Default ON; fires when @@ -2483,6 +2440,12 @@ def main(argv: list[str] | None = None) -> int: if hasattr(sys.stderr, "reconfigure"): sys.stderr.reconfigure(line_buffering=True) + # Absolutise before the parser defaults or any session path derive from it: + # subprocesses run with their own cwd, so a relative value would diverge. + user_data = os.environ.get(ENV_USER_DATA_PATH, "") + if user_data and not Path(user_data).is_absolute(): + os.environ[ENV_USER_DATA_PATH] = str(Path(user_data).expanduser().absolute()) + parser = _build_parser() # Strict on purpose. The platform's prompt FLAGS block is authored by hand, # so a typo is likelier here than in generated argv, and the flags the diff --git a/src/hyperloom/inference_optimizer/cli/backends.py b/src/hyperloom/inference_optimizer/cli/backends.py index 11c10489e8..5f7bfdbdcb 100644 --- a/src/hyperloom/inference_optimizer/cli/backends.py +++ b/src/hyperloom/inference_optimizer/cli/backends.py @@ -273,7 +273,7 @@ def _build_proposal_scorer( deployments (the scorer is OpenAI-compatible only) or when the resolved model list is empty (defaults to :data:`DEFAULT_SCORER_MODELS`). The scorer is purely advisory and never gates anything. The flag is not - persisted across ``--resume``. + persisted across ``--resume-from``. ``session_dir`` is forwarded so the scorer can append its per-model token usage to the full-trace ledger; when omitted trace writes are skipped. @@ -302,47 +302,16 @@ def _build_proposal_scorer( return ProposalScorer(models=models, session_dir=session_dir) -def _robustness_server_configured(args: argparse.Namespace) -> bool: - """Return True when a robustness-server endpoint is configured. - - The server is the only cluster-wide signal source on multi-node; when - wired the agent runs with ``disable_local_probe`` / - ``enable_cluster_pod_metrics`` and the sandbox-local LocalProbe false - positives are silenced. Configured = ``--robustness-server-url`` or - ``ROBUSTNESS_SERVER_URL`` is set. - - Args: - args: Parsed CLI args carrying ``robustness_server_url``. - - Returns: - ``True`` when a robustness-server endpoint is configured via flag or - environment. - """ - url = (getattr(args, "robustness_server_url", None) or "").strip() - if url: - return True - return bool((os.environ.get("ROBUSTNESS_SERVER_URL") or "").strip()) - - -_MULTI_NODE_WORKLOAD_UID_ENV_KEYS: tuple[str, ...] = ( - "ROBUSTNESS_WORKLOAD_UID", - "CLAW_WORKLOAD_UID", - "WORKLOAD_UID", - "KUBE_WORKLOAD_UID", - "RAY_JOB_ID", -) - - def _build_robustness_options(args: argparse.Namespace) -> dict[str, Any]: """Collect non-default ``request.options`` overrides from CLI flags. Only emits keys the operator actually passed so the runtime CLI falls back to its own defaults / env-discovery for the rest. - Multi-node (``--nodes >= 2``): defaults ``disable_local_probe`` + - ``enable_cluster_pod_metrics`` to True, forwards a workload_uid hint, + Multi-node (``--nodes >= 2``): defaults ``disable_local_probe`` to True, disables the 127.0.0.1:8888 auto-probe, and lifts the ``no_levers_found`` - floor to 60 min so cluster-sourced signals replace the local sandbox. + floor to 60 min. The sandbox-local probes would only see one pod, so the + agent runs on the node-agnostic budget / progress / inbox signals instead. Single-node opt-in: ``--robustness-disable-server-probe`` sets ``auto_probe_inference_server=False`` to silence the 127.0.0.1:8888 /health @@ -357,9 +326,6 @@ def _build_robustness_options(args: argparse.Namespace) -> dict[str, Any]: (and multi-node policy); keys the operator did not set are omitted. """ options: dict[str, Any] = {} - server_url = getattr(args, "robustness_server_url", None) - if server_url is not None: - options["robustness_server_url"] = server_url llm_rca = getattr(args, "robustness_llm_rca", None) if llm_rca is not None: options["llm_rca_enabled"] = bool(llm_rca) @@ -369,38 +335,12 @@ def _build_robustness_options(args: argparse.Namespace) -> dict[str, Any]: if nodes > 1: options["nodes"] = nodes - workload_uid = (getattr(args, "robustness_workload_uid", None) or "").strip() - if not workload_uid: - for key in _MULTI_NODE_WORKLOAD_UID_ENV_KEYS: - candidate = (os.environ.get(key) or "").strip() - if candidate: - workload_uid = candidate - break - if workload_uid: - options["workload_uid"] = workload_uid - disable_local = getattr(args, "robustness_disable_local_probe", None) if disable_local is None and multi_node: disable_local = True if disable_local is not None: options["disable_local_probe"] = bool(disable_local) - enable_pod_metrics = getattr(args, "robustness_enable_cluster_pod_metrics", None) - if enable_pod_metrics is None and multi_node: - enable_pod_metrics = True - if enable_pod_metrics is not None: - options["enable_cluster_pod_metrics"] = bool(enable_pod_metrics) - - categories_raw = getattr(args, "robustness_pod_metrics_categories", None) - if categories_raw: - if isinstance(categories_raw, (list, tuple)): - cat_iter = categories_raw - else: - cat_iter = str(categories_raw).split(",") - cat_list = [c.strip() for c in cat_iter if str(c).strip()] - if cat_list: - options["pod_metrics_categories"] = cat_list - # ``auto_probe_inference_server`` controls the 127.0.0.1:8888 /health probe # in LocalProbe. Default it OFF for multi-node (server lives in the head pod) # and scriptable server-less frameworks; single-node operators opt in via diff --git a/src/hyperloom/inference_optimizer/cli/bootstrap.py b/src/hyperloom/inference_optimizer/cli/bootstrap.py index 3584b8da7b..5ec359185e 100644 --- a/src/hyperloom/inference_optimizer/cli/bootstrap.py +++ b/src/hyperloom/inference_optimizer/cli/bootstrap.py @@ -136,8 +136,6 @@ def _seed_shared_state( if getattr(args, "plateau_kernel_lookback", None) is not None: plateau_overrides["kernel_lookback"] = int(args.plateau_kernel_lookback) # EXPLORE hard force-exit thresholds. - if getattr(args, "explore_force_exit_hours_remaining", None) is not None: - plateau_overrides["force_exit_hours_remaining"] = float(args.explore_force_exit_hours_remaining) if getattr(args, "explore_force_exit_budget_pct", None) is not None: plateau_overrides["force_exit_budget_pct"] = float(args.explore_force_exit_budget_pct) @@ -279,7 +277,7 @@ def _resolve_framework_version(args_in: Any) -> str: continue_kernel_after_gemm=bool(getattr(args, "continue_kernel_after_gemm", True)), target_summary=args.target_summary or _default_target_summary(args), baseline_tput=0.0, - cumulative_gain=0.0, + cumulative_gain_validated=0.0, reference_server_args=_ref_args, reference_envs=_ref_envs, reference_model=_ref_model, @@ -383,9 +381,9 @@ def _print_final_summary( """Print the end-of-run summary block to stdout. Reports the stop reason, session id, model, baseline throughput, the - per-round (informational) cumulative gain, the validated cumulative gain - (with a staleness warning when the optimization stack grew after the last - validation), the current best config, pruned families, and crash count. + validated cumulative gain (with a staleness warning when the optimization + stack grew after the last validation), the current best config, pruned + families, and crash count. On ``baseline_failed`` it also surfaces the real terminal root cause from ``reports/final.json``. @@ -418,7 +416,6 @@ def _print_final_summary( ) if failure_summary.get("server_log"): print(f" server_log : {failure_summary.get('server_log')}") - print(f" cumulative_gain : {state.cumulative_gain:.2f}% (per-round sum — informational)") if state.cumulative_gain_validated_ts: stale = ( " ⚠ stack changed since validation" @@ -593,7 +590,7 @@ def _default_target_summary(args: argparse.Namespace) -> str: if args.target_gain: return ( f"Establish baseline on {Path(args.model).name} then drive " - f"cumulative_gain to >= {args.target_gain}% within " + f"cumulative_gain_validated to >= {args.target_gain}% within " f"{args.max_hours}h." ) if args.target_tput: diff --git a/src/hyperloom/inference_optimizer/cli/parser.py b/src/hyperloom/inference_optimizer/cli/parser.py index 506784f566..98080c97ab 100644 --- a/src/hyperloom/inference_optimizer/cli/parser.py +++ b/src/hyperloom/inference_optimizer/cli/parser.py @@ -245,7 +245,7 @@ def _build_parser() -> argparse.ArgumentParser: type=Path, default=None, help="Model path (required for new runs; ignored when " - "--resume is set — model is read from manifest.json/" + "--resume-from is set — model is read from manifest.json/" "state.json)", ) opt.add_argument( @@ -258,7 +258,7 @@ def _build_parser() -> argparse.ArgumentParser: "optimization loop: it drives AMD Quark PTQ from this prompt, " "then rewrites --model to the exported quantized model so the " "rest of the run optimizes the quantized model. Ignored on " - "--resume.", + "--resume-from.", ) from hyperloom.orchestrator.phases.quantization_schemes import QUANT_SCHEME_CHOICES @@ -573,7 +573,12 @@ def _build_parser() -> argparse.ArgumentParser: ), ) grp = opt.add_mutually_exclusive_group() - grp.add_argument("--target-gain", type=float, default=None, help="Stop when cumulative_gain >= N%% over baseline") + grp.add_argument( + "--target-gain", + type=float, + default=None, + help="Stop when cumulative_gain_validated >= N%% over baseline", + ) grp.add_argument( "--target-tput", type=float, @@ -583,41 +588,28 @@ def _build_parser() -> argparse.ArgumentParser: grp.add_argument( "--target-baseline-dir", type=str, default=None, help="Stop when current best matches the baseline in DIR" ) - opt.add_argument( - "--resume", - action="store_true", - default=False, - help="Resume an existing session. Without --resume-from, " - "auto-picks the latest per-session subdir under " - "$USER_DATA_PATH/// (N17 layout) or " - "falls back to $USER_DATA_PATH. " - "USER_DATA_PATH MUST stay at workspace level " - "(/shared/hyperloom-sessions, not the per-session subdir) " - "so runtime/ resolution works. Skips the SharedState " - "seed and lets the Coordinator replay the prior " - "event log + state.json.", - ) opt.add_argument( "--resume-from", type=str, default=None, - help="Explicit per-session subdir to resume from. Use " - "when multiple per-launch ts dirs exist under the " - "same model and the latest is not what you want. " - "Must be an absolute path under $USER_DATA_PATH " - "(workspace_root). Implies --resume.", + help="Per-session subdir to resume; the only way to resume a session. " + "Skips the SharedState seed and lets the Coordinator replay the " + "prior event log + state.json. Must be an absolute path under " + "$USER_DATA_PATH (workspace_root), which MUST stay at workspace " + "level (/shared/hyperloom-sessions, not the per-session subdir) so " + "runtime/ resolution works.", ) opt.add_argument( "--force-resume", action="store_true", default=False, help=( - "Allow ``--resume`` to push past a terminal " + "Allow ``--resume-from`` to push past a terminal " "``stop_reason='target_reached'``. " "Without this flag the resume aborts (Issue-G guard, per " "SKILL.md 'Run-time signals': that terminal requires an " "operator-side workload / strategy change before resuming). " - "No-op outside ``--resume``." + "No-op without ``--resume-from``." ), ) opt.add_argument( @@ -840,15 +832,6 @@ def _build_parser() -> argparse.ArgumentParser: "mirrors critic-agent transport). Requires ROBUSTNESS_AGENT_ROOT " "or a sibling $REPO_ROOT/robustness-agent/ directory.", ) - opt.add_argument( - "--robustness-server-url", - dest="robustness_server_url", - type=str, - default=None, - help="Override the robustness-server base URL forwarded into " - "request.options. Honoured only when --robustness-agent is " - "selected.", - ) opt.add_argument( "--robustness-llm-rca", dest="robustness_llm_rca", @@ -864,16 +847,6 @@ def _build_parser() -> argparse.ArgumentParser: action="store_false", help="Forward llm_rca_enabled=false into request.options.", ) - opt.add_argument( - "--robustness-workload-uid", - dest="robustness_workload_uid", - type=str, - default=None, - help="Forward workload_uid into request.options. The robustness-server " - "resolves it to every pod (head + workers) backing the RayJob via " - "the cluster/workloads/{uid}/hierarchy endpoint. Falls back to " - "$CLAW_WORKLOAD_UID / $WORKLOAD_UID / $RAY_JOB_ID when unset.", - ) opt.add_argument( "--robustness-disable-local-probe", dest="robustness_disable_local_probe", @@ -912,30 +885,6 @@ def _build_parser() -> argparse.ArgumentParser: help="Force auto_probe_inference_server=true (keep the 127.0.0.1:8888 " "/health auto-probe even in multi-node mode).", ) - opt.add_argument( - "--robustness-enable-cluster-pod-metrics", - dest="robustness_enable_cluster_pod_metrics", - action="store_true", - default=None, - help="Force enable_cluster_pod_metrics=true so the robustness-agent " - "fans out per-pod metrics through robustness-server and feeds " - "the local_health rules with cluster-decoded GPU snapshots.", - ) - opt.add_argument( - "--no-robustness-enable-cluster-pod-metrics", - dest="robustness_enable_cluster_pod_metrics", - action="store_false", - help="Force enable_cluster_pod_metrics=false.", - ) - opt.add_argument( - "--robustness-pod-metrics-categories", - dest="robustness_pod_metrics_categories", - type=str, - default=None, - help="Comma-separated metric categories forwarded into " - "pod_metrics_categories (e.g. 'gpu,memory'). Default 'gpu' is " - "applied by the runtime when this flag is omitted.", - ) opt.add_argument( "--orch-prompt", type=str, default=None, help="Override Orchestration system prompt (file path or inline)" ) @@ -1403,16 +1352,7 @@ def _build_parser() -> argparse.ArgumentParser: default=None, help="KERNEL plateau: number of trailing integrate attempts the gain sum is computed over. Default 5.", ) - # IR-6 — EXPLORE hard force-exit thresholds (either condition fires; locked at start). - opt.add_argument( - "--explore-force-exit-hours-remaining", - dest="explore_force_exit_hours_remaining", - type=float, - default=None, - help="EXPLORE force-exit: total wall-clock remaining (hours) " - "below which EXPLORE exits immediately to the next phase, " - "regardless of plateau / steward. Default 3.0 (IR-6).", - ) + # IR-6 — EXPLORE hard force-exit threshold (locked at start). opt.add_argument( "--explore-force-exit-budget-pct", dest="explore_force_exit_budget_pct", diff --git a/src/hyperloom/inference_optimizer/cli/preflight.py b/src/hyperloom/inference_optimizer/cli/preflight.py index ef0d818610..14f97d20e2 100644 --- a/src/hyperloom/inference_optimizer/cli/preflight.py +++ b/src/hyperloom/inference_optimizer/cli/preflight.py @@ -46,7 +46,6 @@ from ..session.paths import ( DEFAULT_SESSION_DIR, ENV_USER_DATA_PATH, - find_latest_per_session_dir, session_dir as _session_dir_resolve, workspace_root as _workspace_root_resolve, ) @@ -1164,8 +1163,8 @@ def _install_pinned_lm_eval(python_exe: str, pip_extra: list[str]) -> None: def _resolved_eval_disabled(args: argparse.Namespace) -> bool: """Effective ``--no-eval`` for this launch, flag or persisted. - Preflight runs before the ``--resume`` block reads ``state.json``, so a - resume that inherits the flag instead of re-passing it must be read here. + Preflight runs before the resume block reads ``state.json``, so a resume + that inherits the flag instead of re-passing it must be read here. Args: args (argparse.Namespace): The parsed ``optimize`` args. @@ -1175,12 +1174,10 @@ def _resolved_eval_disabled(args: argparse.Namespace) -> bool: """ if bool(getattr(args, "no_eval", False)): return True - if not bool(getattr(args, "resume", False)): - return False raw = str(getattr(args, "resume_from", "") or "").strip() - resumed = Path(raw).expanduser() if raw else find_latest_per_session_dir() - if resumed is None: + if not raw: return False + resumed = Path(raw).expanduser() try: state = json.loads((resumed / "state.json").read_text(encoding="utf-8")) except (OSError, ValueError): @@ -1464,7 +1461,7 @@ def _check_tracelens_cli() -> None: """Hard-gate TraceLens CLI presence — abort before Coordinator starts (SKILL IR-2). Pod-local /opt/venv/bin/TraceLens_* console_scripts don't persist across pod restarts, so install.sh - must run before every launch (carve-out: --resume in the same shell). Fail-fast beats a delayed + must run before every launch (carve-out: --resume-from in the same shell). Fail-fast beats a delayed tracelens_cli_missing strike at tick ~6 after baseline burned setup time. """ missing = [name for name in _TRACELENS_REQUIRED_CLIS if shutil.which(name) is None] @@ -1477,7 +1474,7 @@ def _check_tracelens_cli() -> None: f"src/hyperloom/agents/kernel/scripts/install.sh (chained from " f"src/hyperloom/inference_optimizer/assets/install.sh) and do NOT persist " f"across pod restarts. SKILL IR-2 requires running install.sh " - f"before every launch (carve-out applies only to --resume in " + f"before every launch (carve-out applies only to --resume-from in " f"the same shell that earlier ran install.sh). Re-run:\n" f" bash $REPO_ROOT/src/hyperloom/inference_optimizer/assets/install.sh\n" f" . {session_dir}/runtime/kernel-agent.env.sh\n" diff --git a/src/hyperloom/inference_optimizer/cli/quantization.py b/src/hyperloom/inference_optimizer/cli/quantization.py index 5cbed2c878..b33fe94492 100644 --- a/src/hyperloom/inference_optimizer/cli/quantization.py +++ b/src/hyperloom/inference_optimizer/cli/quantization.py @@ -49,8 +49,8 @@ async def _run_quantization_prelude(args: argparse.Namespace) -> None: kernel) optimizes the quantized model instead of the source. Contract: - * Skipped on ``--resume`` (a resumed session already has its model - pinned in the manifest; re-quantizing would diverge from it). + * Reached only on the fresh-launch path; a resumed session takes its + model from the manifest and never re-quantizes. * On a failed/blocked quantization the process exits with code 3 — we must not silently fall through and optimize the un-quantized source model when the user explicitly asked for quantization. @@ -65,8 +65,8 @@ async def _run_quantization_prelude(args: argparse.Namespace) -> None: Args: args: Parsed CLI arguments; reads ``quantize`` / ``quantize_scheme`` / - ``gpu_type`` / ``resume`` and rewrites ``args.model`` in place to - the exported quantized model path on success. + ``gpu_type`` and rewrites ``args.model`` in place to the exported + quantized model path on success. """ # Free-text --quantize wins; otherwise resolve the structured # --quantize-scheme enum (the UI/backend path) to a prompt. @@ -99,9 +99,6 @@ async def _run_quantization_prelude(args: argparse.Namespace) -> None: prompt = resolve_scheme_prompt(scheme) if not prompt: return - if getattr(args, "resume", False): - print("Quantization prelude: skipped (--resume); using model from manifest.") - return # Deterministic master switch: quantization runs ONLY when # $HYPERLOOM_QUANTIZE_ENABLED is truthy, regardless of the flags. Absent / diff --git a/src/hyperloom/inference_optimizer/multi_node/SKILL.md b/src/hyperloom/inference_optimizer/multi_node/SKILL.md index 69796a43ee..7762967fa1 100644 --- a/src/hyperloom/inference_optimizer/multi_node/SKILL.md +++ b/src/hyperloom/inference_optimizer/multi_node/SKILL.md @@ -220,14 +220,17 @@ in-flight launch (`MULTI_NODE_RESTART_RESUME_RUNNING=1`, default). by `state.backend`; on infera GEAK runs on a GPU pod over SSH, installed once per cluster via `install-geak`). The integrate path auto-restarts the server after `apply-patch` — do not restart manually. -* **Robustness auto-downgrades to mock on `nodes >= 2`.** The agent's - LocalProbe only sees sandbox-local resources, so on multi-node every probe - (`ray_head_dead`, `local_server_unreachable`, `gpu_memory_leaked`, …) is a - false positive. The CLI forces `--robustness-mock` (heartbeat-only; warns only - if `--robustness-agent` was explicit) — unless a robustness server is - configured (`--robustness-server-url` / `$ROBUSTNESS_SERVER_URL`), which keeps - the agent backend on its cluster-wide signal. Shell-level health monitoring - (`optimizer_runs/robustness_monitor.sh`, auto-resume on terminal +* **Robustness runs the real agent on `nodes >= 2`, with LocalProbe off.** The + probe only sees sandbox-local resources, so on multi-node every probe-derived + symptom (`ray_head_dead`, `local_server_unreachable`, `gpu_memory_leaked`, …) + would be a false positive; `disable_local_probe` defaults to True there and + swaps the probe for a silent stub. What remains is the node-agnostic set the + agent reads straight off the Coordinator prompt and inbox: the deadline / + budget ladder, `gain_plateau`, `no_levers_found`, crash escalation, + `phase_budget_nearly_exhausted`, `conversation_no_progress`, plus the + inbox-driven `agent_stall` / `repeated_failure` / `repeated_policy_denied` + family. Pass `--robustness-mock` for heartbeat-only. Shell-level health + monitoring (`optimizer_runs/robustness_monitor.sh`, auto-resume on terminal `stop_reason`) is unaffected. ## Exit Codes diff --git a/src/hyperloom/inference_optimizer/multi_node/_internal/infera_support.py b/src/hyperloom/inference_optimizer/multi_node/_internal/infera_support.py index 3a76f96691..f3b1b5a214 100644 --- a/src/hyperloom/inference_optimizer/multi_node/_internal/infera_support.py +++ b/src/hyperloom/inference_optimizer/multi_node/_internal/infera_support.py @@ -161,73 +161,6 @@ def _classify_pod_role( return None -def discover_role_pods( - workload: dict[str, Any], - *, - pd_mode: str = "aggregated", - ssh_port_base: int = DEFAULT_SSH_PORT, -) -> dict[str, list[dict[str, Any]]]: - """Group a SaFE GetWorkloadResponse's pods by role. - - ``pd_mode`` selects the positional serviceRoles used to map a pod's slot - index (resourceId / ``-role-``) to its role. Returns - ``{"frontend": [...], "prefill": [...], "decode": [...], "worker": [...]}``; - each entry is ``{"podId", "podIP", "role", "lwsIndex", "sshPort"}`` for pods - with a non-empty ``podIP``, sorted by LWS ordinal (leader = 0) for - deterministic rank order. - - Args: - workload: A SaFE GetWorkloadResponse mapping with a ``pods`` list. - pd_mode: Deployment topology selecting the positional service roles. - ssh_port_base: Base SSH port the pods were deployed with. - - Returns: - A mapping of role to its list of pod SSH target dicts, sorted by LWS - ordinal then pod id. - """ - service_roles = _service_roles_for(pd_mode) - groups: dict[str, list[dict[str, Any]]] = { - "frontend": [], - "prefill": [], - "decode": [], - "worker": [], - } - for p in workload.get("pods") or []: - if not isinstance(p, dict): - continue - pod_ip = str(p.get("podIP") or "").strip() - if not pod_ip: - continue - # Skip terminal / dead pods. A IDEP role pod that crashed during early - # scheduling lingers in GetWorkload.pods with a stale podIP but no sshd - # (phase=Failed/Succeeded). Including it makes restart-server SSH-fan-out - # to a dead replica -> "Connection refused" rc=1 -> baseline_failed. - # The live replacement replica (same role index) is the one we want. - pod_phase = str(p.get("phase") or "").strip().lower() - if pod_phase in ("failed", "succeeded", "terminating"): - continue - pod_id = str(p.get("podId") or "") - role = _classify_pod_role(pod_id, p.get("resourceId"), service_roles) - if role is None: - continue - lws_idx = _parse_lws_ordinal(pod_id) - groups[role].append( - { - "podId": pod_id, - "podIP": pod_ip, - "role": role, - "lwsIndex": lws_idx, - "sshPort": ssh_port_for_pod(role, lws_idx, ssh_port_base=ssh_port_base), - } - ) - for role in groups: - groups[role].sort( - key=lambda d: ( - d["lwsIndex"] if isinstance(d["lwsIndex"], int) else 1 << 30, - d["podId"], - ) - ) - return groups def pod_targets_from_lists( @@ -296,47 +229,6 @@ def _parse_lws_ordinal(pod_id: str) -> int | None: return int(tail) if tail.isdigit() else None -def frontend_service_url( - workload_id: str, - namespace: str, - service_info: dict[str, Any] | None = None, - *, - port: int = INFERA_FRONTEND_PORT, -) -> str: - """Resolve the Infera frontend base URL for benchmarks. - - Prefers the live service info (clusterIp / dns) when present; falls - back to the conventional ``http://..svc.cluster.local:``. - - Args: - workload_id: The Infera workload id. - namespace: The Kubernetes namespace the workload runs in. - service_info: Optional live service info (internalDomain / dns / - clusterIp / port). - port: Default frontend HTTP port used when none is in ``service_info``. - - Returns: - The resolved frontend base URL. - """ - if service_info: - # Prefer the ready-made internalDomain. - internal = str(service_info.get("internalDomain") or "").strip() - if internal: - internal = internal.split("://", 1)[-1].rstrip("/") - return f"http://{internal}" - # ``port`` may be a nested object {protocol, port, targetPort} or a bare int. - raw_port = service_info.get("port") - if isinstance(raw_port, dict): - svc_port = raw_port.get("port") or raw_port.get("targetPort") or port - else: - svc_port = raw_port or port - dns = str(service_info.get("dns") or service_info.get("dnsName") or "").strip() - cluster_ip = str(service_info.get("clusterIp") or "").strip() - host = dns or cluster_ip - if host: - host = host.split("://", 1)[-1].rstrip("/") - return f"http://{host}:{svc_port}" - return f"http://{workload_id}.{namespace}.svc.cluster.local:{port}" # sglang PD bootstrap rendezvous port (SaFE common.InferaBootstrapPort). diff --git a/src/hyperloom/inference_optimizer/multi_node/_internal/ray_dashboard.py b/src/hyperloom/inference_optimizer/multi_node/_internal/ray_dashboard.py index 146ee464f1..fc23600404 100644 --- a/src/hyperloom/inference_optimizer/multi_node/_internal/ray_dashboard.py +++ b/src/hyperloom/inference_optimizer/multi_node/_internal/ray_dashboard.py @@ -64,20 +64,6 @@ def __init__(self, status: int | None, body: str, *, endpoint: str) -> None: self.endpoint = endpoint -def ray_gcs_address(head_pod_ip: str) -> str: - """Ray driver address for ``ray.init(address=...)`` (GCS on head, default port). - - Args: - head_pod_ip (str): The head pod IP or host. - - Returns: - str: ``:6379`` for a non-empty input, otherwise an empty string. - """ - ip = (head_pod_ip or "").strip() - if not ip: - return "" - return f"{ip}:6379" - def dashboard_url(head_pod_ip: str) -> str: """Build the Ray Dashboard base URL for a given head pod IP. diff --git a/src/hyperloom/inference_optimizer/multi_node/_internal/ssh_client.py b/src/hyperloom/inference_optimizer/multi_node/_internal/ssh_client.py index 0e74a73e29..193ce2d275 100644 --- a/src/hyperloom/inference_optimizer/multi_node/_internal/ssh_client.py +++ b/src/hyperloom/inference_optimizer/multi_node/_internal/ssh_client.py @@ -320,44 +320,3 @@ def ssh_run_bash_with_env( timeout=timeout, ) - -def probe_ssh( - host: str, - *, - key_path: Path | str, - known_hosts: Path, - port: int = DEFAULT_SSH_PORT, - user: str = "root", - timeout: int = 20, -) -> bool: - """Return True iff a trivial SSH command succeeds (pod reachable + key OK). - - Args: - host: The target host/IP. - key_path: Path to the private SSH key. - known_hosts: Session known_hosts file for host-key verification. - port: The remote sshd port. - user: The remote login user. - timeout: Subprocess timeout in seconds. - - Returns: - ``True`` when a trivial probe command succeeds (pod reachable and key - accepted), ``False`` otherwise (including on timeout). - """ - try: - cp = ssh_run( - host, - "echo mn_ssh_ok", - key_path=key_path, - known_hosts=known_hosts, - port=port, - user=user, - timeout=timeout, - ) - except subprocess.TimeoutExpired: - warn(f"ssh probe to {host}:{port} timed out") - return False - ok = cp.returncode == 0 and "mn_ssh_ok" in (cp.stdout or "") - if not ok: - warn(f"ssh probe to {host}:{port} failed rc={cp.returncode} stderr={(cp.stderr or '').strip()[:200]}") - return ok diff --git a/src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py b/src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py index d0f988c3f6..93dd3c93cb 100755 --- a/src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py +++ b/src/hyperloom/inference_optimizer/multi_node/scripts/launch_multinode.py @@ -60,8 +60,9 @@ def _pd_decode_dist_init_port(prefill_dist_init_port: int) -> int: # sglang PD bootstrap (KV transfer rendezvous) port; override via --pd-bootstrap-port. _PD_DEFAULT_BOOTSTRAP_PORT = 8998 -# Max seconds to wait for ray.nodes() to surface every expected pod. -_NODES_DISCOVERY_TIMEOUT_SEC = 120 +# Max seconds to wait for ray.nodes() to surface every expected pod; raise it +# when pods queue behind a busy scheduler or a cold image pull. +_NODES_DISCOVERY_TIMEOUT_SEC = int(os.environ.get("RAY_NODES_DISCOVERY_TIMEOUT_SEC", "120")) # rank-0 /health probe budget (cold MoE can exceed it; --no-wait-health to bypass). _HEALTH_PROBE_TIMEOUT_SEC = int(os.environ.get("SGLANG_HEALTH_PROBE_TIMEOUT_SEC", "1800")) diff --git a/src/hyperloom/inference_optimizer/references/internals.md b/src/hyperloom/inference_optimizer/references/internals.md index cdf8c7f81c..70fee68246 100644 --- a/src/hyperloom/inference_optimizer/references/internals.md +++ b/src/hyperloom/inference_optimizer/references/internals.md @@ -30,10 +30,11 @@ lives in `orchestrator/prompts/orchestration.md`. In brief: (any port that is not the production serving port 8888), profile, autotune, and run real benchmark loops. The one invariant is that they must not touch the production serving process, its cards, or port 8888. -- **IR-6 HARD force-exit**: EXPLORE exits the moment wall-clock remaining < - `--explore-force-exit-hours-remaining` (default 3.0 h) OR phase budget < - `--explore-force-exit-budget-pct` (default 20%). Non-negotiable — leaves - buffer for KERNEL_AGENT → SWEEP → CLOSE + report. +- **IR-6 HARD force-exit**: EXPLORE exits the moment the unspent fraction of its + own phase budget drops to `--explore-force-exit-budget-pct` (default 20%). + Non-negotiable. The buffer for KERNEL_AGENT → SWEEP → CLOSE + report is already + inside that fraction, since charge-back rebuilds the allotment from the time + left at phase entry; there is no session-remaining arm. - **Plateau advisory**: EXPLORE / KERNEL_AGENT / FRAMEWORK plateau signals are computed every tick and rendered as advisory in the orchestration prompt. They do NOT drive phase advance — the LLM may emit diff --git a/src/hyperloom/inference_optimizer/references/operations.md b/src/hyperloom/inference_optimizer/references/operations.md index 091376f1c1..3fe0d70f92 100644 --- a/src/hyperloom/inference_optimizer/references/operations.md +++ b/src/hyperloom/inference_optimizer/references/operations.md @@ -112,7 +112,7 @@ echo $! > "$PID_FILE" `setsid nohup ... &` is required for runs longer than 5 minutes. The `$!` written above is the **setsid wrapper** PID, which exits immediately — the -robustness monitor reads `$PID_FILE` and would misfire a spurious `--resume` if +robustness monitor reads `$PID_FILE` and would misfire a spurious resume if it kept the dead wrapper PID. After launch, reconcile `$PID_FILE` to the **real** optimizer PID, which the CLI records as `.pid` in the launch-info JSON. @@ -143,15 +143,15 @@ test -f "$SESSION_DIR/state.json" && echo "state_exists=true" ## Resume Existing Session -For monitored runs, prefer explicit same-session resume: -`--resume --resume-from "$SESSION_DIR"`. Bare `--resume` auto-picks the latest -`$USER_DATA_PATH///` and is only acceptable for manual recovery -when that is the intended session. Keep `$USER_DATA_PATH` at the workspace root -so `runtime/kernel-agent.env.sh` resolves. The selected session must contain +`--resume-from "$SESSION_DIR"` is the only way to resume; the CLI never +chooses a session for you. Take `$SESSION_DIR` from the launch-info JSON or +the `HYPERLOOM_LAUNCH` line, never from the newest timestamp dir under +`$USER_DATA_PATH//`. Keep `$USER_DATA_PATH` at the workspace root +so `runtime/kernel-agent.env.sh` resolves. The named session must contain `manifest.json` and `state.json`. Reuse the launch template with these diffs: drop `--model`, add -`--resume --resume-from "$SESSION_DIR"`, and set +`--resume-from "$SESSION_DIR"`, and set `RUN_TAG="resume-$(date +%Y%m%d_%H%M%S)"`. Set `$FRAMEWORK` when resuming a non-default session. @@ -160,9 +160,9 @@ non-default session. For runs longer than 5 minutes, start the monitor in its own `setsid nohup` process. It polls every 300s. It reads `$INFERENCE_OPTIMIZER_SESSION_DIR` first, else `.session_dir` from `$LAUNCH_INFO_FILE`. Its only allowed relaunch is the -same session via `optimize --resume --resume-from "$SESSION_DIR"` after the +same session via `optimize --resume-from "$SESSION_DIR"` after the optimizer process disappears without a terminal marker; it must not start a -fresh run or auto-pick the latest session. +fresh run. ```bash export RUN_DIR="${USER_DATA_PATH:-/workspace/hyperloom}/optimizer_runs" diff --git a/src/hyperloom/inference_optimizer/references/paths.md b/src/hyperloom/inference_optimizer/references/paths.md index 8510bd1d29..7b4410aac5 100644 --- a/src/hyperloom/inference_optimizer/references/paths.md +++ b/src/hyperloom/inference_optimizer/references/paths.md @@ -12,7 +12,7 @@ $USER_DATA_PATH/ # workspace_root — set by operator / │ ├── kernel-agent.env.sh ├── logs/ # workspace-shared launcher stdout └── / # e.g. DeepSeek-R1-0528, deepseek-ai-DeepSeek-V3 - └── / # session_dir — manifest.json, state.json, runs/, … + └── -/ # session_dir — manifest.json, state.json, runs/, … ├── manifest.json ├── state.json ├── storage/coordinator.db @@ -29,10 +29,10 @@ $USER_DATA_PATH/ # workspace_root — set by operator / run-scoped path *before* the optimizer starts, e.g. `/hyperloom/users//deepseek-ai-DeepSeek-V3-20260522_034024/`. That outer directory is **platform isolation** (one Claw job). The optimizer then creates -`//` inside it. Full session path example: +`/-/` inside it. Full session path example: /hyperloom/users//deepseek-ai-DeepSeek-V3-20260522_034024/ ← USER_DATA_PATH (Claw) - deepseek-ai-DeepSeek-V3/20260522T035359Z/ ← session_dir (optimizer) + deepseek-ai-DeepSeek-V3/20260522T035359Z-9f3c1a04/ ← session_dir (optimizer) ## Path resolution (do not guess) diff --git a/src/hyperloom/inference_optimizer/references/quantization.md b/src/hyperloom/inference_optimizer/references/quantization.md index 188653adcc..f9dfa71c17 100644 --- a/src/hyperloom/inference_optimizer/references/quantization.md +++ b/src/hyperloom/inference_optimizer/references/quantization.md @@ -34,7 +34,7 @@ Rules: - `mxfp4*` schemes are MI355X-only. - Keep `--precision` consistent with quantization, for example `--quantize-scheme fp8` with `--precision fp8`. -- Quantization is one-shot and skipped on `--resume`. +- Quantization is one-shot and never runs on a resume. - Failed or unusable quantization hard-stops with `SystemExit(3)`; it never silently optimizes the source model. - A scheme/GPU mismatch via `--quantize-scheme` is skipped, emits diff --git a/src/hyperloom/inference_optimizer/references/troubleshooting.md b/src/hyperloom/inference_optimizer/references/troubleshooting.md index de6a5b9489..ed0aa63ff5 100644 --- a/src/hyperloom/inference_optimizer/references/troubleshooting.md +++ b/src/hyperloom/inference_optimizer/references/troubleshooting.md @@ -101,5 +101,5 @@ Bypass with `--critic-mock` for offline / smoke runs. See `=== Recent policy denials ===` block, then change something substantive (a new `params.grid` variant, a different `benchmark_script`, or a sibling action family). Do not hand-edit `state.json`. -- `stop_reason=time_exhausted`: resume same session (`--resume`); do not start - fresh. +- `stop_reason=time_exhausted`: resume same session (`--resume-from`); do not + start fresh. diff --git a/src/hyperloom/inference_optimizer/session/lock.py b/src/hyperloom/inference_optimizer/session/lock.py index 647201500b..b91e9261bb 100644 --- a/src/hyperloom/inference_optimizer/session/lock.py +++ b/src/hyperloom/inference_optimizer/session/lock.py @@ -4,7 +4,7 @@ """Single-optimizer session lock. A long ``python -m hyperloom.inference_optimizer.cli optimize`` run is guarded by a robustness monitor -that re-launches the optimizer via ``--resume`` if it judges the process dead. +that re-launches the optimizer via ``--resume-from`` if it judges the process dead. During the slow serving cold-start that liveness check can misfire and spawn a **second** optimizer on the same ``session_dir``; the two then contend for the shared ``coordinator.db`` leases and both write ``state.json``, corrupting the @@ -125,7 +125,7 @@ class SessionLock: """Exclusive, crash-safe, single-optimizer-per-session lock. Acquire once at optimizer startup (both fresh ``optimize`` and - ``--resume``); hold for the whole run. Use as a context manager or call + ``--resume-from``); hold for the whole run. Use as a context manager or call :meth:`acquire` / :meth:`release` explicitly. """ diff --git a/src/hyperloom/inference_optimizer/session/manifest.py b/src/hyperloom/inference_optimizer/session/manifest.py index b72b6e6faa..6335f468ff 100644 --- a/src/hyperloom/inference_optimizer/session/manifest.py +++ b/src/hyperloom/inference_optimizer/session/manifest.py @@ -425,8 +425,8 @@ def build_manifest( "created_at_utc": now_iso(timespec="seconds"), "session_dir": str(session_dir), # USER_DATA_PATH root snapshotted so a trace-based consumer can locate - # the on-disk artifacts. Falls back to the resolved workspace_root(). - "user_data_path": (os.environ.get("USER_DATA_PATH") or "").strip() or str(_paths.workspace_root()), + # the on-disk artifacts. + "user_data_path": str(_paths.workspace_root()), "model_path": model_path, "model_name": model_name, "framework": framework or "sglang", @@ -496,7 +496,7 @@ def write_manifest( def load_manifest(session_dir: Path) -> dict[str, Any]: """Read ``manifest.json`` for an existing session. Raises - ``FileNotFoundError`` if missing (the signal ``--resume`` uses to refuse a + ``FileNotFoundError`` if missing (the signal ``--resume-from`` uses to refuse a fresh sandbox). Args: @@ -511,7 +511,7 @@ def load_manifest(session_dir: Path) -> dict[str, Any]: p = manifest_path(Path(session_dir)) if not p.exists(): raise FileNotFoundError( - f"manifest.json not found under {session_dir} — the session was never initialised; cannot --resume" + f"manifest.json not found under {session_dir} — the session was never initialised; cannot resume" ) with p.open(encoding="utf-8") as f: return json.load(f) diff --git a/src/hyperloom/inference_optimizer/session/paths.py b/src/hyperloom/inference_optimizer/session/paths.py index d00e58f707..d940ab0c1e 100644 --- a/src/hyperloom/inference_optimizer/session/paths.py +++ b/src/hyperloom/inference_optimizer/session/paths.py @@ -19,6 +19,7 @@ import os import re from pathlib import Path +from uuid import uuid4 from hyperloom.common.timeutil import utc_now_compact @@ -44,7 +45,7 @@ # Per-session directory skeleton mkdir-ed by make_session_dir(). Splits into # workspace-shared roots (runtime/, logs/ — one per $USER_DATA_PATH) and -# per-session roots (one per model+timestamp). +# per-session roots (one per launch). _SESSION_SKELETON: tuple[str, ...] = ( "storage", "personas", @@ -169,53 +170,16 @@ def session_dir() -> Path: return workspace_root() -def find_latest_per_session_dir( - model_name: str | os.PathLike[str] | None = None, -) -> Path | None: - """Latest per-session subdir under :func:`workspace_root` (used by - ``--resume`` without ``--resume-from``). Selects by the - ``%Y%m%dT%H%M%SZ`` timestamp in the directory name (lex sort), not mtime. - Returns None when no matching subdir exists. - - Args: - model_name: Restrict the scan to one model's subtree, or ``None`` to - scan every model basename. - - Returns: - The latest per-session directory, or ``None`` when none match. - """ - ws = workspace_root() - if not ws.is_dir(): - return None - if model_name: - basename = _sanitize_model_basename(model_name) - model_root = ws / basename - if not model_root.is_dir(): - return None - candidates = [p for p in model_root.iterdir() if p.is_dir() and len(p.name) == 16 and p.name.endswith("Z")] - else: - # Scan every model_basename subdir; the timestamp-shaped name check - # skips workspace-shared subdirs. - candidates: list[Path] = [] - for model_dir in ws.iterdir(): - if not model_dir.is_dir() or model_dir.name in ("runtime", "logs"): - continue - for p in model_dir.iterdir(): - if p.is_dir() and len(p.name) == 16 and p.name.endswith("Z"): - candidates.append(p) - if not candidates: - return None - candidates.sort(key=lambda p: p.name) # lex == chronological for ts names - return candidates[-1] - - def make_session_dir(model_name: str | os.PathLike[str] | None = None) -> Path: """Create the session directory + per-session + workspace-shared skeletons. With a ``model_name`` the session_dir is - ``///`` and is pinned via + ``//-/`` and is pinned via ``$INFERENCE_OPTIMIZER_CURRENT_SESSION_DIR``; otherwise it is workspace_root. Idempotent. + The random suffix keeps two same-second launches of one model apart; the + fixed-width timestamp stays first so lexical order remains chronological. + Args: model_name: Model name selecting the per-model subtree, or ``None`` to use workspace_root directly. @@ -230,8 +194,7 @@ def make_session_dir(model_name: str | os.PathLike[str] | None = None) -> Path: if model_name: basename = _sanitize_model_basename(model_name) - ts = utc_now_compact() - sd = ws / basename / ts + sd = ws / basename / f"{utc_now_compact()}-{uuid4().hex[:8]}" else: sd = ws @@ -425,6 +388,5 @@ def mn_profile_trace_root() -> Path: "resolve_dep_dir", "runtime_dir", "session_dir", - "find_latest_per_session_dir", "workspace_root", ] diff --git a/src/hyperloom/inference_optimizer/session/session_paths.py b/src/hyperloom/inference_optimizer/session/session_paths.py index cbe4e84623..109dce402b 100644 --- a/src/hyperloom/inference_optimizer/session/session_paths.py +++ b/src/hyperloom/inference_optimizer/session/session_paths.py @@ -97,15 +97,13 @@ def _validate_action(action: str) -> str: def _validate_id_component(value: str, *, field: str) -> str: - """Reject path-traversal in an LLM-controlled single-segment id. + """Reject blank ids and path-traversal in an LLM-controlled single-segment id. - Legitimate ids (uuid hex, ``k001``) never contain a separator or ``..``; - anything that does could relocate a per-task sandbox and is refused here. + Legitimate ids (uuid hex, ``k001``) are never blank and never contain a + separator or ``..``; either would relocate a per-task sandbox. """ - v = str(value or "").strip() or "unknown" - if v == "unknown": - return v - if v == "." or "/" in v or "\\" in v or ".." in Path(v).parts or Path(v).is_absolute(): + v = str(value or "").strip() + if not v or v == "." or "/" in v or "\\" in v or ".." in Path(v).parts or Path(v).is_absolute(): raise ValueError(f"{field}: unsafe path component {value!r}") return v @@ -134,14 +132,14 @@ def runs_dir(session_dir: Path, action: str, task_id: str) -> Path: session_dir (Path): The session root directory. action (str): The owning action name; validated against the runs-workspace action set. - task_id (str): The task identifier; blank/empty falls back to - ``"unknown"``. + task_id (str): The task identifier; must be non-blank. Returns: Path: The absolute path to ``/runs//``. Raises: - ValueError: If ``action`` is not a recognised runs-workspace action. + ValueError: If ``action`` is not a recognised runs-workspace action, or + ``task_id`` is blank or path-like. """ a = _validate_action(action) tid = _validate_id_component(task_id, field="runs_dir.task_id") @@ -161,14 +159,14 @@ def unique_runs_dir(session_dir: Path, action: str, task_id: str) -> Path: session_dir (Path): The session root directory. action (str): The owning action name; validated against the runs-workspace action set. - task_id (str): The task identifier; blank/empty falls back to - ``"unknown"``. + task_id (str): The task identifier; must be non-blank. Returns: Path: The newly created workspace directory. Raises: - ValueError: If ``action`` is not a recognised runs-workspace action. + ValueError: If ``action`` is not a recognised runs-workspace action, or + ``task_id`` is blank or path-like. RuntimeError: If every suffix up to ``_MAX_RUNS_DIR_ATTEMPTS`` is taken. """ base = runs_dir(session_dir, action, task_id) @@ -202,8 +200,7 @@ def kernel_agent_runs_dir(session_dir: Path, session_id: str) -> Path: Args: session_dir: The session root directory. - session_id: Tool-invocation session id; blank falls back to - ``"unknown"``. + session_id: Tool-invocation session id; must be non-blank. Returns: ``/kernel-agent/runs/``. @@ -218,8 +215,7 @@ def patches_dir(session_dir: Path, kernel_id: str) -> Path: Args: session_dir: The session root directory. - kernel_id: Kernel id keying the patch dir; blank falls back to - ``"unknown"``. + kernel_id: Kernel id keying the patch dir; must be non-blank. Returns: ``/patches/``. diff --git a/src/hyperloom/inference_optimizer/tests/test_breakdown_assembler_merge_units.py b/src/hyperloom/inference_optimizer/tests/test_breakdown_assembler_merge_units.py new file mode 100644 index 0000000000..f85735b9fa --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_breakdown_assembler_merge_units.py @@ -0,0 +1,157 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""Fragment merge semantics in the breakdown assembler. + +Producers write partial fragments from deep inside their own work, and a second +write of the same entity id merges into the first. What that merge keeps, and +what it silently replaces, decides whether a recorded fact survives to the +archive -- so these pin the merge rather than any one caller's output. +""" + +from __future__ import annotations + +from hyperloom.inference_optimizer.breakdown.recorder import assembler as asm + + +# ---- _deep_merge ---- + + +def test_a_partial_update_fills_gaps_without_dropping_prior_fields(): + merged = asm._deep_merge({"a": 1, "b": 2}, {"b": 2, "c": 3}) + assert merged == {"a": 1, "b": 2, "c": 3} + + +def test_nested_dicts_merge_key_by_key_rather_than_wholesale(): + """Whole-value replacement is what loses a sibling the update never mentioned.""" + merged = asm._deep_merge({"m": {"x": 1, "y": 2}}, {"m": {"y": 2, "z": 3}}) + assert merged["m"] == {"x": 1, "y": 2, "z": 3} + + +def test_filling_an_absent_field_is_not_a_conflict(): + conflicts: list[str] = [] + asm._deep_merge({"a": None}, {"a": 5}, conflicts=conflicts) + asm._deep_merge({}, {"b": 5}, conflicts=conflicts) + assert conflicts == [] + + +def test_replacing_an_answer_with_a_different_one_is_reported(): + """Last-writer-wins is the behaviour; the point is that it is visible.""" + conflicts: list[str] = [] + merged = asm._deep_merge({"tput": 5081.01}, {"tput": 5100.76}, conflicts=conflicts) + assert merged["tput"] == 5100.76 + assert conflicts == ["tput"] + + +def test_a_conflict_is_reported_at_its_full_path(): + conflicts: list[str] = [] + asm._deep_merge({"outer": {"inner": 1}}, {"outer": {"inner": 2}}, conflicts=conflicts) + assert conflicts == ["outer.inner"] + + +def test_conflicts_are_only_collected_when_the_caller_asks(): + assert asm._deep_merge({"a": 1}, {"a": 2}) == {"a": 2} + + +# ---- _merge_lists ---- + + +def test_rows_with_the_same_nested_id_merge_instead_of_duplicating(): + merged = asm._merge_lists( + [{"attempt_id": "a1", "status": "running"}], + [{"attempt_id": "a1", "gain_pct": 3.0}], + ) + assert merged == [{"attempt_id": "a1", "status": "running", "gain_pct": 3.0}] + + +def test_a_new_id_appends_rather_than_overwriting(): + merged = asm._merge_lists( + [{"attempt_id": "a1"}], + [{"attempt_id": "a2"}], + ) + assert [row["attempt_id"] for row in merged] == ["a1", "a2"] + + +def test_a_second_row_for_a_new_id_still_merges_onto_the_first(): + """The index has to learn ids the update itself introduced.""" + merged = asm._merge_lists( + [], + [{"attempt_id": "a1", "status": "running"}, {"attempt_id": "a1", "status": "done"}], + ) + assert merged == [{"attempt_id": "a1", "status": "done"}] + + +def test_rows_without_a_known_id_are_appended_once(): + merged = asm._merge_lists([{"note": "x"}], [{"note": "x"}, {"note": "y"}]) + assert merged == [{"note": "x"}, {"note": "y"}] + + +def test_scalar_entries_deduplicate(): + assert asm._merge_lists(["a"], ["a", "b"]) == ["a", "b"] + + +def test_a_list_under_a_key_merges_by_id_through_deep_merge(): + merged = asm._deep_merge( + {"attempts": [{"attempt_id": "a1", "status": "running"}]}, + {"attempts": [{"attempt_id": "a1", "status": "kept"}]}, + ) + assert merged["attempts"] == [{"attempt_id": "a1", "status": "kept"}] + + +def test_the_first_recognised_id_field_decides_identity(): + """A row carrying two id fields is keyed on the first in the registry order.""" + merged = asm._merge_lists( + [{"attempt_id": "a1", "measurement_id": "m1", "n": 1}], + [{"attempt_id": "a1", "measurement_id": "m2", "n": 2}], + ) + assert len(merged) == 1 + assert merged[0]["n"] == 2 + + +# ---- substream composition ---- + + +def test_versions_fold_into_one_entry_per_tool(): + out = {"versions": [{"tool": "TraceLens", "v": "1"}, {"tool": "tracelens", "v": "2"}]} + asm._compose_versions(out) + assert out["versions"] == {"tracelens": {"tool": "tracelens", "v": "2"}} + + +def test_versions_ignore_rows_that_name_no_tool(): + out = {"versions": [{"v": "1"}, "junk"]} + asm._compose_versions(out) + assert out["versions"] == {} + + +def test_versions_are_left_alone_when_nothing_was_recorded(): + out = {} + asm._compose_versions(out) + assert "versions" not in out + + +def test_critic_and_robustness_substreams_fold_into_one_section(): + out = { + "critic_iterations": [{"iteration": 1}], + "robustness_signals": [{"name": "gain_plateau"}], + } + asm._compose_critic_robustness(out) + + assert "critic_iterations" not in out and "robustness_signals" not in out + section = out["critic_robustness"] + assert section["critic_iterations"] == [{"iteration": 1}] + assert section["robustness_signals"] == [{"name": "gain_plateau"}] + assert "kb_writes_summary" in section + + +def test_a_recorded_section_outranks_the_substreams(): + """A producer that wrote the whole section already said what it means.""" + out = {"critic_robustness": {"critic_iterations": ["kept"]}, "critic_iterations": [{"i": 1}]} + asm._compose_critic_robustness(out) + assert out["critic_robustness"] == {"critic_iterations": ["kept"]} + assert "critic_iterations" not in out + + +def test_composing_critic_robustness_is_a_no_op_without_either_substream(): + out = {"other": 1} + asm._compose_critic_robustness(out) + assert out == {"other": 1} diff --git a/src/hyperloom/inference_optimizer/tests/test_breakdown_exporter_unit.py b/src/hyperloom/inference_optimizer/tests/test_breakdown_exporter_unit.py index 547d972af5..ba3b023158 100644 --- a/src/hyperloom/inference_optimizer/tests/test_breakdown_exporter_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_breakdown_exporter_unit.py @@ -191,7 +191,7 @@ def test_write_minimal_final_json_idempotent(tmp_path): def test_write_minimal_final_json_refreshes_stale_fallback(tmp_path): - # A prior crash-safe fallback is stale after --resume and must be + # A prior crash-safe fallback is stale after a resume and must be # overwritten with the current state, NOT preserved. from hyperloom.orchestrator.state.shared_state import SharedState @@ -421,6 +421,56 @@ def test_orchestration_context_is_empty_without_a_census_or_db(tmp_path): assert warnings == [] +def test_recorder_snapshot_leaves_the_workload_contract_intact(tmp_path): + """A recorder fragment replaces its whole section, so it must not own workload.""" + from types import SimpleNamespace + + from hyperloom.inference_optimizer.breakdown.recorder import instrument + + (tmp_path / "state.json").write_text( + json.dumps({"session_id": "s", "framework": "sglang", "model_name": "qwen3-8b"}), + encoding="utf-8", + ) + (tmp_path / "manifest.json").write_text( + json.dumps({"session_id": "s", "framework_version": "0.4.1", "workload": {"conc": 64}}), + encoding="utf-8", + ) + instrument.snapshot_state_sections( + tmp_path, + SimpleNamespace(framework="sglang", model_name="qwen3-8b", model_path="", session_id="s"), + ) + + workload = ex.build(tmp_path)["workload"] + assert workload["framework_name"] == "sglang" + assert workload["framework_version"] == "0.4.1" + assert workload["conc"] == 64 + # Unset knobs stay None; a recorder fragment used to coerce them to 0. + assert workload["tp"] is None + + +def test_recorder_snapshot_leaves_the_sweep_variants_intact(tmp_path): + """Only the collector can scan the variant points off disk.""" + from types import SimpleNamespace + + from hyperloom.inference_optimizer.breakdown.recorder import instrument + + (tmp_path / "state.json").write_text( + json.dumps({"session_id": "s", "last_sweep": {"grid_size": 2}}), + encoding="utf-8", + ) + instrument.snapshot_state_sections( + tmp_path, + SimpleNamespace( + session_id="s", + last_sweep={"grid_size": 2}, + framework="sglang", + model_name="m", + model_path="", + ), + ) + + assert "all_variants" in ex.build(tmp_path)["sweep"] + # ---- session_meta duration ---- diff --git a/src/hyperloom/inference_optimizer/tests/test_bypass_backend.py b/src/hyperloom/inference_optimizer/tests/test_bypass_backend.py index 4c3132aa15..7800c48f69 100644 --- a/src/hyperloom/inference_optimizer/tests/test_bypass_backend.py +++ b/src/hyperloom/inference_optimizer/tests/test_bypass_backend.py @@ -11,7 +11,6 @@ from __future__ import annotations import json -import os import subprocess from pathlib import Path @@ -1293,14 +1292,49 @@ class _P: assert captured.get("num_warmups") == "3" -def test_server_env_injects_rocr_visible_devices(monkeypatch): +def test_server_env_exports_every_materialized_env(monkeypatch): + """Dropping these silently reruns the baseline and scores the noise.""" monkeypatch.setenv("OPENAI_API_KEY", "must-not-reach-benchmark") - env = bypass_runner._server_env(False, None, {"ROCR_VISIBLE_DEVICES": "0,1,2,3"}) + + env = bypass_runner._server_env( + False, + None, + { + "SGLANG_USE_AITER": "1", + "VLLM_ROCM_USE_AITER_MHA": "0", + "PYTORCH_TUNABLEOP_ENABLED": "1", + "ROCR_VISIBLE_DEVICES": "0,1,2,3", + "TP": 8, + "RANDOM_RANGE_RATIO": 0.8, + }, + ) + + assert env["SGLANG_USE_AITER"] == "1" + assert env["VLLM_ROCM_USE_AITER_MHA"] == "0" + assert env["PYTORCH_TUNABLEOP_ENABLED"] == "1" assert env["ROCR_VISIBLE_DEVICES"] == "0,1,2,3" + # YAML scalars arrive as int/float and must not reach putenv unstringified. + assert env["TP"] == "8" + assert env["RANDOM_RANGE_RATIO"] == "0.8" assert "OPENAI_API_KEY" not in env - env2 = bypass_runner._server_env(False, None, {}) - # No pin in bench_envs: whatever the parent env had (may be unset). - assert env2.get("ROCR_VISIBLE_DEVICES") == os.environ.get("ROCR_VISIBLE_DEVICES") + + +def test_server_env_lets_config_path_win_like_magpie(monkeypatch): + """Magpie exports the YAML PATH to the server; bypass matches it.""" + monkeypatch.setenv("PATH", "/inherited/bin") + + env = bypass_runner._server_env(False, None, {"PATH": "/opt/venv/bin:/usr/bin"}) + + assert env["PATH"] == "/opt/venv/bin:/usr/bin" + + +def test_server_env_pins_profiler_dirs_when_profiling(tmp_path): + env = bypass_runner._server_env(True, str(tmp_path), {"SGLANG_USE_AITER": "1"}) + + assert env["VLLM_TORCH_PROFILER_DIR"] == str(tmp_path) + assert env["SGLANG_TORCH_PROFILER_DIR"] == str(tmp_path) + assert env["ATOM_TORCH_PROFILER_DIR"] == str(tmp_path) + assert env["SGLANG_USE_AITER"] == "1" def _eval_client_run(monkeypatch, *, client_rc=0, eval_rc=1): diff --git a/src/hyperloom/inference_optimizer/tests/test_cli_backends_unit.py b/src/hyperloom/inference_optimizer/tests/test_cli_backends_unit.py index 6bdb7df3c2..2987328fd9 100644 --- a/src/hyperloom/inference_optimizer/tests/test_cli_backends_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_cli_backends_unit.py @@ -3,7 +3,7 @@ """Coverage for ``cli_backends``: per-role backend construction (mock/agent choices, kernel selection, validation errors), advisory proposal-scorer -wiring, robustness-server detection, and robustness option overrides.""" +wiring and robustness option overrides.""" from __future__ import annotations @@ -375,74 +375,28 @@ def test_proposal_scorer_models_without_enable_stays_off(monkeypatch) -> None: assert clib._build_proposal_scorer(args) is None -def test_robustness_server_configured_via_arg() -> None: - args = argparse.Namespace(robustness_server_url="http://rob:9000") - assert clib._robustness_server_configured(args) is True - - -def test_robustness_server_configured_via_env(monkeypatch) -> None: - monkeypatch.delenv("ROBUSTNESS_SERVER_URL", raising=False) - args = argparse.Namespace(robustness_server_url=None) - assert clib._robustness_server_configured(args) is False - monkeypatch.setenv("ROBUSTNESS_SERVER_URL", "http://env:9000") - assert clib._robustness_server_configured(args) is True - - -def test_robustness_options_single_node_minimal(monkeypatch) -> None: - for k in clib._MULTI_NODE_WORKLOAD_UID_ENV_KEYS: - monkeypatch.delenv(k, raising=False) +def test_robustness_options_single_node_minimal() -> None: args = argparse.Namespace( - robustness_server_url=None, robustness_llm_rca=None, nodes=1, - robustness_workload_uid=None, robustness_disable_local_probe=None, - robustness_enable_cluster_pod_metrics=None, - robustness_pod_metrics_categories=None, ) opts = clib._build_robustness_options(args) assert "auto_probe_inference_server" not in opts assert "nodes" not in opts -def test_robustness_options_multi_node_defaults(monkeypatch) -> None: - for k in clib._MULTI_NODE_WORKLOAD_UID_ENV_KEYS: - monkeypatch.delenv(k, raising=False) +def test_robustness_options_multi_node_defaults() -> None: args = argparse.Namespace( - robustness_server_url="http://rob", robustness_llm_rca=True, nodes=4, - robustness_workload_uid="wl-1", robustness_disable_local_probe=None, - robustness_enable_cluster_pod_metrics=None, - robustness_pod_metrics_categories="gpu,net", ) opts = clib._build_robustness_options(args) assert opts["nodes"] == 4 - assert opts["robustness_server_url"] == "http://rob" assert opts["llm_rca_enabled"] is True - assert opts["workload_uid"] == "wl-1" assert opts["disable_local_probe"] is True - assert opts["enable_cluster_pod_metrics"] is True - assert opts["pod_metrics_categories"] == ["gpu", "net"] assert opts["auto_probe_inference_server"] is False assert opts["progress_no_levers_min_minutes"] == 60.0 -def test_robustness_options_workload_uid_from_env(monkeypatch) -> None: - for k in clib._MULTI_NODE_WORKLOAD_UID_ENV_KEYS: - monkeypatch.delenv(k, raising=False) - monkeypatch.setenv("RAY_JOB_ID", "ray-42") - args = argparse.Namespace( - robustness_server_url=None, - robustness_llm_rca=None, - nodes=1, - robustness_workload_uid=None, - robustness_disable_local_probe=None, - robustness_enable_cluster_pod_metrics=None, - robustness_pod_metrics_categories=None, - ) - opts = clib._build_robustness_options(args) - assert opts["workload_uid"] == "ray-42" - - diff --git a/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py b/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py index a423bab234..193c15811e 100644 --- a/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py +++ b/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap.py @@ -44,7 +44,6 @@ def _args(**overrides): plateau_kernel_revert_streak=3, plateau_kernel_keep_gain=2.5, plateau_kernel_lookback=5, - explore_force_exit_hours_remaining=1.25, explore_force_exit_budget_pct=0.2, explore_overtime_kill_ratio="bad", explore_variant_timeout_sec="bad", @@ -330,7 +329,6 @@ def test_read_failure_summary_and_final_summary_output(tmp_path: Path, capsys) - session_id="s", model_name="m", baseline_tput=10.0, - cumulative_gain=1.25, cumulative_gain_validated=1.0, cumulative_gain_validated_ts="2026-01-01T00:00:00Z", cumulative_gain_validated_stack_len=0, diff --git a/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap_coverage_unit.py b/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap_coverage_unit.py index 362b7e080f..f4eb741cb3 100644 --- a/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap_coverage_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_cli_bootstrap_coverage_unit.py @@ -41,7 +41,6 @@ def _args(**overrides): plateau_kernel_revert_streak=3, plateau_kernel_keep_gain=2.5, plateau_kernel_lookback=5, - explore_force_exit_hours_remaining=1.25, explore_force_exit_budget_pct=0.2, explore_overtime_kill_ratio="bad", explore_variant_timeout_sec="bad", @@ -398,7 +397,6 @@ def test_read_failure_summary_and_final_summary_output(tmp_path: Path, capsys) - session_id="s", model_name="m", baseline_tput=10.0, - cumulative_gain=1.25, cumulative_gain_validated=1.0, cumulative_gain_validated_ts="2026-01-01T00:00:00Z", cumulative_gain_validated_stack_len=0, diff --git a/src/hyperloom/inference_optimizer/tests/test_common_utils.py b/src/hyperloom/inference_optimizer/tests/test_common_utils.py index 49c1e62c74..c7fcec42cf 100644 --- a/src/hyperloom/inference_optimizer/tests/test_common_utils.py +++ b/src/hyperloom/inference_optimizer/tests/test_common_utils.py @@ -1657,7 +1657,7 @@ def test_paths_helpers(monkeypatch: pytest.MonkeyPatch, tmp_path: Path) -> None: monkeypatch.setenv(paths.ENV_OVERRIDE_ASSET_ROOT, str(tmp_path)) assert paths.asset_root() == tmp_path monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path / "does_not_exist")) - assert paths.find_latest_per_session_dir() is None + assert paths.workspace_root() == tmp_path / "does_not_exist" diff --git a/src/hyperloom/inference_optimizer/tests/test_config_patch_agent_kb.py b/src/hyperloom/inference_optimizer/tests/test_config_patch_agent_kb.py index c463bd9e3a..0352bcb22a 100644 --- a/src/hyperloom/inference_optimizer/tests/test_config_patch_agent_kb.py +++ b/src/hyperloom/inference_optimizer/tests/test_config_patch_agent_kb.py @@ -33,7 +33,6 @@ def _state(stack: list[dict]) -> SimpleNamespace: optimization_stack=stack, current_best={"tput": 130.0}, cumulative_gain_validated=30.0, - cumulative_gain=30.0, gain_per_stack_entry=[], session_id="s1", recipe_kb_session_id="s1", diff --git a/src/hyperloom/inference_optimizer/tests/test_coordinator_async_batch2_unit.py b/src/hyperloom/inference_optimizer/tests/test_coordinator_async_batch2_unit.py index 407b497fcc..69c127b0ef 100644 --- a/src/hyperloom/inference_optimizer/tests/test_coordinator_async_batch2_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_coordinator_async_batch2_unit.py @@ -281,7 +281,7 @@ def _boom() -> None: @pytest.mark.asyncio -async def test_resume_consistency_marks_unvalidated_and_rebuilds_current_best(coord: Coordinator) -> None: +async def test_resume_consistency_marks_unvalidated_keeps(coord: Coordinator) -> None: coord._resumed_from["is_resume"] = True coord.shared_state.optimization_stack = [ { @@ -300,20 +300,50 @@ async def test_resume_consistency_marks_unvalidated_and_rebuilds_current_best(co }, ] coord.shared_state.cumulative_gain_validated_stack_len = 1 - coord.shared_state.current_best = {"extra_server_args": "--stale 1", "extra_envs": {}} + coord.shared_state.current_best = {"extra_server_args": "--a 1 --b 2", "extra_envs": {"A": "1", "B": "2"}} report = await coord._resume_consistency_pass() warning_kinds = {w["kind"] for w in report["warnings"]} assert "resume_unvalidated_keeps" in warning_kinds - assert "resume_inconsistent_current_best" in warning_kinds assert coord.shared_state.resume_pending_revalidation is True - assert coord.shared_state.current_best["extra_server_args"] == "--a 1 --b 2" - assert coord.shared_state.current_best["extra_envs"] == {"A": "1", "B": "2"} - assert "rebuilt_current_best_config_from_stack" in report["fixes"] assert any(isinstance(f, dict) and f.get("kind") == "queued_resume_stack_rebench" for f in report["fixes"]) +@pytest.mark.asyncio +async def test_resume_consistency_leaves_current_best_alone(coord: Coordinator) -> None: + """Resume must not rewrite the config; a stack replay loses ablated envs.""" + coord._resumed_from["is_resume"] = True + coord.shared_state.optimization_stack = [ + { + "action": "explore", + "variant_name": "v1", + "candidate_extra_server_args": "--a 1", + "extra_envs": {"OLD": "1"}, + "tput": 110.0, + }, + { + "action": "explore", + "variant_name": "v2", + "candidate_extra_server_args": "--b 2", + "extra_envs": {"NEW": "1"}, + "unset_envs": ["OLD"], + "tput": 120.0, + }, + ] + coord.shared_state.cumulative_gain_validated_stack_len = 2 + coord.shared_state.current_best = {"extra_server_args": "--b 2", "extra_envs": {"NEW": "1"}} + + report = await coord._resume_consistency_pass() + + assert coord.shared_state.current_best["extra_envs"] == {"NEW": "1"} + assert coord.shared_state.current_best["extra_server_args"] == "--b 2" + assert not any( + isinstance(w, dict) and w.get("kind") == "resume_inconsistent_current_best" for w in report["warnings"] + ) + assert "rebuilt_current_best_config_from_stack" not in report["fixes"] + + @pytest.mark.asyncio async def test_resume_restores_promoted_inferencex_checkout( coord: Coordinator, @@ -324,7 +354,9 @@ async def test_resume_restores_promoted_inferencex_checkout( active.mkdir() coord._resumed_from["is_resume"] = True coord.shared_state.active_inferencex_path = str(active) - monkeypatch.delenv("INFERENCEX_PATH", raising=False) + # setenv, not delenv: delenv of an absent name arms no undo, so the value + # the resume pass exports below would leak into every later test. + monkeypatch.setenv("INFERENCEX_PATH", "") await coord._resume_consistency_pass() @@ -449,6 +481,44 @@ async def test_resume_consistency_clears_stale_pending_integrate(coord: Coordina assert coord.shared_state.pending_integrate == {} +@pytest.mark.asyncio +async def test_resume_consistency_keeps_sentinel_when_event_scan_fails( + coord: Coordinator, + monkeypatch: pytest.MonkeyPatch, +) -> None: + """An unreadable event log must not be treated as 'no KEEP exists'.""" + coord._resumed_from["is_resume"] = True + sentinel = { + "task_id": "ti-unreadable", + "framework_source_root": "/tmp/framework", + "patches": ["/tmp/p.diff"], + } + coord.shared_state.pending_integrate = dict(sentinel) + + async def _boom(*_args, **_kwargs): + raise RuntimeError("database disk image is malformed") + + monkeypatch.setattr(coord.bus, "tail", _boom) + + rolled_back: list[dict] = [] + monkeypatch.setattr( + coord.writeback, + "_resume_rollback_pending_integrate", + lambda pending: rolled_back.append(pending) or {"reversed": [], "failed": []}, + ) + + report = await coord._resume_consistency_pass() + + assert rolled_back == [] + assert coord.shared_state.pending_integrate == sentinel + warning = next(w for w in report["warnings"] if w.get("kind") == "pending_integrate_scan_failed") + assert warning["task_id"] == "ti-unreadable" + assert not any( + isinstance(f, dict) and f.get("kind") in {"rolled_back_pending_integrate", "cleared_stale_pending_integrate"} + for f in report["fixes"] + ) + + @pytest.mark.asyncio async def test_resume_consistency_discards_orphaned_integrate_keep_missing_workspace( coord: Coordinator, @@ -704,6 +774,14 @@ async def test_resume_consistency_enqueues_stack_rebench_for_unvalidated(coord: } ] coord.shared_state.cumulative_gain_validated_stack_len = 0 + # The lift writes both together, so a stack always has a config behind it. + coord.shared_state.current_best = { + "action": "explore", + "variant_name": "v1", + "tput": 110.0, + "extra_server_args": "--a 1", + "extra_envs": {"A": "1"}, + } report = await coord._resume_consistency_pass() diff --git a/src/hyperloom/inference_optimizer/tests/test_coordinator_async_methods_coverage_unit.py b/src/hyperloom/inference_optimizer/tests/test_coordinator_async_methods_coverage_unit.py index 88514af235..0e631b85e0 100644 --- a/src/hyperloom/inference_optimizer/tests/test_coordinator_async_methods_coverage_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_coordinator_async_methods_coverage_unit.py @@ -870,7 +870,7 @@ def test_is_promotable_result_unchanged_for_reverted_integrate_patch(coord: Coor @pytest.mark.asyncio async def test_compose_prompt_orchestration_gain_objective(coord: Coordinator) -> None: coord._current_objective = TargetGainObjective(target_gain_pct=20.0) - coord.shared_state.cumulative_gain = 5.0 + coord.shared_state.cumulative_gain_validated = 5.0 await coord._compose_prompt("orchestration") assert coord.shared_state.target_gap_pct == pytest.approx(15.0) @@ -883,7 +883,7 @@ async def test_compose_prompt_renders_the_gap_it_just_computed(coord: Coordinato a shared-state dump assembled before the recompute, which renders both. """ coord._current_objective = TargetGainObjective(target_gain_pct=20.0) - coord.shared_state.cumulative_gain = 5.0 + coord.shared_state.cumulative_gain_validated = 5.0 text = await coord._compose_prompt("orchestration") assert "target_gap_pct=15.00" in text assert "target_gap_pct=0.00" not in text @@ -892,7 +892,7 @@ async def test_compose_prompt_renders_the_gap_it_just_computed(coord: Coordinato @pytest.mark.asyncio async def test_compose_prompt_time_only_objective_leaves_no_gap(coord: Coordinator) -> None: coord._current_objective = TimeOnlyObjective() - coord.shared_state.cumulative_gain = 5.0 + coord.shared_state.cumulative_gain_validated = 5.0 await coord._compose_prompt("orchestration") assert coord.shared_state.target_gap_pct == 0.0 diff --git a/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py b/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py index 3422ccecbd..afb8f56599 100644 --- a/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py +++ b/src/hyperloom/inference_optimizer/tests/test_coordinator_gemm_promote_units.py @@ -3,9 +3,9 @@ """Unit coverage for the unified GEMM-tuning result handling on Coordinator. -Exercises ``_promote_gemm_tuning_keep`` guard rails plus the forge/geak -promote branches, and the ``_handle_gemm_tuning_result`` routing that keeps -forge results on the per-tuner E2E path while GEAK results promote inline. +Exercises ``_gemm_e2e_candidates`` guard rails for both the forge and GEAK +result shapes, and the ``_handle_gemm_tuning_result`` routing that puts every +backend on the per-tuner E2E path so a promoted gain is always a measurement. """ from __future__ import annotations @@ -69,8 +69,8 @@ def test_syncs_standard_roofline_fallback_into_live_coordinator_state(tmp_path): persisted.last_trace_analyze = { "trace_input": persisted.last_profile_trace, "steady_state_trace": selected_trace, + "roofline_snapshot_id": 3, } - persisted.roofline_snapshot_id = 3 persisted.roofline_snapshots = [{"snapshot_id": 3}] persisted.baseline_eager_fallback = False persisted.save(tmp_path) @@ -173,8 +173,8 @@ async def fake_handler(payload, *, session_dir): state.last_trace_analyze = { "trace_input": state.last_profile_trace, "steady_state_trace": selected_trace, + "roofline_snapshot_id": 7, } - state.roofline_snapshot_id = 7 state.baseline_eager_fallback = False state.save(session_dir) return { @@ -321,59 +321,12 @@ def _write_collective_recovery( return manifest, checkpoint_path -class TestPromoteGemmTuningKeep: - def test_ignores_non_dict(self, tmp_path): - coord = _coord(tmp_path, baseline_tput=100.0) - coord._promote_gemm_tuning_keep("not-a-dict") # type: ignore[arg-type] - assert coord.shared_state.optimization_stack == [] - - def test_ignores_non_ok_status(self, tmp_path): - coord = _coord(tmp_path, baseline_tput=100.0) - coord._promote_gemm_tuning_keep({"status": "failed", "decision": "KEEP"}) - assert coord.shared_state.optimization_stack == [] - - def test_ignores_non_keep_decision(self, tmp_path): - coord = _coord(tmp_path, baseline_tput=100.0) - coord._promote_gemm_tuning_keep({"status": "ok", "decision": "REVERT"}) - assert coord.shared_state.optimization_stack == [] - - def test_ignores_unparseable_speedup(self, tmp_path): - coord = _coord(tmp_path, baseline_tput=100.0) - coord._promote_gemm_tuning_keep({"status": "ok", "decision": "KEEP", "best_speedup": object()}) - assert coord.shared_state.optimization_stack == [] - - def test_ignores_low_speedup_or_baseline(self, tmp_path): - coord = _coord(tmp_path, baseline_tput=0.0) - coord._promote_gemm_tuning_keep({"status": "ok", "decision": "KEEP", "best_speedup": 1.5}) - assert coord.shared_state.optimization_stack == [] - - coord2 = _coord(tmp_path, baseline_tput=100.0) - coord2._promote_gemm_tuning_keep({"status": "ok", "decision": "KEEP", "best_speedup": 1.0}) - assert coord2.shared_state.optimization_stack == [] +class TestGemmE2eCandidates: + """Guard rails deciding which tuning results reach the E2E validator.""" - def test_forge_backend_records_stack_and_current_best(self, tmp_path): - coord = _coord(tmp_path, baseline_tput=100.0) - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.25, - "backend": "forge", - "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, - "workspace": str(tmp_path), - } - ) - stack = coord.shared_state.optimization_stack - assert len(stack) == 1 - assert stack[0]["variant_name"] == "forge_gemm_tuned" - assert stack[0]["backend"] == "forge" - assert coord.shared_state.current_best["engine"] == "forge" - assert coord.shared_state.cumulative_gain == pytest.approx(25.0) - assert coord.shared_state.cumulative_gain_validated == pytest.approx(25.0) - - def test_geak_backend_uses_tuned_file(self, tmp_path): + def test_geak_result_yields_the_tuned_dispatch_csv(self, tmp_path): coord = _coord(tmp_path, baseline_tput=200.0) - coord._promote_gemm_tuning_keep( + cands = coord._gemm_e2e_candidates( { "status": "ok", "decision": "KEEP", @@ -382,48 +335,76 @@ def test_geak_backend_uses_tuned_file(self, tmp_path): "tuned_file": "/tuned/gemm.csv", } ) - stack = coord.shared_state.optimization_stack - assert len(stack) == 1 - assert stack[0]["variant_name"] == "a8w8_blockscale_tuned_gemm" - envs = coord.shared_state.current_best["extra_envs"] - assert envs["AITER_CONFIG_GEMM_A8W8_BLOCKSCALE"] == "/tuned/gemm.csv" + assert len(cands) == 1 + assert cands[0]["tuner"] == "a8w8_blockscale_tuned_gemm" + assert cands[0]["envs"] == {"AITER_CONFIG_GEMM_A8W8_BLOCKSCALE": "/tuned/gemm.csv"} + assert cands[0]["micro_speedup"] == pytest.approx(1.1) - def test_geak_keep_writes_gemm_tuning_journal_event(self, tmp_path): - # Adopted GEMM-tuning run surfaces as a KIND_GEMM_TUNING KEEP journal row - # carrying the serving throughput and originating task_id. + @pytest.mark.parametrize( + "overrides", + [ + {"status": "failed"}, + {"decision": "REVERT"}, + {"best_speedup": 1.0}, + {"best_speedup": object()}, + {"tuned_file": ""}, + ], + ) + def test_geak_result_yields_nothing_without_a_usable_keep(self, tmp_path, overrides): coord = _coord(tmp_path, baseline_tput=200.0) - coord._promote_gemm_tuning_keep( + result = { + "status": "ok", + "decision": "KEEP", + "best_speedup": 1.1, + "backend": "geak", + "tuned_file": "/tuned/gemm.csv", + } + result.update(overrides) + assert coord._gemm_e2e_candidates(result) == [] + + def test_non_dict_result_is_rejected(self, tmp_path): + coord = _coord(tmp_path, baseline_tput=100.0) + assert coord._gemm_e2e_candidates({}) == [] + + def test_forge_result_yields_one_candidate_per_improved_tuner(self, tmp_path): + coord = _coord(tmp_path, baseline_tput=100.0) + cands = coord._gemm_e2e_candidates( { "status": "ok", "decision": "KEEP", - "best_speedup": 1.1, - "backend": "geak", - "tuned_file": "/tuned/gemm.csv", - "task_id": "kernel_entry_gemm_tuning", + "backend": "forge", + "tuners_run": [ + { + "tuner": "fmoe_ck", + "status": "ok", + "candidate": True, + "env_var": "AITER_CONFIG_FMOE", + "env_value": "/cfg/fmoe.csv", + "best_micro_speedup": 1.3, + }, + {"tuner": "skipped_one", "status": "ok", "improved_shapes": 0}, + {"tuner": "failed_one", "status": "failed", "candidate": True}, + ], } ) - rows = [e for e in _journal_entries(tmp_path) if e.get("kind") == "gemm_tuning"] - assert len(rows) == 1 - row = rows[0] - assert row["outcome"] == "KEEP" - assert row["variant_name"] == "a8w8_blockscale_tuned_gemm" - assert row["throughput_after"] == pytest.approx(220.0) - assert row["task_id"] == "kernel_entry_gemm_tuning" - assert row["provenance"] == "gemm_tuning:geak" + assert [c["tuner"] for c in cands] == ["fmoe_ck"] + assert cands[0]["envs"] == {"AITER_CONFIG_FMOE": "/cfg/fmoe.csv"} - def test_forge_keep_dedupes_same_tuned_file(self, tmp_path): + def test_forge_result_ignores_a_tuned_file(self, tmp_path): + """tuned_file is the GEAK shape; forge must come from tuners_run.""" coord = _coord(tmp_path, baseline_tput=100.0) - result = { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "forge", - "artifacts": {"cfg": "/cfg/tuned.json"}, - "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, - } - coord._promote_gemm_tuning_keep(result) - coord._promote_gemm_tuning_keep(result) - assert len(coord.shared_state.optimization_stack) == 1 + assert ( + coord._gemm_e2e_candidates( + { + "status": "ok", + "decision": "KEEP", + "best_speedup": 1.2, + "backend": "forge", + "tuned_file": "/tuned/gemm.csv", + } + ) + == [] + ) class TestPromoteFusionIntegrateKeep: @@ -431,7 +412,6 @@ def test_records_incremental_gain_but_preserves_baseline_total(self, tmp_path): coord = _coord( tmp_path, baseline_tput=100.0, - cumulative_gain=20.0, cumulative_gain_validated=20.0, cumulative_gain_validated_stack_len=1, optimization_stack=[ @@ -479,11 +459,11 @@ def test_records_incremental_gain_but_preserves_baseline_total(self, tmp_path): assert stack[1]["gain_pct"] == pytest.approx(50.0) assert stack[1]["extra_envs"]["SGLANG_USE_AITER"] == "1" assert stack[1]["extra_envs"]["ZAYA_FUSED_HYBRID_RESIDUAL"] == "1" - assert stack[1]["kernel_speedup"] == 3.05 assert coord.shared_state.current_best["action"] == "fusion" - assert coord.shared_state.current_best["backend"] == "forge" - assert coord.shared_state.current_best["engine"] == "forge_fusion" assert coord.shared_state.current_best["tput"] == 180.0 + # current_best is a config record; the forge labels live on the entry. + assert "engine" not in coord.shared_state.current_best + assert "backend" not in coord.shared_state.current_best assert coord.shared_state.cumulative_gain_validated == 80.0 assert coord.shared_state.gain_per_stack_entry == [20.0, 80.0] assert coord.shared_state.cumulative_gain_validated_stack_len == 2 @@ -510,7 +490,8 @@ def test_dedupes_same_fusion_patch(self, tmp_path): phase._promote_fusion_integrate_keep(fusion, integ) assert len(coord.shared_state.optimization_stack) == 1 - assert coord.shared_state.current_best["patch_path"] == "/tmp/fusion.patch" + assert coord.shared_state.optimization_stack[0]["patch_path"] == "/tmp/fusion.patch" + assert coord.shared_state.current_best["variant_name"] == "forge_fusion:fusion.patch" @pytest.mark.asyncio async def test_handle_fusion_result_posts_and_integrates_kept_candidate(self, tmp_path, monkeypatch): @@ -595,7 +576,8 @@ async def _fake_integrate(payload, *, session_dir): assert calls[0]["snapshot_dir"] == str(tmp_path / "snapshot") assert calls[0]["extra_envs"] == {"SGLANG_USE_AITER": "1", "ZAYA_FUSED": "1"} assert coord.shared_state.last_fusion_integrate["decision"] == "KEEP" - assert coord.shared_state.current_best["engine"] == "forge_fusion" + assert coord.shared_state.current_best["action"] == "fusion" + assert coord.shared_state.optimization_stack[-1]["engine"] == "forge_fusion" assert coord.bus.messages[-1].payload["kind"] == "fusion_integrate_done" @pytest.mark.asyncio @@ -712,12 +694,15 @@ def test_promotes_and_deduplicates_collective_keep(self, tmp_path): } phase._promote_collective_integrate_keep(collective, integrate) - assert coord.shared_state.current_best["engine"] == "forge_collective" + # current_best is a pure config record; the engine that produced the + # winner is stack-entry provenance. + assert coord.shared_state.current_best["action"] == "collective" + assert coord.shared_state.current_best["variant_name"] == "forge_collective" + assert coord.shared_state.optimization_stack[0]["engine"] == "forge_collective" coord.shared_state.current_best = { "engine": "later_lane", "tput": 150.0, } - coord.shared_state.cumulative_gain = 50.0 coord.shared_state.cumulative_gain_validated = 50.0 phase._promote_collective_integrate_keep(collective, integrate) @@ -1597,7 +1582,6 @@ async def test_integrate_collective_rolls_back_failed_promotion( tmp_path, baseline_tput=100.0, current_best={"engine": "existing", "tput": 110.0}, - cumulative_gain=10.0, cumulative_gain_validated=10.0, ) coord.bus = _Bus() @@ -1984,7 +1968,7 @@ async def test_does_not_e2e_validate_sparse_aiter_candidate_without_base_configs ], } - await phase._validate_forge_gemm_tuning_e2e(result) + await phase._validate_gemm_tuning_e2e(result) assert fake.calls == [] assert result["e2e_results"]["reverted"][0]["reason"] == ( @@ -2021,13 +2005,68 @@ async def test_does_not_e2e_validate_missing_aiter_candidate( ], } - await phase._validate_forge_gemm_tuning_e2e(result) + await phase._validate_gemm_tuning_e2e(result) assert fake.calls == [] assert result["e2e_results"]["reverted"][0]["reason"] == ( "candidate_artifact_missing" ) + @pytest.mark.asyncio + async def test_a_stopped_run_leaves_its_tuners_unjudged(self, tmp_path, monkeypatch): + """A clock that ran out is not a verdict on the tuners it interrupted.""" + coord = _coord(tmp_path, baseline_tput=100.0, framework="sglang") + phase = KernelPhase(coord) + first = tmp_path / "fmoe.csv" + second = tmp_path / "dense.csv" + first.write_text("token,model_dim\n1,2\n", encoding="utf-8") + second.write_text("M,N,K\n1,2,3\n", encoding="utf-8") + calls: list[dict] = [] + + async def _fake_integrate(payload, *, session_dir): + calls.append(payload) + return { + "status": "failed", + "error_class": "session_time_exhausted", + "decision": "NEEDS_REVIEW", + } + + monkeypatch.setattr(krh_mod, "integrate_handler", _fake_integrate) + monkeypatch.setattr(explore_mod, "_compute_explore_variant_timeout", lambda **_k: 61) + monkeypatch.setattr( + phase, + "_merge_gemm_candidate_with_runtime", + lambda _env_var, env_value: env_value, + ) + + result = { + "backend": "forge", + "tuners_run": [ + { + "status": "ok", + "tuner": "fmoe_ck", + "improved_shapes": 1, + "env_var": "AITER_CONFIG_FMOE", + "env_value": str(first), + }, + { + "status": "ok", + "tuner": "dense_bf16", + "improved_shapes": 1, + "env_var": "AITER_CONFIG_DENSE", + "env_value": str(second), + }, + ], + } + + await phase._validate_gemm_tuning_e2e(result) + + assert len(calls) == 1 + assert result["e2e_results"]["kept"] == [] + assert result["e2e_results"]["reverted"] == [] + assert coord.shared_state.optimization_stack == [] + + @pytest.mark.asyncio async def test_stacks_keeps_and_reverts(self, tmp_path, monkeypatch): coord = _coord( @@ -2089,7 +2128,7 @@ async def _fake_integrate(payload, *, session_dir): ], } - await phase._validate_forge_gemm_tuning_e2e(result) + await phase._validate_gemm_tuning_e2e(result) assert [c["kernel_id"] for c in calls] == [ "gemm_tune_fmoe_ck", @@ -2106,9 +2145,10 @@ async def _fake_integrate(payload, *, session_dir): "AITER_CONFIG_FMOE": str(fmoe_candidate), "AITER_CONFIG_DENSE": str(dense_candidate), } - assert coord.shared_state.current_best["engine"] == "forge" + assert coord.shared_state.current_best["variant_name"] == "forge_fmoe_ck" assert coord.shared_state.current_best["tput"] == 130.0 assert coord.shared_state.optimization_stack[0]["variant_name"] == "forge_fmoe_ck" + assert coord.shared_state.optimization_stack[0]["backend"] == "forge" assert result["decision"] == "KEEP" assert result["recommended_env"] == { "AITER_CONFIG_FMOE": str(fmoe_candidate) @@ -2136,7 +2176,7 @@ async def test_handles_no_candidates_without_rewriting_raw_result(self, tmp_path ], } - await phase._validate_forge_gemm_tuning_e2e(result) + await phase._validate_gemm_tuning_e2e(result) assert result["recommended_env"] == {"AITER_CONFIG": "/raw.csv"} assert coord.shared_state.optimization_stack == [] @@ -2171,7 +2211,7 @@ async def _raise_integrate(*_args, **_kwargs): ], } - await phase._validate_forge_gemm_tuning_e2e(result) + await phase._validate_gemm_tuning_e2e(result) assert result["decision"] == "REVERT" assert result["micro_decision"] == "candidate_no_e2e_gain" @@ -2486,117 +2526,76 @@ def test_non_dict_result_is_not_eligible(self, tmp_path, monkeypatch): assert coord._ck_blockscale_switch_eligible("nope") is False # type: ignore[arg-type] -class TestPromoteInjectsCkBlockscaleEnv: - """Inline-promote safety net: an eligible forge result that reaches - ``_promote_gemm_tuning_keep`` (without the validator) injects - ``SGLANG_FP8_BLOCKSCALE_CK_MAX_M=256``. The GEAK path never injects the - un-validated CK switch.""" +class TestCkBlockscaleCandidateInjection: + """The fp8 block-scale CK switch enters as its own candidate to be measured.""" + + def _forge_result(self, **overrides): + result = { + "status": "ok", + "decision": "KEEP", + "best_speedup": 1.2, + "backend": "forge", + "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, + } + result.update(overrides) + return result + + def _ck_candidates(self, coord, result): + return [c for c in coord._gemm_e2e_candidates(result) if c["env_var"] == "SGLANG_FP8_BLOCKSCALE_CK_MAX_M"] def test_injects_for_forge_eligible_keep(self, tmp_path, monkeypatch): coord = _eligible_coord(tmp_path, monkeypatch) - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "forge", - "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, - } - ) - envs = coord.shared_state.current_best["extra_envs"] - assert envs["SGLANG_FP8_BLOCKSCALE_CK_MAX_M"] == "256" - stack_envs = coord.shared_state.optimization_stack[0]["extra_envs"] - assert stack_envs["SGLANG_FP8_BLOCKSCALE_CK_MAX_M"] == "256" + cands = self._ck_candidates(coord, self._forge_result()) + assert len(cands) == 1 + assert cands[0]["envs"] == {"SGLANG_FP8_BLOCKSCALE_CK_MAX_M": "256"} + assert cands[0]["tuner"] == "ck_blockscale_backend_switch" def test_does_not_inject_for_geak_backend(self, tmp_path, monkeypatch): - # GEAK is not forge, so the CK switch is never stamped inline. coord = _eligible_coord(tmp_path, monkeypatch) - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "geak", - "tuned_file": "/tuned/gemm.csv", - } - ) - envs = coord.shared_state.current_best["extra_envs"] - assert envs["AITER_CONFIG_GEMM_A8W8_BLOCKSCALE"] == "/tuned/gemm.csv" - assert "SGLANG_FP8_BLOCKSCALE_CK_MAX_M" not in envs + result = { + "status": "ok", + "decision": "KEEP", + "best_speedup": 1.2, + "backend": "geak", + "tuned_file": "/tuned/gemm.csv", + } + assert self._ck_candidates(coord, result) == [] + assert [c["tuner"] for c in coord._gemm_e2e_candidates(result)] == ["a8w8_blockscale_tuned_gemm"] def test_does_not_inject_for_bf16_precision(self, tmp_path, monkeypatch): coord = _eligible_coord(tmp_path, monkeypatch, precision="bf16") - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "forge", - "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, - } - ) - envs = coord.shared_state.current_best["extra_envs"] - assert "SGLANG_FP8_BLOCKSCALE_CK_MAX_M" not in envs + assert self._ck_candidates(coord, self._forge_result()) == [] def test_does_not_inject_for_non_sglang_framework(self, tmp_path, monkeypatch): coord = _eligible_coord(tmp_path, monkeypatch, framework="vllm") - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "forge", - "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, - } - ) - envs = coord.shared_state.current_best["extra_envs"] - assert "SGLANG_FP8_BLOCKSCALE_CK_MAX_M" not in envs + assert self._ck_candidates(coord, self._forge_result()) == [] def test_does_not_inject_for_non_gfx942_gpu(self, tmp_path, monkeypatch): coord = _eligible_coord(tmp_path, monkeypatch, gpu_type="mi355x") - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "forge", - "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, - } - ) - envs = coord.shared_state.current_best["extra_envs"] - assert "SGLANG_FP8_BLOCKSCALE_CK_MAX_M" not in envs + assert self._ck_candidates(coord, self._forge_result()) == [] def test_does_not_inject_for_non_block_scale_fp8(self, tmp_path, monkeypatch): coord = _eligible_coord(tmp_path, monkeypatch) monkeypatch.setattr(mcu_mod, "_fp8_is_block_scale", lambda _p: False) - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "forge", - "extra_envs": {"AITER_CONFIG": "/cfg/tuned.json"}, - } - ) - envs = coord.shared_state.current_best["extra_envs"] - assert "SGLANG_FP8_BLOCKSCALE_CK_MAX_M" not in envs + assert self._ck_candidates(coord, self._forge_result()) == [] - def test_respects_preset_value_setdefault(self, tmp_path, monkeypatch): + def test_does_not_double_inject_when_a_tuner_already_carries_the_switch(self, tmp_path, monkeypatch): coord = _eligible_coord(tmp_path, monkeypatch) - coord._promote_gemm_tuning_keep( - { - "status": "ok", - "decision": "KEEP", - "best_speedup": 1.2, - "backend": "forge", - "extra_envs": { - "AITER_CONFIG": "/cfg/tuned.json", - "SGLANG_FP8_BLOCKSCALE_CK_MAX_M": "512", - }, - } + result = self._forge_result( + tuners_run=[ + { + "tuner": "blockscale", + "status": "ok", + "candidate": True, + "env_var": "SGLANG_FP8_BLOCKSCALE_CK_MAX_M", + "env_value": "512", + "best_micro_speedup": 1.4, + } + ] ) - envs = coord.shared_state.current_best["extra_envs"] - assert envs["SGLANG_FP8_BLOCKSCALE_CK_MAX_M"] == "512" + cands = self._ck_candidates(coord, result) + assert len(cands) == 1 + assert cands[0]["env_value"] == "512" class TestHandleGemmTuningResult: @@ -2608,7 +2607,7 @@ async def test_forge_requires_e2e_routes_to_validator(self, tmp_path): async def _fake_validate(result): called["result"] = result - coord.phase_kernel._validate_forge_gemm_tuning_e2e = _fake_validate # type: ignore[assignment] + coord.phase_kernel._validate_gemm_tuning_e2e = _fake_validate # type: ignore[assignment] await coord._handle_gemm_tuning_result( { @@ -2671,7 +2670,7 @@ async def test_forge_no_improvement_but_ck_eligible_routes_to_validator(self, tm async def _fake_validate(result): called["result"] = result - coord.phase_kernel._validate_forge_gemm_tuning_e2e = _fake_validate # type: ignore[assignment] + coord.phase_kernel._validate_gemm_tuning_e2e = _fake_validate # type: ignore[assignment] await coord._handle_gemm_tuning_result( { @@ -2687,8 +2686,17 @@ async def _fake_validate(result): assert coord.shared_state.optimization_stack == [] @pytest.mark.asyncio - async def test_non_forge_routes_to_inline_promote(self, tmp_path): - coord = _coord(tmp_path, baseline_tput=100.0) + async def test_geak_promotes_on_the_measured_tput_not_the_micro_speedup(self, tmp_path, monkeypatch): + coord = _coord(tmp_path, baseline_tput=100.0, framework="sglang") + tuned = tmp_path / "gemm.csv" + tuned.write_text("token,model_dim\n1,2\n", encoding="utf-8") + fake = _make_integrate([{"decision": "KEEP", "new_tput": 150.0, "gain_pct": 50.0}]) + monkeypatch.setattr(krh_mod, "integrate_handler", fake) + monkeypatch.setattr( + KernelPhase, + "_merge_gemm_candidate_with_runtime", + lambda _self, _env_var, env_value: env_value, + ) await coord._handle_gemm_tuning_result( { @@ -2696,12 +2704,45 @@ async def test_non_forge_routes_to_inline_promote(self, tmp_path): "decision": "KEEP", "best_speedup": 1.4, "backend": "geak", - "tuned_file": "/tuned/gemm.csv", + "tuned_file": str(tuned), } ) - assert len(coord.shared_state.optimization_stack) == 1 - assert coord.shared_state.current_best["engine"] == "geak" + assert len(fake.calls) == 1 + stack = coord.shared_state.optimization_stack + assert len(stack) == 1 + assert stack[0]["variant_name"] == "geak_a8w8_blockscale_tuned_gemm" + assert stack[0]["backend"] == "geak" + # 150.0 measured, not baseline * best_speedup (140.0). + assert stack[0]["tput"] == pytest.approx(150.0) + assert coord.shared_state.current_best["tput"] == pytest.approx(150.0) + assert coord.shared_state.cumulative_gain_validated == pytest.approx(50.0) + + @pytest.mark.asyncio + async def test_geak_promotes_nothing_when_the_measurement_reverts(self, tmp_path, monkeypatch): + coord = _coord(tmp_path, baseline_tput=100.0, framework="sglang") + tuned = tmp_path / "gemm.csv" + tuned.write_text("token,model_dim\n1,2\n", encoding="utf-8") + fake = _make_integrate([{"decision": "REVERT", "new_tput": 90.0, "gain_pct": -10.0}]) + monkeypatch.setattr(krh_mod, "integrate_handler", fake) + monkeypatch.setattr( + KernelPhase, + "_merge_gemm_candidate_with_runtime", + lambda _self, _env_var, env_value: env_value, + ) + + await coord._handle_gemm_tuning_result( + { + "status": "ok", + "decision": "KEEP", + "best_speedup": 1.4, + "backend": "geak", + "tuned_file": str(tuned), + } + ) + + assert coord.shared_state.optimization_stack == [] + assert not coord.shared_state.current_best class TestValidateForgeGemmTuningE2E: @@ -2722,7 +2763,7 @@ async def test_no_candidates_returns_early(self, tmp_path, monkeypatch): {"status": "ok", "improved_shapes": 3, "env_var": "", "env_value": ""}, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) assert fake.calls == [] assert result["requires_e2e_validation"] is True @@ -2769,7 +2810,7 @@ async def test_vllm_candidate_without_micro_count_validates_full_env_bundle( ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) assert len(fake.calls) == 1 assert fake.calls[0]["extra_envs"] == { @@ -2827,7 +2868,7 @@ def _merge(env_var, env_value): ], } - await phase._validate_forge_gemm_tuning_e2e(result) + await phase._validate_gemm_tuning_e2e(result) assert set(merge_calls) == { ("AITER_CONFIG_GEMM_A8W8_BLOCKSCALE", str(dense)), @@ -2858,6 +2899,7 @@ async def test_keep_stacks_envs_and_rewrites_result(self, tmp_path, monkeypatch) result = { "workspace": str(tmp_path), + "backend": "forge", "recommended_env": { "AITER_CONFIG_FMOE": str(fmoe_candidate), "AITER_DENSE": "/dense.json", @@ -2886,7 +2928,7 @@ async def test_keep_stacks_envs_and_rewrites_result(self, tmp_path, monkeypatch) }, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) # fmoe_ck on sglang carries the aiter MoE runner arg; dense does not. assert fake.calls[0]["extra_server_args"] == "--moe-runner-backend aiter" @@ -2910,10 +2952,14 @@ async def test_keep_stacks_envs_and_rewrites_result(self, tmp_path, monkeypatch) "gemm_tune_e2e_dense_gemm", } cb = coord.shared_state.current_best - assert cb["engine"] == "forge" + assert cb["variant_name"] == "forge_dense_gemm" assert cb["tput"] == pytest.approx(132.0) assert cb["extra_server_args"] == "--moe-runner-backend aiter" - assert coord.shared_state.cumulative_gain == pytest.approx(32.0) + # Both tuners' envs accumulate onto current_best, one lift each. + assert cb["extra_envs"] == { + "AITER_CONFIG_FMOE": str(fmoe_candidate), + "AITER_DENSE": "/dense.json", + } assert coord.shared_state.cumulative_gain_validated == pytest.approx(32.0) # Result rewritten to the E2E-validated outcome. @@ -2955,7 +3001,7 @@ async def test_injects_synthetic_ck_candidate_when_eligible_no_table_candidates( }, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) assert len(fake.calls) == 1 assert fake.calls[0]["extra_envs"] == {"SGLANG_FP8_BLOCKSCALE_CK_MAX_M": "256"} @@ -2993,7 +3039,7 @@ async def test_no_synthetic_ck_candidate_when_not_eligible(self, tmp_path, monke }, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) assert fake.calls == [] assert coord.shared_state.optimization_stack == [] @@ -3020,7 +3066,7 @@ async def test_keep_only_when_tput_improves(self, tmp_path, monkeypatch): }, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) assert coord.shared_state.optimization_stack == [] assert result["decision"] == "REVERT" @@ -3047,7 +3093,7 @@ async def test_all_revert_resets_and_marks_no_gain(self, tmp_path, monkeypatch): }, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) assert coord.shared_state.optimization_stack == [] assert result["decision"] == "REVERT" @@ -3080,7 +3126,7 @@ async def _boom(payload, *, session_dir): }, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) assert result["decision"] == "REVERT" reverted = result["e2e_results"]["reverted"] @@ -3119,7 +3165,7 @@ async def _fake(payload, *, session_dir): }, ], } - await coord._validate_forge_gemm_tuning_e2e(result) + await coord._validate_gemm_tuning_e2e(result) # Fallback budget is 15 minutes. assert captured["budget"] == 15 diff --git a/src/hyperloom/inference_optimizer/tests/test_coordinator_kb_writes.py b/src/hyperloom/inference_optimizer/tests/test_coordinator_kb_writes.py index 60cfc97292..d9bce75965 100644 --- a/src/hyperloom/inference_optimizer/tests/test_coordinator_kb_writes.py +++ b/src/hyperloom/inference_optimizer/tests/test_coordinator_kb_writes.py @@ -423,7 +423,6 @@ def test_close_does_not_clobber_with_bare_baseline_higher_tput( ss.current_best = {"action": "baseline", "name": "baseline", "tput": 2813.5} ss.optimization_stack = [] ss.cumulative_gain_validated = 0.0 - ss.cumulative_gain = 0.0 coord.finalize_recipe_and_journal() row = coord.recipe_kb.get_recipe(canonical_id=cid) assert row["best_throughput"] == 2532.0, "bare-baseline CLOSE clobbered a validated best_throughput" diff --git a/src/hyperloom/inference_optimizer/tests/test_coverage_boost_unit.py b/src/hyperloom/inference_optimizer/tests/test_coverage_boost_unit.py index 9ee0eafdd2..5e0c30e935 100644 --- a/src/hyperloom/inference_optimizer/tests/test_coverage_boost_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_coverage_boost_unit.py @@ -349,9 +349,9 @@ def test_paths_helpers(monkeypatch, tmp_path) -> None: # asset_root override that exists. monkeypatch.setenv(paths.ENV_OVERRIDE_ASSET_ROOT, str(tmp_path)) assert paths.asset_root() == tmp_path - # find_latest returns None when workspace root is not a dir. + # workspace_root echoes USER_DATA_PATH even when the dir is absent. monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path / "does_not_exist")) - assert paths.find_latest_per_session_dir() is None + assert paths.workspace_root() == tmp_path / "does_not_exist" def test_paths_asset_root_missing_override(monkeypatch, tmp_path) -> None: diff --git a/src/hyperloom/inference_optimizer/tests/test_decision_framework.py b/src/hyperloom/inference_optimizer/tests/test_decision_framework.py index d3388ebbad..a95d0e112f 100644 --- a/src/hyperloom/inference_optimizer/tests/test_decision_framework.py +++ b/src/hyperloom/inference_optimizer/tests/test_decision_framework.py @@ -187,7 +187,8 @@ async def fake(payload, *, session_dir): ), ) - assert c.shared_state.last_gemm_tuning["status"] == "ok" + # The E2E validator rewrites the stored result to its measured outcome. + assert c.shared_state.last_gemm_tuning["status"] == "complete" assert c.shared_state.last_gemm_tuning["best_speedup"] == 1.2 assert c.shared_state.gemm_tuning_attempts assert "last_gemm_tuning=" in c.shared_state.to_prompt_summary() @@ -233,8 +234,11 @@ async def test_kernel_entry_auto_runs_gemm_tuning_for_fp8_sglang( } c.shared_state.continue_kernel_after_gemm = False calls: list[dict] = [] + tuned = session_dir / "tuned.csv" + tuned.write_text("M,N,K,kernelId\n16,512,7168,3\n", encoding="utf-8") from hyperloom.orchestrator.kernel import request_handlers as kernel_request_handlers + from hyperloom.orchestrator.phases.kernel import KernelPhase async def fake_handler(payload, *, session_dir): calls.append(dict(payload)) @@ -242,25 +246,32 @@ async def fake_handler(payload, *, session_dir): "status": "complete", "decision": "KEEP", "best_speedup": 1.28, - "tuned_file": "/tmp/tuned.csv", + "tuned_file": str(tuned), } + async def fake_integrate(payload, *, session_dir): + return {"decision": "KEEP", "new_tput": 900.0, "gain_pct": 12.5} + monkeypatch.setattr( kernel_request_handlers, "run_gemm_tuning_handler", fake_handler, ) + monkeypatch.setattr(kernel_request_handlers, "integrate_handler", fake_integrate) + monkeypatch.setattr( + KernelPhase, + "_merge_gemm_candidate_with_runtime", + lambda _self, _env_var, env_value: env_value, + ) await c._on_enter_kernel(from_phase="EXPLORE") assert calls - assert c.shared_state.last_gemm_tuning["status"] == "complete" - assert c.shared_state.last_gemm_tuning["best_speedup"] == 1.28 assert c.shared_state.gemm_tuning_attempts assert c.shared_state.current_best["action"] == "gemm_tuning" - assert c.shared_state.current_best["tput"] == 1024.0 - assert c.shared_state.cumulative_gain == pytest.approx(28.0) - assert c.shared_state.cumulative_gain_validated == pytest.approx(28.0) + # The measured rebench, not baseline_tput * best_speedup (1024.0). + assert c.shared_state.current_best["tput"] == 900.0 + assert c.shared_state.cumulative_gain_validated == pytest.approx(12.5) assert c.shared_state.optimization_stack[-1]["action"] == "gemm_tuning" assert c._gemm_tuning_required_before_kernel_opt() is False finally: diff --git a/src/hyperloom/inference_optimizer/tests/test_diffusion_latency_final_and_ceiling.py b/src/hyperloom/inference_optimizer/tests/test_diffusion_latency_final_and_ceiling.py index 33edce8f70..7e6282b07d 100644 --- a/src/hyperloom/inference_optimizer/tests/test_diffusion_latency_final_and_ceiling.py +++ b/src/hyperloom/inference_optimizer/tests/test_diffusion_latency_final_and_ceiling.py @@ -6,7 +6,7 @@ * ``framework_registry.primary_metric_name`` (which field is the result). * ``collect_roofline_progress`` independent latency ceiling + ``ceiling_kind``. * ``_normalize_roofline_snapshot`` preserving the latency siblings. -* recorder ``_snapshot_final`` emitting e2el / unit / primary_metric. +* ``collect_final`` emitting e2el / unit / primary_metric. * ``SharedState._backfill_scriptable_latency`` deriving e2el from tput. """ @@ -14,7 +14,7 @@ from hyperloom.inference_optimizer import framework_registry as fr from hyperloom.inference_optimizer.breakdown.collectors import roofline as col -from hyperloom.inference_optimizer.breakdown.recorder import instrument as inst +from hyperloom.inference_optimizer.breakdown.collectors import sessions as sess class TestPrimaryMetricName: @@ -53,7 +53,7 @@ def _state(self, snap: dict) -> dict: return { "framework": "xdit", "baseline_tput": 0.168919, - "cumulative_gain": 550.5, + "cumulative_gain_validated": 550.5, "optimization_stack": [ {"ts": "2026-01-01T00:00:00", "tput": 1.098901, "variant_name": "v", "action": "explore"} ], @@ -104,58 +104,55 @@ def test_no_ceiling_when_latency_partial(self, tmp_path): assert out["latency_ceiling_available"] is False -class _Rec: - def __init__(self): - self.singletons: dict[str, dict] = {} +class TestCollectFinalEmitsLatency: + """``collect_final`` is the single producer of the final section.""" - def record_singleton(self, name, payload): - self.singletons[name] = payload + def _state(self, framework: str, current_best: dict) -> dict: + return { + "framework": framework, + "current_best": current_best, + "optimization_stack": [{"action": current_best.get("action", "explore")}], + "cumulative_gain_validated": 0.0, + "cumulative_gain_validated_ts": "", + } + def test_scriptable_final_surfaces_the_derived_e2el(self, tmp_path): + """``save`` derives the latency; the collector surfaces what it wrote. -class _St: - def __init__(self, **kw): - self.framework = kw.get("framework", "xdit") - self.current_best = kw.get("current_best", {}) - self.optimization_stack = kw.get("optimization_stack", []) - self.cumulative_gain_validated = kw.get("cumulative_gain_validated", 0.0) - self.cumulative_gain = kw.get("cumulative_gain", 0.0) - self.cumulative_gain_validated_ts = kw.get("cumulative_gain_validated_ts", "") + ``_backfill_scriptable_latency`` runs before ``state.json`` is written, + so ``current_best`` already carries ``e2el_mean_ms`` when read back. + """ + from hyperloom.orchestrator.state.shared_state import SharedState + st = SharedState(session_id="s", model_name="m", model_path="/m") + st.framework = "xdit" + st.current_best = {"action": "explore", "tput": 1.098901} + st._backfill_scriptable_latency() -class TestSnapshotFinalEmitsLatency: - def test_scriptable_final_derives_e2el_from_tput(self): - rec = _Rec() - st = _St( - framework="xdit", - current_best={"action": "explore", "tput": 1.098901}, - optimization_stack=[{"action": "explore"}], + final = sess.collect_final( + tmp_path, + self._state("xdit", st.current_best), + [], ) - inst._snapshot_final(rec, st) - final = rec.singletons["final"] assert final["throughput_unit"] == "img/s" assert final["primary_metric"] == "e2el_mean_ms" # 1000 / 1.098901 ~= 910.0 assert final["e2el_mean_ms"] == round(1000.0 / 1.098901, 4) - def test_scriptable_final_prefers_measured_e2el(self): - rec = _Rec() - st = _St( - framework="xdit", - current_best={"action": "explore", "tput": 1.098901, "e2el_mean_ms": 980.0}, - optimization_stack=[{"action": "explore"}], + def test_scriptable_final_prefers_measured_e2el(self, tmp_path): + final = sess.collect_final( + tmp_path, + self._state("xdit", {"action": "explore", "tput": 1.098901, "e2el_mean_ms": 980.0}), + [], ) - inst._snapshot_final(rec, st) - assert rec.singletons["final"]["e2el_mean_ms"] == 980.0 - - def test_serving_final_has_no_derived_e2el(self): - rec = _Rec() - st = _St( - framework="sglang", - current_best={"action": "grid", "tput": 123.4}, - optimization_stack=[{"action": "grid"}], + assert final["e2el_mean_ms"] == 980.0 + + def test_serving_final_has_no_derived_e2el(self, tmp_path): + final = sess.collect_final( + tmp_path, + self._state("sglang", {"action": "grid", "tput": 123.4}), + [], ) - inst._snapshot_final(rec, st) - final = rec.singletons["final"] assert final["throughput_unit"] == "tok/s" assert final["primary_metric"] == "throughput_tok_s_per_gpu" assert final["e2el_mean_ms"] is None diff --git a/src/hyperloom/inference_optimizer/tests/test_drop_scoreboard.py b/src/hyperloom/inference_optimizer/tests/test_drop_scoreboard.py index ecaaf2db69..3db2ae99e7 100644 --- a/src/hyperloom/inference_optimizer/tests/test_drop_scoreboard.py +++ b/src/hyperloom/inference_optimizer/tests/test_drop_scoreboard.py @@ -60,6 +60,7 @@ def _legacy_state_payload() -> dict: "session_id": "legacy-sid", "baseline_tput": 1234.0, "cumulative_gain": 2.5, + "cumulative_gain_validated": 2.0, "action_scores": { "backends": {"base_score": 5.0, "score_mult": 0.8}, "params": {"base_score": 4.0, "score_mult": 1.0}, @@ -79,7 +80,14 @@ def test_from_dict_drops_action_scores_silently(): loaded = SharedState.from_dict(raw) assert not hasattr(loaded, "action_scores") assert loaded.baseline_tput == 1234.0 - assert loaded.cumulative_gain == 2.5 + assert loaded.cumulative_gain_validated == 2.0 + + +def test_from_dict_drops_the_unvalidated_gain(): + """The raw gain was a second copy of the validated one; a resume must not revive it.""" + loaded = SharedState.from_dict(_legacy_state_payload()) + assert not hasattr(loaded, "cumulative_gain") + assert loaded.cumulative_gain_validated == 2.0 def test_load_or_init_roundtrips_through_drop(tmp_path, monkeypatch): @@ -92,6 +100,7 @@ def test_load_or_init_roundtrips_through_drop(tmp_path, monkeypatch): loaded.save(sd) written = json.loads((sd / "state.json").read_text()) assert "action_scores" not in written + assert "cumulative_gain" not in written def test_scoring_module_was_retired(): diff --git a/src/hyperloom/inference_optimizer/tests/test_env_safety.py b/src/hyperloom/inference_optimizer/tests/test_env_safety.py index c5d1359188..cf9f3649ed 100644 --- a/src/hyperloom/inference_optimizer/tests/test_env_safety.py +++ b/src/hyperloom/inference_optimizer/tests/test_env_safety.py @@ -140,6 +140,45 @@ def test_scrub_benchmark_process_env_removes_control_plane_credentials(): } +def test_variant_env_key_allows_workload_pins_and_blocks_hijacks(): + # Sweep, conc-sweep and shape-capture grids set these from code, so an + # allowlist that dropped them would silently flatten every variant. + for pinned in ("CONC", "ISL", "OSL", "NUM_PROMPTS", "RUN_EVAL", "PORT", "TP", "MAX_MODEL_LEN"): + assert common_env_safety.is_allowed_variant_env_key(pinned) + for knob in ("SGLANG_USE_AITER", "VLLM_USE_MTP", "AITER_CONFIG_GEMM_A8W8", "PYTORCH_TUNABLEOP_ENABLED"): + assert common_env_safety.is_allowed_variant_env_key(knob) + # Name-shape matching would read this as a credential; it is the private + # model download token and has to survive. + assert common_env_safety.is_allowed_variant_env_key("HF_TOKEN") + + for hijack in ("LD_PRELOAD", "PATH", "PYTHONPATH", "BASH_ENV", "LD_AUDIT", "PYTHONSTARTUP"): + assert not common_env_safety.is_allowed_variant_env_key(hijack) + for secret in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY", "LLM_GATEWAY_KEY"): + assert not common_env_safety.is_allowed_variant_env_key(secret) + assert not common_env_safety.is_allowed_variant_env_key("bad key") + assert not common_env_safety.is_allowed_variant_env_key("") + + +def test_build_benchmark_env_layers_over_parent_and_normalizes(monkeypatch): + monkeypatch.setenv("INHERITED_KNOB", "from-parent") + monkeypatch.setenv("OPENAI_API_KEY", "must-not-reach-benchmark") + monkeypatch.setenv("SGLANG_USE_AITER", "0") + + env = common_env_safety.build_benchmark_env( + {"SGLANG_USE_AITER": "1", "TP": 8, "RANDOM_RANGE_RATIO": 0.8}, + None, + {"TP": 4, "lowercase_knob": "on"}, + ) + + assert env["INHERITED_KNOB"] == "from-parent" + assert env["SGLANG_USE_AITER"] == "1" + assert env["TP"] == "4" + assert env["RANDOM_RANGE_RATIO"] == "0.8" + # Env names are conventionally upper case; a lower-case key would be inert. + assert env["LOWERCASE_KNOB"] == "on" + assert "OPENAI_API_KEY" not in env + + def test_redact_secret_values_masks_assignments_and_bearer_tokens(): text = "OPENAI_API_KEY=ak-sensitive-value Authorization: Bearer sensitive-token" diff --git a/src/hyperloom/inference_optimizer/tests/test_framework_agent_artifacts.py b/src/hyperloom/inference_optimizer/tests/test_framework_agent_artifacts.py deleted file mode 100644 index 67e9dbfe3e..0000000000 --- a/src/hyperloom/inference_optimizer/tests/test_framework_agent_artifacts.py +++ /dev/null @@ -1,159 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""FRAMEWORK candidate artifacts and outcome classification tests.""" - -from __future__ import annotations - -import json -from pathlib import Path - -from hyperloom.orchestrator.framework.artifacts import ( - candidate_slug, - summarize_candidate_outcomes, - write_decision_json, - write_semantic_audit, -) - - -def test_candidate_slug_sanitizes_url(): - slug = candidate_slug("https://github.com/ROCm/vllm/pull/1234") - assert "/" not in slug and ":" not in slug - assert slug.strip("-") == slug - assert slug - - -def test_candidate_slug_empty_defaults(): - assert candidate_slug("") == "candidate" - assert candidate_slug("///") == "candidate" - - -def test_candidate_slug_caps_length(): - assert len(candidate_slug("x" * 500)) == 96 - - -def test_write_decision_json_roundtrip(tmp_path: Path): - dest = write_decision_json( - tmp_path, - candidate_id="ROCm/vllm#42", - batch_id="batch-001", - status="kept", - kept=True, - provenance="raw_diff", - reason="", - gain_pct=7.5, - accuracy_pass=True, - extra={"workspace": "/tmp/x"}, - ) - assert dest is not None - p = Path(dest) - assert p.name == "decision.json" - assert p.parent.parent.name == "framework_agent" - data = json.loads(p.read_text()) - assert data["candidate_id"] == "ROCm/vllm#42" - assert data["batch_id"] == "batch-001" - assert data["status"] == "kept" - assert data["kept"] is True - assert data["provenance"] == "raw_diff" - assert data["gain_pct"] == 7.5 - assert data["accuracy_pass"] is True - assert data["workspace"] == "/tmp/x" - assert data["ts"] - - -def test_write_decision_json_normalizes_optional_numerics(tmp_path: Path): - dest = write_decision_json( - tmp_path, - candidate_id="cand-x", - status="critic_denied", - gain_pct=None, - accuracy_pass=None, - ) - data = json.loads(Path(dest).read_text()) - assert data["gain_pct"] is None - assert data["accuracy_pass"] is None - assert data["kept"] is False - - -def test_write_decision_json_never_raises_on_bad_session_dir(tmp_path: Path): - bad = tmp_path / "afile" - bad.write_text("x", encoding="utf-8") - out = write_decision_json(bad, candidate_id="c", status="failed") - assert out is None - - -def test_write_semantic_audit_co_located(tmp_path: Path): - verdict = { - "candidate_id": "ROCm/vllm#42", - "semantic_status": "already_equivalent", - "applicability": "not_applicable", - "recommended_next_step": "skip", - "confidence": 0.95, - "evidence": [{"local_file": "vllm/x.py", "symbol": "f", "reason": "present"}], - "risks": [], - } - dest = write_semantic_audit(tmp_path, candidate_id="ROCm/vllm#42", verdict=verdict) - assert dest is not None - p = Path(dest) - assert p.name == "semantic_audit.json" - assert p.parent.parent.name == "framework_agent" - assert (p.parent / "semantic_audit.md").exists() - data = json.loads(p.read_text()) - assert data["semantic_status"] == "already_equivalent" - decision = write_decision_json(tmp_path, candidate_id="ROCm/vllm#42", status="already_present") - assert Path(decision).parent == p.parent - - -def test_write_semantic_audit_empty_verdict_returns_none(tmp_path: Path): - assert write_semantic_audit(tmp_path, candidate_id="c", verdict={}) is None - - -def test_summarize_empty_discovery(): - s = summarize_candidate_outcomes([]) - assert s["outcome_class"] == "empty_discovery" - assert s["total"] == 0 - assert s["keeps"] == 0 - - -def test_summarize_tested_no_keep(): - progress = [ - {"status": "reverted", "kept": False, "batch_id": "b1"}, - {"status": "apply_failed", "kept": False, "batch_id": "b1"}, - {"status": "critic_denied", "kept": False, "batch_id": "b1"}, - ] - s = summarize_candidate_outcomes(progress) - assert s["outcome_class"] == "tested_no_keep" - assert s["keeps"] == 0 - assert s["total"] == 3 - # critic_denied is not a "tested" (applied) status; reverted + apply_failed are. - assert s["tested"] == 2 - assert s["by_status"]["reverted"] == 1 - assert s["by_status"]["critic_denied"] == 1 - - -def test_summarize_tested_with_keep(): - progress = [ - {"status": "kept", "kept": True, "batch_id": "b1"}, - {"status": "reverted", "kept": False, "batch_id": "b1"}, - ] - s = summarize_candidate_outcomes(progress) - assert s["outcome_class"] == "tested_with_keep" - assert s["keeps"] == 1 - - -def test_summarize_filters_by_batch(): - progress = [ - {"status": "kept", "kept": True, "batch_id": "b1"}, - {"status": "reverted", "kept": False, "batch_id": "b2"}, - ] - s_b2 = summarize_candidate_outcomes(progress, batch_id="b2") - assert s_b2["outcome_class"] == "tested_no_keep" - assert s_b2["total"] == 1 - s_b1 = summarize_candidate_outcomes(progress, batch_id="b1") - assert s_b1["outcome_class"] == "tested_with_keep" - - -def test_summarize_ignores_non_dict_rows(): - s = summarize_candidate_outcomes([None, "x", {"status": "kept", "kept": True}]) # type: ignore[list-item] - assert s["total"] == 1 - assert s["keeps"] == 1 diff --git a/src/hyperloom/inference_optimizer/tests/test_framework_agent_critic_gate.py b/src/hyperloom/inference_optimizer/tests/test_framework_agent_critic_gate.py index 605c25b185..2c9fd13f79 100644 --- a/src/hyperloom/inference_optimizer/tests/test_framework_agent_critic_gate.py +++ b/src/hyperloom/inference_optimizer/tests/test_framework_agent_critic_gate.py @@ -154,6 +154,19 @@ async def test_materialize_unknown_route_runs_both_tracks(coord: Coordinator, mo assert len(author) == 1 +@pytest.mark.asyncio +async def test_enqueued_task_rides_the_decaying_keep_curve(coord: Coordinator) -> None: + """framework_agent grades against the same per-cycle bar as explore and integrate_patch.""" + from hyperloom.orchestrator.phases.machine_state import decaying_keep_threshold_pct + + coord.shared_state.macro_cycle = 2 + await coord.phase_framework._enqueue_framework_agent_task(dict(_CANDIDATE)) + + queued = [t for t in await coord.tasks.queued() if t.kind == "framework_agent"] + assert len(queued) == 1 + assert queued[0].params["keep_threshold_pct"] == pytest.approx(decaying_keep_threshold_pct(2)) + + # -- verdict drives materialize/reject through _handle_single_verdict -------- @pytest.mark.asyncio async def test_approve_verdict_materializes(coord: Coordinator, monkeypatch) -> None: diff --git a/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit.py b/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit.py index 670698701d..76dac063f2 100644 --- a/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit.py +++ b/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit.py @@ -18,7 +18,6 @@ from hyperloom.orchestrator.loop import coordinator as coord_mod from hyperloom.orchestrator.framework import client as fa_client_mod from hyperloom.orchestrator.framework import paths as fp_mod -from hyperloom.orchestrator.framework import artifacts as fpa_mod from hyperloom.orchestrator.knowledge import kb_writeback as kb_mod from hyperloom.orchestrator.phases import machine_state as ps_mod from hyperloom.orchestrator.actions.executors import framework_agent as fpr_mod @@ -686,17 +685,6 @@ async def _invoke(*, subcommand, request, session_dir, timeout_sec): assert req["diff_url"] == "http://x/d" -# -------------------------------------------------------------------------- -# framework_agent_artifacts.write_semantic_audit error path -# -------------------------------------------------------------------------- -def test_write_semantic_audit_error_returns_none(tmp_path) -> None: - # Pass a FILE as the session dir so the runs_dir mkdir fails -> exception path. - file_path = tmp_path / "not_a_dir" - file_path.write_text("x") - out = fpa_mod.write_semantic_audit(file_path, candidate_id="c1", verdict={"semantic_status": "x"}) - assert out is None - - # -------------------------------------------------------------------------- # _dispatch_paused_for_phase_budget # -------------------------------------------------------------------------- diff --git a/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit_coverage_unit.py b/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit_coverage_unit.py index c170de0a79..b6d2fcf111 100644 --- a/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit_coverage_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_framework_agent_selection_audit_coverage_unit.py @@ -18,7 +18,6 @@ from hyperloom.orchestrator.loop import coordinator as coord_mod from hyperloom.orchestrator.framework import client as fa_client_mod from hyperloom.orchestrator.framework import paths as fp_mod -from hyperloom.orchestrator.framework import artifacts as fpa_mod from hyperloom.orchestrator.knowledge import kb_writeback as kb_mod from hyperloom.orchestrator.phases import machine_state as ps_mod from hyperloom.orchestrator.actions.executors import framework_agent as fpr_mod @@ -605,17 +604,6 @@ async def _invoke(*, subcommand, request, session_dir, timeout_sec): assert req["diff_url"] == "http://x/d" -# -------------------------------------------------------------------------- -# framework_agent_artifacts.write_semantic_audit error path -# -------------------------------------------------------------------------- -def test_write_semantic_audit_error_returns_none(tmp_path) -> None: - # Pass a FILE as the session dir so the runs_dir mkdir fails -> exception path. - file_path = tmp_path / "not_a_dir" - file_path.write_text("x") - out = fpa_mod.write_semantic_audit(file_path, candidate_id="c1", verdict={"semantic_status": "x"}) - assert out is None - - # -------------------------------------------------------------------------- # _dispatch_paused_for_phase_budget # -------------------------------------------------------------------------- diff --git a/src/hyperloom/inference_optimizer/tests/test_framework_switch_manifest.py b/src/hyperloom/inference_optimizer/tests/test_framework_switch_manifest.py index a74f5a9b05..30478599c8 100644 --- a/src/hyperloom/inference_optimizer/tests/test_framework_switch_manifest.py +++ b/src/hyperloom/inference_optimizer/tests/test_framework_switch_manifest.py @@ -434,15 +434,6 @@ def test_attribution_of_an_unknown_switch_is_ignored(): assert state.record_framework_lever_attribution("HL_GHOST", gain_pct=1.0, source="additive") is False -def test_levers_can_be_selected_by_state(): - """The explore seeder needs the dormant and active sets separately.""" - state = _state() - state.record_authored_framework_levers([_entry_parsed("HL_ON")], default_on=True) - state.record_authored_framework_levers([_entry_parsed("HL_OFF")], default_on=False) - assert [r["switch"] for r in state.framework_levers_by_state(default_on=True)] == ["HL_ON"] - assert [r["switch"] for r in state.framework_levers_by_state(default_on=False)] == ["HL_OFF"] - - def _entry_parsed(switch: str) -> dict[str, Any]: """Return a single parsed manifest entry for ``switch``.""" switches, _ = manifest.parse_manifest([_entry(switch)]) diff --git a/src/hyperloom/inference_optimizer/tests/test_geak_gain_alignment.py b/src/hyperloom/inference_optimizer/tests/test_geak_gain_alignment.py index 0284fc785e..124c9f8ec6 100644 --- a/src/hyperloom/inference_optimizer/tests/test_geak_gain_alignment.py +++ b/src/hyperloom/inference_optimizer/tests/test_geak_gain_alignment.py @@ -213,7 +213,6 @@ async def _fake_sweep(**_kwargs): # Validated == the MEASURED same-harness total (≈+8.8%), NOT the hot A/B (+13.29%). assert ss.cumulative_gain_validated == pytest.approx(expected_pct, abs=1e-6) assert ss.cumulative_gain_validated != pytest.approx(13.29, abs=0.05) - assert ss.cumulative_gain_provenance == "geak_same_harness_geak" assert ss.resume_pending_revalidation is False # Rebench-first writes the headline HERE: current_best.tput == measured, and # the geak_e2e stack entry now exists. @@ -322,23 +321,18 @@ async def _fake_sweep(**_kwargs): ) repeated = assemble_parts(tmp_path)["kernel_journey"]["kernels"] assert len(repeated) == 2 - adoption = rejected["adoptions"][0] - assert adoption["decision"] == "REVERT" - assert adoption["validated"] is False # ── Fix B: report renders a PROVISIONAL gain honestly (not "+0.00% validated") ─ -def _final_breakdown(*, provenance: str, pending: bool, gain_v: float, gain_round: float) -> dict: +def _final_breakdown(*, pending: bool, gain_v: float) -> dict: return { "session": {"image": ""}, "baseline": {"throughput_tok_s_per_gpu": 2844.2}, "final": { "throughput_tok_s_per_gpu": 3236.5, "cumulative_gain_pct_validated": gain_v, - "cumulative_gain_pct_per_round_sum": gain_round, - "cumulative_gain_provenance": provenance, "revalidation_pending": pending, "validated_at_stack_len": 2, "validated_ts": "", @@ -351,16 +345,13 @@ def test_report_shows_provisional_not_zero_validated() -> None: """A cross-harness provisional (validated pending) must not read as +0.00%.""" sec = render_final( _final_breakdown( - provenance="geak_cross_harness_provisional", pending=True, gain_v=0.0, # collectors coerces a pending/unstamped validated to 0.0 - gain_round=13.79, ) ) facts = " ".join(sec.key_facts) warns = " ".join(sec.warnings) - assert "Provisional" in facts - assert "13.79" in facts or "13.8" in facts + assert "PENDING same-harness revalidation" in facts assert "Validated cumulative gain" not in facts # must NOT claim validation assert "+0.00%" not in facts # must NOT read as no-op assert "PROVISIONAL" in warns and "cross-harness" in warns @@ -370,10 +361,8 @@ def test_report_shows_validated_when_same_harness_confirmed() -> None: """A same-harness validated gain renders as authoritative, no provisional tag.""" sec = render_final( _final_breakdown( - provenance="geak_orch_harness_validated", pending=False, gain_v=13.5, - gain_round=13.5, ) ) facts = " ".join(sec.key_facts) @@ -403,7 +392,7 @@ async def test_2b_stamps_validated_from_orchestrator_rebench(tmp_path: Path) -> """decision==validated → validated == (measured − baseline)/baseline, same harness.""" base, measured = 2844.209, 3270.0 # ~+14.97%, engaged + identity matches coord = _coord(tmp_path, baseline=base, best_tput=3236.489) - coord.shared_state.optimization_stack = [{"action": "geak_e2e", "tput": 3236.489}] + coord.shared_state.optimization_stack = [{"action": "geak_e2e", "variant_name": "geak_e2e", "tput": 3236.489}] coord.shared_state.resume_pending_revalidation = True # Guard: the GEAK-harness fallback must NOT be taken on the validated path. @@ -422,7 +411,6 @@ async def _must_not_fallback(**_kwargs): ss = coord.shared_state expected_pct = (measured - base) / base * 100.0 assert ss.cumulative_gain_validated == pytest.approx(expected_pct, abs=1e-6) - assert ss.cumulative_gain_provenance == "geak_orch_harness_validated" assert ss.resume_pending_revalidation is False assert ss.cumulative_gain_validated_stack_len == 1 @@ -432,7 +420,7 @@ async def test_2b_identity_mismatch_defers_to_geak_harness(tmp_path: Path) -> No """decision==fallback (config drift) → NO validated stamp; 2a is invoked.""" base, measured = 2844.209, 3270.0 # engaged, but fingerprint won't match coord = _coord(tmp_path, baseline=base, best_tput=3236.489) - coord.shared_state.optimization_stack = [{"action": "geak_e2e", "tput": 3236.489}] + coord.shared_state.optimization_stack = [{"action": "geak_e2e", "variant_name": "geak_e2e", "tput": 3236.489}] coord.shared_state.resume_pending_revalidation = True coord.shared_state.geak_pending = { "status": "awaiting_rebench", @@ -458,7 +446,6 @@ async def _fallback(**_kwargs): # 2b did NOT stamp validated (still 0); it deferred to the GEAK harness (2a). assert called["n"] == 1 assert ss.cumulative_gain_validated == pytest.approx(0.0) - assert ss.cumulative_gain_provenance != "geak_orch_harness_validated" assert not ss.geak_pending assert ss.resume_pending_revalidation is False assert ss.geak_result["revalidation_status"] == "fallback_failed" @@ -488,7 +475,6 @@ async def _must_not_fallback(**_kwargs): ss = coord.shared_state assert ss.current_best["tput"] == pytest.approx(current_best) assert ss.cumulative_gain_validated == pytest.approx(0.0) - assert ss.cumulative_gain_provenance != "geak_orch_harness_validated" assert not any(e.get("action") == "geak_e2e" for e in ss.optimization_stack) assert ss.resume_pending_revalidation is False assert not ss.geak_pending @@ -523,7 +509,6 @@ def test_record_candidate_writes_pending_not_headline(tmp_path: Path) -> None: ss = coord.shared_state # Headline is UNCHANGED — no premature promote. assert ss.current_best == before_best - assert ss.cumulative_gain == pytest.approx(0.0) assert ss.cumulative_gain_validated == pytest.approx(0.0) assert not any(e.get("action") == "geak_e2e" for e in ss.optimization_stack) # The candidate is recorded as pending with audit-only self-reported numbers. @@ -548,7 +533,6 @@ def test_promote_from_candidate_writes_measured_headline(tmp_path: Path) -> None coord._promote_geak_from_candidate( result, measured_tput=measured, - provenance="geak_orch_harness_validated", ) ss = coord.shared_state expected_pct = (measured - base) / base * 100.0 @@ -557,8 +541,6 @@ def test_promote_from_candidate_writes_measured_headline(tmp_path: Path) -> None assert ss.current_best["extra_server_args"] == "--max-num-batched-tokens 24576" assert ss.current_best["extra_envs"].get("VLLM_ROCM_USE_AITER") == "0" assert ss.cumulative_gain_validated == pytest.approx(expected_pct) - assert ss.cumulative_gain == pytest.approx(expected_pct) - assert ss.cumulative_gain_provenance == "geak_orch_harness_validated" assert ss.resume_pending_revalidation is False assert any(e.get("action") == "geak_e2e" for e in ss.optimization_stack) assert not ss.geak_pending @@ -573,8 +555,6 @@ def test_report_shows_pending_candidate_excluded_from_headline() -> None: "final": { "throughput_tok_s_per_gpu": 2844.2, "cumulative_gain_pct_validated": 0.0, - "cumulative_gain_pct_per_round_sum": 0.0, - "cumulative_gain_provenance": "", "revalidation_pending": False, "action_path": [], "geak_pending": {"status": "awaiting_rebench", "self_reported_gain_pct": 13.79}, @@ -631,7 +611,6 @@ async def _must_not_fallback(**_kwargs): ss = coord.shared_state assert ss.current_best["tput"] == pytest.approx(current_best) assert ss.cumulative_gain_validated == pytest.approx(0.0) - assert ss.cumulative_gain_provenance != "geak_orch_harness_validated" assert not any(e.get("action") == "geak_e2e" for e in ss.optimization_stack) assert ss.resume_pending_revalidation is False assert not ss.geak_pending @@ -681,7 +660,6 @@ async def _must_not_fallback(**_kwargs): expected_pct = (measured - base) / base * 100.0 assert ss.current_best["tput"] == pytest.approx(measured) assert ss.cumulative_gain_validated == pytest.approx(expected_pct) - assert ss.cumulative_gain_provenance == "geak_orch_harness_validated" assert any(e.get("action") == "geak_e2e" for e in ss.optimization_stack) assert not ss.geak_pending @@ -715,7 +693,6 @@ async def _must_not_fallback(**_kwargs): ss = coord.shared_state assert ss.current_best["tput"] == pytest.approx(current_best) assert ss.cumulative_gain_validated == pytest.approx(0.0) - assert ss.cumulative_gain_provenance != "geak_orch_harness_validated" assert not any(e.get("action") == "geak_e2e" for e in ss.optimization_stack) assert ss.resume_pending_revalidation is False assert not ss.geak_pending @@ -878,7 +855,7 @@ async def test_2b_empty_result_with_prior_geak_e2e_still_promotes(tmp_path: Path (the material was proven in the original KERNEL cycle).""" base, current_best, measured = 8668.5946, 8900.0, 9600.0 coord = _coord(tmp_path, baseline=base, best_tput=current_best) - coord.shared_state.optimization_stack = [{"action": "geak_e2e", "tput": current_best}] + coord.shared_state.optimization_stack = [{"action": "geak_e2e", "variant_name": "geak_e2e", "tput": current_best}] coord.shared_state.resume_pending_revalidation = True coord.shared_state.geak_result = {} # lost on resume @@ -897,7 +874,6 @@ async def _must_not_fallback(**_kwargs): ss = coord.shared_state expected_pct = (measured - base) / base * 100.0 assert ss.cumulative_gain_validated == pytest.approx(expected_pct) - assert ss.cumulative_gain_provenance == "geak_orch_harness_validated" assert ss.resume_pending_revalidation is False @@ -917,7 +893,7 @@ async def test_2b_resume_reverify_of_promoted_geak_win_still_promotes(tmp_path: coord.shared_state.current_best["extra_envs"] = {"VLLM_ROCM_USE_AITER": "1"} coord.shared_state.optimization_stack = [ {"action": "explore", "variant_name": "kv-cache-fp8", "tput": 8900.0}, - {"action": "geak_e2e", "tput": current_best}, + {"action": "geak_e2e", "variant_name": "geak_e2e", "tput": current_best}, {"action": "integrate_patch", "variant_name": "kernel-x", "tput": current_best}, ] coord.shared_state.resume_pending_revalidation = True @@ -947,6 +923,5 @@ async def _must_not_fallback(**_kwargs): ss = coord.shared_state expected_pct = (measured - base) / base * 100.0 assert ss.cumulative_gain_validated == pytest.approx(expected_pct) - assert ss.cumulative_gain_provenance == "geak_orch_harness_validated" assert ss.resume_pending_revalidation is False assert ss.geak_result.get("revalidation_status") != "no_material" diff --git a/src/hyperloom/inference_optimizer/tests/test_geak_resume_recovery.py b/src/hyperloom/inference_optimizer/tests/test_geak_resume_recovery.py index cd19da4faa..f4508e33b5 100644 --- a/src/hyperloom/inference_optimizer/tests/test_geak_resume_recovery.py +++ b/src/hyperloom/inference_optimizer/tests/test_geak_resume_recovery.py @@ -100,7 +100,7 @@ def _runner_should_not_be_needed(_name: str) -> Path: assert coord.shared_state.geak_pending["self_reported_tput"] == 116.0 # No premature headline: current_best / gain / stack are untouched. assert coord.shared_state.current_best["action"] == "baseline" - assert coord.shared_state.cumulative_gain == pytest.approx(0.0) + assert coord.shared_state.cumulative_gain_validated == pytest.approx(0.0) assert not any(e.get("action") == "geak_e2e" for e in coord.shared_state.optimization_stack) assert coord.shared_state.pending_escalate_hint == ESCALATE_HINT_SKIP_TO_SWEEP diff --git a/src/hyperloom/inference_optimizer/tests/test_grid_runner.py b/src/hyperloom/inference_optimizer/tests/test_grid_runner.py index 0df03a1044..38b699abc8 100644 --- a/src/hyperloom/inference_optimizer/tests/test_grid_runner.py +++ b/src/hyperloom/inference_optimizer/tests/test_grid_runner.py @@ -430,6 +430,28 @@ def test_used_by_backends_grid_override(self): } assert isinstance(v.fingerprint, str) and len(v.fingerprint) > 0 + def test_drops_hijacking_envs_but_keeps_workload_pins(self): + v = GridVariant( + name="mixed", + extra_envs={ + "SGLANG_USE_AITER": "1", + "CONC": "64", + "ISL": "1024", + "RUN_EVAL": "false", + "LD_PRELOAD": "/tmp/evil.so", + "PATH": "/tmp/bin", + "PYTHONPATH": "/tmp/evil", + "OPENAI_API_KEY": "must-not-reach-benchmark", + }, + ) + + assert v.extra_envs == { + "SGLANG_USE_AITER": "1", + "CONC": "64", + "ISL": "1024", + "RUN_EVAL": "false", + } + # Section 3: per-variant mtime gating + param overrides @@ -707,6 +729,31 @@ def test_build_variant_yaml_can_remove_base_args_and_unset_envs(tmp_path): assert "SGLANG_ENABLE_FOO" not in envs +def test_build_variant_yaml_refuses_to_unset_pinned_envs(tmp_path): + """A variant that unsets TP would silently shrink the Ray lease to one GPU.""" + base = tmp_path / "base.yaml" + _write_baseline_yaml_overrides(base) + cfg = yaml.safe_load(base.read_text()) + cfg["benchmark"]["envs"].update({"TP": 8, "CONC": 64, "SGLANG_ENABLE_FOO": "1"}) + base.write_text(yaml.safe_dump(cfg), encoding="utf-8") + + out = _build_variant_yaml( + base, + base_extra_args="", + variant=GridVariant( + "unset_the_world", + unset_envs=["TP", "CONC", "ROCR_VISIBLE_DEVICES", "SGLANG_ENABLE_FOO"], + ), + output_subdir=tmp_path / "unset_the_world", + ) + + envs = yaml.safe_load(out.read_text())["benchmark"]["envs"] + assert envs["TP"] == 8 + assert envs["CONC"] == 64 + # A plain tuning knob is still removable; only the pins are protected. + assert "SGLANG_ENABLE_FOO" not in envs + + def test_run_magpie_default_result_dir_is_output_dir(tmp_path, monkeypatch): monkeypatch.setenv("PYTEST_CURRENT_TEST", "skip-kill") captured: dict = {} @@ -942,7 +989,9 @@ def fake_run(cmd, *args, **kwargs): ) ctx = ReactorContext( tick_index=0, - shared_state=SharedStateSnapshot(session_id=session_dir.name), + # The session identity the rule matches on comes from LocalHealthConfig + # below; the snapshot no longer carries a second copy of it. + shared_state=SharedStateSnapshot(), inbox=[], now_unix=1.0, ) diff --git a/src/hyperloom/inference_optimizer/tests/test_kernel_fmoe_ck_gate.py b/src/hyperloom/inference_optimizer/tests/test_kernel_fmoe_ck_gate.py index 64106743e0..f9be4b645c 100644 --- a/src/hyperloom/inference_optimizer/tests/test_kernel_fmoe_ck_gate.py +++ b/src/hyperloom/inference_optimizer/tests/test_kernel_fmoe_ck_gate.py @@ -103,7 +103,7 @@ async def test_unsupported_shape_is_skipped_before_the_restart( model = _qwen3_moe_model(tmp_path / "Qwen3-30B-A3B") result = _fmoe_ck_result(tuned_csv) - await _phase(tmp_path, model, tp=8)._validate_forge_gemm_tuning_e2e(result) + await _phase(tmp_path, model, tp=8)._validate_gemm_tuning_e2e(result) reverted = result["e2e_results"]["reverted"] assert [r["reason"] for r in reverted] == [_SKIP_REASON] @@ -120,7 +120,7 @@ async def test_supported_shape_still_reaches_validation( model = _qwen3_moe_model(tmp_path / "Qwen3-30B-A3B") result = _fmoe_ck_result(tuned_csv) - await _phase(tmp_path, model, tp=2)._validate_forge_gemm_tuning_e2e(result) + await _phase(tmp_path, model, tp=2)._validate_gemm_tuning_e2e(result) assert [c["task_id"] for c in integrate_spy] == ["gemm_tune_e2e_fmoe_ck"] assert [k["tuner"] for k in result["e2e_results"]["kept"]] == ["fmoe_ck"] @@ -133,7 +133,7 @@ async def test_gate_only_applies_to_fmoe_ck(tmp_path: Path, integrate_spy, tuned result = _fmoe_ck_result(tuned_csv) result["tuners_run"][0]["tuner"] = "fmoe_asm" - await _phase(tmp_path, model, tp=8)._validate_forge_gemm_tuning_e2e(result) + await _phase(tmp_path, model, tp=8)._validate_gemm_tuning_e2e(result) assert [c["task_id"] for c in integrate_spy] == ["gemm_tune_e2e_fmoe_asm"] assert result["e2e_results"]["reverted"] == [] @@ -144,7 +144,7 @@ async def test_undecidable_model_is_not_skipped(tmp_path: Path, integrate_spy, t """No readable config: leave the call to sglang rather than skip on a guess.""" result = _fmoe_ck_result(tuned_csv) - await _phase(tmp_path, str(tmp_path / "absent"), tp=8)._validate_forge_gemm_tuning_e2e(result) + await _phase(tmp_path, str(tmp_path / "absent"), tp=8)._validate_gemm_tuning_e2e(result) assert [c["task_id"] for c in integrate_spy] == ["gemm_tune_e2e_fmoe_ck"] assert [r["reason"] for r in result["e2e_results"]["reverted"]] != [_SKIP_REASON] diff --git a/src/hyperloom/inference_optimizer/tests/test_kernel_idle_streak.py b/src/hyperloom/inference_optimizer/tests/test_kernel_idle_streak.py index c3a042bff9..afaff0529e 100644 --- a/src/hyperloom/inference_optimizer/tests/test_kernel_idle_streak.py +++ b/src/hyperloom/inference_optimizer/tests/test_kernel_idle_streak.py @@ -220,7 +220,6 @@ def test_fingerprint_ignores_fields_that_are_not_progress(): base = SimpleNamespace( kernel_opt_task_attempts={"k000": {"last_decision": "", "last_micro_speedup": 1.0}}, - kernel_opt_attempts={}, rejected_kernel_ids=[], last_kernel_opt={}, optimization_stack=[], @@ -242,7 +241,6 @@ def test_fingerprint_tracks_inflight_task_ids(): state = SimpleNamespace( kernel_opt_task_attempts={}, - kernel_opt_attempts={}, rejected_kernel_ids=[], last_kernel_opt={}, optimization_stack=[], diff --git a/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py b/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py index 1b59444328..5777f64091 100644 --- a/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py +++ b/src/hyperloom/inference_optimizer/tests/test_kernel_integrate_and_report.py @@ -1434,7 +1434,7 @@ def _fake_run(cmd, *args, **kwargs): assert result["decision"] == "KEEP" assert result["new_tput"] == 900.0 assert c.shared_state.current_best["action"] == "integrate" - assert c.shared_state.current_best["kernel_id"] == "k1" + assert c.shared_state.current_best["variant_name"] == "k1" assert any( item.get("action") == "integrate" and item.get("kernel_id") == "k1" for item in c.shared_state.optimization_stack @@ -1535,7 +1535,7 @@ async def test_report_executor_writes_md_and_json(session_dir): session_id=session_dir.name, model_name="Qwen-Qwen3-8B", model_path="/path/models/Qwen-Qwen3-8B", - cumulative_gain=12.5, + cumulative_gain_validated=12.5, current_best={ "action": "backends", "tput": 900.0, @@ -1600,7 +1600,7 @@ async def test_report_executor_writes_md_and_json(session_dir): summary = json.loads(js.read_text()) assert summary["session_id"] == session_dir.name assert summary["baseline_tput"] == 800.0 - assert summary["cumulative_gain"] == 12.5 + assert summary["cumulative_gain_validated"] == 12.5 assert summary["stop_reason"] == "target_reached" assert summary["event_counts_by_topic"].get("proposal", 0) >= 2 assert summary["event_counts_by_topic"].get("alert", 0) >= 1 diff --git a/src/hyperloom/inference_optimizer/tests/test_kernel_journey.py b/src/hyperloom/inference_optimizer/tests/test_kernel_journey.py index 61e9679027..6ea57a57bc 100644 --- a/src/hyperloom/inference_optimizer/tests/test_kernel_journey.py +++ b/src/hyperloom/inference_optimizer/tests/test_kernel_journey.py @@ -573,5 +573,5 @@ def test_build_phase_timeline_merges_journal_and_kernel_lanes( decisions = {(e.get("action"), e.get("decision")) for e in timeline} assert ("explore", "REVERT") in decisions assert ("integrate", "KEEP") in decisions - # action_timeline aliases phase_timeline. - assert out["action_timeline"] == timeline + # action_timeline alias was removed; phase_timeline is the canonical key. + assert out["phase_timeline"] == timeline diff --git a/src/hyperloom/inference_optimizer/tests/test_langfuse_emitter.py b/src/hyperloom/inference_optimizer/tests/test_langfuse_emitter.py index 94b9af0fc8..b643dcdbd1 100644 --- a/src/hyperloom/inference_optimizer/tests/test_langfuse_emitter.py +++ b/src/hyperloom/inference_optimizer/tests/test_langfuse_emitter.py @@ -1423,7 +1423,7 @@ def test_pair_key_degrades_when_turn_absent(): def _status(**over) -> dict: - base = {"phase": "EXPLORE", "stop_reason": "", "cumulative_gain": 12.5} + base = {"phase": "EXPLORE", "stop_reason": "", "cumulative_gain_validated": 12.5} base.update(over) return base diff --git a/src/hyperloom/inference_optimizer/tests/test_longrun_phase1.py b/src/hyperloom/inference_optimizer/tests/test_longrun_phase1.py index 46c5ccaa89..50a568e2bb 100644 --- a/src/hyperloom/inference_optimizer/tests/test_longrun_phase1.py +++ b/src/hyperloom/inference_optimizer/tests/test_longrun_phase1.py @@ -22,7 +22,7 @@ def _sweep_state( *, macro_cycle: int = 0, - cumulative_gain: float = 5.0, + validated_gain: float = 5.0, gain_at_cycle_start: float = 0.0, no_gain_streak: int = 0, max_minutes: int = 96 * 60, @@ -35,7 +35,7 @@ def _sweep_state( start_ts=(now - timedelta(hours=started_hours_ago)).isoformat(), max_minutes=max_minutes, macro_cycle=macro_cycle, - cumulative_gain_validated=cumulative_gain, + cumulative_gain_validated=validated_gain, gain_at_cycle_start=gain_at_cycle_start, no_gain_cycle_streak=no_gain_streak, ) @@ -46,8 +46,8 @@ def _sweep_state( # compute_next_phase SWEEP back-edge def test_sweep_reloops_to_explore_when_budget_and_leverage(): - st = _sweep_state(macro_cycle=0, cumulative_gain=5.0, gain_at_cycle_start=0.0) - nxt = ps.compute_next_phase(st, max_hours=96.0) + st = _sweep_state(macro_cycle=0, validated_gain=5.0, gain_at_cycle_start=0.0) + nxt = ps.compute_next_phase(st) assert nxt is not None target, reason, evidence = nxt assert target == ps.PHASE_EXPLORE @@ -57,10 +57,10 @@ def test_sweep_reloops_to_explore_when_budget_and_leverage(): def test_sweep_closes_on_failed_conc_sweep_even_when_reloop_available(): - st = _sweep_state(macro_cycle=0, cumulative_gain=5.0, gain_at_cycle_start=0.0) + st = _sweep_state(macro_cycle=0, validated_gain=5.0, gain_at_cycle_start=0.0) st.last_sweep = {} st.last_conc_sweep = {"status": "failed"} - target, reason, evidence = ps.compute_next_phase(st, max_hours=96.0) + target, reason, evidence = ps.compute_next_phase(st) assert target == ps.PHASE_CLOSE assert reason == "conc_sweep_failed" assert evidence.get("conc_sweep_status") == "failed" @@ -71,11 +71,11 @@ def test_sweep_closes_when_globally_converged(): # No gain this cycle + streak at 2 → effective 3 ≥ threshold. st = _sweep_state( macro_cycle=2, - cumulative_gain=5.0, + validated_gain=5.0, gain_at_cycle_start=5.0, no_gain_streak=2, ) - target, reason, evidence = ps.compute_next_phase(st, max_hours=96.0) + target, reason, evidence = ps.compute_next_phase(st) assert target == ps.PHASE_CLOSE assert reason == "global_converged" assert evidence["terminal"] is True @@ -85,7 +85,7 @@ def test_sweep_closes_when_globally_converged(): def test_sweep_closes_when_insufficient_remaining(): # Long run (48h) but only ~10min remain → below the reloop floor. st = _sweep_state(max_minutes=48 * 60, started_hours_ago=48 - 10 / 60.0) - target, reason, evidence = ps.compute_next_phase(st, max_hours=48.0) + target, reason, evidence = ps.compute_next_phase(st) assert target == ps.PHASE_CLOSE assert reason == "sweep_done" assert evidence["reloop_blocked"] == "insufficient_remaining" @@ -101,7 +101,7 @@ def test_sweep_skip_to_close_does_not_override_a_settled_conc_sweep(): "skip_reason": "session_time_budget", } st.set_pending_escalate_hint(ps.ESCALATE_HINT_SKIP_TO_CLOSE) - nxt = ps.compute_next_phase(st, max_hours=3.0) + nxt = ps.compute_next_phase(st) assert nxt is not None target, reason, evidence = nxt assert target == ps.PHASE_CLOSE @@ -115,7 +115,7 @@ def test_sweep_skip_to_close_still_escalates_when_conc_sweep_never_settled(): st.last_sweep = {} st.last_conc_sweep = {} st.set_pending_escalate_hint(ps.ESCALATE_HINT_SKIP_TO_CLOSE) - nxt = ps.compute_next_phase(st, max_hours=3.0) + nxt = ps.compute_next_phase(st) assert nxt is not None target, reason, _evidence = nxt assert target == ps.PHASE_CLOSE @@ -124,7 +124,7 @@ def test_sweep_skip_to_close_still_escalates_when_conc_sweep_never_settled(): def test_sweep_skip_to_close_yields_to_reloop_when_conc_sweep_was_skipped(): """A skipped conc_sweep with budget left must not be aborted by skip_to_close.""" - st = _sweep_state(macro_cycle=0, cumulative_gain=5.0, gain_at_cycle_start=0.0) + st = _sweep_state(macro_cycle=0, validated_gain=5.0, gain_at_cycle_start=0.0) st.last_sweep = {} st.last_conc_sweep = { "status": "skipped", @@ -132,7 +132,7 @@ def test_sweep_skip_to_close_yields_to_reloop_when_conc_sweep_was_skipped(): "skip_reason": "session_time_budget", } st.set_pending_escalate_hint(ps.ESCALATE_HINT_SKIP_TO_CLOSE) - nxt = ps.compute_next_phase(st, max_hours=96.0) + nxt = ps.compute_next_phase(st) assert nxt is not None target, reason, evidence = nxt assert target == ps.PHASE_EXPLORE @@ -146,7 +146,7 @@ def test_short_bounded_run_reloops_when_budget_and_leverage_remain(): st = _sweep_state( max_minutes=12 * 60, started_hours_ago=1.0, - cumulative_gain=5.0, + validated_gain=5.0, gain_at_cycle_start=0.0, ) reloop, ev = ps.should_reloop_to_explore(st) @@ -154,7 +154,7 @@ def test_short_bounded_run_reloops_when_budget_and_leverage_remain(): assert ev["reloop"] is True assert ev["next_cycle"] == 1 - target, reason, evidence = ps.compute_next_phase(st, max_hours=12.0) + target, reason, evidence = ps.compute_next_phase(st) assert target == ps.PHASE_EXPLORE assert reason == "cycle_reloop" assert evidence["loopback"] is True @@ -168,7 +168,7 @@ def test_short_bounded_run_closes_when_insufficient_remaining(): assert reloop is False assert ev["reloop_blocked"] == "insufficient_remaining" - target, reason, evidence = ps.compute_next_phase(st, max_hours=12.0) + target, reason, evidence = ps.compute_next_phase(st) assert target == ps.PHASE_CLOSE assert reason == "sweep_done" assert "loopback" not in evidence @@ -480,7 +480,7 @@ def test_policygate_allows_explore_action_after_loopback(tmp_path, monkeypatch): # Regression — short-run path now uses macro-loop while budget remains. def test_regression_short_run_sweep_evidence_carries_loopback(): st = _sweep_state(max_minutes=12 * 60) - target, reason, evidence = ps.compute_next_phase(st, max_hours=12.0) + target, reason, evidence = ps.compute_next_phase(st) assert (target, reason) == (ps.PHASE_EXPLORE, "cycle_reloop") assert evidence["loopback"] is True assert evidence["next_cycle"] == 1 diff --git a/src/hyperloom/inference_optimizer/tests/test_longrun_phase2.py b/src/hyperloom/inference_optimizer/tests/test_longrun_phase2.py index 7216badc2d..f2bec2356c 100644 --- a/src/hyperloom/inference_optimizer/tests/test_longrun_phase2.py +++ b/src/hyperloom/inference_optimizer/tests/test_longrun_phase2.py @@ -62,7 +62,7 @@ def test_explore_plateau_is_actionable(): def test_compute_next_phase_plateau_routes_explore_to_kernel(): st = _plateaued_explore_state() - target, reason, evidence = ps.compute_next_phase(st, max_hours=96.0) + target, reason, evidence = ps.compute_next_phase(st) # Exhausted explore leverage switches lever to KERNEL. assert target == ps.PHASE_KERNEL_AGENT assert reason == "explore_no_more_leverage" diff --git a/src/hyperloom/inference_optimizer/tests/test_multi_node_scripts.py b/src/hyperloom/inference_optimizer/tests/test_multi_node_scripts.py index a79e479b12..fed8bb31cb 100644 --- a/src/hyperloom/inference_optimizer/tests/test_multi_node_scripts.py +++ b/src/hyperloom/inference_optimizer/tests/test_multi_node_scripts.py @@ -114,7 +114,7 @@ def _kb_env(monkeypatch, tmp_path, **env): def test_resolve_kb_topology_prefers_env_over_state(monkeypatch, tmp_path): - """Env is exported before T0 and stable across --resume, so it outranks state.""" + """Env is exported before T0 and stable across a resume, so it outranks state.""" _kb_env(monkeypatch, tmp_path, TP="8", EP="4", INFERENCE_OPTIMIZER_NODES="2") monkeypatch.setattr(mne, "_read_state", lambda: {"nodes": 2, "tp": 2, "ep": 2}) @@ -931,29 +931,6 @@ def _mono(): assert lm._wait_health(timeout_s=5) is False -def test_infera_discover_role_pods_groups_prefill_decode(): - from hyperloom.inference_optimizer.multi_node._internal import infera_support - - wl = { - "pods": [ - {"podId": "x-frontend-a", "resourceId": 0, "podIP": "10.0.0.9"}, - {"podId": "x-prefillworker-1", "resourceId": 1, "podIP": "10.0.1.1"}, - {"podId": "x-prefillworker-0", "resourceId": 1, "podIP": "10.0.1.0"}, - {"podId": "x-decodeworker-0", "resourceId": 2, "podIP": "10.0.2.0"}, - ] - } - # Ports come off the default base, so assert against the constant: moving - # the default must not need this test edited. - base = infera_support.DEFAULT_SSH_PORT - stride = infera_support.ssh_role_port_offset("decode") - r = infera_support.discover_role_pods(wl) - assert [p["podIP"] for p in r["prefill"]] == ["10.0.1.0", "10.0.1.1"] - assert [p["sshPort"] for p in r["prefill"]] == [base, base + 1] - assert [p["podIP"] for p in r["decode"]] == ["10.0.2.0"] - assert r["decode"][0]["sshPort"] == base + stride - assert r["frontend"] and not r["worker"] - - def test_infera_ssh_port_role_stride_and_idle_entrypoint(): from hyperloom.inference_optimizer.multi_node._internal import infera_support from hyperloom.inference_optimizer.multi_node._internal.ssh_client import DEFAULT_SSH_PORT @@ -995,55 +972,6 @@ def test_infera_disagg_flags_and_launch_args(): # infera_support pure-helper tests (Infera backend SSH fan-out). -def test_infera_discover_worker_pods_excludes_frontend_sorts_by_ordinal(): - from hyperloom.inference_optimizer.multi_node._internal import infera_support - - wl = { - "pods": [ - {"podId": "dyn-frontend-abc", "resourceId": 0, "podIP": "10.0.0.9"}, - {"podId": "dyn-worker-1", "resourceId": 1, "podIP": "10.0.0.2"}, - {"podId": "dyn-worker-0", "resourceId": 1, "podIP": "10.0.0.1"}, - {"podId": "dyn-worker-pending", "resourceId": 1, "podIP": ""}, - ] - } - w = infera_support.discover_role_pods(wl, pd_mode="aggregated")["worker"] - assert [p["podIP"] for p in w] == ["10.0.0.1", "10.0.0.2"] - assert [p["lwsIndex"] for p in w] == [0, 1] - - -def test_infera_frontend_service_url_prefers_live_then_dns(): - from hyperloom.inference_optimizer.multi_node._internal import infera_support - - assert infera_support.frontend_service_url("wid", "ws") == "http://wid.ws.svc.cluster.local:8000" - assert ( - infera_support.frontend_service_url("wid", "ws", {"clusterIp": "10.1.2.3", "port": 8000}) - == "http://10.1.2.3:8000" - ) - - -def test_infera_frontend_service_url_internal_domain_and_nested_port(): - from hyperloom.inference_optimizer.multi_node._internal import infera_support - - # internalDomain wins. - assert ( - infera_support.frontend_service_url( - "w", - "ns", - {"internalDomain": "w.ns.svc.cluster.local:8000", "clusterIp": "1.2.3.4", "port": {"port": 8000}}, - ) - == "http://w.ns.svc.cluster.local:8000" - ) - # Nested port dict (SaFE shape) -> integer port, not the dict repr. - assert ( - infera_support.frontend_service_url( - "w", - "ns", - {"clusterIp": "192.168.154.0", "port": {"protocol": "TCP", "port": 8000, "targetPort": 8000}}, - ) - == "http://192.168.154.0:8000" - ) - - def test_infera_build_node_launch_args_sglang_and_kill_only(): from hyperloom.inference_optimizer.multi_node._internal import infera_support diff --git a/src/hyperloom/inference_optimizer/tests/test_no_legacy_writer_sites.py b/src/hyperloom/inference_optimizer/tests/test_no_legacy_writer_sites.py deleted file mode 100644 index fe97b89643..0000000000 --- a/src/hyperloom/inference_optimizer/tests/test_no_legacy_writer_sites.py +++ /dev/null @@ -1,106 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Static guard for the retired ``extra_sglang_args`` payload field.""" - -from __future__ import annotations - -import re -from pathlib import Path - -import pytest - - -_REPO_ROOT = Path(__file__).resolve().parents[4] - -ALLOWED_FILES: dict[str, str] = { - "src/hyperloom/agents/kernel/tools/_payload_aliases.py": ( - "standalone kernel-agent shim kept for bare subprocess imports" - ), - "src/hyperloom/agents/kernel/tests/test_payload_aliases_shim.py": ( - "tests pin the standalone kernel-agent shim contract" - ), - "src/hyperloom/inference_optimizer/tests/test_no_legacy_writer_sites.py": ( - "this guard names the retired field it scans for" - ), -} - -_LEGACY_PATTERN = re.compile(r"extra_sglang_args") - - -def _git_tracked_files() -> set[str] | None: - """Repo-relative POSIX paths tracked by git, or ``None`` when git is unavailable.""" - import subprocess - - try: - proc = subprocess.run( - ["git", "-C", str(_REPO_ROOT), "ls-files", "-z"], - capture_output=True, - text=True, - timeout=30, - check=False, - ) - except (OSError, ValueError): - return None - if proc.returncode != 0: - return None - return {p for p in (proc.stdout or "").split("\0") if p} - - -def _iter_repo_files() -> list[Path]: - """All non-binary, git-tracked repo files we want to scan.""" - tracked = _git_tracked_files() - out: list[Path] = [] - for path in _REPO_ROOT.rglob("*"): - if not path.is_file(): - continue - rel = path.relative_to(_REPO_ROOT).as_posix() - if tracked is not None and rel not in tracked: - continue - if any(rel.startswith(skip) for skip in (".git/", "node_modules/", "__pycache__/", "build/", ".venv/")): - continue - if path.suffix not in {".py", ".md", ".yaml", ".yml", ".toml", ".sh", ".txt", ".json", ".cfg", ".ini"}: - continue - out.append(path) - return out - - -def _files_with_legacy_key() -> set[str]: - """Repo-relative POSIX paths containing the retired key literal.""" - hits: set[str] = set() - for path in _iter_repo_files(): - try: - text = path.read_text(encoding="utf-8", errors="ignore") - except OSError: - continue - if _LEGACY_PATTERN.search(text): - hits.add(path.relative_to(_REPO_ROOT).as_posix()) - return hits - - -def test_no_legacy_writer_sites_outside_allowlist() -> None: - actual = _files_with_legacy_key() - unexpected = sorted(actual - set(ALLOWED_FILES)) - assert not unexpected, ( - "Files mentioning retired 'extra_sglang_args' outside the residual kernel-agent shim allowlist:\n " - + "\n ".join(unexpected) - ) - - -def test_allowlist_is_minimal() -> None: - actual = _files_with_legacy_key() - dead_entries = sorted(set(ALLOWED_FILES) - actual) - assert not dead_entries, "ALLOWED_FILES entries that no longer contain 'extra_sglang_args':\n " + "\n ".join( - dead_entries - ) - - -def test_allowlist_paths_resolve() -> None: - missing = [p for p in ALLOWED_FILES if not (_REPO_ROOT / p).exists()] - assert not missing, "ALLOWED_FILES entries pointing at non-existent paths:\n " + "\n ".join(missing) - - -@pytest.mark.parametrize("path,_reason", sorted(ALLOWED_FILES.items())) -def test_allowlist_entries_have_justification(path: str, _reason: str) -> None: - reason = ALLOWED_FILES[path] - assert reason and reason.strip(), f"ALLOWED_FILES[{path!r}] needs a justification" diff --git a/src/hyperloom/inference_optimizer/tests/test_objective.py b/src/hyperloom/inference_optimizer/tests/test_objective.py index 8b2815c453..9e045b6a41 100644 --- a/src/hyperloom/inference_optimizer/tests/test_objective.py +++ b/src/hyperloom/inference_optimizer/tests/test_objective.py @@ -56,29 +56,29 @@ def _backends_silent() -> dict[str, object]: # TargetGainObjective def test_target_gain_basic_progress(): obj = TargetGainObjective(target_gain_pct=10.0) - s = SharedState(baseline_tput=1000.0, cumulative_gain=0.0) + s = SharedState(baseline_tput=1000.0, cumulative_gain_validated=0.0) assert obj.kind() == "gain_pct" assert obj.progress(s) == 0.0 assert not obj.reached(s) - s.cumulative_gain = 5.0 + s.cumulative_gain_validated = 5.0 assert obj.progress(s) == 0.5 assert not obj.reached(s) - s.cumulative_gain = 12.0 + s.cumulative_gain_validated = 12.0 assert obj.progress(s) == 1.0 assert obj.reached(s) def test_target_gain_gap_pct_counts_down_to_zero(): obj = TargetGainObjective(target_gain_pct=15.0) - s = SharedState(baseline_tput=1000.0, cumulative_gain=0.0) + s = SharedState(baseline_tput=1000.0, cumulative_gain_validated=0.0) assert obj.gap_pct(s) == pytest.approx(15.0) - s.cumulative_gain = 9.89 + s.cumulative_gain_validated = 9.89 assert obj.gap_pct(s) == pytest.approx(5.11) - s.cumulative_gain = 20.0 + s.cumulative_gain_validated = 20.0 assert obj.gap_pct(s) == 0.0 @@ -162,7 +162,7 @@ def test_target_baseline_missing_report_rejected(tmp_path): # TimeOnlyObjective def test_time_only_never_reached(): obj = TimeOnlyObjective() - s = SharedState(baseline_tput=999.0, current_best={"tput": 9999.0}, cumulative_gain=99.0) + s = SharedState(baseline_tput=999.0, current_best={"tput": 9999.0}, cumulative_gain_validated=99.0) assert obj.kind() == "time_only" assert obj.progress(s) == 0.0 assert not obj.reached(s) @@ -236,7 +236,7 @@ async def test_run_stops_on_objective_reached(session_dir): c = Coordinator(session_dir, backends=_backends_silent()) try: c.shared_state.baseline_tput = 1000.0 - c.shared_state.cumulative_gain = 50.0 + c.shared_state.cumulative_gain_validated = 50.0 c.shared_state.save(session_dir) reason = await c.run( objective=TargetGainObjective(target_gain_pct=10.0), @@ -247,6 +247,31 @@ async def test_run_stops_on_objective_reached(session_dir): await c.stop() +@pytest.mark.asyncio +async def test_run_does_not_stop_on_a_gain_no_rebench_confirmed(session_dir): + """A current_best ahead of the last stack rebench is not evidence of the target.""" + c = Coordinator(session_dir, backends=_backends_silent()) + try: + c.shared_state.baseline_tput = 1000.0 + c.shared_state.current_best = {"action": "explore", "tput": 1500.0} + c.shared_state.cumulative_gain_validated = 2.0 + c.shared_state.save(session_dir) + reason = await c.run( + objective=TargetGainObjective(target_gain_pct=10.0), + max_ticks=3, + ) + assert reason == "max_ticks" + finally: + await c.stop() + + +def test_target_tput_reads_the_shared_grading_anchor(): + """A current_best carrying only ``output_throughput`` still counts as measured.""" + obj = TargetTputObjective(target_tput_per_gpu=900.0) + s = SharedState(baseline_tput=750.0, current_best={"output_throughput": 950.0}) + assert obj.reached(s) + + @pytest.mark.asyncio async def test_run_stops_on_time_exhausted(session_dir): c = Coordinator(session_dir, backends=_backends_silent()) diff --git a/src/hyperloom/inference_optimizer/tests/test_optimization_journal.py b/src/hyperloom/inference_optimizer/tests/test_optimization_journal.py index c301f2cf4d..af5253fd00 100644 --- a/src/hyperloom/inference_optimizer/tests/test_optimization_journal.py +++ b/src/hyperloom/inference_optimizer/tests/test_optimization_journal.py @@ -382,6 +382,18 @@ def test_derive_journal_outcome_integrate_patch_kept_is_keep(): assert out == OUTCOME_KEEP +def test_derive_journal_outcome_refused_promotion_is_no_promote(): + """A KEEP the anchor gate declined to lift adopted nothing, so it is not a KEEP.""" + from hyperloom.orchestrator.state.optimization_journal import PROMOTION_REFUSED_KEY + + out = derive_journal_outcome( + "integrate_patch", + {"status": "kept", "delta_pct": 7.5, PROMOTION_REFUSED_KEY: True}, + promotable=True, + ) + assert out == OUTCOME_NO_PROMOTE + + def test_derive_journal_outcome_accuracy_unavailable_reject_is_revert(): out = derive_journal_outcome( "integrate_patch", diff --git a/src/hyperloom/inference_optimizer/tests/test_phase_force_exit.py b/src/hyperloom/inference_optimizer/tests/test_phase_force_exit.py index 1b9048b435..d76bb3870c 100644 --- a/src/hyperloom/inference_optimizer/tests/test_phase_force_exit.py +++ b/src/hyperloom/inference_optimizer/tests/test_phase_force_exit.py @@ -4,15 +4,10 @@ """IR-6 — EXPLORE HARD force-exit gate tests. Covers ``phase_state.should_force_exit_explore`` and its integration via -``exit_normal_explore`` / ``compute_next_phase``. The gate fires when: - -* total session wall-clock remaining drops below - ``force_exit_hours_remaining`` hours, OR -* EXPLORE phase remaining budget pct drops below - ``force_exit_budget_pct``. - -Either gate alone is sufficient. Both gates feed evidence into the -phase_history audit row regardless of which (or both) fired. +``exit_normal_explore`` / ``compute_next_phase``. The gate fires when the +unspent fraction of EXPLORE's own charge-back budget drops to +``force_exit_budget_pct`` or below; there is no session-wall-clock arm, which +``test_force_exit_is_blind_to_session_length`` guards. """ from __future__ import annotations @@ -66,42 +61,28 @@ def _make_explore_state( return state -def test_force_exit_total_remaining_below_threshold(): - """7.5h elapsed of a 10h budget -> 2.5h remaining < 3h threshold.""" +def test_force_exit_phase_pct_below_threshold(): + """A phase that has spent nearly its whole slice force-exits.""" state = _make_explore_state( max_minutes=600, - started_hours_ago=7.5, - phase_started_hours_ago=4.0, - ) - fired, evidence = phase_state.should_force_exit_explore( - state, - hours_remaining_threshold=3.0, - budget_pct_threshold=0.20, + started_hours_ago=6.0, + phase_started_hours_ago=5.7, ) + fired, evidence = phase_state.should_force_exit_explore(state, budget_pct_threshold=0.20) assert fired is True - assert "session_remaining" in evidence["fired_reasons"] - assert evidence["session_remaining_seconds"] < 3 * 3600 + 60 - assert evidence["hours_remaining_threshold"] == 3.0 + assert evidence["phase_remaining_pct"] <= 0.20 -def test_force_exit_phase_pct_below_threshold(): - """Phase elapsed close to its slice; session_remaining still OK.""" - # EXPLORE slice nearly exhausted (~5% left <= 20%) while session - # remaining (4h) stays above the 3h threshold. +def test_force_exit_late_in_session_still_covered_by_pct_arm(): + """An EXPLORE that overran its slice still force-exits 7.5h into a 10h run.""" state = _make_explore_state( max_minutes=600, - started_hours_ago=6.0, - phase_started_hours_ago=5.7, - ) - fired, evidence = phase_state.should_force_exit_explore( - state, - hours_remaining_threshold=3.0, - budget_pct_threshold=0.20, + started_hours_ago=7.5, + phase_started_hours_ago=4.0, ) + fired, evidence = phase_state.should_force_exit_explore(state) assert fired is True - assert "phase_remaining_pct" in evidence["fired_reasons"] - assert evidence["phase_remaining_pct"] <= 0.20 - assert evidence["session_remaining_seconds"] > 3 * 3600 + assert evidence["phase_remaining_pct"] == 0.0 def test_force_exit_neither_trigger_fires(): @@ -111,39 +92,27 @@ def test_force_exit_neither_trigger_fires(): started_hours_ago=1.0, phase_started_hours_ago=0.5, ) - fired, evidence = phase_state.should_force_exit_explore( - state, - hours_remaining_threshold=3.0, - budget_pct_threshold=0.20, - ) + fired, evidence = phase_state.should_force_exit_explore(state, budget_pct_threshold=0.20) assert fired is False - assert evidence["fired_reasons"] == [] - # Evidence still populated for diagnostics. - assert evidence["session_remaining_seconds"] > 3 * 3600 assert evidence["phase_remaining_pct"] > 0.20 -def test_force_exit_both_triggers_fire(): - """Both gates trigger; evidence lists both reasons.""" +@pytest.mark.parametrize("max_minutes", [60, 120, 180, 600]) +def test_force_exit_is_blind_to_session_length(max_minutes): + """A freshly entered EXPLORE never force-exits, however short the run.""" state = _make_explore_state( - max_minutes=600, - started_hours_ago=7.6, - phase_started_hours_ago=5.7, - ) - fired, evidence = phase_state.should_force_exit_explore( - state, - hours_remaining_threshold=3.0, - budget_pct_threshold=0.20, + max_minutes=max_minutes, + started_hours_ago=0.1, + phase_started_hours_ago=0.0, ) - assert fired is True - assert set(evidence["fired_reasons"]) >= { - "session_remaining", - "phase_remaining_pct", - } + fired, evidence = phase_state.should_force_exit_explore(state) + assert fired is False + assert evidence["phase_remaining_pct"] == pytest.approx(1.0) + assert phase_state.exit_normal_explore(state) is None def test_force_exit_unlimited_run_never_fires(): - """max_minutes=0 -> unlimited; gate cannot fire on session_remaining.""" + """max_minutes=0 -> unlimited; the phase keeps its flat per-window slice.""" state = _make_explore_state( max_minutes=0, started_hours_ago=100.0, @@ -151,64 +120,38 @@ def test_force_exit_unlimited_run_never_fires(): ) fired, evidence = phase_state.should_force_exit_explore(state) assert fired is False - # Without max_minutes nothing is computable. assert "session_remaining_seconds" not in evidence - assert "hours_remaining_gate" not in evidence -def test_force_exit_hours_leavebehind_cannot_cover_the_session(): - """IR-6's 3h default on a 3h session must not skip EXPLORE at first tick. +@pytest.mark.parametrize("started_hours_ago", [0.0, 1.12]) +def test_force_exit_leaves_a_short_session_its_explore_phase(started_hours_ago): + """A 3h run must reach EXPLORE with grids left, at entry and after PRELUDE. - Remaining starts at max_hours, so remaining <= 3h is true as soon as any - time has been spent. CI's 3h smoke (and the 3h example) would otherwise - leave EXPLORE with 0 grids. + The phase is graded on its own charge-back budget, which is rebuilt from + the time left when it starts, so entering late shrinks the allotment + instead of spending it. """ state = _make_explore_state( max_minutes=180, - started_hours_ago=0.0, - phase_started_hours_ago=0.0, - ) - fired, evidence = phase_state.should_force_exit_explore( - state, - hours_remaining_threshold=3.0, - budget_pct_threshold=0.20, - ) - assert fired is False - assert "session_remaining" not in evidence["fired_reasons"] - assert evidence["hours_remaining_gate"] == "disabled_leavebehind_covers_session" - - -def test_force_exit_hours_leavebehind_disabled_after_prelude_on_three_hour_session(): - """CI shape: ~67 min spent before EXPLORE on a 3h budget.""" - state = _make_explore_state( - max_minutes=180, - started_hours_ago=1.12, + started_hours_ago=started_hours_ago, phase_started_hours_ago=0.0, ) - fired, evidence = phase_state.should_force_exit_explore( - state, - hours_remaining_threshold=3.0, - budget_pct_threshold=0.20, - ) + fired, evidence = phase_state.should_force_exit_explore(state) assert fired is False - assert "session_remaining" not in evidence["fired_reasons"] - assert evidence["hours_remaining_gate"] == "disabled_leavebehind_covers_session" + assert evidence["fired_reasons"] == [] + assert evidence["phase_remaining_pct"] == pytest.approx(1.0) -def test_force_exit_explicit_hours_leavebehind_still_fires_on_short_session(): - """An operator-set leave-behind smaller than the session still fires.""" +def test_force_exit_fires_once_the_phase_has_spent_its_own_budget(): + """The unspent fraction is the only thing that fires the hard gate.""" state = _make_explore_state( - max_minutes=180, - started_hours_ago=2.96, - phase_started_hours_ago=0.01, - ) - fired, evidence = phase_state.should_force_exit_explore( - state, - hours_remaining_threshold=0.05, - budget_pct_threshold=0.0, + max_minutes=600, + started_hours_ago=7.6, + phase_started_hours_ago=4.0, ) + fired, evidence = phase_state.should_force_exit_explore(state) assert fired is True - assert "session_remaining" in evidence["fired_reasons"] + assert evidence["fired_reasons"] == ["phase_remaining_pct"] def test_exit_normal_explore_force_exit_takes_priority_over_plateau(): @@ -270,11 +213,8 @@ def test_force_exit_thresholds_routed_through_overrides(): started_hours_ago=2.0, phase_started_hours_ago=1.0, ) - state.plateau_overrides = { - "force_exit_hours_remaining": 9.0, - "force_exit_budget_pct": 0.95, - } - # With absurd thresholds, any in-progress session triggers. + state.plateau_overrides = {"force_exit_budget_pct": 0.95} + # With an absurd threshold, any in-progress phase triggers. nxt = phase_state.compute_next_phase(state, kernel_enabled=True) assert nxt is not None target, reason, _ = nxt diff --git a/src/hyperloom/inference_optimizer/tests/test_phase_state_framework_agent.py b/src/hyperloom/inference_optimizer/tests/test_phase_state_framework_agent.py index d430f4ba88..f3401a01c5 100644 --- a/src/hyperloom/inference_optimizer/tests/test_phase_state_framework_agent.py +++ b/src/hyperloom/inference_optimizer/tests/test_phase_state_framework_agent.py @@ -55,12 +55,18 @@ def test_framework_exit_reasons_registered(): reasons = { "framework_agent_phase_done", "framework_agent_plateau", - "framework_agent_force_exit_low_budget", } assert reasons <= phase_state.PHASE_EXIT_REASONS assert reasons <= phase_state.STOP_REASON_VOCAB +def test_framework_agent_force_exit_low_budget_is_retired_vocab(): + """``framework_agent_budget_cap`` replaced the session-remaining force-exit.""" + assert "framework_agent_force_exit_low_budget" not in phase_state.PHASE_EXIT_REASONS + assert "framework_agent_force_exit_low_budget" not in phase_state.STOP_REASON_VOCAB + assert "framework_agent_budget_cap" in phase_state.PHASE_EXIT_REASONS + + def test_framework_agent_skipped_is_not_a_registered_reason(): """``framework_agent_skipped`` is dead vocab — never emitted, must not be registered.""" assert "framework_agent_skipped" not in phase_state.PHASE_EXIT_REASONS @@ -99,33 +105,16 @@ def test_exit_normal_framework_agent_returns_none_when_nothing_to_do(): assert phase_state.exit_normal_framework_agent(state) is None -def test_exit_normal_framework_agent_force_exit_when_remaining_below_ratio(): - # remaining 30min < 72min force-exit floor → fires. - state = _State(remaining_minutes_value=30.0) - out = phase_state.exit_normal_framework_agent(state, max_hours=2.0) - assert out is not None - reason, ev = out - assert reason == "framework_agent_force_exit_low_budget" - assert ev["evidence"] == "force_exit" - assert ev["remaining_minutes"] == 30.0 - - -def test_exit_normal_framework_agent_no_force_exit_when_remaining_above_ratio(): - # remaining 80min > 72min force-exit floor → no force exit. - state = _State(remaining_minutes_value=80.0) - assert phase_state.exit_normal_framework_agent(state, max_hours=2.0) is None +def test_exit_normal_framework_agent_ignores_session_remaining(): + """Low session-remaining alone must not evict a phase that has done nothing. - -def test_exit_normal_framework_agent_accepts_positional_remaining_minutes(): - class PositionalRemainingState(_State): - def remaining_minutes(self) -> float: # type: ignore[override] - return 30.0 - - out = phase_state.exit_normal_framework_agent(PositionalRemainingState(), max_hours=2.0) - - assert out is not None - assert out[0] == "framework_agent_force_exit_low_budget" - assert out[1]["remaining_minutes"] == 30.0 + FRAMEWORK now leaves only on its own charge-back cap, a plateau, or an + exhausted candidate list — never on the session clock. + """ + assert phase_state.exit_normal_framework_agent(_State(remaining_minutes_value=30.0)) is None + below_streak = [{"candidate_id": f"c{i}", "status": "reverted", "kept": False} for i in range(3)] + state = _State(framework_agent_phase_progress=below_streak, remaining_minutes_value=10.0) + assert phase_state.exit_normal_framework_agent(state) is None def test_exit_normal_framework_agent_exits_on_consecutive_reject_plateau(): @@ -232,27 +221,25 @@ def test_framework_batch_plateau_counts_current_cycle_audit_skips(): assert triggered is True -def test_exit_normal_framework_agent_force_exit_evidence_carries_pending_count(): - """Regression: force-exit evidence surfaces ``pending_candidate_count``.""" +def test_exit_normal_framework_agent_exit_evidence_carries_pending_count(): + """Regression: an early exit surfaces how many candidates it left behind.""" + n = phase_state.DEFAULT_FRAMEWORK_PLATEAU_NO_KEEP_STREAK batches = [ { "batch_id": "b1", "max_gain_pct_observed_in_batch": 0.0, - "candidates": [{"id": "c1a"}, {"id": "c1b"}, {"id": "c1c"}], + "candidates": [{"id": f"c{i}"} for i in range(n + 2)], }, ] - progress = [ - {"batch_id": "b1", "candidate_id": "c1a", "status": "reject"}, - ] + progress = [{"batch_id": "b1", "candidate_id": f"c{i}", "status": "reverted"} for i in range(n)] state = _State( framework_agent_batches=batches, framework_agent_phase_progress=progress, - remaining_minutes_value=10.0, ) - out = phase_state.exit_normal_framework_agent(state, max_hours=2.0) + out = phase_state.exit_normal_framework_agent(state) assert out is not None reason, ev = out - assert reason == "framework_agent_force_exit_low_budget" + assert reason == "framework_agent_plateau" assert ev["pending_candidate_count"] == 2 @@ -377,22 +364,6 @@ def test_exit_normal_framework_agent_plateau_mixed_terminal_no_keep_rows(): assert out[1]["consecutive_no_keep"] == phase_state.DEFAULT_FRAMEWORK_PLATEAU_NO_KEEP_STREAK -def test_exit_normal_framework_agent_force_exit_beats_plateau(): - """Priority order: force-exit > plateau.""" - progress = [ - {"candidate_id": "c1", "status": "reverted", "kept": False}, - {"candidate_id": "c2", "status": "reverted", "kept": False}, - {"candidate_id": "c3", "status": "reverted", "kept": False}, - ] - state = _State( - framework_agent_phase_progress=progress, - remaining_minutes_value=10.0, - ) - out = phase_state.exit_normal_framework_agent(state, max_hours=2.0) - assert out is not None - assert out[0] == "framework_agent_force_exit_low_budget" - - def test_exit_normal_framework_agent_plateau_beats_phase_done(): """Priority order: plateau > phase_done.""" progress = [ @@ -442,8 +413,8 @@ def test_framework_agent_plateau_streak_threshold_is_default(): ) -def test_exit_normal_framework_agent_force_exit_beats_phase_done(): - """Priority order: force-exit > phase_done.""" +def test_exit_normal_framework_agent_phase_done_wins_when_low_on_session_time(): + """A nearly spent session no longer masks the honest ``phase_done`` reason.""" batches = [ {"max_gain_pct_observed_in_batch": 0.1}, {"max_gain_pct_observed_in_batch": 0.1}, @@ -454,9 +425,9 @@ def test_exit_normal_framework_agent_force_exit_beats_phase_done(): framework_agent_phase_done=True, remaining_minutes_value=10.0, ) - out = phase_state.exit_normal_framework_agent(state, max_hours=2.0) + out = phase_state.exit_normal_framework_agent(state) assert out is not None - assert out[0] == "framework_agent_force_exit_low_budget" + assert out[0] == "framework_agent_phase_done" def test_compute_next_phase_prelude_to_framework_when_enabled(): @@ -491,21 +462,17 @@ def test_compute_next_phase_framework_does_not_advance_on_plateau(): assert phase_state.compute_next_phase(state, framework_agent_phase_enabled=True) is None -def test_compute_next_phase_framework_agent_force_exit_passes_max_hours_through(): +def test_compute_next_phase_framework_holds_on_a_nearly_spent_session(): + """A reloop into FRAMEWORK late in the run must not bounce straight out. + + FRAMEWORK is the preferred macro-cycle reloop target, and every reloop + happens once most of the session is gone. + """ state = _State( phase=phase_state.PHASE_FRAMEWORK_AGENT, remaining_minutes_value=30.0, ) - out = phase_state.compute_next_phase( - state, - framework_agent_phase_enabled=True, - max_hours=2.0, - ) - assert out is not None - next_phase, reason, ev = out - assert next_phase == phase_state.PHASE_EXPLORE - assert reason == "framework_agent_force_exit_low_budget" - assert ev["max_hours"] == 2.0 + assert phase_state.compute_next_phase(state, framework_agent_phase_enabled=True) is None def test_compute_next_phase_framework_stays_when_no_signal(): @@ -513,7 +480,6 @@ def test_compute_next_phase_framework_stays_when_no_signal(): out = phase_state.compute_next_phase( state, framework_agent_phase_enabled=True, - max_hours=10.0, ) assert out is None diff --git a/src/hyperloom/inference_optimizer/tests/test_phase_state_machine.py b/src/hyperloom/inference_optimizer/tests/test_phase_state_machine.py index 364976e7e6..c632041836 100644 --- a/src/hyperloom/inference_optimizer/tests/test_phase_state_machine.py +++ b/src/hyperloom/inference_optimizer/tests/test_phase_state_machine.py @@ -133,7 +133,7 @@ def test_phase_exit_reasons_includes_required_vocab(): "kernel_no_more_leverage", "framework_agent_phase_done", "framework_agent_plateau", - "framework_agent_force_exit_low_budget", + "framework_agent_budget_cap", "cycle_reloop", "global_converged", "robustness_escalated", @@ -450,7 +450,6 @@ def test_exit_normal_explore_uses_budget_exhaustion(): ) out = phase_state.exit_normal_explore( state, - force_exit_hours_remaining=0.0, force_exit_budget_pct=0.0, ) assert out is not None and out[0] == "explore_phase_budget_exhausted" diff --git a/src/hyperloom/inference_optimizer/tests/test_phase_state_plateau.py b/src/hyperloom/inference_optimizer/tests/test_phase_state_plateau.py index 70114b558b..d432ac1b20 100644 --- a/src/hyperloom/inference_optimizer/tests/test_phase_state_plateau.py +++ b/src/hyperloom/inference_optimizer/tests/test_phase_state_plateau.py @@ -350,7 +350,7 @@ def test_exit_normal_kernel_after_gemm_does_not_exit(): max_minutes=0, phase_budget_pct={}, kernel_integrate_attempts={}, - kernel_opt_attempts={}, + kernel_opt_task_attempts={}, continue_kernel_after_gemm=False, rejected_kernel_ids=[], last_gemm_tuning={ @@ -444,7 +444,7 @@ def test_kernel_skip_to_sweep_waits_for_pending_keep(): def test_kernel_skip_to_sweep_waits_for_partial_kernel_attempt(): state = _skip_to_sweep_state("KERNEL_AGENT") - state.kernel_opt_attempts = { + state.kernel_opt_task_attempts = { "k009": { "last_decision": "PARTIAL", "last_status": "ok", @@ -517,7 +517,7 @@ def test_collective_only_waits_for_pending_integration(): def test_kernel_skip_to_sweep_waits_for_retryable_failed_kernel(): state = _skip_to_sweep_state("KERNEL_AGENT") - state.kernel_opt_attempts = { + state.kernel_opt_task_attempts = { "k018": { "attempts": 1, "failure_count": 1, @@ -536,7 +536,7 @@ def test_kernel_skip_to_sweep_ignores_rejected_or_integrated_attempts(): state = _skip_to_sweep_state("KERNEL_AGENT") state.rejected_kernel_ids = ["k001"] state.optimization_stack = [{"action": "integrate", "kernel_id": "k002"}] - state.kernel_opt_attempts = { + state.kernel_opt_task_attempts = { "k001": { "last_decision": "REVERT", "last_status": "ok", diff --git a/src/hyperloom/inference_optimizer/tests/test_preflight_auth_override.py b/src/hyperloom/inference_optimizer/tests/test_preflight_auth_override.py index da2942f014..0b702a28be 100644 --- a/src/hyperloom/inference_optimizer/tests/test_preflight_auth_override.py +++ b/src/hyperloom/inference_optimizer/tests/test_preflight_auth_override.py @@ -1014,6 +1014,14 @@ def test_resolve_robustness_choice_explicit_mock_wins(): assert cli._resolve_robustness_choice(args) == "mock" +def test_resolve_robustness_choice_keeps_the_agent_on_multi_node(): + """Multi-node runs the agent on its node-agnostic signals; the local probe + is what gets disabled, not the whole backend.""" + args = _make_args(robustness_backend=None, nodes=4) + + assert cli._resolve_robustness_choice(args) == "agent" + + def test_resolve_robustness_choice_env_override_still_works(monkeypatch): monkeypatch.setenv("INFERENCE_OPTIMIZER_DEFAULT_ROBUSTNESS_BACKEND", "mock") reloaded_cli = importlib.reload(cli) diff --git a/src/hyperloom/inference_optimizer/tests/test_preflight_lm_eval_dep.py b/src/hyperloom/inference_optimizer/tests/test_preflight_lm_eval_dep.py index 295cd5087b..5cca74f261 100644 --- a/src/hyperloom/inference_optimizer/tests/test_preflight_lm_eval_dep.py +++ b/src/hyperloom/inference_optimizer/tests/test_preflight_lm_eval_dep.py @@ -256,7 +256,7 @@ def _boom(*_a, **_k): # --- _resolved_eval_disabled: preflight runs before the resume block -------- def _args(**kw): - return SimpleNamespace(**{"no_eval": False, "resume": False, "resume_from": "", **kw}) + return SimpleNamespace(**{"no_eval": False, "resume_from": "", **kw}) def test_resolved_eval_disabled_reads_the_flag(): @@ -266,11 +266,22 @@ def test_resolved_eval_disabled_reads_the_flag(): def test_resolved_eval_disabled_reads_the_resumed_session(tmp_path): (tmp_path / "state.json").write_text('{"eval_disabled": true}', encoding="utf-8") - assert preflight._resolved_eval_disabled(_args(resume=True, resume_from=str(tmp_path))) is True + assert preflight._resolved_eval_disabled(_args(resume_from=str(tmp_path))) is True def test_resolved_eval_disabled_without_a_readable_state(tmp_path): - assert preflight._resolved_eval_disabled(_args(resume=True, resume_from=str(tmp_path))) is False + assert preflight._resolved_eval_disabled(_args(resume_from=str(tmp_path))) is False + + +def test_resolved_eval_disabled_reads_nothing_without_a_named_session(tmp_path, monkeypatch): + """No session dir named means no session state is consulted, whatever sits in the workspace.""" + from hyperloom.inference_optimizer.session import paths + + monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) + foreign = tmp_path / "SomeOtherModel" / "29990101T000000Z" + foreign.mkdir(parents=True) + (foreign / "state.json").write_text('{"eval_disabled": true}', encoding="utf-8") + assert preflight._resolved_eval_disabled(_args()) is False def test_unprobeable_interpreter_is_left_untouched(monkeypatch, capsys): diff --git a/src/hyperloom/inference_optimizer/tests/test_prelude_roofline.py b/src/hyperloom/inference_optimizer/tests/test_prelude_roofline.py index 6120bf5cd8..25185619e7 100644 --- a/src/hyperloom/inference_optimizer/tests/test_prelude_roofline.py +++ b/src/hyperloom/inference_optimizer/tests/test_prelude_roofline.py @@ -217,22 +217,37 @@ async def test_profile_kind_keeps_the_plain_key(coord: Coordinator): assert task.idempotency_key == "internal-analysis-prelude_initial" +@pytest.mark.parametrize( + "named_state,reopens", + [ + ("succeeded", True), + ("cancelled", True), + # A watchdog-reclaimed roofline reports no result, so the gate release + # is the only thing that can ever clear the marker it left. + ("failed", True), + ("running", False), + ("queued", False), + ], +) @pytest.mark.asyncio -async def test_watermark_gate_reopens_after_the_roofline_it_names_finished( +async def test_watermark_gate_reopens_exactly_when_the_roofline_it_names_finished( coord: Coordinator, + named_state: str, + reopens: bool, ): - """A marker left on a finished task gated the watermark forever, and it is - persisted state, so resuming the session inherited the wedge.""" + """The gate exists so two rooflines never run at once, so it must hold for + every live state and release for every finished one. It is persisted state: + a marker left on a finished task is a wedge the next resume inherits.""" state = coord.shared_state state.baseline_tput = 100.0 state.cumulative_gain_validated = 50.0 state.last_roofline_tput = 0.0 - stale = await coord._enqueue_internal_analysis_task( + named = await coord._enqueue_internal_analysis_task( reason="integrate_keep_watermark", ) - coord.tasks._tasks[stale.task_id].state = "succeeded" - state.auto_roofline_pending_task_id = stale.task_id + coord.tasks._tasks[named.task_id].state = named_state + state.auto_roofline_pending_task_id = named.task_id state.roofline_failure_streak = 1 # its trace analysis failed assert coord._needs_roofline_for_watermark() is False @@ -240,34 +255,8 @@ async def test_watermark_gate_reopens_after_the_roofline_it_names_finished( reason="integrate_keep_watermark", ) - assert enqueued is True - assert state.auto_roofline_pending_task_id != stale.task_id - - -@pytest.mark.asyncio -async def test_watermark_gate_holds_while_a_roofline_is_genuinely_running( - coord: Coordinator, -): - """The gate exists so two rooflines never run at once; releasing it must - depend on the task being finished, not merely on being asked.""" - state = coord.shared_state - state.baseline_tput = 100.0 - state.cumulative_gain_validated = 50.0 - state.last_roofline_tput = 0.0 - state.roofline_failure_streak = 1 - - live = await coord._enqueue_internal_analysis_task( - reason="integrate_keep_watermark", - ) - coord.tasks._tasks[live.task_id].state = "running" - state.auto_roofline_pending_task_id = live.task_id - - enqueued = await coord._maybe_enqueue_watermark_roofline( - reason="integrate_keep_watermark", - ) - - assert enqueued is False - assert state.auto_roofline_pending_task_id == live.task_id + assert enqueued is reopens + assert (state.auto_roofline_pending_task_id != named.task_id) is reopens def test_watermark_stops_re_arming_once_retries_are_spent(coord: Coordinator): diff --git a/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py b/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py index 2ba535a310..def3f61150 100644 --- a/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py +++ b/src/hyperloom/inference_optimizer/tests/test_profile_and_kernel_handlers.py @@ -511,7 +511,7 @@ def test_materialize_profile_bounds_survive_a_replacing_candidate( assert "--profiler-config.max_iterations 128" in extra, extra # The frontend profiler tracks no iterations, so it has to come back too. assert "--profiler-config.ignore_frontend True" in extra, extra - assert "--profiler-config.capture_torch_profiler True" in extra, extra + assert "--profiler-config.detailed_trace_annotation True" in extra, extra # The candidate's own flags must still take effect, JSON value unmangled. assert "--no-enable-prefix-caching" in extra, extra assert '--compilation-config {"cudagraph_capture_sizes":[17,34,1088]}' in extra, extra @@ -612,11 +612,11 @@ def test_materialize_profile_cap_wins_over_a_max_iterations_pinned_in_the_yaml( assert extra.rindex("max_iterations 128") > extra.rindex("max_iterations 100000"), extra -def test_materialize_profile_capture_flag_wins_over_a_stale_yaml_value( +def test_materialize_profile_annotation_flag_wins_over_a_stale_yaml_value( tmp_path, monkeypatch, ): - """A YAML that disables the capture would leave the run with no sidecar traces, + """A YAML that disables the annotation would leave the trace unlabelled, so the injected value has to land after it and win the last-wins resolution.""" import yaml @@ -629,14 +629,14 @@ def test_materialize_profile_capture_flag_wins_over_a_stale_yaml_value( "CONC": 32, "ISL": 256, "OSL": 1024, - "EXTRA_VLLM_ARGS": "--profiler-config.capture_torch_profiler False", + "EXTRA_VLLM_ARGS": "--profiler-config.detailed_trace_annotation False", }, ) out = _materialize_config_with_envs(src, tmp_path) extra = yaml.safe_load(out.read_text())["benchmark"]["envs"]["EXTRA_VLLM_ARGS"] - assert "--profiler-config.capture_torch_profiler True" in extra, extra - assert extra.rindex("capture_torch_profiler True") > extra.rindex( - "capture_torch_profiler False" + assert "--profiler-config.detailed_trace_annotation True" in extra, extra + assert extra.rindex("detailed_trace_annotation True") > extra.rindex( + "detailed_trace_annotation False" ), extra @@ -731,7 +731,6 @@ def test_materialize_profile_restore_accepts_a_bound_that_already_holds( "--profiler-config.delay_iterations 6080 " "--profiler-config.max_iterations 64 " "--profiler-config.ignore_frontend True " - "--profiler-config.capture_torch_profiler True " "--profiler-config.detailed_trace_annotation True" ), }, @@ -796,7 +795,7 @@ def test_materialize_profile_restores_max_iterations_even_when_delay_survives( extra = yaml.safe_load(out.read_text())["benchmark"]["envs"]["EXTRA_VLLM_ARGS"] assert "--profiler-config.max_iterations 128" in extra, extra assert "--profiler-config.ignore_frontend True" in extra, extra - assert "--profiler-config.capture_torch_profiler True" in extra, extra + assert "--profiler-config.detailed_trace_annotation True" in extra, extra # The candidate's own delay value is left alone; only the missing flags return. assert extra.count("--profiler-config.delay_iterations") == 1, extra @@ -983,7 +982,7 @@ def test_materialize_profile_vllm_injects_tracelens_flags_when_patched( tmp_path, monkeypatch, ): - """Patcher True for vLLM ⇒ EXTRA_VLLM_ARGS gains capture_torch_profiler + detailed_trace_annotation on top of the default iteration count.""" + """Patcher True for vLLM ⇒ EXTRA_VLLM_ARGS gains detailed_trace_annotation on top of the default iteration count.""" import yaml _clear_workload_env(monkeypatch) @@ -993,7 +992,7 @@ def test_materialize_profile_vllm_injects_tracelens_flags_when_patched( extra = yaml.safe_load(out.read_text())["benchmark"]["envs"]["EXTRA_VLLM_ARGS"] assert "--profiler-config.delay_iterations 6080" in extra, extra assert "--profiler-config.max_iterations 128" in extra, extra - assert "--profiler-config.capture_torch_profiler True" in extra, extra + assert "--profiler-config.detailed_trace_annotation True" in extra, extra assert "--profiler-config.detailed_trace_annotation True" in extra, extra # Per-framework dispatch: the SGLang patcher must NOT run for a vLLM YAML. assert counts == {"vllm": 1, "sglang": 0}, counts @@ -1015,7 +1014,6 @@ def test_materialize_profile_vllm_omits_tracelens_flags_when_patch_fails( envs = yaml.safe_load(out.read_text())["benchmark"]["envs"] extra = envs["EXTRA_VLLM_ARGS"] assert "--profiler-config.delay_iterations 6080" in extra, extra - assert "capture_torch_profiler" not in extra, extra assert "detailed_trace_annotation" not in extra, extra assert envs["HYPERLOOM_TRACELENS_PATCH_STATUS"] == "unavailable" assert envs["HYPERLOOM_PROFILE_DEGRADED_REASON"] == "tracelens_runtime_patch_unavailable" @@ -1079,7 +1077,6 @@ def test_materialize_profile_kill_switch_skips_patcher_entirely( # Safe profiler flags still present. assert "--profiler-config.delay_iterations 6080" in extra, extra # TraceLens-only flags absent. - assert "capture_torch_profiler" not in extra, extra assert "detailed_trace_annotation" not in extra, extra # Patchers never invoked. assert counts == {"vllm": 0, "sglang": 0}, counts diff --git a/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py b/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py index 26ffe16240..6b886f76f4 100644 --- a/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py +++ b/src/hyperloom/inference_optimizer/tests/test_promote_shared_state_lock.py @@ -219,14 +219,10 @@ async def test_promote_profile_writes_state_and_audit(session_dir): "max_model_len": 4096, } ) - # +1% rule met (150 vs 100): current_best re-lifted to profile. - assert s.current_best["action"] == "profile" - assert s.current_best["tput"] == 150.0 - assert s.current_best["engine"] == "sglang" - assert s.current_best["extra_server_args"] == "--attention-backend aiter" - assert s.current_best["extra_envs"] == { - "AITER_CONFIG_GEMM_A8W8_BLOCKSCALE": "/tmp/tuned.csv" - } + # A profiler-on measurement never moves current_best, however high it reads. + assert s.current_best["action"] == "explore" + assert s.current_best["tput"] == 100.0 + assert s.cumulative_gain_validated == 0.0 # Audit row. assert s.last_profile["decision"] == "promoted" assert s.last_profile["status"] == "succeeded" @@ -456,6 +452,31 @@ async def test_promote_integrate_patch_kept_lifts_and_clears_pending(session_dir assert not hasattr(s, "last_integrate_patch") +@pytest.mark.asyncio +async def test_promote_integrate_patch_marks_a_refused_keep(session_dir): + """A KEEP measured below the live anchor is not adopted, and must not journal as one.""" + from hyperloom.orchestrator.state.optimization_journal import ( + OUTCOME_NO_PROMOTE, + derive_journal_outcome, + ) + + coord = _coord(session_dir) + s = coord.shared_state + s.baseline_tput = 100.0 + s.current_best = {"action": "explore", "tput": 200.0, "extra_server_args": "", "extra_envs": {}} + + result = { + "status": "kept", + "output_throughput": 140.0, + "specialist_task_id": "spec-1", + "delta_pct": 40.0, + } + await coord._promote_to_shared_state("integrate_patch", result, task=_task("integrate_patch", task_id="t1")) + + assert s.current_best["tput"] == 200.0 + assert derive_journal_outcome("integrate_patch", result, promotable=True) == OUTCOME_NO_PROMOTE + + @pytest.mark.asyncio async def test_integrate_patch_preserves_proposal_owner_across_phase_change( session_dir, @@ -599,7 +620,6 @@ async def test_prebaseline_enablement_patch_is_config_only_not_gain(session_dir) assert entry["baseline_enablement"] is True assert entry["attribution_eligible"] is False assert s.gain_per_stack_entry == [None] - assert s.cumulative_gain == 0.0 assert s.cumulative_gain_validated == 0.0 assert s.pending_integrate == {} @@ -855,10 +875,18 @@ def test_outbox_dead_letters_missing_patch_without_blocking_close( @pytest.mark.asyncio -async def test_resume_reconciles_state_before_draining_kb_outbox( +async def test_resume_settles_state_before_draining_kb_outbox( session_dir, monkeypatch, ): + """The outbox drains after the recovery pass, from the durable config. + + Resume no longer rebuilds ``current_best`` from an ``optimization_stack`` + replay: the lift writes both together, so the replay could only ever + reintroduce an env a later ablation removed. What still has to hold is the + ordering — recovery settles and saves, and only then does the outbox stage + whatever ``current_best`` durably holds. + """ coord = _coord(session_dir) coord._resumed_from = {"is_resume": True} coord.shared_state.optimization_stack = [ @@ -871,8 +899,8 @@ async def test_resume_reconciles_state_before_draining_kb_outbox( } ] coord.shared_state.current_best = { - "extra_server_args": "--stale", - "extra_envs": {"STALE_ENV": "1"}, + "extra_server_args": "--new", + "extra_envs": {"NEW_ENV": "1"}, "tput": 120.0, } coord.shared_state.cumulative_gain_validated_stack_len = 1 @@ -935,7 +963,7 @@ async def test_promote_framework_agent_kept_without_a_candidate_key_is_skipped_l undecorated key is falsy, so ``_lift_to_current_best``'s guard now skips the append instead. - Only the append: current_best and cumulative_gain are set unconditionally + Only the append: current_best is set unconditionally further down, so the win still counts — it just is not recorded as a step anything can later reconcile, dedupe or replay. Pinned because that split is easy to misread in either direction, and because a KEEP that leaves no trace @@ -1202,6 +1230,152 @@ def test_lift_applies_unset_envs_before_new_envs(session_dir): } +def test_lift_is_the_only_writer_so_an_ablated_env_stays_gone(session_dir): + """A later winner that drops an inherited env must not see it come back.""" + coord = _coord(session_dir) + s = coord.shared_state + s.baseline_tput = 1000.0 + + coord._lift_to_current_best( + "explore", + 1100.0, + {"name": "adds-env", "extra_server_args": "--flag-a 1", "extra_envs": {"SGLANG_OLD": "1"}}, + ) + coord._lift_to_current_best( + "explore", + 1200.0, + { + "name": "drops-env", + "extra_server_args": "--flag-a 1", + "extra_envs": {"SGLANG_NEW": "1"}, + "unset_envs": ["SGLANG_OLD"], + }, + ) + + assert s.current_best["extra_envs"] == {"SGLANG_NEW": "1"} + assert [e["variant_name"] for e in s.optimization_stack] == ["adds-env", "drops-env"] + + +def test_lift_refuses_a_winner_that_does_not_beat_the_anchor(session_dir): + """A measurement below current_best must leave config and stack untouched.""" + coord = _coord(session_dir) + s = coord.shared_state + s.baseline_tput = 1000.0 + coord._lift_to_current_best( + "explore", + 1500.0, + {"name": "good", "extra_server_args": "--flag-a 1", "extra_envs": {"A": "1"}}, + ) + + lifted = coord._lift_to_current_best( + "gemm_tuning", + 1100.0, + {"name": "worse", "extra_server_args": "--flag-b 2", "extra_envs": {"B": "2"}}, + ) + + assert lifted is False + assert s.current_best["tput"] == 1500.0 + assert s.current_best["extra_envs"] == {"A": "1"} + assert [e["variant_name"] for e in s.optimization_stack] == ["good"] + + +def test_lift_keeps_entry_extra_off_current_best(session_dir): + """Artifact and provenance handles belong to the stack entry, not the config.""" + coord = _coord(session_dir) + s = coord.shared_state + s.baseline_tput = 1000.0 + + coord._lift_to_current_best( + "gemm_tuning", + 1200.0, + {"name": "geak_a8w8", "extra_server_args": "", "extra_envs": {"AITER_CONFIG": "/tuned.csv"}}, + entry_extra={"tuned_file": "/tuned.csv", "backend": "geak", "empty": "", "absent": None}, + ) + + entry = s.optimization_stack[-1] + assert entry["tuned_file"] == "/tuned.csv" + assert entry["backend"] == "geak" + assert "empty" not in entry + assert "absent" not in entry + assert "tuned_file" not in s.current_best + assert "backend" not in s.current_best + + +def test_env_spec_reports_the_config_current_best_was_measured_on(session_dir): + """The GEAK handoff must describe current_best, ablations included.""" + coord = _coord(session_dir) + s = coord.shared_state + s.baseline_tput = 1000.0 + + coord._lift_to_current_best( + "explore", + 1100.0, + {"name": "v1", "extra_server_args": "--flag-a 1", "extra_envs": {"OLD": "1"}}, + ) + coord._lift_to_current_best( + "explore", + 1200.0, + { + "name": "v2", + "extra_server_args": "--flag-a 1", + "extra_envs": {"NEW": "1"}, + "unset_envs": ["OLD"], + "final_overlay": "/overlay/build", + }, + ) + + spec = coord.build_env_spec() + + assert spec["config"]["extra_envs"] == {"NEW": "1"} + assert spec["config"]["extra_server_args"] == "--flag-a 1" + assert spec["overlay_pythonpath"] == "/overlay/build" + + +def test_env_spec_routes_a_flag_stored_under_extra_envs_back_into_args(session_dir): + """A ``-``-prefixed env key is a server arg; exporting it would drop it.""" + coord = _coord(session_dir) + s = coord.shared_state + s.baseline_tput = 1000.0 + coord._lift_to_current_best( + "integrate_patch", + 1200.0, + { + "name": "patch-1", + "extra_server_args": "--flag-a 1", + "extra_envs": {"REAL_ENV": "1", "--compilation-config": "3"}, + }, + ) + + spec = coord.build_env_spec() + + assert spec["config"]["extra_envs"] == {"REAL_ENV": "1"} + args = spec["config"]["extra_server_args"].split() + assert args[args.index("--compilation-config") + 1] == "3" + assert "--flag-a" in args + + +def test_lift_carries_the_active_overlay_forward(session_dir): + """An authored-kernel overlay outlives the KEEP that built it.""" + coord = _coord(session_dir) + s = coord.shared_state + s.baseline_tput = 1000.0 + + coord._lift_to_current_best( + "geak_e2e", + 1200.0, + {"name": "geak", "extra_server_args": "", "extra_envs": {}, "final_overlay": "/overlay/build"}, + ) + assert s.current_best["final_overlay"] == "/overlay/build" + assert s.optimization_stack[-1]["final_overlay"] == "/overlay/build" + + coord._lift_to_current_best( + "explore", + 1300.0, + {"name": "flags-only", "extra_server_args": "--flag-a 1", "extra_envs": {}}, + ) + assert s.current_best["final_overlay"] == "/overlay/build" + + @pytest.mark.asyncio async def test_lift_copies_source_snapshot_into_stack_entry(session_dir): """Source snapshot manifest and changed files reach the stack entry.""" diff --git a/src/hyperloom/inference_optimizer/tests/test_protocol_layer.py b/src/hyperloom/inference_optimizer/tests/test_protocol_layer.py index be4aa88190..57185ebab2 100644 --- a/src/hyperloom/inference_optimizer/tests/test_protocol_layer.py +++ b/src/hyperloom/inference_optimizer/tests/test_protocol_layer.py @@ -317,6 +317,8 @@ def test_task_states_and_terminals(): assert "queued" in TASK_STATES assert "succeeded" in TERMINAL_STATES assert "running" not in TERMINAL_STATES + # A retry takes a fresh idempotency key, never a re-run of the failed row. + assert "failed" in TERMINAL_STATES @pytest.mark.asyncio diff --git a/src/hyperloom/inference_optimizer/tests/test_quantization_prelude.py b/src/hyperloom/inference_optimizer/tests/test_quantization_prelude.py index 5caaf3cc89..964743e684 100644 --- a/src/hyperloom/inference_optimizer/tests/test_quantization_prelude.py +++ b/src/hyperloom/inference_optimizer/tests/test_quantization_prelude.py @@ -9,8 +9,8 @@ * Adapter — ``run_quantization_prelude_async`` maps quantization_agent's QuantSkillRunResult status -> decision (return dir vs SystemExit(3)). * CLI hook — ``cli_quantization._run_quantization_prelude`` is a no-op without the flag, - skipped on --resume, gated on $HYPERLOOM_QUANTIZE_ENABLED, and - rewrites args.model otherwise. + gated on $HYPERLOOM_QUANTIZE_ENABLED, and rewrites args.model + otherwise. ``hyperloom.agents.quantization.quantize_via_prompt`` is monkeypatched so nothing real runs. """ @@ -284,11 +284,10 @@ def test_adapter_failed_exits_3(tmp_path, monkeypatch): class _Args: - def __init__(self, *, model, quantize=None, quantize_scheme=None, resume=False, gpu_type=None): + def __init__(self, *, model, quantize=None, quantize_scheme=None, gpu_type=None): self.model = Path(model) self.quantize = quantize self.quantize_scheme = quantize_scheme - self.resume = resume self.gpu_type = gpu_type @@ -306,20 +305,6 @@ async def _should_not_run(**kwargs): # pragma: no cover - asserts non-call assert str(args.model) == "/models/src" # unchanged -def test_prelude_skipped_on_resume(monkeypatch): - called = {"n": 0} - - async def _should_not_run(**kwargs): # pragma: no cover - asserts non-call - called["n"] += 1 - return "x" - - monkeypatch.setattr(qrh, "run_quantization_prelude_async", _should_not_run) - args = _Args(model="/models/src", quantize="fp8", resume=True) - asyncio.run(cli_quantization._run_quantization_prelude(args)) - assert called["n"] == 0 - assert str(args.model) == "/models/src" # unchanged - - def test_prelude_rewrites_model_on_success(tmp_path, monkeypatch): import hyperloom.inference_optimizer.session.paths as paths diff --git a/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py b/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py index 75260e735d..127ddef7dc 100644 --- a/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_ray_backend_unit.py @@ -927,9 +927,9 @@ def test_gpu_specialist_lease_is_alive_false_before_start(): def test_gpu_specialist_lease_start_async_poll_and_pending(monkeypatch: pytest.MonkeyPatch): - """§3.3 non-blocking start: start_async submits without blocking; poll_started - returns None while pending (ray.wait empty) and the pid once ready; - pending_seconds is > 0 while pending and 0 after the pid is obtained.""" + """§3.3 non-blocking start: start_async submits without blocking, and + poll_started returns None while pending (ray.wait empty) and the pid once + ready.""" class _FakeRayWait(_FakeRayP2): def __init__(self): @@ -953,13 +953,11 @@ def wait(self, refs, num_returns=1, timeout=None): assert lease._start_ref is not None assert lease.poll_started() is None assert lease.pid() is None - assert lease.pending_seconds() >= 0.0 - # Scheduled: wait reports ready -> pid resolves, pending resets to 0. + # Scheduled: wait reports ready -> pid resolves. fake.ready = True assert lease.poll_started() == 4242 assert lease.pid() == 4242 - assert lease.pending_seconds() == 0.0 lease.close() diff --git a/src/hyperloom/inference_optimizer/tests/test_record_trace_analyze_analysis_md.py b/src/hyperloom/inference_optimizer/tests/test_record_trace_analyze_analysis_md.py index 51cae5b939..e6f1da373c 100644 --- a/src/hyperloom/inference_optimizer/tests/test_record_trace_analyze_analysis_md.py +++ b/src/hyperloom/inference_optimizer/tests/test_record_trace_analyze_analysis_md.py @@ -122,7 +122,7 @@ def test_snapshot_id_starts_at_one_after_empty_state() -> None: def test_baseline_gain_captured_at_snapshot_time() -> None: - """``roofline_baseline_gain_at_snapshot`` is a point-in-time capture of cumulative_gain.""" + """``roofline_baseline_gain_at_snapshot`` is a point-in-time capture of the validated gain.""" state = SharedState() state.cumulative_gain_validated = 0.0 state.record_trace_analyze( diff --git a/src/hyperloom/inference_optimizer/tests/test_regression_locks.py b/src/hyperloom/inference_optimizer/tests/test_regression_locks.py index 60b730f65e..d7af593f88 100644 --- a/src/hyperloom/inference_optimizer/tests/test_regression_locks.py +++ b/src/hyperloom/inference_optimizer/tests/test_regression_locks.py @@ -48,7 +48,7 @@ async def test_report_resolves_session_dir_from_env(tmp_path, monkeypatch): """ReportExecutor resolves session_dir from $USER_DATA_PATH.""" sd = tmp_path / "real-session" sd.mkdir() - state = SharedState(session_id=sd.name, model_name="qwen3-8b", baseline_tput=800.0, cumulative_gain=2.5) + state = SharedState(session_id=sd.name, model_name="qwen3-8b", baseline_tput=800.0, cumulative_gain_validated=2.5) state.save(sd) from hyperloom.orchestrator.bus.storage.connection import SqliteConnection diff --git a/src/hyperloom/inference_optimizer/tests/test_remote_recipe_v2.py b/src/hyperloom/inference_optimizer/tests/test_remote_recipe_v2.py index 8895de5229..fd698ebac9 100644 --- a/src/hyperloom/inference_optimizer/tests/test_remote_recipe_v2.py +++ b/src/hyperloom/inference_optimizer/tests/test_remote_recipe_v2.py @@ -98,7 +98,6 @@ def _state(tmp_path: Path) -> SimpleNamespace: "extra_envs": {"FINAL": "1"}, }, cumulative_gain_validated=30.0, - cumulative_gain=30.0, optimization_stack=[ { "action": "explore", @@ -164,7 +163,6 @@ def _state(tmp_path: Path) -> SimpleNamespace: "last_source_file": str(source), } }, - kernel_opt_attempts={}, last_action_failures=[{"action": "explore", "reason": "OOM"}], gaps=[{"description": "attention remains bound"}], warm_start_lessons=[{"statement": "use page size 32"}], @@ -1937,7 +1935,6 @@ def test_nonfinite_built_metrics_are_normalized(tmp_path: Path) -> None: state = _state(tmp_path) state.current_best["tput"] = float("nan") state.cumulative_gain_validated = float("inf") - state.cumulative_gain = float("-inf") bundle = build_remote_knowledge(state, tmp_path / "finite-knowledge") assert bundle.knowledge["optimized_throughput"] == 0.0 assert bundle.knowledge["validated_e2e_gain"] == 0.0 diff --git a/src/hyperloom/inference_optimizer/tests/test_report.py b/src/hyperloom/inference_optimizer/tests/test_report.py index d65fd9db09..718c20e337 100644 --- a/src/hyperloom/inference_optimizer/tests/test_report.py +++ b/src/hyperloom/inference_optimizer/tests/test_report.py @@ -63,7 +63,6 @@ def test_format_md_shows_validated_gain_when_timestamp_missing(): "report_generated_at": "2026-06-23T00:00:00+00:00", "baseline_tput": 100.0, "current_best": {"action": "warm_replay", "tput": 136.146}, - "cumulative_gain": 36.146, "cumulative_gain_validated": 36.146, "cumulative_gain_validated_ts": "", "cumulative_gain_validated_stack_len": 1, @@ -320,7 +319,6 @@ def test_format_md_renders_stop_explanation(): "framework": "sglang", "current_best": {}, "baseline_tput": 100.0, - "cumulative_gain": 0.0, "cumulative_gain_validated": 0.0, "cumulative_gain_validated_stack_len": 0, "optimization_stack_len": 0, diff --git a/src/hyperloom/inference_optimizer/tests/test_report_helpers_coverage_unit.py b/src/hyperloom/inference_optimizer/tests/test_report_helpers_coverage_unit.py index 08fe572d31..33c378abe9 100644 --- a/src/hyperloom/inference_optimizer/tests/test_report_helpers_coverage_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_report_helpers_coverage_unit.py @@ -63,7 +63,6 @@ def test_format_md_shows_validated_gain_when_timestamp_missing(): "report_generated_at": "2026-06-23T00:00:00+00:00", "baseline_tput": 100.0, "current_best": {"action": "warm_replay", "tput": 136.146}, - "cumulative_gain": 36.146, "cumulative_gain_validated": 36.146, "cumulative_gain_validated_ts": "", "cumulative_gain_validated_stack_len": 1, @@ -218,7 +217,6 @@ def test_format_md_renders_stop_explanation(): "framework": "sglang", "current_best": {}, "baseline_tput": 100.0, - "cumulative_gain": 0.0, "cumulative_gain_validated": 0.0, "cumulative_gain_validated_stack_len": 0, "optimization_stack_len": 0, diff --git a/src/hyperloom/inference_optimizer/tests/test_reporters_smoke.py b/src/hyperloom/inference_optimizer/tests/test_reporters_smoke.py index ec22a60e06..c1ece7ed3d 100644 --- a/src/hyperloom/inference_optimizer/tests/test_reporters_smoke.py +++ b/src/hyperloom/inference_optimizer/tests/test_reporters_smoke.py @@ -51,7 +51,6 @@ def _fixture_breakdown(**overrides: Any) -> dict[str, Any]: "final": { "throughput_tok_s_per_gpu": 2447.5, "cumulative_gain_pct_validated": 10.99, - "cumulative_gain_pct_per_round_sum": 10.99, "validated_at_stack_len": 1, "validated_ts": "2026-05-12T11:54:00Z", "stack_changed_after_validation": False, diff --git a/src/hyperloom/inference_optimizer/tests/test_resume.py b/src/hyperloom/inference_optimizer/tests/test_resume.py index 1a5c06c43c..471745d562 100644 --- a/src/hyperloom/inference_optimizer/tests/test_resume.py +++ b/src/hyperloom/inference_optimizer/tests/test_resume.py @@ -431,7 +431,7 @@ async def test_tick_lazily_runs_replay_on_resume(session_dir): class TestN23ResumePerSession: - """``--resume`` understands the N17 per-session layout, exercising ``find_latest_per_session_dir``.""" + """``--resume-from`` addresses a session inside the N17 per-session layout.""" @pytest.fixture(autouse=True) def _isolate_env(self, monkeypatch, tmp_path): @@ -440,20 +440,6 @@ def _isolate_env(self, monkeypatch, tmp_path): monkeypatch.setenv(_paths.ENV_USER_DATA_PATH, str(tmp_path)) monkeypatch.delenv(_paths.ENV_CURRENT_SESSION_DIR, raising=False) - def test_resume_picks_latest_subdir_after_two_launches(self, tmp_path): - from hyperloom.inference_optimizer.session import paths as _paths - - sd1 = _paths.make_session_dir(model_name="DeepSeek-R1-0528") - assert _paths.find_latest_per_session_dir() == sd1 - assert _paths.find_latest_per_session_dir(model_name="DeepSeek-R1-0528") == sd1 - - later_ts = "29990101T000000Z" - sd2 = tmp_path / "DeepSeek-R1-0528" / later_ts - sd2.mkdir(parents=True) - - assert _paths.find_latest_per_session_dir() == sd2 - assert _paths.find_latest_per_session_dir(model_name="DeepSeek-R1-0528") == sd2 - def test_resume_does_not_mutate_user_data_path(self, tmp_path): from hyperloom.inference_optimizer.session import paths as _paths @@ -465,11 +451,6 @@ def test_resume_does_not_mutate_user_data_path(self, tmp_path): assert _paths.workspace_root() == tmp_path assert tmp_path in sd.parents - def test_resume_falls_back_to_flat_when_no_per_session_subdir(self, tmp_path): - from hyperloom.inference_optimizer.session import paths as _paths - - assert _paths.find_latest_per_session_dir() is None - def test_resume_from_explicit_path_must_be_under_workspace_root( self, tmp_path, @@ -488,36 +469,22 @@ def test_resume_from_explicit_path_must_be_under_workspace_root( except ValueError: pass - def test_latest_picks_across_models_when_model_name_omitted(self, tmp_path): - from hyperloom.inference_optimizer.session import paths as _paths - - (tmp_path / "ModelA").mkdir() - (tmp_path / "ModelA" / "20260101T000000Z").mkdir() - (tmp_path / "ModelB").mkdir() - (tmp_path / "ModelB" / "20260520T000000Z").mkdir() - (tmp_path / "ModelC").mkdir() - (tmp_path / "ModelC" / "20260315T000000Z").mkdir() - - picked = _paths.find_latest_per_session_dir() - assert picked is not None - assert picked.parent.name == "ModelB" - assert picked.name == "20260520T000000Z" + @pytest.mark.parametrize( + "argv", + [ + ["optimize", "--resume"], + # The command line already-deployed robustness monitor copies send. + ["optimize", "--resume", "--resume-from", "/tmp/sess"], + ], + ) + def test_no_session_can_be_resumed_without_naming_it(self, argv): + """``--resume`` cannot start a run; it exits instead of choosing a session.""" + from hyperloom.inference_optimizer.cli.parser import _build_parser - def test_workspace_shared_dirs_never_picked_as_session(self, tmp_path): - from hyperloom.inference_optimizer.session import paths as _paths + with pytest.raises(SystemExit) as exc: + _build_parser().parse_args(argv) + assert exc.value.code == 2 - (tmp_path / "runtime").mkdir() - (tmp_path / "runtime" / "20990101T000000Z").mkdir() - (tmp_path / "logs").mkdir() - (tmp_path / "logs" / "20990101T000000Z").mkdir() - (tmp_path / "RealModel").mkdir() - (tmp_path / "RealModel" / "20260518T100000Z").mkdir() - - picked = _paths.find_latest_per_session_dir() - assert picked is not None - assert picked.parent.name == "RealModel" - assert "runtime" not in str(picked) - assert "logs" not in str(picked) # _load_kernel_agent_env_fallback hard-fails on bad state diff --git a/src/hyperloom/inference_optimizer/tests/test_robustness_agent_e2e.py b/src/hyperloom/inference_optimizer/tests/test_robustness_agent_e2e.py index 4f66616b22..086b1f48ca 100644 --- a/src/hyperloom/inference_optimizer/tests/test_robustness_agent_e2e.py +++ b/src/hyperloom/inference_optimizer/tests/test_robustness_agent_e2e.py @@ -76,7 +76,6 @@ async def test_robustness_agent_real_runtime_heartbeat( # No runtime_caller_factory: use the real subprocess path. Disable probes # so an inert CI host doesn't fire HIGH alerts that mask the heartbeat. options={ - "robustness_server_url": "", "auto_probe_inference_server": False, "ray_probe_enabled": False, "external_deps_enabled": False, @@ -130,7 +129,7 @@ async def test_robustness_agent_real_runtime_emits_alert_on_high_crash( backend = RobustnessAgentBackend( robustness_agent_root=robustness_agent_root, session_dir=session_dir, - options={"robustness_server_url": ""}, + options={}, ) backends = { @@ -178,7 +177,7 @@ async def test_robustness_agent_workdir_is_per_turn( backend = RobustnessAgentBackend( robustness_agent_root=robustness_agent_root, session_dir=session_dir, - options={"robustness_server_url": ""}, + options={}, ) backends = { diff --git a/src/hyperloom/inference_optimizer/tests/test_robustness_monitor.py b/src/hyperloom/inference_optimizer/tests/test_robustness_monitor.py index 770b70daca..dafb0f0343 100644 --- a/src/hyperloom/inference_optimizer/tests/test_robustness_monitor.py +++ b/src/hyperloom/inference_optimizer/tests/test_robustness_monitor.py @@ -14,6 +14,7 @@ import json import os +import re import subprocess import time from pathlib import Path @@ -247,9 +248,10 @@ def test_monitor_handles_leading_zero_wait_sec(tmp_path): def test_monitor_resume_is_pinned_to_resolved_session_dir(): - """Crash recovery must never use bare --resume, which auto-picks the latest session.""" + """Crash recovery must name the session explicitly, never let the CLI choose one.""" text = MONITOR.read_text(encoding="utf-8") - assert '--resume --resume-from "$session_dir"' in text + assert '--resume-from "$session_dir"' in text + assert re.search(r"--resume(?!-from)", text) is None @pytest.mark.skipif(not _HAS_PROC, reason="liveness probe reads /proc (Linux)") diff --git a/src/hyperloom/inference_optimizer/tests/test_robustness_storm_and_mix.py b/src/hyperloom/inference_optimizer/tests/test_robustness_storm_and_mix.py index 11cd3b2a3d..039b4bb5c9 100644 --- a/src/hyperloom/inference_optimizer/tests/test_robustness_storm_and_mix.py +++ b/src/hyperloom/inference_optimizer/tests/test_robustness_storm_and_mix.py @@ -74,13 +74,6 @@ def test_specialist_dispatch_counter_increments(): assert s.bump_specialist_dispatched(3) == 4 -def test_specialist_dispatch_counter_resets(): - s = SharedState() - s.bump_specialist_dispatched(5) - s.reset_specialist_dispatched() - assert s.explore_specialist_dispatched_count == 0 - - # 4. Coordinator hook: explore KEEP → config; integrate_patch kept → code_patch def test_coordinator_intervention_hook_records_config_for_explore(): from hyperloom.orchestrator.loop.coordinator import Coordinator diff --git a/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling.py b/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling.py deleted file mode 100644 index 195dbb0cb1..0000000000 --- a/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling.py +++ /dev/null @@ -1,2980 +0,0 @@ -# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Tests for ``orchestrator.kernel.roofline_ceiling`` (formula correctness, graceful degrade, HF metadata parsing).""" - -from __future__ import annotations - -import json -from pathlib import Path -from types import SimpleNamespace - -import pytest - -from hyperloom.orchestrator.kernel import roofline_ceiling as _rc -from hyperloom.orchestrator.kernel.roofline_ceiling import ( - HW_SPECS, - ModelMeta, - RuntimeWorkload, - _read_diffusion_dit_meta, - _resolve_achievable_tflops, - _resolve_dtype_bytes, - _resolve_peak_tflops, - apply_runtime_dtype, - compute_compute_bound_ceiling_tok_per_sec, - compute_diffusion_compute_img_per_sec, - compute_diffusion_mem_img_per_sec, - compute_kv_bytes_per_token, - compute_peak_from_state, - compute_roofline_breakdown_from_state, - compute_roofline_from_perfmodel, - compute_theoretical_peak_output_tok_per_sec, - load_model_meta, - resolve_runtime_dtype, - resolve_runtime_workload, -) - - -# Formula correctness against published numbers. -class TestPeakFormulaAgainstPublishedNumbers: - """Worked examples from ITK Research and arXiv 2402.16363 (KV-cache treated as negligible via num_layers=0).""" - - def test_itk_b200_70b_fp8_matches_cited_114_tok_s(self, monkeypatch): - # ITK Research B200: 8e12 / 70e9 ≈ 114.28 tok/s. Inject the B200 spec for this test only. - monkeypatch.setitem(HW_SPECS, "b200_test", {"hbm_gb": 192.0, "hbm_bw_gbps": 8000.0}) - peak = compute_theoretical_peak_output_tok_per_sec( - gpu_type="b200_test", - num_gpus=1, - weight_bytes=70_000_000_000, - num_layers=0, - num_kv_heads=0, - head_dim=0, - kv_dtype_bytes=1.0, - isl=0, - osl=0, - concurrency=1, - ) - assert peak == pytest.approx(114.28, rel=0.01) - - def test_itk_b200_70b_fp4_doubles_to_228_tok_s(self, monkeypatch): - """FP8 → FP4 halves bytes/param so peak doubles, per ITK Research.""" - monkeypatch.setitem(HW_SPECS, "b200_test", {"hbm_gb": 192.0, "hbm_bw_gbps": 8000.0}) - peak_fp4 = compute_theoretical_peak_output_tok_per_sec( - gpu_type="b200_test", - num_gpus=1, - weight_bytes=35_000_000_000, - num_layers=0, - num_kv_heads=0, - head_dim=0, - kv_dtype_bytes=0.5, - isl=0, - osl=0, - concurrency=1, - ) - assert peak_fp4 == pytest.approx(228.57, rel=0.01) - - -# MI300X realistic sanity (Llama-70B-style dense model). -class TestMI300XRealistic: - """Llama-3-70B BF16 on 1×MI300X: single-stream decode ≈ 30-40 tok/s (5.3 TB/s / 140 GB ≈ 37.8).""" - - def test_llama70b_bf16_single_mi300x(self): - peak = compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - weight_bytes=140_000_000_000, - num_layers=80, - num_kv_heads=8, - head_dim=128, - kv_dtype_bytes=2.0, - isl=2048, - osl=512, - concurrency=1, - ) - assert 25.0 < peak < 45.0 - - def test_concurrency_scales_throughput_in_weight_dominated_regime(self): - """With small isl/osl, batching N requests amortizes weight reads N× so peak is near-linearly higher.""" - kwargs = dict( - gpu_type="mi300x", - num_gpus=8, - weight_bytes=140_000_000_000, - num_layers=80, - num_kv_heads=8, - head_dim=128, - kv_dtype_bytes=2.0, - isl=128, - osl=64, - ) - p1 = compute_theoretical_peak_output_tok_per_sec(**kwargs, concurrency=1) - p20 = compute_theoretical_peak_output_tok_per_sec(**kwargs, concurrency=20) - # 20× concurrency gives >10× peak (KV term doesn't amortize, so not exactly 20×). - assert p20 / max(p1, 1e-9) > 10.0 - - -# Graceful degrade. -class TestGracefulDegrade: - def test_unknown_gpu_type_returns_zero(self): - peak = compute_theoretical_peak_output_tok_per_sec( - gpu_type="rtx4090", - num_gpus=1, - weight_bytes=10**9, - num_layers=12, - num_kv_heads=8, - head_dim=64, - kv_dtype_bytes=2.0, - isl=128, - osl=128, - concurrency=1, - ) - assert peak == 0.0 - - def test_empty_gpu_type_returns_zero(self): - peak = compute_theoretical_peak_output_tok_per_sec( - gpu_type="", - num_gpus=1, - weight_bytes=10**9, - num_layers=12, - num_kv_heads=8, - head_dim=64, - kv_dtype_bytes=2.0, - isl=128, - osl=128, - concurrency=1, - ) - assert peak == 0.0 - - def test_zero_concurrency_clamps_to_one(self): - peak = compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - weight_bytes=10**9, - num_layers=12, - num_kv_heads=8, - head_dim=64, - kv_dtype_bytes=2.0, - isl=128, - osl=128, - concurrency=0, - ) - assert peak > 0.0 - - def test_zero_weight_and_kv_returns_zero(self): - # Pathological all-zero inputs must not raise / divide-by-zero. - peak = compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - weight_bytes=0, - num_layers=0, - num_kv_heads=0, - head_dim=0, - kv_dtype_bytes=0.0, - isl=0, - osl=0, - concurrency=1, - ) - assert peak == 0.0 - - -# Dtype + KV helpers. -class TestResolveDtypeBytes: - @pytest.mark.parametrize( - "tag,expected", - [ - ("bfloat16", 2.0), - ("BF16", 2.0), - ("float16", 2.0), - ("float32", 4.0), - ("float8_e4m3fn", 1.0), - ("float8_e5m2", 1.0), - ("fp8", 1.0), - ("fp4", 0.5), - ("", 2.0), - (None, 2.0), - ("unknown_dtype", 2.0), - ], - ) - def test_canonical_mapping(self, tag, expected): - assert _resolve_dtype_bytes(tag) == expected - - -class TestComputeKVBytesPerToken: - def test_factor_two_for_k_plus_v(self): - # 2 (K+V) × 80 layers × 8 KV heads × 128 head_dim × 2 bytes = 327_680 bytes/token. - assert ( - compute_kv_bytes_per_token( - num_layers=80, - num_kv_heads=8, - head_dim=128, - kv_dtype_bytes=2.0, - ) - == 327_680 - ) - - def test_fp8_kv_halves_volume(self): - assert ( - compute_kv_bytes_per_token( - num_layers=80, - num_kv_heads=8, - head_dim=128, - kv_dtype_bytes=1.0, - ) - == 163_840 - ) - - -# HF metadata extraction. -def _write_synthetic_model( - model_dir: Path, - *, - total_size: int, - num_layers: int = 80, - num_kv_heads: int | None = 8, - hidden_size: int = 8192, - num_attention_heads: int = 64, - torch_dtype: str = "bfloat16", - head_dim: int | None = None, - num_experts: int | None = None, - num_experts_per_tok: int | None = None, - moe_intermediate_size: int | None = None, - n_routed_experts: int | None = None, - num_local_experts: int | None = None, - quant_method: str | None = None, - dtype: str | None = None, - expert_dtype: str | None = None, -) -> None: - """Lay down a minimal HF-shaped model dir; optional kwargs emit MHA / MoE / quant / alias variants.""" - model_dir.mkdir(parents=True, exist_ok=True) - config: dict = { - "num_hidden_layers": num_layers, - "num_attention_heads": num_attention_heads, - "hidden_size": hidden_size, - "torch_dtype": torch_dtype, - } - if num_kv_heads is not None: - config["num_key_value_heads"] = num_kv_heads - if head_dim is not None: - config["head_dim"] = head_dim - if num_experts is not None: - config["num_experts"] = num_experts - if num_experts_per_tok is not None: - config["num_experts_per_tok"] = num_experts_per_tok - if moe_intermediate_size is not None: - config["moe_intermediate_size"] = moe_intermediate_size - if n_routed_experts is not None: - config["n_routed_experts"] = n_routed_experts - if num_local_experts is not None: - config["num_local_experts"] = num_local_experts - if quant_method is not None: - config["quantization_config"] = { - "quant_method": quant_method, - "weight_block_size": [128, 128], - "activation_scheme": "dynamic", - } - if dtype is not None: - config["dtype"] = dtype - if expert_dtype is not None: - config["expert_dtype"] = expert_dtype - (model_dir / "config.json").write_text(json.dumps(config)) - (model_dir / "model.safetensors.index.json").write_text( - json.dumps({"metadata": {"total_size": total_size}, "weight_map": {}}) - ) - - -class TestLoadModelMeta: - def test_reads_total_size_and_geometry(self, tmp_path): - _write_synthetic_model( - tmp_path / "m", - total_size=70_000_000_000, - num_layers=80, - num_kv_heads=8, - hidden_size=8192, - num_attention_heads=64, - torch_dtype="float8_e4m3fn", - ) - meta = load_model_meta(tmp_path / "m") - assert meta is not None - assert isinstance(meta, ModelMeta) - assert meta.weight_bytes == 70_000_000_000 - assert meta.num_layers == 80 - assert meta.num_kv_heads == 8 - assert meta.head_dim == 128 - assert meta.weight_dtype_bytes == 1.0 - - def test_mha_fallback_when_num_kv_heads_absent(self, tmp_path): - _write_synthetic_model( - tmp_path / "m", - total_size=1_000_000_000, - num_layers=12, - num_kv_heads=None, - num_attention_heads=16, - hidden_size=2048, - ) - meta = load_model_meta(tmp_path / "m") - assert meta is not None - assert meta.num_kv_heads == 16 - assert meta.head_dim == 128 - - def test_head_dim_directly_from_config(self, tmp_path): - _write_synthetic_model( - tmp_path / "m", - total_size=1_000_000_000, - num_attention_heads=64, - hidden_size=8192, - head_dim=200, - ) - meta = load_model_meta(tmp_path / "m") - assert meta is not None - assert meta.head_dim == 200 - - def test_nested_text_config_moe_is_read(self, tmp_path): - # Multimodal wrappers (e.g. Kimi-K2 kimi_k25) nest the decoder shape/MoE - # config under text_config; the ceiling reader must flatten it or it - # degrades to a dense full-weight roofline (num_experts=0 -> PerfModel - # falls back to legacy, active_weight_bytes = full weight_bytes). - d = tmp_path / "m" - d.mkdir() - (d / "config.json").write_text( - json.dumps( - { - "model_type": "kimi_k25", - "text_config": { - "num_hidden_layers": 4, - "num_attention_heads": 16, - "num_key_value_heads": 8, - "hidden_size": 2048, - "num_experts": 64, - "num_experts_per_tok": 8, - "moe_intermediate_size": 1024, - "torch_dtype": "bfloat16", - }, - } - ) - ) - (d / "model.safetensors.index.json").write_text( - json.dumps({"metadata": {"total_size": 10_000_000_000}, "weight_map": {}}) - ) - meta = load_model_meta(d) - assert meta is not None - assert meta.num_experts == 64 - assert meta.experts_per_tok == 8 - assert meta.hidden_size == 2048 - assert meta.num_kv_heads == 8 - # MoE decomposition applied -> per-token active bytes below full weights. - assert 0 < meta.active_weight_bytes < meta.weight_bytes - - def test_missing_safetensors_index_uses_safetensor_file_sizes(self, tmp_path): - d = tmp_path / "m" - d.mkdir() - (d / "config.json").write_text( - json.dumps( - { - "num_hidden_layers": 12, - "num_key_value_heads": 4, - "num_attention_heads": 8, - "hidden_size": 1024, - "torch_dtype": "bfloat16", - } - ) - ) - (d / "model-00001-of-00002.safetensors").write_bytes(b"x" * 13) - (d / "model-00002-of-00002.safetensors").write_bytes(b"y" * 17) - - meta = load_model_meta(d) - assert meta is not None - assert meta.weight_bytes == 30 - assert meta.num_layers == 12 - assert meta.num_kv_heads == 4 - assert meta.head_dim == 128 - - def test_missing_safetensors_index_and_files_returns_none(self, tmp_path): - d = tmp_path / "m" - d.mkdir() - (d / "config.json").write_text(json.dumps({"num_hidden_layers": 12, "torch_dtype": "bfloat16"})) - assert load_model_meta(d) is None - - def test_safetensors_index_without_total_size_uses_file_sizes(self, tmp_path): - d = tmp_path / "m" - _write_synthetic_model(d, total_size=1_000_000_000) - (d / "model.safetensors.index.json").write_text(json.dumps({"metadata": {}, "weight_map": {}})) - (d / "model-00001-of-00001.safetensors").write_bytes(b"z" * 23) - - meta = load_model_meta(d) - assert meta is not None - assert meta.weight_bytes == 23 - - def test_missing_safetensors_uses_pytorch_bin_file_sizes(self, tmp_path): - d = tmp_path / "m" - d.mkdir() - (d / "config.json").write_text( - json.dumps( - { - "num_hidden_layers": 12, - "num_key_value_heads": 4, - "num_attention_heads": 8, - "hidden_size": 1024, - "torch_dtype": "float16", - } - ) - ) - (d / "pytorch_model-00001-of-00002.bin").write_bytes(b"a" * 11) - (d / "pytorch_model-00002-of-00002.bin").write_bytes(b"b" * 19) - - meta = load_model_meta(d) - assert meta is not None - assert meta.weight_bytes == 30 - assert meta.weight_dtype_bytes == 2.0 - - def test_missing_config_returns_none(self, tmp_path): - d = tmp_path / "m" - d.mkdir() - (d / "model.safetensors.index.json").write_text(json.dumps({"metadata": {"total_size": 1_000_000_000}})) - assert load_model_meta(d) is None - - def test_nonexistent_model_path_returns_none(self, tmp_path): - assert load_model_meta(tmp_path / "does_not_exist") is None - - def test_empty_model_path_returns_none(self): - assert load_model_meta("") is None - - def test_precision_hint_used_when_torch_dtype_missing(self, tmp_path): - d = tmp_path / "m" - d.mkdir() - (d / "config.json").write_text( - json.dumps( - { - "num_hidden_layers": 80, - "num_key_value_heads": 8, - "num_attention_heads": 64, - "hidden_size": 8192, - } - ) - ) - (d / "model.safetensors.index.json").write_text(json.dumps({"metadata": {"total_size": 70_000_000_000}})) - meta = load_model_meta(d, precision_hint="fp8") - assert meta is not None - assert meta.weight_dtype_bytes == 1.0 - - -# State-driven entry point. -class TestComputePeakFromState: - def test_happy_path_yields_positive(self, tmp_path): - _write_synthetic_model( - tmp_path / "m", - total_size=140_000_000_000, - num_layers=80, - num_kv_heads=8, - hidden_size=8192, - num_attention_heads=64, - torch_dtype="bfloat16", - ) - state = SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="mi300x", - tp=8, - precision="bf16", - conc=20, - isl=2048, - osl=512, - ) - peak = compute_peak_from_state(state) - assert peak > 0.0 - assert peak < 1e6 - - def test_missing_model_path_returns_zero(self): - state = SimpleNamespace( - model_path="/no/such/dir", - gpu_type="mi300x", - tp=8, - precision="bf16", - conc=20, - isl=2048, - osl=512, - ) - assert compute_peak_from_state(state) == 0.0 - - def test_unknown_gpu_returns_zero(self, tmp_path): - _write_synthetic_model(tmp_path / "m", total_size=10**9) - state = SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="rtx4090", - tp=1, - precision="bf16", - conc=1, - isl=128, - osl=128, - ) - assert compute_peak_from_state(state) == 0.0 - - -# MoE active weight bytes (PR: MoE-aware decode ceiling). -def _write_qwen3_moe_model(model_dir: Path, *, total_size: int) -> None: - """Lay down a Qwen3-30B-A3B-shaped MoE HF dir for ceiling tests.""" - model_dir.mkdir(parents=True, exist_ok=True) - (model_dir / "config.json").write_text( - json.dumps( - { - "architectures": ["Qwen3MoeForCausalLM"], - "model_type": "qwen3_moe", - "num_hidden_layers": 48, - "num_attention_heads": 32, - "num_key_value_heads": 4, - "hidden_size": 2048, - "head_dim": 128, - "intermediate_size": 6144, - "moe_intermediate_size": 768, - "num_experts": 128, - "num_experts_per_tok": 8, - "torch_dtype": "bfloat16", - } - ) - ) - (model_dir / "model.safetensors.index.json").write_text( - json.dumps({"metadata": {"total_size": total_size}, "weight_map": {}}) - ) - - -def _write_deepseek_v4_model(model_dir: Path, *, total_size: int, expert_dtype: str | None) -> None: - """Lay down a DeepSeek-V4-Pro-shaped MoE dir: fp8 attention + (optional) fp4 experts. - - The routed-expert weights, sized at the *global* fp8 dtype, exceed the whole - on-disk checkpoint (1547 GB computed vs 865 GB real), which trips the - ``total_expert_bytes >= weight_bytes`` safe-degrade. Reading ``expert_dtype`` - (fp4) is what keeps the decomposition alive. - """ - model_dir.mkdir(parents=True, exist_ok=True) - config: dict = { - "architectures": ["DeepseekV4ForCausalLM"], - "model_type": "deepseek_v4", - "num_hidden_layers": 61, - "num_attention_heads": 128, - "num_key_value_heads": 1, - "hidden_size": 7168, - "head_dim": 512, - "moe_intermediate_size": 3072, - "n_routed_experts": 384, - "num_experts_per_tok": 6, - "torch_dtype": "bfloat16", - "quantization_config": { - "quant_method": "fp8", - "weight_block_size": [128, 128], - "activation_scheme": "dynamic", - }, - } - if expert_dtype is not None: - config["expert_dtype"] = expert_dtype - (model_dir / "config.json").write_text(json.dumps(config)) - (model_dir / "model.safetensors.index.json").write_text( - json.dumps({"metadata": {"total_size": total_size}, "weight_map": {}}) - ) - - -class TestSeparateExpertDtype: - """DeepSeek-V4 stores routed experts at ``expert_dtype`` (fp4) distinct from the fp8 attention; sizing experts with the global dtype over-counts and drops the whole MoE from the ceiling.""" - - _TOTAL = 864_704_792_696 # real DeepSeek-V4-Pro on-disk total_size (bytes) - - def test_fp4_expert_dtype_keeps_moe_decomposition(self, tmp_path): - """expert_dtype=fp4 → total expert bytes (~774 GB) < checkpoint → decomposition survives.""" - _write_deepseek_v4_model(tmp_path / "m", total_size=self._TOTAL, expert_dtype="fp4") - meta = load_model_meta(tmp_path / "m") - assert meta is not None - assert meta.num_experts == 384 - assert meta.experts_per_tok == 6 - assert meta.expert_weight_dtype_bytes == 0.5 - assert 0 < meta.active_weight_bytes < meta.weight_bytes - assert 0 < meta.expert_weight_bytes < meta.weight_bytes - - def test_global_fp8_dtype_trips_safe_degrade(self, tmp_path): - """Without expert_dtype the experts are sized at fp8 → computed bytes exceed the checkpoint → MoE silently dropped (the bug this fix targets).""" - _write_deepseek_v4_model(tmp_path / "m", total_size=self._TOTAL, expert_dtype=None) - meta = load_model_meta(tmp_path / "m") - assert meta is not None - # Safe-degrade path: no expert decomposition, active == total. - assert meta.num_experts == 0 - assert meta.expert_weight_bytes == 0 - assert meta.active_weight_bytes == meta.weight_bytes - - def test_perfmodel_counts_moe_and_lowers_peak(self, tmp_path): - """With fp4 experts the PerfModel emits a moe_fused op and the peak drops far below the attention-only (safe-degraded) ceiling.""" - state_kwargs = dict( - gpu_type="mi355x", - tp=8, - precision="fp8", - framework="vllm", - conc=64, - isl=1024, - osl=1024, - ) - _write_deepseek_v4_model(tmp_path / "fp4", total_size=self._TOTAL, expert_dtype="fp4") - _write_deepseek_v4_model(tmp_path / "degraded", total_size=self._TOTAL, expert_dtype=None) - - meta_fp4 = load_model_meta(tmp_path / "fp4") - assert meta_fp4 is not None - pm = compute_roofline_from_perfmodel( - meta=meta_fp4, - gpu_type="mi355x", - concurrency=64, - isl=1024, - osl=1024, - num_gpus=8, - precision_tag="fp8", - ) - assert pm is not None - assert any(op.name == "moe_fused" for op in pm.ops) - - fp4_peak = compute_peak_from_state(SimpleNamespace(model_path=str(tmp_path / "fp4"), **state_kwargs)) - degraded_peak = compute_peak_from_state(SimpleNamespace(model_path=str(tmp_path / "degraded"), **state_kwargs)) - assert fp4_peak > 0.0 - # Counting the fp4 expert IO must pull the ceiling well below the - # attention-only (MoE-dropped) estimate that produced within% ~= 2.6%. - assert fp4_peak < degraded_peak * 0.5 - - -class TestMoEActiveWeightBytes: - """MoE models route a subset of experts per token; the decode divisor must use the active subset (else within_roofline_pct > 100%).""" - - def test_qwen3_30b_a3b_active_is_small_fraction_of_total(self, tmp_path): - """Qwen3-30B-A3B: 128 experts, 8 active → active weight bytes land in the ~8-15% range of total.""" - _write_qwen3_moe_model(tmp_path / "m", total_size=61_064_245_248) - meta = load_model_meta(tmp_path / "m") - assert meta is not None - assert meta.weight_bytes == 61_064_245_248 - assert 0 < meta.active_weight_bytes < meta.weight_bytes - ratio = meta.active_weight_bytes / meta.weight_bytes - assert 0.05 < ratio < 0.20, f"Qwen3-30B-A3B active ratio {ratio:.3f} outside expected window" - - def test_dense_model_active_equals_total(self, tmp_path): - """Dense (no num_experts) → active = total.""" - _write_synthetic_model( - tmp_path / "m", - total_size=16_381_470_720, - num_layers=36, - num_kv_heads=8, - num_attention_heads=32, - hidden_size=4096, - ) - meta = load_model_meta(tmp_path / "m") - assert meta is not None - assert meta.active_weight_bytes == meta.weight_bytes - - def test_moe_ceiling_higher_than_dense_equivalent(self, tmp_path): - """Same total bytes, MoE active routing → ceiling several × higher than naive dense (fixes within_roofline_pct > 100%).""" - _write_qwen3_moe_model(tmp_path / "m", total_size=61_064_245_248) - meta = load_model_meta(tmp_path / "m") - assert meta is not None - kwargs = dict( - gpu_type="mi355x", - num_gpus=1, - num_layers=meta.num_layers, - num_kv_heads=meta.num_kv_heads, - head_dim=meta.head_dim, - kv_dtype_bytes=meta.weight_dtype_bytes, - isl=1024, - osl=1024, - concurrency=8, - ) - dense_peak = compute_theoretical_peak_output_tok_per_sec( - weight_bytes=meta.weight_bytes, - **kwargs, - ) - moe_peak = compute_theoretical_peak_output_tok_per_sec( - weight_bytes=meta.weight_bytes, - active_weight_bytes=meta.active_weight_bytes, - **kwargs, - ) - assert moe_peak > dense_peak * 2.0, ( - f"MoE ceiling {moe_peak:.0f} should be much higher than dense-treated ceiling {dense_peak:.0f}" - ) - - def test_active_weight_bytes_zero_falls_back_to_weight_bytes(self): - """Backward-compat: active_weight_bytes=0 behaves as before (uses weight_bytes).""" - without = compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - weight_bytes=140_000_000_000, - num_layers=80, - num_kv_heads=8, - head_dim=128, - kv_dtype_bytes=2.0, - isl=2048, - osl=512, - concurrency=1, - ) - with_zero = compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - weight_bytes=140_000_000_000, - active_weight_bytes=0, - num_layers=80, - num_kv_heads=8, - head_dim=128, - kv_dtype_bytes=2.0, - isl=2048, - osl=512, - concurrency=1, - ) - assert without == with_zero - - def test_moe_geometry_overshoots_safe_degrade(self, tmp_path): - """When computed expert bytes >= total_size, the helper clamps to weight_bytes (no negative non_expert_bytes).""" - model_dir = tmp_path / "m" - model_dir.mkdir() - # huge num_experts but tiny total_size → expert_bytes > total - (model_dir / "config.json").write_text( - json.dumps( - { - "num_hidden_layers": 4, - "num_attention_heads": 32, - "num_key_value_heads": 4, - "hidden_size": 4096, - "moe_intermediate_size": 4096, - "num_experts": 256, - "num_experts_per_tok": 2, - "torch_dtype": "bfloat16", - } - ) - ) - (model_dir / "model.safetensors.index.json").write_text(json.dumps({"metadata": {"total_size": 100_000_000}})) - meta = load_model_meta(model_dir) - assert meta is not None - # Safe degrade: active equals total (no inflation, no negatives). - assert meta.active_weight_bytes == meta.weight_bytes - - -# HW_SPECS table sanity. -class TestResolveEffectiveConcurrency: - """Concurrency fallback chain (PR-A): state.conc -> baseline yaml envs.CONC -> 1.""" - - def test_state_conc_wins_when_positive(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - _resolve_effective_concurrency, - ) - - state = SimpleNamespace(conc=32, last_baseline={}) - assert _resolve_effective_concurrency(state) == 32 - - def test_falls_back_to_baseline_yaml_envs_conc(self, tmp_path): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - _resolve_effective_concurrency, - ) - - yaml_path = tmp_path / "baseline_config.with_envs.yaml" - yaml_path.write_text( - "benchmark:\n envs:\n CONC: 64\n ISL: 256\n", - encoding="utf-8", - ) - state = SimpleNamespace( - conc=0, # SharedState default — drives the fallback - last_baseline={ - "extras": {"materialized_config": str(yaml_path)}, - }, - ) - assert _resolve_effective_concurrency(state) == 64 - - def test_baseline_yaml_conc_is_authoritative_over_stale_state_conc( - self, - tmp_path, - ): - """The materialized baseline yaml's ``CONC`` wins over ``state.conc``.""" - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - _resolve_effective_concurrency, - ) - - yaml_path = tmp_path / "baseline_config.with_envs.yaml" - yaml_path.write_text( - "benchmark:\n envs:\n CONC: 64\n", - encoding="utf-8", - ) - state = SimpleNamespace( - conc=8, # stale SharedState default - last_baseline={ - "extras": {"materialized_config": str(yaml_path)}, - }, - ) - assert _resolve_effective_concurrency(state) == 64 - - def test_missing_yaml_falls_back_to_one(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - _resolve_effective_concurrency, - ) - - state = SimpleNamespace( - conc=0, - last_baseline={ - "extras": {"materialized_config": "/no/such/file.yaml"}, - }, - ) - assert _resolve_effective_concurrency(state) == 1 - - def test_malformed_yaml_falls_back_to_one(self, tmp_path): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - _resolve_effective_concurrency, - ) - - bad = tmp_path / "broken.yaml" - bad.write_text("not: [valid yaml at all", encoding="utf-8") - state = SimpleNamespace( - conc=0, - last_baseline={"extras": {"materialized_config": str(bad)}}, - ) - assert _resolve_effective_concurrency(state) == 1 - - def test_yaml_without_conc_falls_back_to_one(self, tmp_path): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - _resolve_effective_concurrency, - ) - - yaml_path = tmp_path / "no_conc.yaml" - yaml_path.write_text( - "benchmark:\n envs:\n ISL: 256\n", - encoding="utf-8", - ) - state = SimpleNamespace( - conc=0, - last_baseline={"extras": {"materialized_config": str(yaml_path)}}, - ) - assert _resolve_effective_concurrency(state) == 1 - - def test_no_last_baseline_falls_back_to_one(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - _resolve_effective_concurrency, - ) - - state = SimpleNamespace(conc=0, last_baseline=None) - assert _resolve_effective_concurrency(state) == 1 - - def test_compute_peak_from_state_uses_yaml_fallback(self, tmp_path): - """End-to-end: state.conc=0 + yaml envs.CONC=64 → peak computed with batch=64.""" - _write_synthetic_model( - tmp_path / "m", - total_size=140_000_000_000, - num_layers=80, - num_kv_heads=8, - hidden_size=8192, - num_attention_heads=64, - torch_dtype="bfloat16", - ) - yaml_path = tmp_path / "bl.yaml" - yaml_path.write_text( - "benchmark:\n envs:\n CONC: 64\n", - encoding="utf-8", - ) - state = SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="mi300x", - tp=1, - precision="bf16", - conc=0, # default — forces yaml fallback - isl=2048, - osl=512, - last_baseline={"extras": {"materialized_config": str(yaml_path)}}, - ) - peak_with_yaml = compute_peak_from_state(state) - # Drop the yaml so resolution falls through to state.conc=1. - state.last_baseline = {} - state.conc = 1 - peak_with_conc_1 = compute_peak_from_state(state) - # batch=64 amortizes weight reads, so the yaml-resolved peak is much higher. - assert peak_with_yaml > 10 * peak_with_conc_1 - - -class TestMoEBatchSaturation: - """The MoE weight-read term grows with batch as activated experts saturate toward all experts (a constant active_weight_bytes over-amortizes at high batch).""" - - _COMMON = dict( - gpu_type="mi355x", - num_gpus=1, - weight_bytes=60_000_000_000, - num_layers=48, - num_kv_heads=4, - head_dim=128, - kv_dtype_bytes=2.0, - isl=256, - osl=256, - ) - - def test_saturates_to_dense_at_high_batch(self): - # 128 experts top-8: activated_fraction = 1-(1-8/128)^B (coupon union) asymptotes to dense as B grows (~dense at B=512); strictly below dense at mid batch (see TestMoEUnionUpperBound). - moe = compute_theoretical_peak_output_tok_per_sec( - **self._COMMON, - concurrency=512, - num_experts=128, - experts_per_tok=8, - expert_weight_bytes=53_000_000_000, - ) - dense = compute_theoretical_peak_output_tok_per_sec( - **self._COMMON, - concurrency=512, # no expert fields -> weight_bytes - ) - assert moe == pytest.approx(dense, rel=1e-6) - - def test_saturation_lowers_ceiling_vs_constant_active(self): - # Old behaviour: constant active_weight_bytes regardless of batch. - old = compute_theoretical_peak_output_tok_per_sec( - **self._COMMON, - concurrency=64, - active_weight_bytes=10_000_000_000, - ) - # Fixed: expert union saturates at B=64 -> effective weight ~ full. - new = compute_theoretical_peak_output_tok_per_sec( - **self._COMMON, - concurrency=64, - num_experts=128, - experts_per_tok=8, - expert_weight_bytes=53_000_000_000, - ) - # Larger effective weight -> lower ceiling -> within% rises to a sensible value. - assert new < old - - def test_batch1_matches_active(self): - # At B=1 the saturated weight == non_expert + (k/n)*expert == active. - non_expert = 60_000_000_000 - 53_000_000_000 - active = non_expert + int((8 / 128) * 53_000_000_000) - sat = compute_theoretical_peak_output_tok_per_sec( - **self._COMMON, - concurrency=1, - num_experts=128, - experts_per_tok=8, - expert_weight_bytes=53_000_000_000, - ) - constant_active = compute_theoretical_peak_output_tok_per_sec( - **self._COMMON, - concurrency=1, - active_weight_bytes=active, - ) - assert sat == pytest.approx(constant_active, rel=1e-3) - - -class TestMoEUnionUpperBound: - """Decode ceiling must stay an upper bound on real throughput at every batch; the coupon union ``1-(1-k/n)^B`` keeps the ceiling above the measurement where the linear ``min(1,B*k/n)`` bound under-estimates it.""" - - # Real Qwen3-30B-A3B geometry (config.json) + safetensors total_size. - _NUM_LAYERS = 48 - _NUM_KV_HEADS = 4 - _HEAD_DIM = 128 - _HIDDEN = 2048 - _MOE_INTER = 768 - _NUM_EXPERTS = 128 - _EXPERTS_PER_TOK = 8 - _DTYPE_BYTES = 2.0 - _WEIGHT_BYTES = 61_064_245_248 - _EXPERT_WEIGHT_BYTES = _NUM_LAYERS * _NUM_EXPERTS * 3 * _HIDDEN * _MOE_INTER * int(_DTYPE_BYTES) - _MEASURED_CONC16_TOK_S = 1754.16 # InferenceX sweep, real measurement - - def _ceiling(self, concurrency: int) -> float: - return compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - weight_bytes=self._WEIGHT_BYTES, - num_experts=self._NUM_EXPERTS, - experts_per_tok=self._EXPERTS_PER_TOK, - expert_weight_bytes=self._EXPERT_WEIGHT_BYTES, - num_layers=self._NUM_LAYERS, - num_kv_heads=self._NUM_KV_HEADS, - head_dim=self._HEAD_DIM, - kv_dtype_bytes=self._DTYPE_BYTES, - isl=1024, - osl=1024, - concurrency=concurrency, - ) - - def test_ceiling_is_upper_bound_at_mid_batch(self): - # conc=16: real 1754 tok/s must not exceed the theoretical ceiling. - # Linear union-bound gives ~1336 (<1754) and FAILS; the coupon - # union gives ~1980 (>1754) and PASSES. - ceiling = self._ceiling(16) - assert ceiling >= self._MEASURED_CONC16_TOK_S, ( - f"ceiling {ceiling:.0f} < measured " - f"{self._MEASURED_CONC16_TOK_S} at conc=16 — roofline upper " - "bound violated (expert union over-saturated)" - ) - - def test_union_fraction_below_one_at_mid_batch(self): - # At B=16 the union of activated experts is ~64% (coupon), not the - # 100% the linear bound assumes; back it out from the ceiling and - # check it sits strictly below the dense (all-experts) ceiling. - dense = compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - weight_bytes=self._WEIGHT_BYTES, - num_layers=self._NUM_LAYERS, - num_kv_heads=self._NUM_KV_HEADS, - head_dim=self._HEAD_DIM, - kv_dtype_bytes=self._DTYPE_BYTES, - isl=1024, - osl=1024, - concurrency=16, - ) - # Un-saturated union → smaller effective weight → higher ceiling. - assert self._ceiling(16) > dense - - -# HW_SPECS table sanity -class TestHWSpecsTable: - def test_mi_series_present(self): - for key in ("mi300x", "mi325x", "mi355x"): - assert key in HW_SPECS, f"missing {key} in HW_SPECS" - spec = HW_SPECS[key] - assert spec["hbm_bw_gbps"] > 0 - assert spec["hbm_gb"] > 0 - - def test_bw_monotonic_across_generations(self): - # MI300X (HBM3) < MI325X (HBM3e) < MI355X (HBM3e refresh). - assert HW_SPECS["mi300x"]["hbm_bw_gbps"] < HW_SPECS["mi325x"]["hbm_bw_gbps"] < HW_SPECS["mi355x"]["hbm_bw_gbps"] - - def test_peak_tflops_present_for_supported_precisions(self): - # Every entry must carry a peak_tflops dict with at least bf16. - for key in ("mi300x", "mi325x", "mi355x"): - tbl = HW_SPECS[key].get("peak_tflops") - assert isinstance(tbl, dict) - assert tbl.get("bf16", 0) > 0 - assert tbl.get("fp8", 0) > 0 - - def test_mi355x_doubles_mi300x_bf16(self): - # CDNA4 ≈ 2× CDNA3 matrix peak at the same precision. - m3 = HW_SPECS["mi300x"]["peak_tflops"]["bf16"] - m5 = HW_SPECS["mi355x"]["peak_tflops"]["bf16"] - assert 1.8 < m5 / m3 < 2.2 - - def test_mi325x_compute_equals_mi300x(self): - # MI325X reuses the CDNA3 die — same matrix peak, larger HBM only. - assert HW_SPECS["mi300x"]["peak_tflops"]["bf16"] == HW_SPECS["mi325x"]["peak_tflops"]["bf16"] - - -# Two-sided roofline: T_cmp formula + RooflineBreakdown classification. -class TestResolvePeakTFLOPS: - """``_resolve_peak_tflops`` lookup with safe degrade on miss.""" - - def test_hits_known_mi355x_bf16(self): - assert _resolve_peak_tflops("mi355x", "bf16") == 2516.6 - - def test_alias_bfloat16_same_as_bf16(self): - assert _resolve_peak_tflops("mi355x", "bfloat16") == _resolve_peak_tflops("mi355x", "bf16") - - def test_mi300x_fp8_dense(self): - # Dense FP8 (not sparse-doubled). - assert _resolve_peak_tflops("mi300x", "fp8") == 2614.9 - - def test_unknown_gpu_zero(self): - assert _resolve_peak_tflops("nvidia_h100", "bf16") == 0.0 - - def test_unknown_precision_zero(self): - # MI300X does not support MXFP4 — table miss must degrade to 0. - assert _resolve_peak_tflops("mi300x", "mxfp4") == 0.0 - - def test_empty_inputs_zero(self): - assert _resolve_peak_tflops("", "bf16") == 0.0 - assert _resolve_peak_tflops("mi355x", "") == 0.0 - - -class TestComputePeakProvenance: - """roofline_provenance exposes the compute-peak convention/value/source.""" - - def test_achievable_convention_and_value(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import resolve_compute_peak_provenance - - prov = resolve_compute_peak_provenance("mi300x", "bf16") - assert prov["compute_peak_convention"] == "achievable" - assert prov["compute_peak_tflops"] == _resolve_achievable_tflops("mi300x", "bf16") # 708 - assert "achievable" in prov["compute_peak_source"].lower() - - def test_unknown_gpu_is_unknown_convention(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import resolve_compute_peak_provenance - - prov = resolve_compute_peak_provenance("h100", "bf16") - assert prov["compute_peak_convention"] == "unknown" - assert prov["compute_peak_tflops"] == 0.0 - - -class TestDiffusionComputeCeiling: - """xDiT compute-bound ceiling (config-analytical DiT FLOP model) + min() wiring.""" - - def test_compute_ceiling_hand_calc_sana_like(self): - # Sana-like DiT: L=20, H=2240, dit_params=12*L*H^2, T=1024, 20 steps, mi325x bf16. - L, H, T, steps = 20, 2240, 1024, 20 - dit_params = 12 * L * H * H - img_s = compute_diffusion_compute_img_per_sec( - gpu_type="mi325x", - num_gpus=1, - precision_tag="bf16", - dit_params=dit_params, - latent_tokens=T, - num_layers=L, - hidden_size=H, - num_steps=steps, - ) - peak = _resolve_achievable_tflops("mi325x", "bf16") * 1e12 # 843e12 - flops_per_image = steps * (2.0 * dit_params * T + 4.0 * L * T * T * H) - assert img_s == pytest.approx(peak / flops_per_image, rel=1e-9) - assert img_s == pytest.approx(15.88, rel=1e-2) # ~16 img/s, far tighter than mem ~93 - - def test_compute_ceiling_uses_achievable_not_vendor(self): - common = dict( - gpu_type="mi300x", - num_gpus=1, - precision_tag="bf16", - dit_params=1_000_000_000, - latent_tokens=1024, - num_layers=20, - hidden_size=2048, - num_steps=20, - ) - img_s = compute_diffusion_compute_img_per_sec(**common) - flops = 20 * (2.0 * 1e9 * 1024 + 4.0 * 20 * 1024**2 * 2048) - assert img_s == pytest.approx((_resolve_achievable_tflops("mi300x", "bf16") * 1e12) / flops, rel=1e-9) - assert img_s != pytest.approx((_resolve_peak_tflops("mi300x", "bf16") * 1e12) / flops, rel=1e-3) - - def test_compute_ceiling_zero_on_degenerate(self): - base = dict( - gpu_type="mi300x", - num_gpus=1, - precision_tag="bf16", - dit_params=1_000_000_000, - latent_tokens=1024, - num_layers=20, - hidden_size=2048, - num_steps=20, - ) - assert compute_diffusion_compute_img_per_sec(**{**base, "dit_params": 0}) == 0.0 - assert compute_diffusion_compute_img_per_sec(**{**base, "num_steps": 0}) == 0.0 - assert compute_diffusion_compute_img_per_sec(**{**base, "gpu_type": "h100"}) == 0.0 - - def test_read_dit_meta_from_transformer_config(self, tmp_path): - import json as _json - - td = tmp_path / "transformer" - td.mkdir() - (td / "config.json").write_text( - _json.dumps( - { - "num_layers": 20, - "num_attention_heads": 70, - "attention_head_dim": 32, - "patch_size": 1, - "sample_size": 32, - } - ) - ) - dit = _read_diffusion_dit_meta(str(tmp_path)) - assert dit is not None - dit_params, latent_tokens, num_layers, hidden = dit - assert num_layers == 20 and hidden == 2240 - assert latent_tokens == 1024 - assert dit_params == 12 * 20 * 2240 * 2240 - - def test_read_dit_meta_none_when_missing(self, tmp_path): - assert _read_diffusion_dit_meta(str(tmp_path)) is None - - def _write_dit_config(self, tmp_path, patch_size): - import json as _json - - td = tmp_path / "transformer" - td.mkdir() - (td / "config.json").write_text( - _json.dumps( - { - "num_layers": 20, - "num_attention_heads": 70, - "attention_head_dim": 32, - "patch_size": patch_size, - "sample_size": 32, - } - ) - ) - return _read_diffusion_dit_meta(str(tmp_path)) - - def test_read_dit_meta_abstains_on_tokens_for_a_3d_patch_size(self, tmp_path): - """A (t, h, w) patch marks a video denoiser and nothing here models frames, - so the sequence length is withheld -- but the weights still resolve, which - is what keeps the caller's memory bound DiT-only rather than whole-model.""" - dit = self._write_dit_config(tmp_path, [1, 2, 2]) - assert dit is not None - dit_params, latent_tokens, num_layers, hidden = dit - assert latent_tokens == 0 - assert dit_params > 0 and num_layers == 20 and hidden == 2240 - - @pytest.mark.parametrize("patch_size", [[4, 2], [2, 4]]) - def test_read_dit_meta_uses_both_axes_of_a_2d_patch_size(self, tmp_path, patch_size): - """A (h, w) patch need not be square, so collapsing it to one axis is a - 2x error either way -- the grid is (sample/ph) * (sample/pw).""" - dit = self._write_dit_config(tmp_path, patch_size) - assert dit is not None - assert dit[1] == (32 // 4) * (32 // 2) - - def test_read_dit_meta_squares_a_scalar_patch_size(self, tmp_path): - dit = self._write_dit_config(tmp_path, 2) - assert dit is not None - assert dit[1] == (32 // 2) ** 2 - - def test_read_dit_meta_survives_an_empty_patch_size(self, tmp_path): - dit = self._write_dit_config(tmp_path, []) - assert dit is not None - assert dit[1] == 32**2 - - @pytest.mark.parametrize("patch_size", [["a"], {"h": 2}, "xx", [[2]], ["a", "b", "c"], [1, 2, "c"]]) - def test_read_dit_meta_declines_a_non_numeric_patch_size(self, tmp_path, patch_size): - """``Never raises`` is the documented contract of the public entry point. - - The 3-element cases matter separately: a video patch withholds only the - token count, but a broken one is still a broken config.""" - assert self._write_dit_config(tmp_path, patch_size) is None - - def test_a_withheld_token_count_keeps_the_dit_only_memory_bound(self, monkeypatch): - """Returning ``None`` instead would fall the memory bound back to the whole - checkpoint -- text encoder and VAE included, neither of which runs per step. - Those extra bytes understate the ceiling, which in turn overstates how - close the measured run is to it.""" - import types - - import hyperloom.orchestrator.kernel.roofline_ceiling as rc - - dit_params = 12 * 20 * 2240**2 - dit_bytes = dit_params * 2 # bf16 - whole_checkpoint = 10 * dit_bytes - monkeypatch.setattr(rc, "load_model_meta", lambda *a, **k: types.SimpleNamespace(weight_bytes=whole_checkpoint)) - monkeypatch.setattr(rc, "_read_diffusion_num_steps", lambda state: 20) - - monkeypatch.setattr(rc, "_read_diffusion_dit_meta", lambda mp, **k: (dit_params, 0, 20, 2240)) - withheld = rc._compute_diffusion_breakdown_from_state(object(), self._rt()) - monkeypatch.setattr(rc, "_read_diffusion_dit_meta", lambda mp, **k: None) - declined = rc._compute_diffusion_breakdown_from_state(object(), self._rt()) - - # Compute abstains either way; only the memory bound differs. - assert withheld.cmp_tok_per_sec == 0.0 - assert withheld.mem_tok_per_sec > 0.0 - # Fewer bytes per step => a higher, and here correct, img/s ceiling. The - # 10x checkpoint gives 10x fewer images per second. - assert withheld.mem_tok_per_sec == pytest.approx(declined.mem_tok_per_sec * 10, rel=1e-6) - - def test_read_dit_meta_declines_a_non_numeric_layer_count(self, tmp_path): - """The sibling int() conversions carry the same exposure as patch_size.""" - import json as _json - - td = tmp_path / "transformer" - td.mkdir() - (td / "config.json").write_text(_json.dumps({"num_layers": ["nope"], "hidden_size": 64, "sample_size": 32})) - assert _read_diffusion_dit_meta(str(tmp_path)) is None - - def _rt(self): - return RuntimeWorkload( - model_path="/x", - gpu_type="mi325x", - precision="bf16", - framework="xdit", - tp=1, - concurrency=1, - isl=0, - osl=0, - server_args="", - ) - - def test_breakdown_takes_min_compute_bound(self, monkeypatch): - import types - import hyperloom.orchestrator.kernel.roofline_ceiling as rc - - monkeypatch.setattr(rc, "load_model_meta", lambda *a, **k: types.SimpleNamespace(weight_bytes=3_200_000_000)) - monkeypatch.setattr(rc, "_read_diffusion_num_steps", lambda state: 20) - monkeypatch.setattr(rc, "_read_diffusion_dit_meta", lambda mp, **k: (12 * 20 * 2240**2, 1024, 20, 2240)) - bd = rc._compute_diffusion_breakdown_from_state(object(), self._rt()) - assert bd.cmp_tok_per_sec > 0 - assert bd.mem_tok_per_sec > bd.cmp_tok_per_sec # memory ceiling is looser - assert bd.bound_kind == "compute" - assert bd.peak_tok_per_sec == pytest.approx(bd.cmp_tok_per_sec, rel=1e-9) - - def test_breakdown_degrades_to_memory_without_dit_config(self, monkeypatch): - import types - import hyperloom.orchestrator.kernel.roofline_ceiling as rc - - monkeypatch.setattr(rc, "load_model_meta", lambda *a, **k: types.SimpleNamespace(weight_bytes=3_200_000_000)) - monkeypatch.setattr(rc, "_read_diffusion_num_steps", lambda state: 20) - monkeypatch.setattr(rc, "_read_diffusion_dit_meta", lambda mp, **k: None) - bd = rc._compute_diffusion_breakdown_from_state(object(), self._rt()) - assert bd.cmp_tok_per_sec == 0.0 - assert bd.bound_kind == "memory" - assert bd.peak_tok_per_sec == pytest.approx(bd.mem_tok_per_sec, rel=1e-9) - - # FLUX has no sample_size; latent tokens come from the runtime resolution. - def _write_flux_configs(self, tmp_path): - import json as _json - - td = tmp_path / "transformer" - td.mkdir() - (td / "config.json").write_text( - _json.dumps( - { - "num_layers": 19, - "num_single_layers": 38, - "num_attention_heads": 24, - "attention_head_dim": 128, - "patch_size": 1, - "in_channels": 64, # 16 latent ch x 2x2 pack - } - ) - ) - vd = tmp_path / "vae" - vd.mkdir() - (vd / "config.json").write_text( - _json.dumps( - { - "block_out_channels": [128, 256, 512, 512], # 4 stages -> vae_scale 8 - "latent_channels": 16, - } - ) - ) - - def test_read_dit_meta_flux_no_sample_size_uses_resolution(self, tmp_path): - self._write_flux_configs(tmp_path) - dit = _read_diffusion_dit_meta(str(tmp_path), height=1024, width=1024) - assert dit is not None # resolution stands in for the absent sample_size - dit_params, latent_tokens, num_layers, hidden = dit - assert hidden == 24 * 128 # 3072 - # (1024 / (vae_scale 8 * pack 2))^2 = 64^2 - assert latent_tokens == 4096 - # attention term spans every block (double + single stream) - assert num_layers == 19 + 38 - # weighted params: double-stream blocks count 2x, single-stream 1x - assert dit_params == 12 * (2 * 19 + 38) * 3072 * 3072 - - def test_read_dit_meta_flux_without_resolution_is_none(self, tmp_path): - self._write_flux_configs(tmp_path) - # no sample_size AND no resolution -> degrade to memory-only (None) - assert _read_diffusion_dit_meta(str(tmp_path)) is None - - def test_read_dit_meta_sana_unchanged(self, tmp_path): - import json as _json - - td = tmp_path / "transformer" - td.mkdir() - (td / "config.json").write_text( - _json.dumps( - { - "num_layers": 20, - "num_attention_heads": 70, - "attention_head_dim": 32, - "patch_size": 1, - "sample_size": 32, - } - ) - ) - dit = _read_diffusion_dit_meta(str(tmp_path), height=1024, width=1024) - assert dit is not None - dit_params, latent_tokens, num_layers, hidden = dit - # sample_size present -> latent tokens from it (resolution ignored); no - # single-stream blocks -> params unchanged from the original estimate. - assert latent_tokens == 1024 and num_layers == 20 and hidden == 2240 - assert dit_params == 12 * 20 * 2240 * 2240 - - def test_breakdown_flux_gets_compute_ceiling(self, tmp_path, monkeypatch): - import types - import hyperloom.orchestrator.kernel.roofline_ceiling as rc - - self._write_flux_configs(tmp_path) - monkeypatch.setattr(rc, "load_model_meta", lambda *a, **k: types.SimpleNamespace(weight_bytes=24_000_000_000)) - monkeypatch.setattr(rc, "_read_diffusion_num_steps", lambda state: 28) - monkeypatch.setattr(rc, "_read_diffusion_resolution", lambda state: (1024, 1024)) - rt = RuntimeWorkload( - model_path=str(tmp_path), - gpu_type="mi325x", - precision="bf16", - framework="xdit", - tp=1, - concurrency=1, - isl=0, - osl=0, - server_args="", - ) - bd = rc._compute_diffusion_breakdown_from_state(object(), rt) - # FLUX resolves its DiT meta from the runtime resolution, so a compute - # ceiling exists instead of degrading to memory-only. - assert bd.cmp_tok_per_sec > 0 - - # Per denoising step only the DiT runs; memory ceiling uses DiT-only bytes. - def test_breakdown_flux_ceiling_when_load_model_meta_fails(self, tmp_path, monkeypatch): - # When load_model_meta returns None, the resolution-derived DiT meta - # alone still drives the compute + DiT-only memory ceiling. - import hyperloom.orchestrator.kernel.roofline_ceiling as rc - - self._write_flux_configs(tmp_path) - monkeypatch.setattr(rc, "load_model_meta", lambda *a, **k: None) - monkeypatch.setattr(rc, "_read_diffusion_num_steps", lambda state: 20) - monkeypatch.setattr(rc, "_read_diffusion_resolution", lambda state: (1024, 1024)) - rt = RuntimeWorkload( - model_path=str(tmp_path), - gpu_type="mi325x", - precision="bf16", - framework="xdit", - tp=1, - concurrency=1, - isl=0, - osl=0, - server_args="", - ) - bd = rc._compute_diffusion_breakdown_from_state(object(), rt) - assert bd.cmp_tok_per_sec > 0 # compute ceiling from DiT meta, no full-weight needed - assert bd.mem_tok_per_sec > 0 # memory ceiling from DiT-only bytes - assert bd.bound_kind in ("compute", "memory") - - def test_breakdown_memory_uses_dit_only_bytes(self, monkeypatch): - import types - import hyperloom.orchestrator.kernel.roofline_ceiling as rc - - dit_params = 12 * 20 * 2240**2 - monkeypatch.setattr(rc, "load_model_meta", lambda *a, **k: types.SimpleNamespace(weight_bytes=20_000_000_000)) - monkeypatch.setattr(rc, "_read_diffusion_num_steps", lambda state: 20) - monkeypatch.setattr(rc, "_read_diffusion_resolution", lambda state: (0, 0)) - monkeypatch.setattr(rc, "_read_diffusion_dit_meta", lambda mp, **k: (dit_params, 1024, 20, 2240)) - rt = RuntimeWorkload( - model_path="/x", - gpu_type="mi325x", - precision="bf16", - framework="xdit", - tp=1, - concurrency=1, - isl=0, - osl=0, - server_args="", - ) - bd = rc._compute_diffusion_breakdown_from_state(object(), rt) - dit_bytes = int(dit_params * 2) # bf16 - expected = rc.compute_diffusion_mem_img_per_sec( - gpu_type="mi325x", num_gpus=1, weight_bytes=dit_bytes, num_steps=20 - ) - full = rc.compute_diffusion_mem_img_per_sec( - gpu_type="mi325x", num_gpus=1, weight_bytes=20_000_000_000, num_steps=20 - ) - # DiT-only bytes, not the full 20GB checkpoint, drive the memory ceiling. - assert bd.mem_tok_per_sec == pytest.approx(expected, rel=1e-9) - assert bd.mem_tok_per_sec > full - - -class TestComputeBoundCeiling: - """T_cmp = (F_peak * G * dtype_bytes) / (2 * active_weight_bytes_B1); the divisor is the B=1 active weight (per-token compute is batch-invariant).""" - - def test_matches_hand_calculation_mi355x_bf16_a3b_like(self): - # A3B: max-achievable 1686.0 TFLOPS * 2 bytes / (2 * 6.7 GB) ≈ 251 642 tok/s. - # (uses the sustained/achievable peak, unified with the PerfModel path, - # NOT the vendor dense 2516.6 which would give ~375 612.) - cmp = compute_compute_bound_ceiling_tok_per_sec( - gpu_type="mi355x", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=6_700_000_000, - weight_bytes=61_000_000_000, - weight_dtype_bytes=2.0, - ) - assert cmp == pytest.approx(251_641.79, rel=1e-3) - - def test_uses_achievable_not_vendor_peak_no_fallback_jump(self): - # The top-down compute ceiling must use max-achievable TFLOPS (708 bf16 - # mi300x), the SAME convention as the bottom-up PerfModel path, NOT the - # vendor dense peak (1307.4) — otherwise within%/gap jump ~1.85x when the - # run falls back to this path on an incomplete config. - ach = _resolve_achievable_tflops("mi300x", "bf16") - vendor = _resolve_peak_tflops("mi300x", "bf16") - assert 0 < ach < vendor # distinct conventions (708 < 1307.4) - kwargs = dict( - gpu_type="mi300x", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=2_000_000_000, - weight_bytes=2_000_000_000, - weight_dtype_bytes=2.0, - ) - cmp = compute_compute_bound_ceiling_tok_per_sec(**kwargs) - flops_per_token = 2.0 * 2_000_000_000 / 2.0 - assert cmp == pytest.approx((ach * 1e12) / flops_per_token, rel=1e-6) - assert cmp != pytest.approx((vendor * 1e12) / flops_per_token, rel=1e-3) - - def test_scales_linearly_with_num_gpus(self): - common = dict( - gpu_type="mi355x", - precision_tag="bf16", - active_weight_bytes=10_000_000_000, - weight_bytes=10_000_000_000, - weight_dtype_bytes=2.0, - ) - one = compute_compute_bound_ceiling_tok_per_sec(num_gpus=1, **common) - eight = compute_compute_bound_ceiling_tok_per_sec(num_gpus=8, **common) - assert eight == pytest.approx(8 * one, rel=1e-9) - - def test_dense_uses_weight_bytes_when_active_missing(self): - # active_weight_bytes=0 must fall back to weight_bytes (dense). - cmp_zero = compute_compute_bound_ceiling_tok_per_sec( - gpu_type="mi355x", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=0, - weight_bytes=10_000_000_000, - weight_dtype_bytes=2.0, - ) - cmp_explicit = compute_compute_bound_ceiling_tok_per_sec( - gpu_type="mi355x", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=10_000_000_000, - weight_bytes=10_000_000_000, - weight_dtype_bytes=2.0, - ) - assert cmp_zero == pytest.approx(cmp_explicit, rel=1e-9) - - def test_moe_uses_active_b1_not_batch_saturated(self): - # Anti-regression: T_cmp must divide by active_weight_bytes (6.7G), not a batch-saturated 61G. - cmp_active = compute_compute_bound_ceiling_tok_per_sec( - gpu_type="mi355x", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=6_700_000_000, - weight_bytes=61_000_000_000, - weight_dtype_bytes=2.0, - ) - cmp_full = compute_compute_bound_ceiling_tok_per_sec( - gpu_type="mi355x", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=61_000_000_000, - weight_bytes=61_000_000_000, - weight_dtype_bytes=2.0, - ) - assert cmp_active > 8 * cmp_full - - def test_zero_on_unknown_gpu(self): - assert ( - compute_compute_bound_ceiling_tok_per_sec( - gpu_type="h100", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=10_000_000_000, - weight_bytes=10_000_000_000, - weight_dtype_bytes=2.0, - ) - == 0.0 - ) - - def test_zero_on_unknown_precision(self): - # MXFP4 on MI300X — not in HW_SPECS, must degrade. - assert ( - compute_compute_bound_ceiling_tok_per_sec( - gpu_type="mi300x", - num_gpus=1, - precision_tag="mxfp4", - active_weight_bytes=10_000_000_000, - weight_bytes=10_000_000_000, - weight_dtype_bytes=0.5, - ) - == 0.0 - ) - - def test_zero_on_zero_dtype_bytes(self): - assert ( - compute_compute_bound_ceiling_tok_per_sec( - gpu_type="mi355x", - num_gpus=1, - precision_tag="bf16", - active_weight_bytes=10_000_000_000, - weight_bytes=10_000_000_000, - weight_dtype_bytes=0.0, - ) - == 0.0 - ) - - -class TestRooflineBreakdownClassification: - """``compute_roofline_breakdown_from_state`` routes the correct ``bound_kind`` for every (mem, cmp) ordering and degrades safely.""" - - def _mock_state_and_helpers(self, monkeypatch, mem_val, cmp_val): - """Stub ``load_model_meta`` + both ceilings to control the (mem, cmp) ordering directly.""" - from hyperloom.orchestrator.kernel import roofline_ceiling - - meta = ModelMeta( - weight_bytes=10_000_000_000, - num_layers=48, - num_kv_heads=4, - head_dim=128, - weight_dtype_bytes=2.0, - active_weight_bytes=5_000_000_000, - ) - monkeypatch.setattr(roofline_ceiling, "load_model_meta", lambda *a, **kw: meta) - monkeypatch.setattr( - roofline_ceiling, - "compute_theoretical_peak_output_tok_per_sec", - lambda **kw: mem_val, - ) - monkeypatch.setattr( - roofline_ceiling, - "compute_compute_bound_ceiling_tok_per_sec", - lambda **kw: cmp_val, - ) - return SimpleNamespace( - model_path="/fake", - gpu_type="mi355x", - tp=1, - precision="bf16", - conc=8, - isl=256, - osl=256, - last_baseline={}, - ) - - def test_memory_bound_when_cmp_higher(self, monkeypatch): - state = self._mock_state_and_helpers(monkeypatch, 8000.0, 40_000.0) - br = compute_roofline_breakdown_from_state(state) - assert br.mem_tok_per_sec == 8000.0 - assert br.cmp_tok_per_sec == 40_000.0 - assert br.peak_tok_per_sec == 8000.0 - assert br.bound_kind == "memory" - - def test_compute_bound_when_cmp_lower(self, monkeypatch): - state = self._mock_state_and_helpers(monkeypatch, 8000.0, 2000.0) - br = compute_roofline_breakdown_from_state(state) - assert br.peak_tok_per_sec == 2000.0 - assert br.bound_kind == "compute" - - def test_unknown_when_both_zero(self, monkeypatch): - state = self._mock_state_and_helpers(monkeypatch, 0.0, 0.0) - br = compute_roofline_breakdown_from_state(state) - assert br.peak_tok_per_sec == 0.0 - assert br.bound_kind == "unknown" - - def test_degrades_to_memory_when_cmp_unavailable(self, monkeypatch): - # T_cmp == 0 (precision missing from HW_SPECS) → keep T_mem as the ceiling, label memory-bound. - state = self._mock_state_and_helpers(monkeypatch, 8000.0, 0.0) - br = compute_roofline_breakdown_from_state(state) - assert br.peak_tok_per_sec == 8000.0 - assert br.cmp_tok_per_sec == 0.0 - assert br.bound_kind == "memory" - - def test_compute_only_when_mem_zero(self, monkeypatch): - state = self._mock_state_and_helpers(monkeypatch, 0.0, 2000.0) - br = compute_roofline_breakdown_from_state(state) - assert br.peak_tok_per_sec == 2000.0 - assert br.bound_kind == "compute" - - def test_quantized_legacy_fallback_uses_activation_kv_dtype(self, monkeypatch): - from hyperloom.orchestrator.kernel import roofline_ceiling - - meta = ModelMeta( - weight_bytes=10_000_000_000, - num_layers=48, - num_kv_heads=4, - head_dim=128, - weight_dtype_bytes=1.0, - active_weight_bytes=5_000_000_000, - ) - captured: dict[str, float] = {} - - def _fake_mem(**kw): - captured["kv_dtype_bytes"] = kw["kv_dtype_bytes"] - return 8000.0 - - monkeypatch.setattr(roofline_ceiling, "load_model_meta", lambda *a, **kw: meta) - monkeypatch.setattr( - roofline_ceiling, - "compute_theoretical_peak_output_tok_per_sec", - _fake_mem, - ) - monkeypatch.setattr( - roofline_ceiling, - "compute_compute_bound_ceiling_tok_per_sec", - lambda **kw: 40_000.0, - ) - state = SimpleNamespace( - model_path="/fake", - gpu_type="mi355x", - tp=1, - precision="fp8", - conc=8, - isl=256, - osl=256, - last_baseline={}, - ) - br = compute_roofline_breakdown_from_state(state) - - assert br.peak_tok_per_sec == 8000.0 - assert captured["kv_dtype_bytes"] == 2.0 - - def test_backward_compat_compute_peak_from_state_returns_min( - self, - monkeypatch, - ): - # Old API must keep returning a scalar float == breakdown.peak. - state = self._mock_state_and_helpers(monkeypatch, 8000.0, 2000.0) - assert compute_peak_from_state(state) == 2000.0 - - def test_returns_empty_breakdown_on_missing_model(self, monkeypatch): - from hyperloom.orchestrator.kernel import roofline_ceiling - - monkeypatch.setattr(roofline_ceiling, "load_model_meta", lambda *a, **kw: None) - state = SimpleNamespace(model_path="", gpu_type="mi355x", tp=1) - br = compute_roofline_breakdown_from_state(state) - assert br.peak_tok_per_sec == 0.0 - assert br.bound_kind == "unknown" - - -class TestPhysicalInterpretation095726Z: - """Decode-stage MoE stays memory-bound so adding T_cmp doesn't change T_peak (within% ~77%).""" - - _A3B_META = dict( - gpu_type="mi355x", - num_gpus=1, - precision_tag="bf16", - weight_bytes=61_000_000_000, - active_weight_bytes=6_700_000_000, - weight_dtype_bytes=2.0, - ) - _ACHIEVED_TOK_S = 6244.3 - - def test_t_cmp_is_far_above_t_mem_at_decode(self): - cmp = compute_compute_bound_ceiling_tok_per_sec(**self._A3B_META) - mem = compute_theoretical_peak_output_tok_per_sec( - gpu_type="mi355x", - num_gpus=1, - weight_bytes=61_000_000_000, - active_weight_bytes=6_700_000_000, - num_experts=128, - experts_per_tok=8, - expert_weight_bytes=58_000_000_000, - num_layers=48, - num_kv_heads=4, - head_dim=128, - kv_dtype_bytes=2.0, - isl=256, - osl=256, - concurrency=64, - ) - # T_cmp ≈ 251 642 ; T_mem ≈ 8 194 ; ratio ~ 31×. - assert cmp / mem > 10 - - def test_breakdown_stays_memory_bound_with_cmp_visible(self, tmp_path): - # Real HF-like model dir so load_model_meta exercises the full code path. - _write_synthetic_model( - tmp_path / "a3b", - total_size=61_000_000_000, - num_layers=48, - num_kv_heads=4, - hidden_size=2048, - num_attention_heads=32, - torch_dtype="bfloat16", - num_experts=128, - num_experts_per_tok=8, - moe_intermediate_size=768, - ) - yaml_path = tmp_path / "bl.yaml" - yaml_path.write_text( - "benchmark:\n envs:\n CONC: 64\n", - encoding="utf-8", - ) - state = SimpleNamespace( - model_path=str(tmp_path / "a3b"), - gpu_type="mi355x", - tp=1, - precision="bf16", - conc=8, # stale SharedState default; yaml CONC=64 wins. - isl=256, - osl=256, - last_baseline={"extras": {"materialized_config": str(yaml_path)}}, - ) - br = compute_roofline_breakdown_from_state(state) - # T_cmp must be present but must NOT cap the ceiling at decode (bound stays memory). - assert br.cmp_tok_per_sec > br.mem_tok_per_sec - # PerfModel (FusedMoE coupon formula) drives peak_tok_per_sec and must stay > measured. - assert br.bound_kind == "memory" - assert br.peak_tok_per_sec >= self._ACHIEVED_TOK_S - # within% recomputed from achieved must still match the ~77% anchor. - within_pct = 100.0 * self._ACHIEVED_TOK_S / br.peak_tok_per_sec - assert 70.0 < within_pct < 85.0 - - -class TestDeepSeekV3ConfigAliases: - """DeepSeek-V3-derived models use HF aliases (``n_routed_experts``, ``quant_method=fp8`` + bf16 ``dtype``) that must be resolved so ceilings stay accurate.""" - - def test_n_routed_experts_alias_equivalent_to_num_experts(self, tmp_path): - # Identical geometry differing only in the routed-expert field name must give an identical breakdown. - common = dict( - total_size=26_000_000_000, - num_layers=26, - num_kv_heads=32, - hidden_size=1536, - num_attention_heads=32, - torch_dtype="bfloat16", - num_experts_per_tok=4, - moe_intermediate_size=1280, - ) - _write_synthetic_model(tmp_path / "qwen_alias", num_experts=64, **common) - _write_synthetic_model(tmp_path / "ds_alias", n_routed_experts=64, **common) - qwen = load_model_meta(str(tmp_path / "qwen_alias")) - ds = load_model_meta(str(tmp_path / "ds_alias")) - assert qwen is not None and ds is not None - # Identical MoE decomposition for both alias variants. - assert qwen.num_experts == ds.num_experts == 64 - assert qwen.experts_per_tok == ds.experts_per_tok == 4 - assert qwen.expert_weight_bytes == ds.expert_weight_bytes - assert qwen.active_weight_bytes == ds.active_weight_bytes - - def test_fp8_quantization_config_drives_weight_dtype(self, tmp_path): - # quant_method=fp8 + dtype=bfloat16 (activation): without the quant short-circuit, load_model_meta over-counts weight bytes 2x. - _write_synthetic_model( - tmp_path / "gigachat", - total_size=12_000_000_000, - num_layers=26, - num_kv_heads=32, - hidden_size=1536, - num_attention_heads=32, - torch_dtype="bfloat16", # HF-standard activation dtype - dtype="bfloat16", # DeepSeek-V3 redundancy - quant_method="fp8", # block-fp8 weight quant - n_routed_experts=64, - num_experts_per_tok=4, - moe_intermediate_size=1280, - ) - meta = load_model_meta(str(tmp_path / "gigachat")) - assert meta is not None - # fp8 -> 1 byte per param (not 2 from the bf16 activation dtype). - assert meta.weight_dtype_bytes == 1.0 - # MoE decomposition still fires via n_routed_experts. - assert meta.num_experts == 64 - assert meta.experts_per_tok == 4 - assert meta.expert_weight_bytes > 0 - - def test_quant_method_overrides_torch_dtype(self, tmp_path): - # quant_method wins even when torch_dtype is inconsistent. - _write_synthetic_model( - tmp_path / "fp8_model", - total_size=12_000_000_000, - num_layers=4, - num_kv_heads=8, - hidden_size=1024, - num_attention_heads=16, - torch_dtype="bfloat16", # would yield 2.0 if read directly - quant_method="fp8", - ) - meta = load_model_meta(str(tmp_path / "fp8_model")) - assert meta is not None - assert meta.weight_dtype_bytes == 1.0 # fp8 wins - - def test_dtype_field_fallback_when_torch_dtype_missing(self, tmp_path): - # With no torch_dtype/quant_method, the DeepSeek-style ``dtype`` field is consulted before the precision_hint. - _write_synthetic_model( - tmp_path / "ds_no_qd", - total_size=4_000_000_000, - num_layers=4, - num_kv_heads=4, - hidden_size=512, - num_attention_heads=8, - torch_dtype="bfloat16", - ) - # Strip torch_dtype manually so we exercise the dtype fallback. - import json as _json - - cfg_path = tmp_path / "ds_no_qd" / "config.json" - cfg = _json.loads(cfg_path.read_text()) - cfg.pop("torch_dtype", None) - cfg["dtype"] = "float16" - cfg_path.write_text(_json.dumps(cfg)) - meta = load_model_meta(str(tmp_path / "ds_no_qd")) - assert meta is not None - assert meta.weight_dtype_bytes == 2.0 # fp16 - - def test_mxfp4_alias_in_dtype_table(self): - # ``mxfp4`` must map to 0.5 byte/param to match the HW_SPECS key (else the ceiling arithmetic falls back to bf16). - assert _resolve_dtype_bytes("mxfp4") == 0.5 - - def test_num_local_experts_alias_equivalent_to_num_experts(self, tmp_path): - # gpt-oss (GptOssForCausalLM) writes the routed-expert count under ``num_local_experts``; the alias must resolve so it isn't treated as dense. - # total_size is the ~260 GB bf16-equivalent (above the routed pool) so the MoE decomposition path executes. - common = dict( - total_size=260_000_000_000, - num_layers=36, - num_kv_heads=8, - hidden_size=2880, - num_attention_heads=64, - torch_dtype="bfloat16", - num_experts_per_tok=4, - moe_intermediate_size=2880, - ) - _write_synthetic_model(tmp_path / "qwen_alias", num_experts=128, **common) - _write_synthetic_model(tmp_path / "gptoss_alias", num_local_experts=128, **common) - qwen = load_model_meta(str(tmp_path / "qwen_alias")) - gptoss = load_model_meta(str(tmp_path / "gptoss_alias")) - assert qwen is not None and gptoss is not None - assert qwen.num_experts == gptoss.num_experts == 128 - assert qwen.experts_per_tok == gptoss.experts_per_tok == 4 - assert qwen.expert_weight_bytes == gptoss.expert_weight_bytes - assert qwen.active_weight_bytes == gptoss.active_weight_bytes - # And both must be < weight_bytes (MoE shrinks the divisor). - assert qwen.active_weight_bytes < qwen.weight_bytes - - -# --------------------------------------------------------------------------- -# PerfModel bottom-up breakdown. -# --------------------------------------------------------------------------- - - -class TestPerfModelBreakdown: - """Smoke tests for compute_roofline_from_perfmodel and related helpers.""" - - def _make_meta(self) -> "ModelMeta": - """Minimal dense Llama-style ModelMeta.""" - from hyperloom.orchestrator.kernel.roofline_ceiling import ModelMeta - - return ModelMeta( - weight_bytes=int(13e9), # ~13 GB weight - num_layers=32, - num_kv_heads=8, - head_dim=128, - weight_dtype_bytes=2.0, - active_weight_bytes=int(13e9), - hidden_size=4096, - intermediate_size=11008, - vocab_size=32000, - num_attention_heads=32, - ) - - def test_returns_none_for_unknown_gpu(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_from_perfmodel, - ) - - meta = self._make_meta() - result = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="unknown_gpu_xyz", - concurrency=8, - isl=1024, - osl=512, - ) - assert result is None - - def test_returns_none_when_meta_missing_hidden(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - ModelMeta, - compute_roofline_from_perfmodel, - ) - - meta = ModelMeta( - weight_bytes=int(13e9), - num_layers=32, - num_kv_heads=8, - head_dim=128, - weight_dtype_bytes=2.0, - # hidden_size=0 -> insufficient - ) - result = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi300x", - concurrency=8, - isl=1024, - osl=512, - ) - assert result is None - - def test_hw_specs_achievable_coverage(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - HW_SPECS_ACHIEVABLE, - _resolve_achievable_tflops, - ) - - for gpu in ("mi300x", "mi325x", "mi355x"): - assert gpu in HW_SPECS_ACHIEVABLE - # MI300X bf16 achievable must be less than vendor-quoted (708 < 1307) - assert _resolve_achievable_tflops("mi300x", "bf16") == 708.0 - assert _resolve_achievable_tflops("mi355x", "bf16") == 1686.0 - assert _resolve_achievable_tflops("mi300x", "fp8") == 1273.0 - - def test_perfmodel_breakdown_when_traceLens_available(self): - """When model metadata is complete the result is a valid PerfModelBreakdown.""" - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_from_perfmodel, - PerfModelBreakdown, - ) - - meta = self._make_meta() - result = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi300x", - concurrency=16, - isl=1024, - osl=1024, - ) - assert result is not None - assert isinstance(result, PerfModelBreakdown) - # Decode ceiling must be positive - assert result.decode_tok_per_s > 0 - # Prefill is compute-bound for S=1024, so also > 0 - assert result.prefill_tok_per_s > 0 - # Must have ops (at least q/k/v/o + sdpa) - assert len(result.ops) >= 5 - # Achievable peaks must match MI300X bf16 - assert result.peak_achievable_tflops == pytest.approx(708.0 * 1) - assert result.hbm_bw_gbps == pytest.approx(5300.0) - # All ops have non-negative pct_time summing to ~1 - assert all(0.0 <= op.pct_time <= 1.0 + 1e-6 for op in result.ops) - total_pct = sum(op.pct_time for op in result.ops) - assert total_pct == pytest.approx(1.0, abs=1e-6) - # bound_kind must be one of the valid values - assert result.bound_kind in ("memory", "compute", "unknown") - - -# --------------------------------------------------------------------------- -# PerfModel MoE formula correctness -# --------------------------------------------------------------------------- - - -class TestPerfModelMoE: - """Verify that compute_roofline_from_perfmodel uses moe_intermediate_size - for MoE models and gives a valid upper bound on measured throughput.""" - - def _make_qwen3_a3b_meta(self) -> "ModelMeta": - """Qwen3-30B-A3B geometry: 128 experts, top-8, moe_inter=768.""" - from hyperloom.orchestrator.kernel.roofline_ceiling import ModelMeta - - return ModelMeta( - weight_bytes=61_064_245_248, - num_layers=48, - num_kv_heads=4, - head_dim=128, - weight_dtype_bytes=2.0, - active_weight_bytes=6_700_000_000, - num_experts=128, - experts_per_tok=8, - expert_weight_bytes=int(48 * 128 * 3 * 2048 * 768 * 2), - hidden_size=2048, - intermediate_size=6144, # present in config but not used for MoE FFN - moe_intermediate_size=768, - vocab_size=151936, - num_attention_heads=32, - ) - - def test_moe_ffn_uses_moe_intermediate_size(self): - """With moe_intermediate_size set, PerfModel uses FusedMoE formula - (coupon E_active) not the per-token fixed-topk formula.""" - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_from_perfmodel, - ) - - meta = self._make_qwen3_a3b_meta() - result = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi300x", - concurrency=1, - isl=1024, - osl=1024, - ) - assert result is not None - # At batch=1: E_active ≈ topk = 8 (coupon formula == B=1 estimate), - # so decode is memory-bound (weight IO dominates). - assert result.bound_kind == "memory" - # The "moe_fused" op should appear in the op breakdown - moe_ops = [op for op in result.ops if op.name == "moe_fused"] - assert len(moe_ops) == 1 - - def test_moe_ceiling_is_upper_bound_conc16(self): - """PerfModel (FusedMoE coupon formula) ceiling for Qwen3-30B-A3B at conc=16 - must be >= 1754 tok/s (real InferenceX measurement on MI300X).""" - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_from_perfmodel, - ) - - meta = self._make_qwen3_a3b_meta() - result = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi300x", - concurrency=16, - isl=1024, - osl=1024, - ) - assert result is not None - assert result.decode_tok_per_s >= 1754.16, ( - f"PerfModel ceiling {result.decode_tok_per_s:.1f} < measured 1754.16 tok/s" - ) - - def test_moe_ceiling_is_upper_bound_conc64(self): - """At conc=64, FusedMoE coupon formula should give a tighter (lower) ceiling - than the B=1 per-token estimate because E_active grows toward num_experts.""" - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_from_perfmodel, - ) - - meta = self._make_qwen3_a3b_meta() - result_1 = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi300x", - concurrency=1, - isl=512, - osl=512, - ) - result_64 = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi300x", - concurrency=64, - isl=512, - osl=512, - ) - assert result_1 is not None and result_64 is not None - # At conc=64, more expert weights are loaded (E_active > topk), - # so bytes grow → ceiling drops vs conc=1. - assert result_64.decode_tok_per_s < result_1.decode_tok_per_s * 64, ( - "conc=64 ceiling should not be 64× higher than conc=1 (MoE weight saturation)" - ) - - -class TestPerfModelTransparentReplacement: - """compute_roofline_breakdown_from_state must use PerfModel peak when - model config is complete, and fall back to legacy when it is not.""" - - def test_uses_perfmodel_peak_when_config_available(self, tmp_path): - """For a known GPU + complete config, peak comes from PerfModel.""" - from types import SimpleNamespace - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_breakdown_from_state, - compute_roofline_from_perfmodel, - load_model_meta, - ) - - _write_synthetic_model( - tmp_path / "m", - total_size=int(13e9), - num_layers=32, - num_kv_heads=8, - hidden_size=4096, - num_attention_heads=32, - torch_dtype="bfloat16", - ) - state = SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="mi300x", - tp=1, - precision="bf16", - conc=8, - isl=256, - osl=256, - last_baseline={}, - ) - br = compute_roofline_breakdown_from_state(state) - meta = load_model_meta(str(tmp_path / "m")) - pm_bd = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi300x", - concurrency=8, - isl=256, - osl=256, - precision_tag="bf16", - ) - assert pm_bd is not None - assert br.peak_tok_per_sec == pytest.approx(pm_bd.decode_tok_per_s, rel=1e-6) - assert br.mem_tok_per_sec == pytest.approx(pm_bd.decode_mem_tok_per_s, rel=1e-6) - assert br.cmp_tok_per_sec == pytest.approx(pm_bd.decode_cmp_tok_per_s, rel=1e-6) - assert br.bound_kind == pm_bd.bound_kind - - def test_moe_uses_perfmodel_peak(self, tmp_path): - """MoE models now also use PerfModel (FusedMoE coupon formula). - For Qwen3-30B-A3B at conc=64, the FusedMoE ceiling must be above - the measured 6244 tok/s (from TestPhysicalInterpretation095726Z).""" - from types import SimpleNamespace - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_breakdown_from_state, - compute_roofline_from_perfmodel, - load_model_meta, - ) - - _write_qwen3_moe_model(tmp_path / "a3b", total_size=61_000_000_000) - yaml_path = tmp_path / "bl.yaml" - yaml_path.write_text("benchmark:\n envs:\n CONC: 64\n", encoding="utf-8") - state = SimpleNamespace( - model_path=str(tmp_path / "a3b"), - gpu_type="mi355x", - tp=1, - precision="bf16", - conc=8, - isl=256, - osl=256, - last_baseline={"extras": {"materialized_config": str(yaml_path)}}, - ) - br = compute_roofline_breakdown_from_state(state) - meta = load_model_meta(str(tmp_path / "a3b")) - pm_bd = compute_roofline_from_perfmodel( - meta=meta, - gpu_type="mi355x", - concurrency=64, - isl=256, - osl=256, - precision_tag="bf16", - ) - assert pm_bd is not None - assert br.peak_tok_per_sec == pytest.approx(pm_bd.decode_tok_per_s, rel=1e-6) - assert br.mem_tok_per_sec == pytest.approx(pm_bd.decode_mem_tok_per_s, rel=1e-6) - assert br.cmp_tok_per_sec == pytest.approx(pm_bd.decode_cmp_tok_per_s, rel=1e-6) - # FusedMoE ceiling must remain above the measured 6244.3 tok/s - assert br.peak_tok_per_sec >= 6244.3, f"MoE PerfModel ceiling {br.peak_tok_per_sec:.1f} < measured 6244.3 tok/s" - - def test_falls_back_to_legacy_for_unknown_gpu(self, tmp_path): - """For an unknown GPU (PerfModel returns None), legacy ceiling is used.""" - from types import SimpleNamespace - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - compute_roofline_breakdown_from_state, - ) - - _write_synthetic_model( - tmp_path / "m", - total_size=int(13e9), - num_layers=32, - num_kv_heads=8, - hidden_size=4096, - num_attention_heads=32, - torch_dtype="bfloat16", - ) - state = SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="unknown_gpu_xyz", - tp=1, - precision="bf16", - conc=8, - isl=256, - osl=256, - last_baseline={}, - ) - # Legacy ceiling for unknown GPU also returns 0 (no HW spec), so just - # verify the function returns without raising. - br = compute_roofline_breakdown_from_state(state) - assert br.peak_tok_per_sec >= 0.0 - - -class TestRuntimeDtypeResolution: - """Roofline ceiling must use the dtype the run actually read, not the - on-disk ``torch_dtype``. Covers the ``within_roofline_pct > 100`` bug - where a float32 checkpoint served with ``--quantization fp8`` produced - a ceiling below the measured throughput.""" - - def _dense_fp32_state(self, tmp_path, extra_args: str): - # float32 dense checkpoint (4 B/param on disk), served fp8 at runtime. - # Non-empty args model an accepted optimized variant, so current_best - # carries a tput (the arm the ceiling is compared against). - _write_synthetic_model( - tmp_path / "m", - total_size=32_000_000_000, - num_layers=48, - num_kv_heads=8, - hidden_size=5120, - num_attention_heads=40, - torch_dtype="float32", - ) - current_best = {"extra_server_args": extra_args} - if extra_args: - current_best["tput"] = 1000.0 - return SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="mi300x", - tp=1, - precision="fp8", - conc=64, - isl=1024, - osl=1024, - last_baseline={}, - current_best=current_best, - ) - - def test_quantization_fp8_arg_scales_weight_to_one_byte(self, tmp_path): - state = self._dense_fp32_state(tmp_path, "--quantization fp8") - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 1.0 - assert rt.quantization == "fp8" - assert rt.source == "server_args_quantization" - assert rt.compute_precision_tag == "fp8" - - def test_weight_bytes_rescaled_by_runtime_dtype(self, tmp_path): - state = self._dense_fp32_state(tmp_path, "--quantization fp8") - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - scaled = apply_runtime_dtype(meta, rt) - # float32 (4 B) -> fp8 (1 B): weight bytes drop 4x. - assert scaled.weight_dtype_bytes == 1.0 - assert scaled.weight_bytes == meta.weight_bytes // 4 - - def test_fp8_ceiling_exceeds_no_quant_ceiling(self, tmp_path): - """Runtime fp8 ceiling must be strictly higher than the no-quant - (bf16-weight) ceiling, so within% drops back below 100.""" - st_fp8 = self._dense_fp32_state(tmp_path, "--quantization fp8") - st_cfg = self._dense_fp32_state(tmp_path, "") # no quant -> bf16 weight - peak_fp8 = compute_roofline_breakdown_from_state(st_fp8).peak_tok_per_sec - peak_cfg = compute_roofline_breakdown_from_state(st_cfg).peak_tok_per_sec - assert peak_fp8 > peak_cfg > 0 - - def test_precision_fp8_without_quant_does_not_shrink_weight(self, tmp_path): - """A workload tagged precision=fp8 whose run did NOT pass - --quantization must NOT be modeled as fp8 weights: the server - serves bf16, so the weight term stays 2 B (regression for the - baseline within% under-report).""" - state = self._dense_fp32_state(tmp_path, "") # no quant args - assert state.precision == "fp8" # workload tag still fp8 - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - # Weight floored at bf16 (fp32 downcast), NOT fp8. - assert rt.weight_dtype_bytes == 2.0 - assert rt.quantization == "none" - assert rt.source == "config_torch_dtype" - - def test_baseline_arm_ignores_optimized_quant_args(self, tmp_path): - """When achieved comes from the baseline arm (no current_best tput), - an optimized current_best's --quantization fp8 must NOT leak onto - the baseline ceiling.""" - state = self._dense_fp32_state(tmp_path, "") - # current_best has fp8 args but no tput -> not the achieved arm. - state.current_best = {"extra_server_args": "--quantization fp8"} - state.last_baseline = {} - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.quantization == "none" - assert rt.weight_dtype_bytes == 2.0 - - def test_no_quantization_keeps_config_dtype(self, tmp_path): - # bf16 checkpoint, no quant args -> weight stays 2 B. - _write_synthetic_model( - tmp_path / "m", - total_size=16_000_000_000, - num_layers=48, - num_kv_heads=8, - hidden_size=5120, - num_attention_heads=40, - torch_dtype="bfloat16", - ) - state = SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="mi300x", - tp=1, - precision="", - conc=64, - isl=1024, - osl=1024, - last_baseline={}, - current_best={}, - ) - meta = load_model_meta(state.model_path) - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 2.0 - assert rt.quantization == "none" - - def test_prequantized_moe_fp8_not_double_scaled(self, tmp_path): - # On-disk fp8 MoE checkpoint: total_size already reflects fp8. - _write_synthetic_model( - tmp_path / "m", - total_size=480_000_000_000, - num_layers=62, - num_kv_heads=8, - hidden_size=6144, - num_attention_heads=48, - num_experts=256, - num_experts_per_tok=8, - moe_intermediate_size=1536, - quant_method="fp8", - ) - state = SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="mi300x", - tp=4, - precision="fp8", - conc=64, - isl=1024, - osl=1024, - last_baseline={}, - current_best={}, - ) - meta = load_model_meta(state.model_path, precision_hint="fp8") - assert meta.weight_dtype_bytes == 1.0 # quant_method already fp8 - rt = resolve_runtime_dtype(state, meta) - assert rt.source == "quantization_config" - scaled = apply_runtime_dtype(meta, rt) - # No double-scaling: weight bytes unchanged. - assert scaled.weight_bytes == meta.weight_bytes - - def test_dtype_arg_without_quant_sets_weight_dtype(self, tmp_path): - state = self._dense_fp32_state(tmp_path, "--dtype bfloat16") - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 2.0 - assert rt.source == "server_args_dtype" - assert rt.activation_dtype_bytes == 2.0 - - def test_activation_dtype_floored_at_bf16(self, tmp_path): - # Even with fp8 weights, activations stay >= 2 B. - state = self._dense_fp32_state(tmp_path, "--quantization fp8") - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.activation_dtype_bytes >= 2.0 - - def test_eq_form_arg_parsing(self, tmp_path): - # --quantization=fp8 (= form) parses identically to space form. - state = self._dense_fp32_state(tmp_path, "--quantization=fp8") - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 1.0 - assert rt.quantization == "fp8" - - def _baseline_yaml( - self, - tmp_path, - env_key: str, - args: str, - *, - model: str = "", - precision: str = "fp8", - tp: int = 1, - conc: int = 64, - isl: int = 1024, - osl: int = 1024, - framework: str = "sglang", - runner_type: str = "mi300x", - ) -> str: - """Write a materialized baseline yaml carrying ``EXTRA_*_ARGS`` and - return its path (the shape the executor stamps post-baseline).""" - import yaml as _yaml # type: ignore[reportMissingModuleSource] - - envs = { - "TP": tp, - "CONC": conc, - "ISL": isl, - "OSL": osl, - env_key: args, - } - benchmark = { - "framework": framework, - "precision": precision, - "runner_type": runner_type, - "envs": envs, - } - if model: - benchmark["model"] = model - cfg = {"benchmark": benchmark} - path = tmp_path / "baseline.yaml" - path.write_text(_yaml.safe_dump(cfg), encoding="utf-8") - return str(path) - - def test_baseline_yaml_args_resolved_when_current_best_empty(self, tmp_path): - """Baseline-only run: --quantization fp8 lives only in the yaml's - EXTRA_SGLANG_ARGS (current_best carries no extra_server_args), so - dtype resolution must read it from the materialized config.""" - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_SGLANG_ARGS", - "--quantization fp8", - ) - state = self._dense_fp32_state(tmp_path, "") # no top-level args - state.current_best = {"action": "baseline", "tput": 100.0} - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 1.0 - assert rt.quantization == "fp8" - assert rt.source == "server_args_quantization" - - def test_baseline_yaml_args_vllm_env_key(self, tmp_path): - # The vllm framework routes flags through EXTRA_VLLM_ARGS. - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_VLLM_ARGS", - "--quantization fp8", - ) - state = self._dense_fp32_state(tmp_path, "") - state.current_best = {} - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 1.0 - - def test_top_level_args_win_over_baseline_yaml(self, tmp_path): - """An optimized current_best (real extra_server_args) takes - precedence over the baseline yaml fallback.""" - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_SGLANG_ARGS", - "--dtype bfloat16", - ) - state = self._dense_fp32_state(tmp_path, "--quantization fp8") - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - # current_best's fp8 wins; the yaml's bf16 is never consulted. - assert rt.weight_dtype_bytes == 1.0 - assert rt.quantization == "fp8" - - def test_optimized_overlay_overrides_same_baseline_flag(self, tmp_path): - """When both baseline and current_best set the same flag, the - optimized overlay must win.""" - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_SGLANG_ARGS", - "--dtype float16", - ) - state = self._dense_fp32_state(tmp_path, "--dtype bfloat16") - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.source == "server_args_dtype" - assert rt.weight_dtype_tag == "bfloat16" - - def test_optimized_extra_envs_server_args_are_resolved(self, tmp_path): - """Accepted env-only configs can carry server args in EXTRA_*_ARGS.""" - state = self._dense_fp32_state(tmp_path, "") - state.current_best = { - "tput": 1000.0, - "extra_envs": {"EXTRA_SGLANG_ARGS": "--quantization fp8"}, - } - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 1.0 - assert rt.quantization == "fp8" - assert rt.source == "server_args_quantization" - - def test_optimized_extra_envs_override_top_level_server_args(self, tmp_path): - """materialize_config_with_envs applies extra_envs after - extra_server_args, so an EXTRA_*_ARGS env wins for dtype resolution.""" - state = self._dense_fp32_state(tmp_path, "--dtype bfloat16") - state.current_best["extra_envs"] = {"EXTRA_SGLANG_ARGS": "--quantization fp8"} - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.weight_dtype_bytes == 1.0 - assert rt.quantization == "fp8" - - def test_optimized_extra_envs_replace_baseline_server_args(self, tmp_path): - """extra_envs.EXTRA_*_ARGS replaces the YAML server-args env.""" - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_SGLANG_ARGS", - "--quantization fp8", - ) - state = self._dense_fp32_state(tmp_path, "") - state.current_best = { - "tput": 1000.0, - "extra_envs": {"EXTRA_SGLANG_ARGS": "--dtype bfloat16"}, - } - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - meta = load_model_meta(state.model_path, precision_hint="fp8") - rt = resolve_runtime_dtype(state, meta) - assert rt.source == "server_args_dtype" - assert rt.weight_dtype_tag == "bfloat16" - assert rt.quantization == "none" - - def test_runtime_workload_uses_baseline_yaml_fields(self, tmp_path): - state = self._dense_fp32_state(tmp_path, "") - runtime_model = state.model_path - state.model_path = "/wrong/model" - state.tp = 99 - state.conc = 8 - state.isl = 1 - state.osl = 1 - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_SGLANG_ARGS", - "--dtype bfloat16", - model=runtime_model, - precision="bf16", - tp=4, - conc=64, - isl=1024, - osl=2048, - framework="sglang", - ) - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - - runtime = resolve_runtime_workload(state) - assert runtime.model_path == runtime_model - assert runtime.precision == "bf16" - assert runtime.framework == "sglang" - assert runtime.tp == 4 - assert runtime.concurrency == 64 - assert runtime.isl == 1024 - assert runtime.osl == 2048 - assert runtime.server_args == "--dtype bfloat16" - - def test_runtime_workload_uses_real_gpu_over_magpie_runner(self, tmp_path): - """MI325X runs via Magpie's mi300x runner but roofline uses MI325X.""" - state = self._dense_fp32_state(tmp_path, "") - state.gpu_type = "mi325x" - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_SGLANG_ARGS", - "--dtype bfloat16", - runner_type="mi300x", - ) - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - - runtime = resolve_runtime_workload(state) - assert runtime.gpu_type == "mi325x" - - def test_ceiling_uses_baseline_yaml_model_when_state_is_stale(self, tmp_path): - state = self._dense_fp32_state(tmp_path, "") - runtime_model = state.model_path - state.model_path = "/wrong/model" - state.tp = 0 - state.conc = 0 - state.isl = 0 - state.osl = 0 - cfg_path = self._baseline_yaml( - tmp_path, - "EXTRA_SGLANG_ARGS", - "--dtype bfloat16", - model=runtime_model, - precision="bf16", - tp=1, - conc=32, - isl=512, - osl=512, - ) - state.last_baseline = {"extras": {"materialized_config": cfg_path}} - - bd = compute_roofline_breakdown_from_state(state) - assert bd.peak_tok_per_sec > 0 - assert bd.bound_kind in {"memory", "compute"} - - def test_missing_baseline_yaml_degrades_safely(self, tmp_path): - # Unreadable materialized_config -> no crash, falls back to config. - state = self._dense_fp32_state(tmp_path, "") - state.precision = "" - state.current_best = {} - state.last_baseline = {"extras": {"materialized_config": "/no/such.yaml"}} - meta = load_model_meta(state.model_path) - rt = resolve_runtime_dtype(state, meta) - assert rt.quantization == "none" - - -class TestArmPinnedPrecision: - """``arm`` pins ceiling precision: baseline keeps baseline dtype even after a fp8 current_best is promoted.""" - - def _bf16_baseline_with_fp8_best(self, tmp_path): - _write_synthetic_model( - tmp_path / "m", - total_size=140_000_000_000, - num_layers=80, - num_kv_heads=8, - hidden_size=8192, - num_attention_heads=64, - torch_dtype="bfloat16", - ) - # Baseline yaml carries NO quantization (bf16 weights). - yaml_path = tmp_path / "bl.yaml" - yaml_path.write_text( - "benchmark:\n envs:\n CONC: 32\n", - encoding="utf-8", - ) - return SimpleNamespace( - model_path=str(tmp_path / "m"), - gpu_type="mi300x", - tp=1, - precision="bf16", - conc=32, - isl=2048, - osl=512, - last_baseline={"extras": {"materialized_config": str(yaml_path)}}, - # Promoted optimized arm quantizes weights to fp8. - current_best={ - "tput": 2000.0, - "extra_server_args": "--quantization fp8", - }, - ) - - def test_baseline_arm_ignores_current_best_fp8(self, tmp_path): - state = self._bf16_baseline_with_fp8_best(tmp_path) - meta = load_model_meta(state.model_path) - rt = resolve_runtime_dtype(state, meta, arm="baseline") - # Baseline yaml had no --quantization -> weights stay bf16 (2 bytes). - assert rt.weight_dtype_bytes == 2.0 - assert rt.quantization == "none" - - def test_current_best_arm_picks_up_fp8(self, tmp_path): - state = self._bf16_baseline_with_fp8_best(tmp_path) - meta = load_model_meta(state.model_path) - rt = resolve_runtime_dtype(state, meta, arm="current_best") - assert rt.weight_dtype_bytes == 1.0 - assert rt.quantization == "fp8" - - def test_baseline_arm_ceiling_below_fp8_arm(self, tmp_path): - state = self._bf16_baseline_with_fp8_best(tmp_path) - baseline_peak = compute_roofline_breakdown_from_state( - state, - arm="baseline", - ).peak_tok_per_sec - best_peak = compute_roofline_breakdown_from_state( - state, - arm="current_best", - ).peak_tok_per_sec - # fp8 halves weight IO, so the best-arm ceiling must be higher. - assert baseline_peak > 0 - assert best_peak > baseline_peak - - -class TestReadBaselineServerArgs: - """Public ``read_baseline_server_args`` reads the baseline yaml's flags.""" - - def test_reads_server_args_from_yaml(self, tmp_path): - import yaml as _yaml # type: ignore[reportMissingModuleSource] - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - read_baseline_server_args, - ) - - yaml_path = tmp_path / "bl.yaml" - yaml_path.write_text( - _yaml.safe_dump( - { - "benchmark": { - "envs": { - "EXTRA_SGLANG_ARGS": "--attention-backend AITER", - }, - }, - } - ), - encoding="utf-8", - ) - state = SimpleNamespace( - last_baseline={"extras": {"materialized_config": str(yaml_path)}}, - ) - assert read_baseline_server_args(state) == "--attention-backend AITER" - - def test_returns_empty_when_yaml_missing(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - read_baseline_server_args, - ) - - state = SimpleNamespace( - last_baseline={"extras": {"materialized_config": "/no/such.yaml"}}, - ) - assert read_baseline_server_args(state) == "" - - def test_falls_back_to_last_baseline_payload_when_yaml_missing(self): - from hyperloom.orchestrator.kernel.roofline_ceiling import ( - read_baseline_server_args, - ) - - state = SimpleNamespace( - last_baseline={ - "extras": {"materialized_config": "/no/such.yaml"}, - "extra_envs": { - "EXTRA_VLLM_ARGS": "--kv-cache-dtype fp8_e4m3 --gpu-memory-utilization 0.85", - }, - }, - ) - - assert read_baseline_server_args(state) == "--kv-cache-dtype fp8_e4m3 --gpu-memory-utilization 0.85" - - -class TestPreludeRooflineWarmReplayIntegration: - """End-to-end-ish: a delayed PRELUDE roofline running AFTER warm-replay has - promoted an fp8 current_best must still record snapshot[0] as the baseline - arm — real RooflineExecutor + real SharedState + real ceiling math, only - profile/trace_analyze are faked. This reproduces the original bug scenario. - """ - - def _bf16_baseline_yaml(self, tmp_path) -> str: - import yaml as _yaml # type: ignore[reportMissingModuleSource] - - yaml_path = tmp_path / "baseline.yaml" - yaml_path.write_text( - _yaml.safe_dump( - { - "benchmark": { - "framework": "sglang", - "precision": "bf16", - "runner_type": "mi300x", - "model": str(tmp_path / "m"), - "envs": {"TP": 1, "CONC": 32, "ISL": 2048, "OSL": 512}, - }, - } - ), - encoding="utf-8", - ) - return str(yaml_path) - - def _run_prelude_after_fp8_promotion(self, tmp_path): - import asyncio - from unittest.mock import patch - - from hyperloom.orchestrator.actions.executors.roofline import ( - RooflineExecutor, - ) - from hyperloom.orchestrator.state.shared_state import SharedState - from hyperloom.orchestrator.loop.sub_agent_runner import ( - RunnerContext, - ) - from hyperloom.orchestrator.state.task_registry import Task - - _write_synthetic_model( - tmp_path / "m", - total_size=140_000_000_000, - num_layers=80, - num_kv_heads=8, - hidden_size=8192, - num_attention_heads=64, - torch_dtype="bfloat16", - ) - yaml_path = self._bf16_baseline_yaml(tmp_path) - md = tmp_path / "analysis.md" - md.write_text("# Executive Summary\nbody\n", encoding="utf-8") - - # Real SharedState: baseline landed (bf16), then warm-replay promoted - # an optimized fp8 current_best BEFORE the deferred prelude runs. - state = SharedState() - state.baseline_tput = 527.5 - state.gpu_type = "mi300x" - state.last_baseline = { - "extras": {"materialized_config": yaml_path}, - "output_throughput": 527.5, - } - state.current_best = { - "action": "warm_replay", - "tput": 900.0, - "extra_server_args": "--quantization fp8", - } - - async def fake_profile(ctx): - return { - "status": "succeeded", - "main_trace_path": str(tmp_path / "trace.json.gz"), - "workspace": str(tmp_path), - "output_throughput": 527.5, - } - - async def fake_ta(payload, *, session_dir): - return { - "status": "ok", - "candidates_path": str(tmp_path / "kc.json"), - "trace_report_path": str(md), - "hot_kernels": [], - "trace_health_warnings": [], - } - - task = Task( - task_id="t-prelude-1", - kind="roofline", - state="running", - params={"base_extra_args": "", "reason": "prelude_initial"}, - idempotency_key="roofline:prelude-1", - requires_lanes=["profile_lane"], - ) - ctx = RunnerContext( - task=task, - lease=None, - extra={"session_dir": str(tmp_path)}, - ) - executor = RooflineExecutor(shared_state=state) - with ( - patch( - "hyperloom.orchestrator.actions.executors.profile.profile_executor", - new=fake_profile, - ), - patch( - "hyperloom.orchestrator.kernel.request_handlers.trace_analyze_handler", - new=fake_ta, - ), - ): - result = asyncio.run(executor(ctx)) - assert result["status"] == "succeeded" - return state - - def test_delayed_prelude_records_baseline_not_fp8_current_best( - self, - tmp_path, - ): - state = self._run_prelude_after_fp8_promotion(tmp_path) - assert len(state.roofline_snapshots) == 1 - snap = state.roofline_snapshots[0] - # achieved is the baseline arm's tput, NOT the promoted current_best. - assert snap["achieved_tok_per_sec"] == 527.5 - - def test_delayed_prelude_ceiling_stays_baseline_dtype(self, tmp_path): - state = self._run_prelude_after_fp8_promotion(tmp_path) - snap = state.roofline_snapshots[0] - baseline_peak = snap["theoretical_peak_tok_per_sec"] - # The fp8 current_best ceiling would be strictly higher (half weight - # IO); the recorded baseline ceiling must NOT be inflated to it. - fp8_peak = compute_roofline_breakdown_from_state( - state, - arm="current_best", - ).peak_tok_per_sec - assert baseline_peak is not None and baseline_peak > 0 - assert fp8_peak > baseline_peak - - -class TestDiffusionCeiling: - """xDiT (diffusion) images/sec memory-bound roofline ceiling.""" - - def test_mem_img_per_sec_formula(self): - # per-step = weight_bytes / (bw * gpus); img/s = 1 / (steps * per-step). - monkey_bw = HW_SPECS["mi300x"]["hbm_bw_gbps"] * 1e9 - weight_bytes = 6_000_000_000 - steps = 28 - expected = 1.0 / (steps * (weight_bytes / monkey_bw)) - got = compute_diffusion_mem_img_per_sec( - gpu_type="mi300x", num_gpus=1, weight_bytes=weight_bytes, num_steps=steps - ) - assert got == pytest.approx(expected, rel=1e-9) - assert 25 < got < 40 # ~31.5 img/s sanity band - - def test_num_gpus_scales_ceiling(self): - one = compute_diffusion_mem_img_per_sec(gpu_type="mi300x", num_gpus=1, weight_bytes=6e9, num_steps=28) - eight = compute_diffusion_mem_img_per_sec(gpu_type="mi300x", num_gpus=8, weight_bytes=6e9, num_steps=28) - assert eight == pytest.approx(one * 8, rel=1e-9) - - def test_degenerate_inputs_return_zero(self): - assert compute_diffusion_mem_img_per_sec(gpu_type="mi300x", num_gpus=1, weight_bytes=0, num_steps=28) == 0.0 - assert compute_diffusion_mem_img_per_sec(gpu_type="mi300x", num_gpus=1, weight_bytes=6e9, num_steps=0) == 0.0 - assert compute_diffusion_mem_img_per_sec(gpu_type="nope", num_gpus=1, weight_bytes=6e9, num_steps=28) == 0.0 - - def test_xdit_framework_routes_to_diffusion_branch(self, monkeypatch): - # Avoid disk: stub the model meta + step-count readers. - monkeypatch.setattr( - _rc, - "load_model_meta", - lambda *a, **k: ModelMeta( - weight_bytes=6_000_000_000, - weight_dtype_bytes=1.0, - num_layers=30, - num_kv_heads=0, - head_dim=0, - hidden_size=0, - ), - ) - monkeypatch.setattr(_rc, "_read_diffusion_num_steps", lambda state: 28) - state = SimpleNamespace( - framework="xdit", - gpu_type="mi300x", - model_path="/fake/dit", - precision="bf16", - tp=1, - conc=1, - isl=0, - osl=0, - last_baseline=None, - ) - bd = compute_roofline_breakdown_from_state(state, arm="baseline") - assert bd.bound_kind == "memory" - assert bd.peak_tok_per_sec > 0 - assert bd.cmp_tok_per_sec == 0.0 - assert bd.peak_tok_per_sec == pytest.approx(bd.mem_tok_per_sec, rel=1e-9) - - def test_xdit_missing_steps_returns_empty(self, monkeypatch): - monkeypatch.setattr( - _rc, - "load_model_meta", - lambda *a, **k: ModelMeta( - weight_bytes=6_000_000_000, - weight_dtype_bytes=1.0, - num_layers=30, - num_kv_heads=0, - head_dim=0, - hidden_size=0, - ), - ) - monkeypatch.setattr(_rc, "_read_diffusion_num_steps", lambda state: 0) - state = SimpleNamespace( - framework="xdit", - gpu_type="mi300x", - model_path="/fake/dit", - precision="bf16", - tp=1, - conc=1, - isl=0, - osl=0, - last_baseline=None, - ) - bd = compute_roofline_breakdown_from_state(state, arm="baseline") - assert bd.bound_kind == "unknown" - assert bd.peak_tok_per_sec == 0.0 diff --git a/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling_diffusion_units.py b/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling_diffusion_units.py new file mode 100644 index 0000000000..98ac83ef97 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling_diffusion_units.py @@ -0,0 +1,340 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""The diffusion (xDiT) arm of the roofline ceiling. + +Diffusion is measured in images/sec, and its sequence length comes from the +latent grid rather than a token count, so the DiT geometry readers are what +decide whether the ceiling is meaningful at all. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from hyperloom.orchestrator.kernel import roofline_ceiling as rc + + +def _model(tmp_path: Path, *, transformer: dict | None = None, vae: dict | None = None) -> str: + """Write a diffusers-layout model dir with the given sub-configs.""" + root = tmp_path / "diffusion-model" + for name, cfg in (("transformer", transformer), ("vae", vae)): + if cfg is None: + continue + d = root / name + d.mkdir(parents=True, exist_ok=True) + (d / "config.json").write_text(json.dumps(cfg), encoding="utf-8") + root.mkdir(parents=True, exist_ok=True) + return str(root) + + +# ---- VAE geometry ---- + + +def test_vae_scale_is_one_stride_two_stage_per_extra_block(tmp_path): + path = _model(tmp_path, vae={"block_out_channels": [128, 256, 512, 512], "latent_channels": 16}) + assert rc._read_vae_geometry(path) == (2 ** 3, 16) + + +@pytest.mark.parametrize( + "vae", + [None, {"block_out_channels": [], "latent_channels": 4}, {"latent_channels": 4}], +) +def test_vae_geometry_falls_back_to_the_standard_downscale(tmp_path, vae): + """8x is the SD/FLUX VAE; an unreadable config must not change the scale.""" + path = _model(tmp_path, vae=vae) + assert rc._read_vae_geometry(path)[0] == 8 + + +def test_vae_geometry_survives_a_malformed_config(tmp_path): + root = tmp_path / "m" + (root / "vae").mkdir(parents=True) + (root / "vae" / "config.json").write_text("[]", encoding="utf-8") + assert rc._read_vae_geometry(str(root)) == (8, 0) + + +# ---- latent token grid ---- + + +def test_latent_tokens_divide_the_resolution_by_vae_and_patch_pack(tmp_path): + """FLUX packs a 2x2 latent patch into channels, so in_channels / latent = 4.""" + path = _model(tmp_path, vae={"block_out_channels": [1, 2, 3, 4], "latent_channels": 16}) + tokens = rc._diffusion_latent_tokens_from_resolution(path, {"in_channels": 64}, 1024, 1024) + # downscale = vae 8 * pack 2 + assert tokens == (1024 // 16) * (1024 // 16) + + +def test_latent_tokens_default_the_pack_when_it_is_not_a_square_ratio(tmp_path): + path = _model(tmp_path, vae={"block_out_channels": [1, 2, 3, 4], "latent_channels": 16}) + # 48/16 = 3, not a perfect square -> keep the FLUX/SD3 default pack of 2. + tokens = rc._diffusion_latent_tokens_from_resolution(path, {"in_channels": 48}, 1024, 1024) + assert tokens == (1024 // 16) * (1024 // 16) + + +@pytest.mark.parametrize("height, width", [(0, 1024), (1024, 0), (0, 0)]) +def test_latent_tokens_need_a_resolution(tmp_path, height, width): + path = _model(tmp_path, vae={"block_out_channels": [1, 2], "latent_channels": 4}) + assert rc._diffusion_latent_tokens_from_resolution(path, {"in_channels": 16}, height, width) == 0 + + +# ---- DiT metadata ---- + + +def test_dit_meta_counts_dual_stream_blocks_twice(tmp_path): + """A FLUX dual-stream block runs separate image and text projections.""" + hidden = 256 + path = _model( + tmp_path, + transformer={ + "num_layers": 2, + "num_single_layers": 3, + "num_attention_heads": 4, + "attention_head_dim": 64, + "sample_size": 64, + "patch_size": 2, + "in_channels": 16, + }, + ) + meta = rc._read_diffusion_dit_meta(path) + assert meta is not None + params, tokens, layers, h = meta + assert h == hidden + assert layers == 2 + 3 + assert params == 12 * hidden**2 * (2 * 2 + 3) + assert tokens == (64 // 2) * (64 // 2) + + +def test_dit_meta_reads_a_non_square_patch(tmp_path): + path = _model( + tmp_path, + transformer={ + "num_layers": 1, + "num_attention_heads": 2, + "attention_head_dim": 32, + "sample_size": 64, + "patch_size": [2, 4], + }, + ) + assert rc._read_diffusion_dit_meta(path)[1] == (64 // 2) * (64 // 4) + + +def test_dit_meta_falls_back_to_the_runtime_resolution(tmp_path): + """FLUX/SD3 carry no sample_size; the runtime resolution sets the grid.""" + path = _model( + tmp_path, + transformer={"num_layers": 2, "num_attention_heads": 4, "attention_head_dim": 64, "in_channels": 64}, + vae={"block_out_channels": [1, 2, 3, 4], "latent_channels": 16}, + ) + params, tokens, _layers, _h = rc._read_diffusion_dit_meta(path, height=1024, width=512) + assert tokens == (1024 // 16) * (512 // 16) + assert params > 0 + + +def test_dit_meta_declines_an_unusable_transformer(tmp_path): + assert rc._read_diffusion_dit_meta(str(tmp_path / "absent")) is None + # Present but missing the block count / model dim. + assert rc._read_diffusion_dit_meta(_model(tmp_path, transformer={"num_layers": 0})) is None + no_hidden = _model(tmp_path / "b", transformer={"num_layers": 2}) + assert rc._read_diffusion_dit_meta(no_hidden) is None + + +# ---- ceilings ---- + + +def test_diffusion_memory_ceiling_reads_the_weights_once_per_step(): + gb = 1024**3 + img_s = rc.compute_diffusion_mem_img_per_sec( + gpu_type="mi300x", num_gpus=1, weight_bytes=10 * gb, num_steps=25 + ) + bw = rc.HW_SPECS["mi300x"]["hbm_bw_gbps"] * 1e9 + assert img_s == pytest.approx(1.0 / (25 * (10 * gb / bw))) + + +def test_diffusion_memory_ceiling_scales_with_the_gpu_count(): + kw = dict(gpu_type="mi300x", weight_bytes=1024**3, num_steps=10) + assert rc.compute_diffusion_mem_img_per_sec(num_gpus=4, **kw) == pytest.approx( + 4 * rc.compute_diffusion_mem_img_per_sec(num_gpus=1, **kw) + ) + + +@pytest.mark.parametrize( + "over", + [ + {"gpu_type": "not-a-gpu"}, + {"weight_bytes": 0}, + {"num_steps": 0}, + ], +) +def test_diffusion_memory_ceiling_declines_degenerate_input(over): + kw = dict(gpu_type="mi300x", num_gpus=1, weight_bytes=1024**3, num_steps=10) + kw.update(over) + assert rc.compute_diffusion_mem_img_per_sec(**kw) == 0.0 + + +def test_diffusion_compute_ceiling_sums_linear_and_attention_flops(): + kw = dict( + gpu_type="mi300x", + num_gpus=1, + precision_tag="bf16", + dit_params=1_000_000, + latent_tokens=1024, + num_layers=8, + hidden_size=256, + num_steps=20, + ) + linear = 2.0 * kw["dit_params"] * kw["latent_tokens"] + attn = 4.0 * kw["num_layers"] * kw["latent_tokens"] ** 2 * kw["hidden_size"] + peak = rc._resolve_achievable_tflops("mi300x", "bf16") * 1e12 + assert rc.compute_diffusion_compute_img_per_sec(**kw) == pytest.approx( + peak / (kw["num_steps"] * (linear + attn)) + ) + + +def test_diffusion_compute_ceiling_scales_with_the_gpu_count(): + kw = dict( + gpu_type="mi300x", + precision_tag="bf16", + dit_params=1_000_000, + latent_tokens=512, + num_layers=4, + hidden_size=128, + num_steps=10, + ) + assert rc.compute_diffusion_compute_img_per_sec(num_gpus=8, **kw) == pytest.approx( + 8 * rc.compute_diffusion_compute_img_per_sec(num_gpus=1, **kw) + ) + + +@pytest.mark.parametrize("over", [{"dit_params": 0}, {"latent_tokens": 0}, {"num_steps": 0}, {"gpu_type": "xpu"}]) +def test_diffusion_compute_ceiling_declines_degenerate_input(over): + kw = dict( + gpu_type="mi300x", + num_gpus=1, + precision_tag="bf16", + dit_params=1_000, + latent_tokens=64, + num_layers=2, + hidden_size=64, + num_steps=4, + ) + kw.update(over) + assert rc.compute_diffusion_compute_img_per_sec(**kw) == 0.0 + + +# ---- state-level diffusion breakdown ---- + + +def _xdit_state(tmp_path: Path, model_dir: str, **envs): + """An xDiT run state whose baseline yaml carries the denoising geometry.""" + import yaml + from types import SimpleNamespace + + base = {"XDIT_NUM_STEPS": "25", "XDIT_HEIGHT": "1024", "XDIT_WIDTH": "1024", "TP": "1"} + base.update({k: str(v) for k, v in envs.items()}) + cfg = tmp_path / "baseline.yaml" + cfg.write_text( + yaml.safe_dump({"benchmark": {"model": model_dir, "framework": "xdit", "envs": base}}), + encoding="utf-8", + ) + return SimpleNamespace( + last_baseline={"extras": {"materialized_config": str(cfg)}}, + gpu_type="mi300x", + model_path=model_dir, + precision="bf16", + framework="xdit", + tp=0, + conc=0, + isl=0, + osl=0, + current_best={}, + optimization_stack=[], + ) + + +def _flux_like(tmp_path: Path) -> str: + """A diffusers dir with DiT geometry, a VAE, and a weight shard.""" + root = tmp_path / "flux" + (root / "transformer").mkdir(parents=True) + (root / "transformer" / "config.json").write_text( + json.dumps( + { + "num_layers": 2, + "num_single_layers": 4, + "num_attention_heads": 8, + "attention_head_dim": 64, + "in_channels": 64, + } + ), + encoding="utf-8", + ) + (root / "vae").mkdir(parents=True) + (root / "vae" / "config.json").write_text( + json.dumps({"block_out_channels": [1, 2, 3, 4], "latent_channels": 16}), encoding="utf-8" + ) + (root / "config.json").write_text(json.dumps({"num_hidden_layers": 6}), encoding="utf-8") + (root / "model.safetensors").write_bytes(b"\0" * (4 * 1024**2)) + return str(root) + + +def test_diffusion_step_count_and_resolution_come_from_the_baseline_envs(tmp_path): + st = _xdit_state(tmp_path, str(tmp_path / "m")) + assert rc._read_diffusion_num_steps(st) == 25 + assert rc._read_diffusion_resolution(st) == (1024, 1024) + + +def test_diffusion_geometry_accepts_the_custom_workload_aliases(tmp_path): + """An operator-supplied diffusion workload feeds the same ceiling.""" + import yaml + from types import SimpleNamespace + + cfg = tmp_path / "b.yaml" + cfg.write_text( + yaml.safe_dump( + { + "benchmark": { + "envs": {"CUSTOM_NUM_STEPS": "30", "CUSTOM_HEIGHT": "512", "CUSTOM_WIDTH": "768"} + } + } + ), + encoding="utf-8", + ) + st = SimpleNamespace(last_baseline={"extras": {"materialized_config": str(cfg)}}) + assert rc._read_diffusion_num_steps(st) == 30 + assert rc._read_diffusion_resolution(st) == (512, 768) + + +def test_diffusion_breakdown_binds_on_the_slower_of_its_two_ceilings(tmp_path): + bd = rc.compute_roofline_breakdown_from_state(_xdit_state(tmp_path, _flux_like(tmp_path))) + + assert bd.peak_tok_per_sec > 0 + assert bd.mem_tok_per_sec > 0 and bd.cmp_tok_per_sec > 0 + assert bd.peak_tok_per_sec == pytest.approx(min(bd.mem_tok_per_sec, bd.cmp_tok_per_sec)) + assert bd.bound_kind in {"compute", "memory"} + + +def test_diffusion_breakdown_needs_a_step_count(tmp_path): + """Without denoising steps there is no per-image work to bound.""" + st = _xdit_state(tmp_path, _flux_like(tmp_path), XDIT_NUM_STEPS="0") + assert rc.compute_roofline_breakdown_from_state(st) == rc._EMPTY_BREAKDOWN + + +def test_diffusion_breakdown_degrades_to_memory_only_without_dit_geometry(tmp_path): + """No transformer config: the full checkpoint still bounds the step read.""" + root = tmp_path / "plain" + root.mkdir() + (root / "config.json").write_text(json.dumps({"num_hidden_layers": 4}), encoding="utf-8") + (root / "model.safetensors").write_bytes(b"\0" * (2 * 1024**2)) + + bd = rc.compute_roofline_breakdown_from_state(_xdit_state(tmp_path, str(root))) + assert bd.mem_tok_per_sec > 0 + assert bd.cmp_tok_per_sec == 0.0 + assert bd.bound_kind == "memory" + + +def test_diffusion_breakdown_is_empty_without_weights_or_geometry(tmp_path): + empty = tmp_path / "nothing" + empty.mkdir() + assert rc.compute_roofline_breakdown_from_state(_xdit_state(tmp_path, str(empty))) == rc._EMPTY_BREAKDOWN diff --git a/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling_perfmodel_units.py b/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling_perfmodel_units.py new file mode 100644 index 0000000000..8f3194ab04 --- /dev/null +++ b/src/hyperloom/inference_optimizer/tests/test_roofline_ceiling_perfmodel_units.py @@ -0,0 +1,443 @@ +# SPDX-FileCopyrightText: 2026 Advanced Micro Devices, Inc. +# SPDX-License-Identifier: MIT + +"""The bottom-up PerfModel roofline and the HF metadata it reads. + +The op formulas mirror TraceLens PerfModel, so they are pinned against the +arithmetic in their own docstrings rather than against recorded outputs: a +recorded number cannot tell a corrected formula apart from a broken one. +""" + +from __future__ import annotations + +import json +from pathlib import Path + +import pytest + +from hyperloom.orchestrator.kernel import roofline_ceiling as rc + + +# ---- op formulas ---- + + +def test_gemm_flops_is_two_mnk(): + assert rc._gemm_flops(4, 8, 16) == 2.0 * 4 * 8 * 16 + + +def test_gemm_bytes_separates_activation_from_weight_precision(): + """A quantized weight is read at its own width; activations stay bf16.""" + m, n, k = 4, 8, 16 + both_fp8 = rc._gemm_bytes(m, n, k, weight_bpe=1.0) + assert both_fp8 == m * k * 1.0 + k * n * 1.0 + m * n * 1.0 + + split = rc._gemm_bytes(m, n, k, weight_bpe=1.0, act_bpe=2.0) + assert split == m * k * 2.0 + k * n * 1.0 + m * n * 2.0 + # Only the weight read stays narrow, so the split total is the larger one. + assert split > both_fp8 + + +def test_sdpa_flops_counts_both_matmuls(): + b, n_q, h_q, n_kv, h_kv, d = 2, 3, 8, 5, 2, 64 + expected = b * h_q * (2.0 * n_q * n_kv * d) * 2 + assert rc._sdpa_flops(b, n_q, h_q, n_kv, h_kv, d, d, causal=False) == expected + + +def test_causal_masking_halves_prefill_attention_only(): + """Halved when the two lengths match; decode (N_Q=1) is untouched.""" + args = (2, 7, 8, 7, 2, 64, 64) + assert rc._sdpa_flops(*args, causal=True) == rc._sdpa_flops(*args, causal=False) / 2.0 + + decode = (2, 1, 8, 7, 2, 64, 64) + assert rc._sdpa_flops(*decode, causal=True) == rc._sdpa_flops(*decode, causal=False) + + +def test_sdpa_bytes_reads_kv_at_the_kv_head_count(): + """GQA is the point of the split: K/V are sized by H_KV, Q and out by H_Q.""" + b, n_q, h_q, n_kv, h_kv, d, bpe = 2, 3, 8, 5, 2, 64, 2.0 + expected = (b * n_q * h_q * d + b * n_kv * h_kv * d * 2 + b * n_q * h_q * d) * bpe + assert rc._sdpa_bytes(b, n_q, h_q, n_kv, h_kv, d, d, False, bpe) == expected + + +def test_sdpa_bytes_ignores_causal(): + """Causal masking skips compute, not the KV read.""" + args = (2, 7, 8, 7, 2, 64, 64) + assert rc._sdpa_bytes(*args, True, 2.0) == rc._sdpa_bytes(*args, False, 2.0) + + +def test_fused_moe_flops_counts_gate_up_down_and_aggregation(): + m, k, n, topk = 4, 16, 32, 2 + expected = 2.0 * m * k * n * topk * 2 + 2.0 * m * k * n * topk + m * k * (2 * topk - 1) + assert rc._fused_moe_flops(m, k, n, topk) == expected + + +def test_fused_moe_active_experts_saturate_with_batch_size(): + """Coupon collector: one token touches topk experts, a large batch touches all.""" + k, n, num_experts, topk, bpe = 16, 32, 8, 2, 2.0 + + def _expert_bytes(m): + # Subtract the activation terms to leave the expert-weight reads. + return rc._fused_moe_bytes(m, k, n, num_experts, topk, bpe) - 2 * m * k * bpe + + one_token = _expert_bytes(1) + assert one_token == pytest.approx(topk * n * k * bpe * 3) + + all_experts = num_experts * n * k * bpe * 3 + assert _expert_bytes(4096) == pytest.approx(all_experts) + assert one_token < _expert_bytes(8) < all_experts + + +def test_fused_moe_bytes_defaults_activations_to_the_weight_width(): + args = (4, 16, 32, 8, 2) + assert rc._fused_moe_bytes(*args, 1.0) == rc._fused_moe_bytes(*args, 1.0, act_bpe=1.0) + + +# ---- compute_roofline_from_perfmodel ---- + + +def _dense_meta(**over) -> rc.ModelMeta: + """A small dense model whose every PerfModel input is populated.""" + base = dict( + weight_bytes=16 * 1024**3, + num_layers=4, + num_kv_heads=2, + head_dim=64, + weight_dtype_bytes=2.0, + hidden_size=512, + intermediate_size=1024, + vocab_size=32000, + num_attention_heads=8, + ) + base.update(over) + return rc.ModelMeta(**base) + + +_UNSET = object() + + +def _perfmodel(meta=_UNSET, **over): + kw = dict( + meta=_dense_meta() if meta is _UNSET else meta, + gpu_type="mi300x", + concurrency=8, + isl=128, + osl=64, + ) + kw.update(over) + return rc.compute_roofline_from_perfmodel(**kw) + + +def test_perfmodel_breaks_a_dense_forward_into_its_operators(): + out = _perfmodel() + assert out is not None + names = [op.name for op in out.ops] + assert names == ["q_proj", "k_proj", "v_proj", "o_proj", "gate_proj", "up_proj", "down_proj", "lm_head", "sdpa"] + assert out.bound_kind in {"compute", "memory"} + assert out.decode_tok_per_s > 0 + assert out.prefill_tok_per_s > 0 + + +def test_perfmodel_op_shares_are_normalised_over_the_forward(): + out = _perfmodel() + assert sum(op.pct_time for op in out.ops) == pytest.approx(1.0) + assert all(op.time_s > 0 and op.flops > 0 and op.bytes_moved > 0 for op in out.ops) + assert all(op.ai == pytest.approx(op.flops / op.bytes_moved) for op in out.ops) + + +def test_perfmodel_decode_sits_between_its_own_memory_and_compute_ceilings(): + """The roofline takes the slower side, so its rate is the lower of the two.""" + out = _perfmodel() + assert out.decode_tok_per_s == pytest.approx(min(out.decode_mem_tok_per_s, out.decode_cmp_tok_per_s)) + slower = "memory" if out.decode_mem_tok_per_s <= out.decode_cmp_tok_per_s else "compute" + assert out.bound_kind == slower + + +def test_perfmodel_routes_a_moe_model_through_the_fused_expert_op(): + """A MoE model replaces the three dense FFN GEMMs with one fused op.""" + out = _perfmodel(_dense_meta(num_experts=8, experts_per_tok=2, moe_intermediate_size=256)) + names = [op.name for op in out.ops] + assert "moe_fused" in names + assert not {"gate_proj", "up_proj", "down_proj"} & set(names) + + +def test_perfmodel_scales_the_hardware_by_the_gpu_count(): + one, four = _perfmodel(num_gpus=1), _perfmodel(num_gpus=4) + assert four.hbm_bw_gbps == one.hbm_bw_gbps * 4 + assert four.peak_achievable_tflops == one.peak_achievable_tflops * 4 + assert four.decode_tok_per_s > one.decode_tok_per_s + + +def test_perfmodel_declines_what_it_cannot_model(): + assert _perfmodel(None) is None + assert _perfmodel(_dense_meta(hidden_size=0)) is None + assert _perfmodel(_dense_meta(num_attention_heads=0)) is None + assert _perfmodel(_dense_meta(num_layers=0)) is None + # Unknown GPU, and a GPU with no achievable TFLOPS at this precision. + assert _perfmodel(gpu_type="h100") is None + assert _perfmodel(precision_tag="int3") is None + + +def test_perfmodel_drops_the_lm_head_when_the_vocab_is_unknown(): + out = _perfmodel(_dense_meta(vocab_size=0)) + assert "lm_head" not in [op.name for op in out.ops] + + +# ---- load_model_meta ---- + + +def _write_model(dir_path: Path, cfg: dict, *, weight_bytes: int = 4096) -> Path: + """Write a minimal local HF model dir: config.json + one weight shard.""" + dir_path.mkdir(parents=True, exist_ok=True) + (dir_path / "config.json").write_text(json.dumps(cfg), encoding="utf-8") + (dir_path / "model.safetensors").write_bytes(b"\0" * weight_bytes) + return dir_path + + +_DENSE_CFG = { + "num_hidden_layers": 4, + "hidden_size": 512, + "num_attention_heads": 8, + "num_key_value_heads": 2, + "intermediate_size": 1024, + "vocab_size": 32000, + "torch_dtype": "bfloat16", +} + + +def test_load_model_meta_reads_the_dense_shape(tmp_path): + meta = rc.load_model_meta(_write_model(tmp_path / "m", _DENSE_CFG, weight_bytes=8192)) + + assert meta.weight_bytes == 8192 + assert (meta.num_layers, meta.num_kv_heads, meta.hidden_size) == (4, 2, 512) + assert meta.head_dim == 512 // 8 + assert meta.weight_dtype_bytes == 2.0 + # Dense: no expert decomposition, so the whole weight set is active. + assert (meta.num_experts, meta.expert_weight_bytes) == (0, 0) + assert meta.active_weight_bytes == meta.weight_bytes + + +def test_load_model_meta_prefers_the_safetensors_index_over_the_shard_sizes(tmp_path): + """The index records the byte-exact total; the shards on disk may be sparse.""" + d = _write_model(tmp_path / "m", _DENSE_CFG, weight_bytes=10) + (d / "model.safetensors.index.json").write_text( + json.dumps({"metadata": {"total_size": 123456}}), encoding="utf-8" + ) + assert rc.load_model_meta(d).weight_bytes == 123456 + + +def test_load_model_meta_falls_back_when_the_index_is_unusable(tmp_path): + d = _write_model(tmp_path / "m", _DENSE_CFG, weight_bytes=777) + (d / "model.safetensors.index.json").write_text("{not json", encoding="utf-8") + assert rc.load_model_meta(d).weight_bytes == 777 + + +def test_load_model_meta_takes_the_head_dim_the_config_states(tmp_path): + """An explicit head_dim wins over hidden_size / heads, which need not divide.""" + cfg = {**_DENSE_CFG, "head_dim": 128} + assert rc.load_model_meta(_write_model(tmp_path / "m", cfg)).head_dim == 128 + + +def test_load_model_meta_treats_a_missing_kv_head_count_as_multi_head(tmp_path): + cfg = {k: v for k, v in _DENSE_CFG.items() if k != "num_key_value_heads"} + assert rc.load_model_meta(_write_model(tmp_path / "m", cfg)).num_kv_heads == 8 + + +def test_load_model_meta_sizes_weights_by_the_quantization_method(tmp_path): + """quant_method outranks torch_dtype: the checkpoint is stored quantized.""" + cfg = {**_DENSE_CFG, "quantization_config": {"quant_method": "fp8"}} + assert rc.load_model_meta(_write_model(tmp_path / "m", cfg)).weight_dtype_bytes == 1.0 + + +def test_load_model_meta_decomposes_a_moe_checkpoint(tmp_path): + """Only the routed experts a token activates count toward its weight IO.""" + cfg = { + **_DENSE_CFG, + "num_experts": 8, + "num_experts_per_tok": 2, + "moe_intermediate_size": 256, + } + meta = rc.load_model_meta(_write_model(tmp_path / "m", cfg, weight_bytes=64 * 1024**2)) + + assert (meta.num_experts, meta.experts_per_tok) == (8, 2) + assert 0 < meta.expert_weight_bytes < meta.weight_bytes + # Non-expert weights, plus the 2-of-8 share of the expert weights. + assert meta.active_weight_bytes == ( + meta.weight_bytes - meta.expert_weight_bytes + int(2 / 8 * meta.expert_weight_bytes) + ) + assert meta.active_weight_bytes < meta.weight_bytes + + +def test_load_model_meta_sizes_routed_experts_at_their_own_precision(tmp_path): + """fp4 experts under an fp8 model: the global dtype would over-count them.""" + cfg = { + **_DENSE_CFG, + "num_experts": 8, + "num_experts_per_tok": 2, + "moe_intermediate_size": 256, + "quantization_config": {"quant_method": "fp8"}, + "expert_dtype": "fp4", + } + meta = rc.load_model_meta(_write_model(tmp_path / "m", cfg, weight_bytes=64 * 1024**2)) + + assert meta.weight_dtype_bytes == 1.0 + assert meta.expert_weight_dtype_bytes == 0.5 + assert meta.expert_weight_bytes > 0 + + +def test_moe_decomposition_degrades_when_the_experts_exceed_the_checkpoint(): + """An implausible decomposition is dropped rather than published.""" + cfg = { + "num_experts": 8, + "num_experts_per_tok": 2, + "hidden_size": 512, + "num_hidden_layers": 4, + "moe_intermediate_size": 256, + } + active, total, experts, per_tok = rc._compute_expert_decomposition( + cfg, weight_bytes=1024, dtype_bytes=2.0 + ) + assert (active, total, experts, per_tok) == (1024, 0, 0, 0) + + +@pytest.mark.parametrize( + "cfg", + [ + {"num_experts": 0, "num_experts_per_tok": 2}, + {"num_experts": 8, "num_experts_per_tok": 0}, + {"num_experts": 8, "num_experts_per_tok": 2, "hidden_size": 0}, + ], +) +def test_moe_decomposition_needs_a_complete_config(cfg): + assert rc._compute_expert_decomposition(cfg, weight_bytes=999, dtype_bytes=2.0) == (999, 0, 0, 0) + + +def test_load_model_meta_declines_an_unreadable_model(tmp_path): + assert rc.load_model_meta("") is None + # A dir with a config but no weights, and one with weights but no config. + no_weights = tmp_path / "a" + no_weights.mkdir() + (no_weights / "config.json").write_text(json.dumps(_DENSE_CFG), encoding="utf-8") + assert rc.load_model_meta(no_weights) is None + + no_config = tmp_path / "b" + no_config.mkdir() + (no_config / "model.safetensors").write_bytes(b"\0" * 16) + assert rc.load_model_meta(no_config) is None + + +def test_load_model_meta_declines_a_config_that_is_not_a_mapping(tmp_path): + d = tmp_path / "m" + d.mkdir() + (d / "config.json").write_text(json.dumps([1, 2, 3]), encoding="utf-8") + (d / "model.safetensors").write_bytes(b"\0" * 16) + assert rc.load_model_meta(d) is None + + +# ---- state-level entry points ---- + + +def _state(tmp_path: Path, benchmark: dict, **attrs): + """A run state whose baseline provenance points at a materialized yaml.""" + import yaml + + cfg = tmp_path / "baseline.yaml" + cfg.write_text(yaml.safe_dump({"benchmark": benchmark}), encoding="utf-8") + from types import SimpleNamespace + + base = dict( + last_baseline={"extras": {"materialized_config": str(cfg)}}, + gpu_type="mi300x", + model_path="", + precision="", + framework="", + tp=0, + conc=0, + isl=0, + osl=0, + current_best={}, + optimization_stack=[], + ) + base.update(attrs) + return SimpleNamespace(**base) + + +def _serving_benchmark(model_dir: Path, **envs) -> dict: + base = {"TP": "1", "CONC": "8", "ISL": "128", "OSL": "64"} + base.update({k: str(v) for k, v in envs.items()}) + return { + "model": str(model_dir), + "framework": "sglang", + "envs": base, + } + + +def test_runtime_workload_prefers_the_benchmark_envs_over_state(tmp_path): + """The yaml is the geometry of record; state attrs only fill its gaps.""" + st = _state(tmp_path, _serving_benchmark(tmp_path / "m"), tp=8, conc=99, isl=1, osl=2) + rt = rc.resolve_runtime_workload(st) + assert (rt.tp, rt.concurrency, rt.isl, rt.osl) == (1, 8, 128, 64) + assert rt.gpu_type == "mi300x" + assert rt.framework == "sglang" + + +def test_runtime_workload_falls_back_to_state_when_the_envs_are_silent(tmp_path): + st = _state(tmp_path, {"model": "/m", "envs": {}}, tp=4, conc=16, isl=512, osl=32) + rt = rc.resolve_runtime_workload(st) + assert (rt.tp, rt.concurrency, rt.isl, rt.osl) == (4, 16, 512, 32) + + +def test_runtime_workload_defaults_concurrency_to_one(tmp_path): + """Every other geometry field may be unknown; a batch of zero cannot be.""" + rt = rc.resolve_runtime_workload(_state(tmp_path, {"envs": {}})) + assert rt.concurrency == 1 + assert rt.tp == 0 + + +def test_runtime_workload_survives_an_unreadable_baseline_yaml(tmp_path): + from types import SimpleNamespace + + st = SimpleNamespace(last_baseline={"extras": {"materialized_config": str(tmp_path / "gone.yaml")}}) + assert rc.resolve_runtime_workload(st).concurrency == 1 + + +def test_breakdown_from_state_reports_the_decode_ceiling(tmp_path): + model = _write_model(tmp_path / "m", _DENSE_CFG, weight_bytes=8 * 1024**2) + bd = rc.compute_roofline_breakdown_from_state(_state(tmp_path, _serving_benchmark(model))) + + assert bd.peak_tok_per_sec > 0 + assert bd.bound_kind in {"compute", "memory"} + # The roofline takes the slower side, so the peak is the lower projection. + assert bd.peak_tok_per_sec == pytest.approx(min(bd.mem_tok_per_sec, bd.cmp_tok_per_sec)) + + +def test_breakdown_from_state_is_empty_when_the_model_is_unreadable(tmp_path): + st = _state(tmp_path, _serving_benchmark(tmp_path / "absent")) + assert rc.compute_roofline_breakdown_from_state(st) == rc._EMPTY_BREAKDOWN + + +def test_breakdown_from_state_routes_a_diffusion_run_to_the_image_ceiling(tmp_path): + """xDiT is measured in images/sec, so it never reaches the token ceiling.""" + bench = {"model": str(tmp_path / "m"), "framework": "xdit", "envs": {"TP": "1"}} + bd = rc.compute_roofline_breakdown_from_state(_state(tmp_path, bench)) + # No transformer config on disk, so the diffusion arm has nothing to model. + assert bd == rc._EMPTY_BREAKDOWN + + +def test_peak_from_state_is_the_breakdown_peak(tmp_path): + model = _write_model(tmp_path / "m", _DENSE_CFG, weight_bytes=8 * 1024**2) + st = _state(tmp_path, _serving_benchmark(model)) + assert rc.compute_peak_from_state(st) == pytest.approx( + rc.compute_roofline_breakdown_from_state(st).peak_tok_per_sec + ) + + +def test_select_peak_and_bound_takes_the_lower_side(): + assert rc.select_peak_and_bound(100.0, 250.0) == (100.0, "memory") + assert rc.select_peak_and_bound(400.0, 250.0) == (250.0, "compute") + + +@pytest.mark.parametrize("mem, cmp", [(0.0, 250.0), (100.0, 0.0), (0.0, 0.0)]) +def test_select_peak_and_bound_ignores_a_projection_it_could_not_compute(mem, cmp): + """A zero is 'unknown', not 'infinitely slow'; it must not win the min.""" + peak, _kind = rc.select_peak_and_bound(mem, cmp) + assert peak == max(mem, cmp) diff --git a/src/hyperloom/inference_optimizer/tests/test_roofline_comparison_pipeline.py b/src/hyperloom/inference_optimizer/tests/test_roofline_comparison_pipeline.py index 1f224e6f40..0e23862669 100644 --- a/src/hyperloom/inference_optimizer/tests/test_roofline_comparison_pipeline.py +++ b/src/hyperloom/inference_optimizer/tests/test_roofline_comparison_pipeline.py @@ -76,7 +76,6 @@ def _mock_state( baseline_tput=4309.2, baseline_accuracy=0.0, current_best={"action": "params", "tput": 4357.27}, - cumulative_gain=1.12, cumulative_gain_validated=0.33, cumulative_gain_validated_ts="2026-05-24T13:47:22+00:00", cumulative_gain_validated_stack_len=2, diff --git a/src/hyperloom/inference_optimizer/tests/test_roofline_executor.py b/src/hyperloom/inference_optimizer/tests/test_roofline_executor.py index c3d8d2599a..c923c92630 100644 --- a/src/hyperloom/inference_optimizer/tests/test_roofline_executor.py +++ b/src/hyperloom/inference_optimizer/tests/test_roofline_executor.py @@ -547,11 +547,11 @@ def _roofline_result(snapshot_id: int = 1) -> dict: def test_shared_state_has_roofline_audit_fields_by_default(): + """Both audit mirrors are declared fields, so both survive a save.""" s = SharedState() - assert hasattr(s, "last_roofline") - assert hasattr(s, "roofline_attempts") - assert s.last_roofline == {} assert s.roofline_attempts == [] + assert s.last_roofline == {} + assert {"roofline_attempts", "last_roofline"} <= set(s.to_dict()) def test_audit_actions_includes_roofline_in_both_modules(): diff --git a/src/hyperloom/inference_optimizer/tests/test_session_layout.py b/src/hyperloom/inference_optimizer/tests/test_session_layout.py index 0cae842e84..d61092016d 100644 --- a/src/hyperloom/inference_optimizer/tests/test_session_layout.py +++ b/src/hyperloom/inference_optimizer/tests/test_session_layout.py @@ -7,6 +7,7 @@ import json import logging +import re from pathlib import Path import pytest @@ -83,15 +84,37 @@ def test_workspace_root_independent_of_session_pin(tmp_path, monkeypatch): assert paths.workspace_root() == tmp_path +def test_relative_user_data_path_resolves_identically_from_every_cwd(tmp_path, monkeypatch): + """A relative $USER_DATA_PATH must not follow each subprocess's cwd. + + Absolutising on read is not enough — every process would re-expand the + relative value against its own cwd — so the CLI rewrites the env var itself. + """ + from hyperloom.inference_optimizer import cli + + (tmp_path / "nested").mkdir() + monkeypatch.chdir(tmp_path) + monkeypatch.setenv(paths.ENV_USER_DATA_PATH, "relws") + with pytest.raises(SystemExit): + cli.main([]) # no subcommand; the boundary normalises before parsing + ws = paths.workspace_root() + assert ws == Path.cwd() / "relws" + sd = paths.make_session_dir(model_name="DeepSeek-R1-0528") + + monkeypatch.chdir(tmp_path / "nested") + assert paths.workspace_root() == ws + assert paths.session_dir() == sd + + def test_make_session_dir_per_model_ts_layout(tmp_path, monkeypatch): """Default: per-model/per-launch subdir + pin propagation.""" monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) sd = paths.make_session_dir(model_name="/path/models/DeepSeek-R1-0528") - # Layout: /DeepSeek-R1-0528// + # Layout: /DeepSeek-R1-0528/-/ assert sd.parent.parent == tmp_path assert sd.parent.name == "DeepSeek-R1-0528" - # Timestamp shape: YYYYMMDDTHHMMSSZ - assert len(sd.name) == 16 and sd.name.endswith("Z") and "T" in sd.name + # Name shape: YYYYMMDDTHHMMSSZ-<8 hex> + assert re.fullmatch(r"\d{8}T\d{6}Z-[0-9a-f]{8}", sd.name), sd.name import os as _os assert _os.environ[paths.ENV_CURRENT_SESSION_DIR] == str(sd) @@ -104,6 +127,20 @@ def test_make_session_dir_per_model_ts_layout(tmp_path, monkeypatch): assert not (sd / sub).exists() +def test_make_session_dir_same_second_launches_get_distinct_dirs(tmp_path, monkeypatch): + """Two launches of one model inside the same UTC second must not share a dir. + + ``session_dir.name`` is also the de-facto session id (KB fact writes, + per-session sinks), so a shared dir merges two runs' identity as well. + """ + monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) + monkeypatch.setattr(paths, "utc_now_compact", lambda: "20260814T073026Z") + first = paths.make_session_dir(model_name="DeepSeek-R1-0528") + second = paths.make_session_dir(model_name="DeepSeek-R1-0528") + assert first != second + assert first.name[:16] == second.name[:16] == "20260814T073026Z" + + def test_make_session_dir_sanitises_model_basename(tmp_path, monkeypatch): """HF ids, absolute paths, and unsafe chars all reduce to a filename-safe basename.""" monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) @@ -132,79 +169,6 @@ def test_make_session_dir_overwrites_stale_pin(tmp_path, monkeypatch): assert paths.session_dir() == sd2 -def test_find_latest_per_session_dir_returns_none_on_empty( - tmp_path, - monkeypatch, -): - """No per-session subdir under workspace_root -> None.""" - monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) - assert paths.find_latest_per_session_dir() is None - assert paths.find_latest_per_session_dir(model_name="DSR1") is None - - -def test_find_latest_per_session_dir_picks_lex_latest_ts( - tmp_path, - monkeypatch, -): - """Lex-sort on the YYYYMMDDTHHMMSSZ name picks the latest ts (robust to mtime touches).""" - monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) - (tmp_path / "MyModel").mkdir() - (tmp_path / "MyModel" / "20260101T000000Z").mkdir() - (tmp_path / "MyModel" / "20260520T120000Z").mkdir() - (tmp_path / "MyModel" / "20260315T080000Z").mkdir() - picked = paths.find_latest_per_session_dir(model_name="MyModel") - assert picked is not None - assert picked.name == "20260520T120000Z" - - -def test_find_latest_per_session_dir_no_model_scans_all( - tmp_path, - monkeypatch, -): - """No model_name -> scan all model_basename subdirs for the latest ts across the workspace.""" - monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) - (tmp_path / "Qwen-7B").mkdir() - (tmp_path / "Qwen-7B" / "20260101T000000Z").mkdir() - (tmp_path / "DSR1").mkdir() - (tmp_path / "DSR1" / "20260520T120000Z").mkdir() - picked = paths.find_latest_per_session_dir() - assert picked is not None - assert picked.name == "20260520T120000Z" - assert picked.parent.name == "DSR1" - - -def test_find_latest_per_session_dir_skips_workspace_shared( - tmp_path, - monkeypatch, -): - """workspace-shared subdirs (runtime/, logs/) must not be mistaken for model_basename subdirs.""" - monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) - (tmp_path / "runtime").mkdir() - (tmp_path / "runtime" / "20260520T120000Z").mkdir() # decoy - (tmp_path / "logs").mkdir() - (tmp_path / "MyModel").mkdir() - (tmp_path / "MyModel" / "20260518T100000Z").mkdir() - picked = paths.find_latest_per_session_dir() - assert picked is not None - assert picked.parent.name == "MyModel" - assert "runtime" not in str(picked) - - -def test_find_latest_per_session_dir_ignores_non_ts_dirs( - tmp_path, - monkeypatch, -): - """Only YYYYMMDDTHHMMSSZ-shaped dir names count as ts subdirs.""" - monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) - (tmp_path / "MyModel").mkdir() - (tmp_path / "MyModel" / "scratch").mkdir() # ignored - (tmp_path / "MyModel" / "backup-2026").mkdir() # ignored - (tmp_path / "MyModel" / "20260520T120000Z").mkdir() # picked - picked = paths.find_latest_per_session_dir(model_name="MyModel") - assert picked is not None - assert picked.name == "20260520T120000Z" - - def test_runtime_dir_is_workspace_shared(tmp_path, monkeypatch): """runtime/ lives under workspace_root, not the per-session subdir.""" monkeypatch.setenv(paths.ENV_USER_DATA_PATH, str(tmp_path)) diff --git a/src/hyperloom/inference_optimizer/tests/test_session_lock.py b/src/hyperloom/inference_optimizer/tests/test_session_lock.py index c5d4c5bc23..6993466919 100644 --- a/src/hyperloom/inference_optimizer/tests/test_session_lock.py +++ b/src/hyperloom/inference_optimizer/tests/test_session_lock.py @@ -3,7 +3,7 @@ """Tests for the single-optimizer session lock. -The lock guarantees that a second ``optimize`` / ``--resume`` attaching to the +The lock guarantees that a second ``optimize`` / ``--resume-from`` attaching to the same ``session_dir`` cannot run, so a misfiring robustness monitor can never spawn a duplicate optimizer that corrupts the shared leases / ``state.json``. """ diff --git a/src/hyperloom/inference_optimizer/tests/test_session_paths_unit.py b/src/hyperloom/inference_optimizer/tests/test_session_paths_unit.py index 91c1c85722..f58cd0bd87 100644 --- a/src/hyperloom/inference_optimizer/tests/test_session_paths_unit.py +++ b/src/hyperloom/inference_optimizer/tests/test_session_paths_unit.py @@ -24,8 +24,6 @@ def test_runs_root_and_dir(): assert sp.runs_root(SD) == SD / "runs" p = sp.runs_dir(SD, "baseline", "t1") assert p == SD / "runs" / "baseline" / "t1" - # blank task_id falls back to "unknown" - assert sp.runs_dir(SD, "baseline", "").name == "unknown" def test_runs_dir_rejects_unknown_action(): @@ -36,6 +34,8 @@ def test_runs_dir_rejects_unknown_action(): @pytest.mark.parametrize( "bad_task_id", [ + "", + " ", "../escape", "..", ".", @@ -45,9 +45,8 @@ def test_runs_dir_rejects_unknown_action(): "x/y", ], ) -def test_runs_dir_rejects_task_id_traversal(bad_task_id): - # Legitimate task ids are single path components; anything path-like must - # be rejected so it cannot relocate the sandbox. +def test_runs_dir_rejects_bad_task_id(bad_task_id): + # Blank or path-like ids would relocate the sandbox onto a shared path. with pytest.raises(ValueError): sp.runs_dir(SD, "baseline", bad_task_id) @@ -57,14 +56,14 @@ def test_runs_dir_accepts_uuid_hex_task_id(): assert sp.runs_dir(SD, "baseline", tid) == SD / "runs" / "baseline" / tid -def test_kernel_agent_runs_dir_rejects_traversal(): - for bad in ("../x", ".", "a/b", "/abs"): +def test_kernel_agent_runs_dir_rejects_bad_id(): + for bad in ("", " ", "../x", ".", "a/b", "/abs"): with pytest.raises(ValueError): sp.kernel_agent_runs_dir(SD, bad) -def test_patches_dir_rejects_traversal(): - for bad in ("../x", ".", "a/b", "/abs"): +def test_patches_dir_rejects_bad_id(): + for bad in ("", " ", "../x", ".", "a/b", "/abs"): with pytest.raises(ValueError): sp.patches_dir(SD, bad) @@ -75,9 +74,7 @@ def test_validate_action_strips(): def test_kernel_and_patch_paths(): assert sp.kernel_agent_runs_dir(SD, "s1") == SD / "kernel-agent" / "runs" / "s1" - assert sp.kernel_agent_runs_dir(SD, "").name == "unknown" assert sp.patches_dir(SD, "k1") == SD / "patches" / "k1" - assert sp.patches_dir(SD, "").name == "unknown" def test_reports_dir(): @@ -95,12 +92,13 @@ def test_trace_paths(): def test_enablement_paths(): assert sp.enablement_dir(SD) == SD / "reports" / "enablement" assert sp.enablement_round_dir(SD, "abc123") == SD / "reports" / "enablement" / "abc123" - assert sp.enablement_round_dir(SD, "").name == "unknown" -def test_enablement_round_dir_rejects_traversal(): +@pytest.mark.parametrize("task_id", ["../evil", ""]) +def test_enablement_round_dir_refuses_an_unusable_id(task_id): + """A blank id would put every round in one directory; the caller skips those.""" with pytest.raises(ValueError): - sp.enablement_round_dir(SD, "../evil") + sp.enablement_round_dir(SD, task_id) def test_research_and_competitor_paths(): diff --git a/src/hyperloom/inference_optimizer/tests/test_shared_state_evolution.py b/src/hyperloom/inference_optimizer/tests/test_shared_state_evolution.py index d7e740f177..d87f3e90f1 100644 --- a/src/hyperloom/inference_optimizer/tests/test_shared_state_evolution.py +++ b/src/hyperloom/inference_optimizer/tests/test_shared_state_evolution.py @@ -44,7 +44,7 @@ def test_v06_state_without_schema_version_is_migrated(tmp_path): "session_id": "legacy-sid", "baseline_tput": 800.0, "current_best": {"variant_name": "warm-mla", "tput": 880.0}, - "cumulative_gain": 10.0, + "cumulative_gain_validated": 10.0, "optimization_stack": [], "action_scores": {"backends": {"base_score": 5.0}}, "cooldown_until_tick": {"backends": 12}, @@ -66,7 +66,6 @@ def test_v06_state_without_schema_version_is_migrated(tmp_path): "extra_server_args": "--mla", "extra_envs": {"FOO": "bar"}, }, - "cumulative_gain": 17.5, "cumulative_gain_validated": 15.0, "cumulative_gain_validated_ts": "2025-01-01T00:00:00+00:00", "cumulative_gain_validated_stack_len": 2, @@ -155,23 +154,18 @@ def test_migration_is_idempotent(tmp_path): assert second.schema_version == third.schema_version == LATEST_STATE_SCHEMA_VERSION -def test_v2_kernel_keep_migrates_to_stable_task_and_pending_patch(): - state = SharedState.from_dict( - { - "schema_version": 2, - "kernel_opt_attempts": { - "k002": { - "kernel_id": "k002", - "task_group_key": "legacy-task", - "last_decision": "KEEP", - "last_source_file": "/repo/operator.py", - "last_artifact_path": "/artifacts/operator.py", - "last_micro_speedup": 1.2, - } - }, - } - ) - +def test_v2_kernel_keep_populates_stable_task_and_pending_patch(): + state = SharedState() + state.kernel_opt_task_attempts["legacy-task"] = { + "kernel_id": "k002", + "current_kernel_id": "k002", + "stable_task_key": "legacy-task", + "task_group_key": "legacy-task", + "last_decision": "KEEP", + "last_source_file": "/repo/operator.py", + "last_artifact_path": "/artifacts/operator.py", + "last_micro_speedup": 1.2, + } assert state.kernel_opt_task_attempts["legacy-task"]["current_kernel_id"] == "k002" pending = state.pending_kernel_integration_records() assert len(pending) == 1 @@ -179,19 +173,16 @@ def test_v2_kernel_keep_migrates_to_stable_task_and_pending_patch(): assert pending[0]["artifact_path"] == "/artifacts/operator.py" -def test_v2_ungrouped_kernel_uses_runtime_legacy_task_key(): - state = SharedState.from_dict( +def test_v2_ungrouped_kernel_record_opt_accumulates(): + state = SharedState() + state.record_kernel_opt( { - "schema_version": 2, - "kernel_opt_attempts": { - "k001": { - "kernel_id": "k001", - "last_decision": "KEEP", - "last_source_file": "/repo/operator.py", - "last_artifact_path": "/artifacts/operator.py", - "last_micro_speedup": 1.2, - } - }, + "status": "ok", + "kernel_id": "k001", + "source_file": "/repo/operator.py", + "proposal": {"decision": "PARTIAL"}, + "verification": {"micro_speedup": 1.0}, + "attempts": [], } ) state.record_kernel_opt( @@ -200,13 +191,12 @@ def test_v2_ungrouped_kernel_uses_runtime_legacy_task_key(): "kernel_id": "k001", "source_file": "/repo/operator.py", "proposal": {"decision": "PARTIAL"}, - "verification": {"micro_speedup": 1.0}, + "verification": {"micro_speedup": 1.1}, "attempts": [], } ) assert len(state.kernel_opt_task_attempts) == 1 - assert len(state.pending_kernel_integrations) == 1 # 4. --reset-state behavior @@ -272,7 +262,6 @@ def test_core_state_fields_contains_v08_new_additions(): "phase_history", "phase_budget_pct", "recipe_kb_session_id", - "recipe_kb_session_summary", "warm_start_recipe", "warm_start_pitfalls", "warm_start_lessons", diff --git a/src/hyperloom/inference_optimizer/tests/test_shared_state_kernel_opt.py b/src/hyperloom/inference_optimizer/tests/test_shared_state_kernel_opt.py index 1f71b372ca..920dc098cf 100644 --- a/src/hyperloom/inference_optimizer/tests/test_shared_state_kernel_opt.py +++ b/src/hyperloom/inference_optimizer/tests/test_shared_state_kernel_opt.py @@ -876,6 +876,23 @@ def test_integrate_fault_does_not_consume_revert_quota(state: SharedState): assert "k001" not in state.rejected_kernel_ids +@pytest.mark.parametrize("error_class", ["session_time_exhausted", "orchestrator_cancelled"]) +def test_a_run_stopped_integrate_does_not_consume_revert_quota(state: SharedState, error_class): + """A patch the run never measured must not be counted as one that lost.""" + entry = state.record_kernel_integrate_result( + _integrate_result( + "k001", + decision="NEEDS_REVIEW", + status="failed", + error_class=error_class, + ), + ) + assert entry is not None + assert entry["verdict_attempt_count"] == 0 + assert entry.get("retryable") is True + assert state.rejected_kernel_patches == [] + + def test_integrate_attempt_is_stamped_with_macro_cycle(state: SharedState): state.macro_cycle = 2 entry = state.record_kernel_integrate_result( diff --git a/src/hyperloom/inference_optimizer/tests/test_shared_state_persistence.py b/src/hyperloom/inference_optimizer/tests/test_shared_state_persistence.py index 2a1824d505..bb1fc727bb 100644 --- a/src/hyperloom/inference_optimizer/tests/test_shared_state_persistence.py +++ b/src/hyperloom/inference_optimizer/tests/test_shared_state_persistence.py @@ -45,7 +45,7 @@ def test_shared_state_defaults_blank(): s = SharedState() assert s.session_id == "" assert s.baseline_tput == 0.0 - assert s.cumulative_gain == 0.0 + assert s.cumulative_gain_validated == 0.0 assert s.crash_count == 0 assert s.pruned_families == [] assert s.current_best == {} @@ -56,7 +56,7 @@ def test_save_load_round_trip(tmp_path): session_id="abc", model_name="meta-llama/Llama-3.1-8B-Instruct", baseline_tput=1840.0, - cumulative_gain=12.5, + cumulative_gain_validated=12.5, pruned_families=["deep_kernel"], current_best={"action": "backends", "tput": 2010.0}, last_fusion={"status": "complete", "kept": False}, @@ -67,7 +67,7 @@ def test_save_load_round_trip(tmp_path): assert s2.session_id == "abc" assert s2.model_name == "meta-llama/Llama-3.1-8B-Instruct" assert s2.baseline_tput == 1840.0 - assert s2.cumulative_gain == 12.5 + assert s2.cumulative_gain_validated == 12.5 assert s2.pruned_families == ["deep_kernel"] assert s2.current_best == {"action": "backends", "tput": 2010.0} assert s2.last_fusion == {"status": "complete", "kept": False} @@ -132,12 +132,12 @@ def test_from_dict_drops_unknown_fields(): def test_apply_changes_only_known_fields(): s = SharedState() applied = s.apply_changes( - {"current_action": "baseline", "bogus": 1, "cumulative_gain": 5.0}, + {"current_action": "baseline", "bogus": 1, "cumulative_gain_validated": 5.0}, allow_core=True, ) - assert applied == {"current_action": "baseline", "cumulative_gain": 5.0} + assert applied == {"current_action": "baseline", "cumulative_gain_validated": 5.0} assert s.current_action == "baseline" - assert s.cumulative_gain == 5.0 + assert s.cumulative_gain_validated == 5.0 def test_add_pruned_family_idempotent(): @@ -165,7 +165,7 @@ def test_to_prompt_summary_contains_key_fields(): session_id="s1", model_name="Llama-3", baseline_tput=1840.0, - cumulative_gain=10.0, + cumulative_gain_validated=10.0, current_action="backends", pruned_families=["deep_kernel"], ) diff --git a/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py b/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py index 7adf31e710..f9d98fae0d 100644 --- a/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py +++ b/src/hyperloom/inference_optimizer/tests/test_shared_state_units.py @@ -190,14 +190,12 @@ def test_profile_workload_context_tracks_serving_config(self): framework="vllm", precision="fp8", current_best={ - "engine": "forge", "extra_server_args": " --attention-backend AITER ", "extra_envs": {"B": 2, "A": 1}, }, ) assert state.profile_workload_context()["serving_config"] == { - "engine": "forge", "extra_server_args": "--attention-backend AITER", "extra_envs": {"A": "1", "B": "2"}, } @@ -305,14 +303,14 @@ def test_known_field_set(self): def test_core_field_dropped_when_allow_core_false(self): # A non-privileged (allow_core=False) changes dict must not write a core field. s = SharedState() - before = s.cumulative_gain # cumulative_gain is a core field + before = s.cumulative_gain_validated # cumulative_gain_validated is a core field applied = s.apply_changes( - {"current_action": "baseline", "cumulative_gain": 999.0}, + {"current_action": "baseline", "cumulative_gain_validated": 999.0}, allow_core=False, ) assert applied == {"current_action": "baseline"} assert s.current_action == "baseline" - assert s.cumulative_gain == before # core write dropped + assert s.cumulative_gain_validated == before # core write dropped def test_a_stop_time_cannot_be_written_apart_from_its_reason(self): # stop_reason is a core field, so a changes dict that carries both must @@ -331,9 +329,9 @@ def test_a_stop_time_cannot_be_written_apart_from_its_reason(self): def test_core_field_written_when_allow_core_true(self): s = SharedState() - applied = s.apply_changes({"cumulative_gain": 999.0}, allow_core=True) - assert applied == {"cumulative_gain": 999.0} - assert s.cumulative_gain == 999.0 + applied = s.apply_changes({"cumulative_gain_validated": 999.0}, allow_core=True) + assert applied == {"cumulative_gain_validated": 999.0} + assert s.cumulative_gain_validated == 999.0 class TestKernelPatchIdentity: diff --git a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py index 0a2e71c549..f83654e52b 100644 --- a/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py +++ b/src/hyperloom/inference_optimizer/tests/test_specialist_subprocess.py @@ -1004,9 +1004,6 @@ def start_async( def poll_started(self) -> int | None: return getattr(self, "_pid", None) - def pending_seconds(self) -> float: - return 0.0 - def is_alive(self) -> bool: return self.alive diff --git a/src/hyperloom/inference_optimizer/tests/test_split_config_changes.py b/src/hyperloom/inference_optimizer/tests/test_split_config_changes.py deleted file mode 100644 index e0a1fc3ea8..0000000000 --- a/src/hyperloom/inference_optimizer/tests/test_split_config_changes.py +++ /dev/null @@ -1,141 +0,0 @@ -# SPDX-FileCopyrightText: 2025 Advanced Micro Devices, Inc. -# SPDX-License-Identifier: MIT - -"""Tests for split_config_changes: server-flag vs env routing.""" - -from __future__ import annotations - -import yaml -from pathlib import Path - -from hyperloom.orchestrator.actions.executors._grid_server_args import split_config_changes -from hyperloom.orchestrator.actions.executors._grid_runner import ( - GridVariant, - _build_variant_yaml, -) - - -# --------------------------------------------------------------------------- -# Unit tests for split_config_changes -# --------------------------------------------------------------------------- - -def test_dash_prefixed_keys_become_server_args(): - args, envs = split_config_changes({"--tokenizer-mode": "deepseek_v4"}) - assert "deepseek_v4" in args - assert "--tokenizer-mode" in args - assert "tokenizer" not in envs - - -def test_env_keys_stay_in_envs(): - args, envs = split_config_changes({"VLLM_ROCM_USE_AITER": "1"}) - assert envs == {"VLLM_ROCM_USE_AITER": "1"} - assert args == "" - - -def test_mixed_dict_splits_correctly(): - args, envs = split_config_changes({ - "--tokenizer-mode": "deepseek_v4", - "VLLM_ROCM_USE_AITER": "1", - }) - assert "--tokenizer-mode" in args - assert "deepseek_v4" in args - assert envs == {"VLLM_ROCM_USE_AITER": "1"} - - -def test_bare_flag_no_value(): - args, envs = split_config_changes({"--enable-mtp": ""}) - assert "--enable-mtp" in args - assert envs == {} - - -def test_empty_input(): - args, envs = split_config_changes({}) - assert args == "" - assert envs == {} - - -def test_legacy_flag_not_lost(): - """A --key is rebuilt into server_args, not silently dropped.""" - args, envs = split_config_changes({"--speculative-num-steps": "3"}) - assert "3" in args - assert "--speculative-num-steps" in args - assert not any(k.startswith("-") for k in envs) - - -# --------------------------------------------------------------------------- -# Integration test: flag reaches materialized YAML benchmark.envs -# --------------------------------------------------------------------------- - -def _minimal_base_yaml(tmp_path: Path, framework: str = "vllm") -> Path: - cfg = { - "benchmark": { - "framework": framework, - "model": "/model", - "envs": {}, - } - } - p = tmp_path / "base.yaml" - p.write_text(yaml.safe_dump(cfg), encoding="utf-8") - return p - - -def test_tokenizer_flag_reaches_extra_vllm_args_in_yaml(tmp_path): - """--tokenizer-mode deepseek_v4 must appear in EXTRA_VLLM_ARGS in the - materialized YAML, not be silently dropped by valid_env_key.""" - base_yaml = _minimal_base_yaml(tmp_path, framework="vllm") - - # Simulate the split that now happens in _bench_patch/_confirm_stack_rebench - from hyperloom.orchestrator.actions.executors._grid_server_args import ( - merge_server_args, - split_config_changes, - ) - config_changes = { - "--tokenizer-mode": "deepseek_v4", - "VLLM_ROCM_USE_AITER": "1", - } - cc_args, cc_envs = split_config_changes(config_changes) - variant = GridVariant( - name="test-tokenizer", - extra_server_args=merge_server_args("", cc_args), - extra_envs=cc_envs, - ) - - out_yaml = _build_variant_yaml( - base_yaml_path=base_yaml, - base_extra_args="", - variant=variant, - output_subdir=tmp_path / "out", - ) - - with out_yaml.open(encoding="utf-8") as f: - materialized = yaml.safe_load(f) - - envs = materialized["benchmark"]["envs"] - extra_vllm = envs.get("EXTRA_VLLM_ARGS", "") - assert "--tokenizer-mode" in extra_vllm, f"flag not in EXTRA_VLLM_ARGS: {envs}" - assert "deepseek_v4" in extra_vllm, f"value not in EXTRA_VLLM_ARGS: {envs}" - assert envs.get("VLLM_ROCM_USE_AITER") == "1", f"env var missing: {envs}" - - -def test_env_key_not_in_server_args_yaml(tmp_path): - """Pure env keys must NOT appear in EXTRA_VLLM_ARGS.""" - base_yaml = _minimal_base_yaml(tmp_path, framework="vllm") - - from hyperloom.orchestrator.actions.executors._grid_server_args import split_config_changes - _, cc_envs = split_config_changes({"MY_ENV": "value"}) - variant = GridVariant(name="test-env", extra_server_args="", extra_envs=cc_envs) - - out_yaml = _build_variant_yaml( - base_yaml_path=base_yaml, - base_extra_args="", - variant=variant, - output_subdir=tmp_path / "out2", - ) - - with out_yaml.open(encoding="utf-8") as f: - materialized = yaml.safe_load(f) - - envs = materialized["benchmark"]["envs"] - extra_vllm = envs.get("EXTRA_VLLM_ARGS", "") - assert "MY_ENV" not in extra_vllm - assert envs.get("MY_ENV") == "value" diff --git a/src/hyperloom/inference_optimizer/tests/test_ssh_client.py b/src/hyperloom/inference_optimizer/tests/test_ssh_client.py index 53f87ccbe2..41f385ce3e 100644 --- a/src/hyperloom/inference_optimizer/tests/test_ssh_client.py +++ b/src/hyperloom/inference_optimizer/tests/test_ssh_client.py @@ -14,7 +14,6 @@ import base64 import os import shlex -import subprocess import pytest @@ -187,32 +186,6 @@ def _run(argv, **kwargs): assert "pass_fds" not in captured["kwargs"] -def test_probe_ssh_true_on_marker(monkeypatch, known_hosts): - monkeypatch.setattr( - ssh_client, - "ssh_run", - lambda *a, **kw: _FakeCompleted(0, "mn_ssh_ok\n", ""), - ) - assert ssh_client.probe_ssh("h", key_path="/k", known_hosts=known_hosts) is True - - -def test_probe_ssh_false_on_bad_rc(monkeypatch, known_hosts): - monkeypatch.setattr( - ssh_client, - "ssh_run", - lambda *a, **kw: _FakeCompleted(255, "", "conn refused"), - ) - assert ssh_client.probe_ssh("h", key_path="/k", known_hosts=known_hosts) is False - - -def test_probe_ssh_false_on_timeout(monkeypatch, known_hosts): - def _boom(*a, **kw): - raise subprocess.TimeoutExpired(cmd="ssh", timeout=1) - - monkeypatch.setattr(ssh_client, "ssh_run", _boom) - assert ssh_client.probe_ssh("h", key_path="/k", known_hosts=known_hosts) is False - - def test_generate_session_keypair_idempotent_reuse(tmp_path, monkeypatch): priv = tmp_path / "mn_id_ed25519" pub = tmp_path / "mn_id_ed25519.pub" diff --git a/src/hyperloom/inference_optimizer/tests/test_sweep_phase_auto.py b/src/hyperloom/inference_optimizer/tests/test_sweep_phase_auto.py index 0266178e15..14621e1291 100644 --- a/src/hyperloom/inference_optimizer/tests/test_sweep_phase_auto.py +++ b/src/hyperloom/inference_optimizer/tests/test_sweep_phase_auto.py @@ -162,7 +162,7 @@ async def _noop_roofline(*, reason: str): assert c.shared_state.kernel_integrate_attempts assert c.shared_state.next_pending_keep_kernel_id() == "" assert c.shared_state.current_best["action"] == "integrate" - assert c.shared_state.current_best["kernel_id"] == "k004" + assert c.shared_state.current_best["variant_name"] == "k004" def test_pending_keep_kernel_ids_prioritize_trace_impact_over_micro(): @@ -393,7 +393,7 @@ async def _noop_roofline(*, reason: str): await c._maybe_validate_positive_needs_review_stack() assert c.shared_state.current_best["action"] == "integrate" - assert c.shared_state.current_best["kernel_id"] == "k001+k004" + assert c.shared_state.current_best["variant_name"] == "k001+k004" assert c.shared_state.cumulative_gain_validated == pytest.approx(2.0) assert validation_calls == 1 resolved_entries = [ @@ -474,7 +474,7 @@ async def _noop_roofline(*, reason: str): await c._recover_interrupted_stack_validation() assert validation_calls == 0 - assert c.shared_state.current_best["kernel_id"] == "k001+k004" + assert c.shared_state.current_best["variant_name"] == "k001+k004" assert not c.shared_state.pending_stack_validation_result resolved = [ entry @@ -569,7 +569,7 @@ async def _noop_roofline(*, reason: str): await c._on_enter_sweep(from_phase="KERNEL") assert len(validation_calls) == 1 - assert c.shared_state.current_best["kernel_id"] == "k001+k004" + assert c.shared_state.current_best["variant_name"] == "k001+k004" @pytest.mark.asyncio @@ -1032,7 +1032,7 @@ async def test_phase_transition_into_sweep_enqueues_conc_sweep_e2e(tmp_path: Pat coord.shared_state.phase = "KERNEL" coord.shared_state.kernel_enabled = True coord.shared_state.baseline_tput = 100.0 - coord.shared_state.cumulative_gain = 12.0 + coord.shared_state.cumulative_gain_validated = 12.0 coord.shared_state.last_profile_trace = "/tmp/dummy.trace.json.gz" coord.shared_state.phase_history = [ {"to_phase": "EXPLORE", "evidence": {}, "reason": "prelude_done"}, diff --git a/src/hyperloom/inference_optimizer/tests/test_task_group_dispatch_accounting.py b/src/hyperloom/inference_optimizer/tests/test_task_group_dispatch_accounting.py index fb6651dd1f..2fd324a98c 100644 --- a/src/hyperloom/inference_optimizer/tests/test_task_group_dispatch_accounting.py +++ b/src/hyperloom/inference_optimizer/tests/test_task_group_dispatch_accounting.py @@ -576,10 +576,11 @@ def result(kernel_id: str, task_group_key: str) -> dict: state.record_kernel_opt(result("k002", "task-b")) state.record_kernel_opt(result("k002", "task-a")) + # k002 moved to task-a; the stable entry for task-a now belongs to k002. assert state.kernel_opt_attempts["k002"]["task_group_key"] == "task-a" - assert state.kernel_opt_attempts["k001"]["task_group_key"] == "task-b" assert state.kernel_opt_attempts["k002"]["attempts"] == 2 - assert state.kernel_opt_attempts["k001"]["attempts"] == 1 + # k001 still has its own stable record (task-a was its starting key). + assert len(state.kernel_opt_task_attempts) >= 2 def test_single_way_ordinal_reuse_preserves_pending_keep(tmp_path): diff --git a/src/hyperloom/inference_optimizer/tests/test_untried_hot_dedup.py b/src/hyperloom/inference_optimizer/tests/test_untried_hot_dedup.py index db29568f8b..33f0ec3d44 100644 --- a/src/hyperloom/inference_optimizer/tests/test_untried_hot_dedup.py +++ b/src/hyperloom/inference_optimizer/tests/test_untried_hot_dedup.py @@ -22,8 +22,7 @@ def _state(hot, *, rejected=(), attempts=None): last_trace_analyze={"hot_kernels_top15": hot, "task_groups": []}, optimization_stack=[], rejected_kernel_ids=list(rejected), - kernel_opt_attempts=attempts or {}, - kernel_opt_task_attempts=None, # populated by _ensure_kernel_task_state + kernel_opt_task_attempts=dict(attempts or {}), ) diff --git a/src/hyperloom/inference_optimizer/tests/test_warm_replay.py b/src/hyperloom/inference_optimizer/tests/test_warm_replay.py index 8dd1fe525d..5405e3affc 100644 --- a/src/hyperloom/inference_optimizer/tests/test_warm_replay.py +++ b/src/hyperloom/inference_optimizer/tests/test_warm_replay.py @@ -44,7 +44,6 @@ class _StubSharedState: explore_search: dict = field(default_factory=dict) optimization_stack: list = field(default_factory=list) gain_per_stack_entry: list = field(default_factory=list) - cumulative_gain: float = 0.0 cumulative_gain_validated: float = 0.0 cumulative_gain_validated_ts: str = "" cumulative_gain_validated_stack_len: int = 0 @@ -59,6 +58,13 @@ class _StubSharedState: def save(self, *args, **kwargs): # noqa: D401 — stub pass + def append_stack_gain_entry(self, *, action, variant_name, new_tput, extra_server_args="", ts=None): + from hyperloom.common.gain_math import gain_pct + + entry_gain_pct = gain_pct(float(new_tput or 0.0), float(self.baseline_tput or 0.0)) + self.gain_per_stack_entry.append(entry_gain_pct) + return entry_gain_pct + def set_stop_reason(self, reason: str) -> None: self.stop_reason = reason @@ -856,7 +862,7 @@ async def test_warm_replay_falls_back_to_recipe_when_context_not_hit(tmp_path): def test_promote_warm_replay_reproduced_pushes_stack_and_updates_gain( tmp_path, ): - """When measured gain ≥ expected × min_reproduce, push the warm config onto the stack and bump cumulative_gain.""" + """When measured gain ≥ expected × min_reproduce, push the warm config onto the stack and bump the validated gain.""" coord = _make_coord(tmp_path, warm_start_recipe=_warm_recipe_t1()) coord.shared_state.warm_replay_outcome = { "status": "in_flight", @@ -886,11 +892,10 @@ def test_promote_warm_replay_reproduced_pushes_stack_and_updates_gain( assert entry["extra_envs"] == {"VLLM_ROCM_USE_AITER": "1"} assert entry["tput"] == 738.0 assert coord.shared_state.gain_per_stack_entry == [23.0] - assert coord.shared_state.cumulative_gain == 23.0 assert coord.shared_state.cumulative_gain_validated == 23.0 assert coord.shared_state.cumulative_gain_validated_ts assert coord.shared_state.cumulative_gain_validated_stack_len == 1 - assert coord.shared_state.current_best["action"] == "warm_replay" + assert coord.shared_state.current_best["action"] == "replay_warm_recipe" assert coord.shared_state.current_best["tput"] == 738.0 @@ -923,7 +928,7 @@ def test_promote_warm_replay_keeps_prebaseline_enablement_as_zero_gain_anchor( "replay_warm_recipe", ] assert coord.shared_state.gain_per_stack_entry == [None, 23.0] - assert coord.shared_state.cumulative_gain == 23.0 + assert coord.shared_state.cumulative_gain_validated == 23.0 assert coord.shared_state.cumulative_gain_validated_stack_len == 2 @@ -965,7 +970,7 @@ def test_promote_warm_replay_rejected_by_failed_quality_gate(tmp_path): assert outcome["quality_gate"]["passed"] is False assert coord.shared_state.optimization_stack == [] assert coord.shared_state.current_best == {} - assert coord.shared_state.cumulative_gain == 0.0 + assert coord.shared_state.cumulative_gain_validated == 0.0 @pytest.mark.parametrize( @@ -1033,7 +1038,7 @@ def test_promote_warm_replay_passes_quality_gate_is_promoted(tmp_path): outcome = coord.shared_state.warm_replay_outcome assert outcome["status"] == "reproduced" assert len(coord.shared_state.optimization_stack) == 1 - assert coord.shared_state.current_best["action"] == "warm_replay" + assert coord.shared_state.current_best["action"] == "replay_warm_recipe" def test_promote_warm_replay_double_run_uses_hot_measure_round(tmp_path): @@ -1063,16 +1068,16 @@ def test_promote_warm_replay_double_run_uses_hot_measure_round(tmp_path): coord._promote_warm_replay(result, task=task) cb = coord.shared_state.current_best - assert cb["action"] == "warm_replay" + assert cb["action"] == "replay_warm_recipe" assert cb["tput"] == 738.0 - assert cb["hot_tput"] == 738.0 - assert cb["cold_tput"] == 690.0 + # The measured rounds are audit metadata on the stack entry, not config. + assert "hot_tput" not in cb + assert "cold_tput" not in cb entry = coord.shared_state.optimization_stack[0] assert entry["tput"] == 738.0 assert entry["hot_tput"] == 738.0 assert entry["cold_tput"] == 690.0 assert entry["gain_pct"] == 23.0 - assert coord.shared_state.cumulative_gain == 23.0 assert coord.shared_state.cumulative_gain_validated == 23.0 @@ -1099,7 +1104,7 @@ def test_promote_warm_replay_adopts_on_any_positive_gain(tmp_path): assert outcome["actual_gain_pct"] == 10.0 assert outcome.get("below_historical_reproduce_pct") is True assert len(coord.shared_state.optimization_stack) == 1 - assert coord.shared_state.current_best["action"] == "warm_replay" + assert coord.shared_state.current_best["action"] == "replay_warm_recipe" def test_promote_warm_replay_no_gain_is_drift(tmp_path): @@ -1122,7 +1127,7 @@ def test_promote_warm_replay_no_gain_is_drift(tmp_path): outcome = coord.shared_state.warm_replay_outcome assert outcome["status"] == "drift" assert coord.shared_state.optimization_stack == [] - assert coord.shared_state.cumulative_gain == 0.0 + assert coord.shared_state.cumulative_gain_validated == 0.0 def test_promote_warm_replay_succeeded_but_zero_gain_is_drift(tmp_path): @@ -1709,7 +1714,6 @@ def test_promote_warm_replay_cumulative_gain_uses_tput_ratio(tmp_path): # baseline 600, measured 738 -> gain = 23% via tput ratio. result = {"status": "succeeded", "output_throughput": 738.0} coord._promote_warm_replay(result, task=task) - assert coord.shared_state.cumulative_gain == 23.0 assert coord.shared_state.cumulative_gain_validated == 23.0 diff --git a/src/hyperloom/inference_optimizer/tests/test_workload_envs.py b/src/hyperloom/inference_optimizer/tests/test_workload_envs.py index 35ce8dedd4..59ec16d0b7 100644 --- a/src/hyperloom/inference_optimizer/tests/test_workload_envs.py +++ b/src/hyperloom/inference_optimizer/tests/test_workload_envs.py @@ -116,6 +116,25 @@ def test_materialize_remove_args_and_string_unset_env(tmp_path, monkeypatch): assert envs["SGLANG_REMOVE_ME"] == "override" +def test_materialize_refuses_to_unset_pinned_workload_envs(tmp_path, monkeypatch): + """Unsetting TP/CONC would retarget the benchmark, not toggle a knob.""" + _clear_env(monkeypatch) + src = tmp_path / "base.yaml" + _write(src, envs={"TP": 1, "CONC": 64, "SGLANG_TUNING_KNOB": "1"}) + bench = _materialize( + src, + tmp_path / "out", + unset_envs=["TP", "CONC", "RUN_EVAL", "SGLANG_TUNING_KNOB"], + ) + envs = bench["envs"] + + assert "TP" in envs + assert "CONC" in envs + assert "RUN_EVAL" in envs + # A plain tuning knob is still removable; only the pins are protected. + assert "SGLANG_TUNING_KNOB" not in envs + + def test_materialize_pd_forces_string_prompts_for_lm_eval(tmp_path, monkeypatch): # PD-disaggregated: force lm_eval string prompts so the sglang_router's # /v1/completions (StringOrArray) does not 422 on token-id prompts. diff --git a/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py b/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py index 2d87065ff8..2e8c807b18 100644 --- a/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py +++ b/src/hyperloom/inference_optimizer/tools/read_optimizer_state.py @@ -21,7 +21,7 @@ SUMMARY_KEYS = ( "stop_reason", "baseline_tput", - "cumulative_gain", + "cumulative_gain_validated", "current_best", "last_kernel_opt", "last_trace_analyze", diff --git a/src/hyperloom/inference_optimizer/tools/robustness_monitor.sh.example b/src/hyperloom/inference_optimizer/tools/robustness_monitor.sh.example index 12904400e6..0cc4d0073c 100644 --- a/src/hyperloom/inference_optimizer/tools/robustness_monitor.sh.example +++ b/src/hyperloom/inference_optimizer/tools/robustness_monitor.sh.example @@ -7,7 +7,7 @@ # Polls ``$INFERENCE_OPTIMIZER_SESSION_DIR/state.json`` every 300s. On a # terminal session (any ``stop_reason`` in ``STOP_REASON_VOCAB``, ``phase=CLOSE``, # or ``reports/final.md`` present), exits 0 without resuming. Otherwise it only -# re-launches the session via ``optimize --resume --resume-from "$session_dir"`` +# re-launches the session via ``optimize --resume-from "$session_dir"`` # when the optimizer is judged dead by a ROBUST, multi-signal liveness check # (issue #592). The session is considered ALIVE when ANY of the following hold: # * the optimizer owner pid published in ``$session_dir/runtime/optimizer.lock`` @@ -34,7 +34,7 @@ # after MAX_HOURS + 1 buffer) # TARGET_GAIN 10 --target-gain forwarded to # ``python -m hyperloom.inference_optimizer.cli optimize -# --resume --resume-from "$session_dir"`` +# --resume-from "$session_dir"`` # INFERENCE_OPTIMIZER_SESSION_DIR (unset) session-dir holding state.json; # when set it wins outright # LAUNCH_INFO_FILE (unset) launch-info JSON written by @@ -336,14 +336,14 @@ while [ "$(date +%s)" -lt "$deadline" ]; do mkdir -p "$resume_log_dir" resume_log="$resume_log_dir/resume_$(date +%Y%m%d_%H%M%S).log" MAGPIE_PYTHON="${MAGPIE_PYTHON}" setsid nohup python3 -m hyperloom.inference_optimizer.cli --verbose optimize \ - --resume --resume-from "$session_dir" \ + --resume-from "$session_dir" \ --target-gain "${TARGET_GAIN:-10}" --max-hours "${MAX_HOURS:-24}" \ --tick-interval-sec 30 \ > "$resume_log" 2>&1 < /dev/null & # Record the REAL resumed optimizer pid, not the setsid wrapper ($!), which # exits immediately and would make the next loop misfire another resume. sleep 5 - resumed_pid="$(pgrep -f 'hyperloom.inference_optimizer.cli .*--verbose optimize --resume --resume-from '"$session_dir" | head -1)" + resumed_pid="$(pgrep -f 'hyperloom.inference_optimizer.cli .*--verbose optimize --resume-from '"$session_dir" | head -1)" if [ -n "$resumed_pid" ]; then echo "$resumed_pid" > "$PID_FILE" echo "[robustness] resumed optimizer pid=$resumed_pid log=$resume_log" diff --git a/src/hyperloom/orchestrator/actions/executors/_geak_sweep.py b/src/hyperloom/orchestrator/actions/executors/_geak_sweep.py index 65bf504e07..827eacb5f1 100644 --- a/src/hyperloom/orchestrator/actions/executors/_geak_sweep.py +++ b/src/hyperloom/orchestrator/actions/executors/_geak_sweep.py @@ -21,7 +21,7 @@ from pathlib import Path from typing import Any -from hyperloom.common.env_safety import scrub_benchmark_process_env +from hyperloom.common.env_safety import build_benchmark_env from hyperloom.common.jsonio import read_json from ._grid_base import pareto_front @@ -163,8 +163,7 @@ async def sweep_via_geak( variant_idx += 1 out_dir = output_root / variant_name out_dir.mkdir(parents=True, exist_ok=True) - env = scrub_benchmark_process_env(dict(os.environ)) - env.update( + env = build_benchmark_env( { "BACKEND": backend, "OUT_DIR": str(out_dir), diff --git a/src/hyperloom/orchestrator/actions/executors/_grid_base.py b/src/hyperloom/orchestrator/actions/executors/_grid_base.py index 63d42af2a6..20ab253255 100644 --- a/src/hyperloom/orchestrator/actions/executors/_grid_base.py +++ b/src/hyperloom/orchestrator/actions/executors/_grid_base.py @@ -17,7 +17,7 @@ from typing import Any from hyperloom.common.coerce import to_str_list -from hyperloom.common.env_safety import filter_benchmark_env_mapping +from hyperloom.common.env_safety import filter_untrusted_env_mapping, is_allowed_variant_env_key from ._canonical_fingerprint import canonical_fingerprint log = logging.getLogger(__name__) @@ -77,8 +77,8 @@ class GridVariant: name (str): Human-readable label for the variant. extra_server_args (str): Backend server args appended via ``EXTRA_{SGLANG,VLLM,ATOM}_ARGS``. Defaults to ``""``. - extra_envs (dict[str, str]): Per-variant environment overrides. - Defaults to an empty dict. + extra_envs (dict[str, str]): Per-variant environment overrides, minus + any name in ``BLOCKED_VARIANT_ENV_NAMES``. Defaults to an empty dict. remove_args (list[str]): Base/server flags to remove before appending this variant's args. Defaults to ``[]``. unset_envs (list[str]): Inherited environment keys to remove before @@ -116,7 +116,7 @@ def __init__( Args: name: Variant name. extra_server_args: Extra server CLI args for this variant. - extra_envs: Extra environment variables for this variant. + extra_envs: Extra environment variables; unsafe names are dropped. note: Optional reason/category note. remove_args: Base/server args to remove before appending this variant's args. @@ -126,7 +126,12 @@ def __init__( """ self.name = name self.extra_server_args = extra_server_args - self.extra_envs = filter_benchmark_env_mapping(extra_envs) + self.extra_envs, dropped_envs = filter_untrusted_env_mapping( + extra_envs, + allow_predicate=is_allowed_variant_env_key, + ) + if dropped_envs: + log.warning("Variant %s: dropping unsafe extra_envs %s", name, ", ".join(sorted(dropped_envs))) self.remove_args = to_str_list(remove_args) self.unset_envs = to_str_list(unset_envs) mode = str(args_mode or "append").strip().lower() diff --git a/src/hyperloom/orchestrator/actions/executors/_grid_runner.py b/src/hyperloom/orchestrator/actions/executors/_grid_runner.py index 491a15c916..b83f1780f9 100644 --- a/src/hyperloom/orchestrator/actions/executors/_grid_runner.py +++ b/src/hyperloom/orchestrator/actions/executors/_grid_runner.py @@ -26,6 +26,7 @@ from hyperloom.common.env import is_truthy from hyperloom.common.env_safety import ( BLOCKED_CHILD_ENV_NAMES, + BLOCKED_EXTERNAL_ENV_NAMES, _ENV_KEY_RE, is_python_package_root, redact_secret_values, @@ -595,7 +596,8 @@ def _build_variant_yaml( Args: base_yaml_path (Path): Path to the base Magpie YAML to template from. base_extra_args (str): Server args merged ahead of the variant's args. - variant (GridVariant): The variant whose flags/envs are applied. + variant (GridVariant): The variant whose flags/envs are applied; its + ``unset_envs`` may not remove a workload pin. output_subdir (Path): Directory the per-variant ``config.yaml`` is written into. model_path (str | None): Overrides ``benchmark.model`` when set. @@ -646,6 +648,10 @@ def _build_variant_yaml( elif extra_args_env in envs: envs.pop(extra_args_env, None) for k in getattr(variant, "unset_envs", []) or []: + # Unsetting a pin retargets the benchmark rather than toggling a knob. + if str(k).strip().upper() in BLOCKED_EXTERNAL_ENV_NAMES: + log.warning("grid: refusing to unset pinned env %s for variant %s", k, variant.name) + continue envs.pop(str(k), None) for k, v in variant.extra_envs.items(): envs[str(k)] = str(v) @@ -2737,9 +2743,6 @@ def _not_run_skip_result(variant: GridVariant, stopped: StoppedByTheRun) -> Vari ) -SINGLE_NODE_DEFAULT_KEEP_THRESHOLD_PCT = 1.0 -MULTI_NODE_DEFAULT_KEEP_THRESHOLD_PCT = 2.0 - def _existing_log_path(path: Path) -> str | None: """Return ``path`` as a string when it exists, else ``None``. @@ -2868,12 +2871,10 @@ def _write_variant_abort_marker_impl( __all__ = [ "DEFAULT_SGLANG_WATCHDOG_TIMEOUT_SEC", "GridVariant", - "MULTI_NODE_DEFAULT_KEEP_THRESHOLD_PCT", "ORCHESTRATOR_CANCELLED_CLASS", "SESSION_TIME_EXHAUSTED_CLASS", "StoppedByTheRun", "SGLANG_WATCHDOG_TIMEOUT_ENV", - "SINGLE_NODE_DEFAULT_KEEP_THRESHOLD_PCT", "VariantResult", "apply_multi_node_invalid_variants", "apply_aiter_moe_pin_filter", diff --git a/src/hyperloom/orchestrator/actions/executors/_grid_server_args.py b/src/hyperloom/orchestrator/actions/executors/_grid_server_args.py index cc440eebc8..688d795bb4 100644 --- a/src/hyperloom/orchestrator/actions/executors/_grid_server_args.py +++ b/src/hyperloom/orchestrator/actions/executors/_grid_server_args.py @@ -187,47 +187,6 @@ def compose_server_args( return merge_server_args(pruned, variant_extra_args) -def split_config_changes( - config_changes: dict[str, str], -) -> tuple[str, dict[str, str]]: - """Split a flat config_changes dict into (server_args_str, envs_dict). - - Keys starting with ``-`` are CLI flags: they are rebuilt into an argv - string (``--flag value`` or bare ``--flag``) and returned as - ``server_args``. All other keys are env vars and returned as ``envs``. - This translates the legacy flat representation produced by - ``_framework_config_levers_from_done`` into the structured form that - ``GridVariant`` expects so ``--``-prefixed flags reach ``EXTRA_{FW}_ARGS`` - instead of being silently dropped by ``valid_env_key``. - - Args: - config_changes: Flat dict from a framework specialist deliverable, - mixing ``--flag: value`` server-arg keys with ``ENV_VAR: value`` - env keys. - - Returns: - A ``(server_args, envs)`` tuple where ``server_args`` is an argv-like - string safe to assign to ``GridVariant.extra_server_args`` and - ``envs`` is a ``dict[str, str]`` for ``GridVariant.extra_envs``. - """ - from ._grid_base import coerce_extra_envs - - arg_tokens: list[str] = [] - env_items: dict[str, str] = {} - for k, v in (config_changes or {}).items(): - key = str(k).strip() - val = str(v).strip() - if key.startswith("-"): - if val: - arg_tokens.append(f"{key}={val}" if "=" not in key else f"{key} {val}") - else: - arg_tokens.append(key) - else: - if key: - env_items[key] = val - server_args = merge_server_args(*arg_tokens) if arg_tokens else "" - return server_args, coerce_extra_envs(env_items) - # A JSON "bareword": an identifier-like token that appears where a double-quoted # JSON key or string value should be (letters/digits/underscore plus the ``.``, diff --git a/src/hyperloom/orchestrator/actions/executors/_multi_node_env.py b/src/hyperloom/orchestrator/actions/executors/_multi_node_env.py index 23bb7ce8fd..af3aac92f4 100644 --- a/src/hyperloom/orchestrator/actions/executors/_multi_node_env.py +++ b/src/hyperloom/orchestrator/actions/executors/_multi_node_env.py @@ -69,7 +69,7 @@ def mn_bench_warmup_enabled() -> bool: def is_multi_node() -> bool: """True iff the optimizer is operating on a >=2-node RayJob cluster. - State file wins over env so ``--resume`` works: state ``nodes`` >= 2 wins, + State file wins over env so a resume works: state ``nodes`` >= 2 wins, else fall back to ``$INFERENCE_OPTIMIZER_NODES``. Returns: @@ -93,7 +93,7 @@ def resolve_kb_topology() -> dict[str, Any]: """Resolve the node/GPU and PD-disaggregation topology for the KB hardware suffix. Mirrors :func:`is_multi_node`'s source priority so the recipe KB key stays - stable across ``--resume``: the ``multi_node_state.json`` values win + stable across a resume: the ``multi_node_state.json`` values win (persisted), then the ``INFERENCE_OPTIMIZER_NODES`` / ``INFERENCE_OPTIMIZER_GPUS_PER_NODE`` env fallbacks. The CLI exports both before the T0 anchor, so a fresh run (where the state file is not written @@ -157,7 +157,7 @@ def _pd_nodes(env_key: str, *state_keys: str) -> int: # Parallel formation (tp / ep) is fixed at launch, not explored, so it # belongs in the KB key: a best_config tuned at one split is invalid at - # another. The CLI exports TP / EP before T0 (stable across --resume), so + # another. The CLI exports TP / EP before T0 (stable across a resume), so # env wins; state fields are the resume fallback. def _int_pref_env(env_key: str, *state_keys: str, default: int = 1) -> int: raw = (os.environ.get(env_key, "") or "").strip() diff --git a/src/hyperloom/orchestrator/actions/executors/_multi_node_server_lifecycle.py b/src/hyperloom/orchestrator/actions/executors/_multi_node_server_lifecycle.py index a89c9eaa16..7e11f2f966 100644 --- a/src/hyperloom/orchestrator/actions/executors/_multi_node_server_lifecycle.py +++ b/src/hyperloom/orchestrator/actions/executors/_multi_node_server_lifecycle.py @@ -211,7 +211,7 @@ def _intf(kw, sk, ek): pn = _intf(pd_prefill_nodes, "last_restart_pd_prefill_nodes", "PD_PREFILL_NODES") dn = _intf(pd_decode_nodes, "last_restart_pd_decode_nodes", "PD_DECODE_NODES") # Resume fallback: the restart path launches with ``pn or len(pods)`` but - # persists the raw arg (often 0), so a --resume that also lost the + # persists the raw arg (often 0), so a resume that also lost the # ``$PD_*_NODES`` env would leave pn/dn at 0 and wrongly fail the # disaggregated gate below (e.g. auto-roofline after resume). Recover the # group sizes from the discovered per-role pod lists the hand-off diff --git a/src/hyperloom/orchestrator/actions/executors/_ray_serving.py b/src/hyperloom/orchestrator/actions/executors/_ray_serving.py index 635d9558eb..0d9b077c7e 100644 --- a/src/hyperloom/orchestrator/actions/executors/_ray_serving.py +++ b/src/hyperloom/orchestrator/actions/executors/_ray_serving.py @@ -854,17 +854,6 @@ def poll_started(self) -> int | None: self._start_ref = None return self._pid - def pending_seconds(self) -> float: - """Seconds the actor has been PENDING (submitted, not yet scheduled). - - Zero before :meth:`start_async` and after the pid is obtained. Used by - the caller to enforce a pending-time deadline separate from the running - wall budget (§3.3 / invariant §6.4). - """ - if self._pending_started_monotonic is None or self._pid is not None: - return 0.0 - return max(0.0, time.monotonic() - self._pending_started_monotonic) - def pid(self) -> int | None: """Return the launched pid, or ``None`` before it has been resolved. diff --git a/src/hyperloom/orchestrator/actions/executors/_server_patcher.py b/src/hyperloom/orchestrator/actions/executors/_server_patcher.py index f37511b852..a6b471b08b 100644 --- a/src/hyperloom/orchestrator/actions/executors/_server_patcher.py +++ b/src/hyperloom/orchestrator/actions/executors/_server_patcher.py @@ -51,7 +51,7 @@ # Required in ``vllm/config/profiler.py`` for the server to accept the flags. _VLLM_PROFILER_SENTINELS: tuple[str, ...] = ( - "capture_torch_profiler", + "capture_torch_profiler_dir", "detailed_trace_annotation", ) diff --git a/src/hyperloom/orchestrator/actions/executors/_workload_envs.py b/src/hyperloom/orchestrator/actions/executors/_workload_envs.py index 6e83bc2a2e..bb2b1f015d 100644 --- a/src/hyperloom/orchestrator/actions/executors/_workload_envs.py +++ b/src/hyperloom/orchestrator/actions/executors/_workload_envs.py @@ -31,7 +31,8 @@ from hyperloom.common.coerce import to_str_list from hyperloom.common.env_safety import ( - filter_benchmark_env_mapping, + BENCHMARK_SECRET_ENV_NAMES, + BLOCKED_EXTERNAL_ENV_NAMES, filter_untrusted_env_mapping, valid_env_key, ) @@ -753,7 +754,7 @@ def materialize_config_with_envs( extra_envs: Overrides applied last over any computed env values. remove_args: Inherited framework server args to remove before launch. unset_envs: Inherited env names to remove before applying - ``extra_envs``. + ``extra_envs``; workload pins are refused. args_mode: ``"append"`` (default) or ``"replace"`` for ``extra_server_args``. model_path: Model path/id; overrides ``benchmark.model`` when set. @@ -1069,12 +1070,6 @@ def materialize_config_with_envs( ("max_iterations", f"--profiler-config.max_iterations {max_iters}"), ] if tracelens_patch_ok: - profiler_flags.append( - ( - "capture_torch_profiler", - "--profiler-config.capture_torch_profiler True", - ) - ) profiler_flags.append(("detailed_trace_annotation", "--profiler-config.detailed_trace_annotation True")) # vLLM's AsyncLLM-side profiler tracks no iterations and captures the # whole start_profile..stop_profile range, so it has to stay off @@ -1510,6 +1505,10 @@ def materialize_config_with_envs( if remove_list: envs[framework_env] = remove_server_args(envs.get(framework_env, ""), remove_list) for key in unset_list: + # Unsetting a pin retargets the benchmark rather than toggling a knob. + if str(key).strip().upper() in BLOCKED_EXTERNAL_ENV_NAMES: + log.warning("Refusing to unset pinned benchmark env %s", key) + continue envs.pop(str(key), None) for key in unset_list: if isinstance(extra_envs, dict) and key in extra_envs: @@ -1549,12 +1548,15 @@ def materialize_config_with_envs( # _finalize_framework_server_args already ran, so re-apply the sink-side # guard it ends with rather than shipping an unvalidated string. envs[framework_env] = validate_server_args_shell_safe(merged) - filtered_envs = filter_benchmark_env_mapping(envs) - dropped_credentials = sorted(set(envs) - set(filtered_envs)) + # The rendered YAML is persisted, so credentials must not reach it. + filtered_envs, dropped_credentials = filter_untrusted_env_mapping( + envs, + allow_predicate=lambda key: key not in BENCHMARK_SECRET_ENV_NAMES, + ) if dropped_credentials: log.warning( "Dropping control-plane credentials from benchmark envs: %s", - ", ".join(dropped_credentials), + ", ".join(sorted(dropped_credentials)), ) envs.clear() envs.update(filtered_envs) diff --git a/src/hyperloom/orchestrator/actions/executors/benchmark_result.py b/src/hyperloom/orchestrator/actions/executors/benchmark_result.py index 1643089a90..7f77e66f86 100644 --- a/src/hyperloom/orchestrator/actions/executors/benchmark_result.py +++ b/src/hyperloom/orchestrator/actions/executors/benchmark_result.py @@ -955,6 +955,7 @@ def is_valid_measurement(result: dict[str, Any] | None) -> bool: # Fraction of the leading warmup samples dropped before averaging so the # estimate reflects sustained decode rather than the cold-start climb. +# ``bypass_analysis`` parses the same log with 0.2 and a different clamp bound. _DEFAULT_WARMUP_SKIP_FRAC: float = 0.25 diff --git a/src/hyperloom/orchestrator/actions/executors/bypass_analysis.py b/src/hyperloom/orchestrator/actions/executors/bypass_analysis.py index c7267383c3..cbb8257437 100644 --- a/src/hyperloom/orchestrator/actions/executors/bypass_analysis.py +++ b/src/hyperloom/orchestrator/actions/executors/bypass_analysis.py @@ -77,7 +77,8 @@ def steady_state_mean(samples: list[float], *, warmup_skip_frac: float = 0.2) -> """Average the steady-state portion of throughput samples. Drops the leading ``warmup_skip_frac`` of samples before averaging; falls - back to the full set when the trim would empty it. + back to the full set when the trim would empty it. ``benchmark_result`` + parses the same log with 0.25 and a different clamp bound. Args: samples: Positive throughput samples in log order. diff --git a/src/hyperloom/orchestrator/actions/executors/bypass_runner.py b/src/hyperloom/orchestrator/actions/executors/bypass_runner.py index 632994b2d3..883f07c294 100644 --- a/src/hyperloom/orchestrator/actions/executors/bypass_runner.py +++ b/src/hyperloom/orchestrator/actions/executors/bypass_runner.py @@ -34,7 +34,7 @@ import yaml -from hyperloom.common.env_safety import scrub_benchmark_process_env +from hyperloom.common.env_safety import build_benchmark_env from . import bypass_analysis from . import bypass_engine @@ -837,19 +837,17 @@ def _server_env( profile_dir: str | None, bench_envs: dict | None = None, ) -> dict[str, str]: - """Build the server subprocess env (parent + profiler dirs + GPU pin).""" - env = scrub_benchmark_process_env(os.environ.copy()) - # GPU pin: the materializer writes ROCR_VISIBLE_DEVICES into benchmark.envs - # (reconciled against TP). Inject it so the server binds the same cards - # Magpie would; missing on single-GPU pods (harmless). - rocr = str((bench_envs or {}).get("ROCR_VISIBLE_DEVICES") or "").strip() - if rocr: - env["ROCR_VISIBLE_DEVICES"] = rocr - if profile and profile_dir: - env["VLLM_TORCH_PROFILER_DIR"] = profile_dir - env["SGLANG_TORCH_PROFILER_DIR"] = profile_dir - env["ATOM_TORCH_PROFILER_DIR"] = profile_dir - return env + """Build the server subprocess env from the materialized benchmark envs. + + The whole mapping is exported, so an env-only candidate is a real experiment + rather than a rerun of the baseline. + """ + profiler_dirs = ( + dict.fromkeys(("VLLM_TORCH_PROFILER_DIR", "SGLANG_TORCH_PROFILER_DIR", "ATOM_TORCH_PROFILER_DIR"), profile_dir) + if profile and profile_dir + else None + ) + return build_benchmark_env(bench_envs, profiler_dirs) def _launch_server(cmd: list[str], env: dict[str, str], server_log: Path) -> subprocess.Popen: @@ -904,7 +902,7 @@ def _run_subprocess(cmd: list[str], timeout_s: float, workspace: Path, tag: str) capture_output=True, text=True, timeout=timeout_s, - env=scrub_benchmark_process_env(os.environ.copy()), + env=build_benchmark_env(), ) except subprocess.TimeoutExpired: _append_log(workspace, tag, "", f"{tag} timed out after {timeout_s}s") diff --git a/src/hyperloom/orchestrator/actions/executors/bypass_scriptable.py b/src/hyperloom/orchestrator/actions/executors/bypass_scriptable.py index e4e7c141a3..7cd7cb3670 100644 --- a/src/hyperloom/orchestrator/actions/executors/bypass_scriptable.py +++ b/src/hyperloom/orchestrator/actions/executors/bypass_scriptable.py @@ -26,7 +26,7 @@ from pathlib import Path from typing import Any -from hyperloom.common.env_safety import scrub_benchmark_process_env +from hyperloom.common.env_safety import build_benchmark_env from hyperloom.inference_optimizer.session.paths import asset_root @@ -97,25 +97,23 @@ def build_scriptable_env( Returns: The environment mapping for the scriptable subprocess. """ - env = scrub_benchmark_process_env(os.environ.copy()) - env["MODEL"] = str(bench.get("model") or env.get("MODEL", "")) + # Defaults are overridable by the YAML envs; run-scoped values are not. + defaults: dict[str, str] = {"MODEL": str(bench.get("model") or os.environ.get("MODEL", ""))} if bench.get("precision"): - env["PRECISION"] = str(bench["precision"]) - for key, value in (bench.get("envs") or {}).items(): - env[str(key).upper()] = str(value) - env["RUNNER_TYPE"] = runner_type - env["RESULT_FILENAME"] = "inferencex_result" - env["RESULT_DIR"] = str(workspace) - # Profiler: scriptable scripts (e.g. xDiT) gate tracing on PROFILE=1 and - # read the trace dir from VLLM/SGLANG_TORCH_PROFILER_DIR (mirrors the - # serving path's _server_env). Only set when enabled so default runs are - # untouched. + defaults["PRECISION"] = str(bench["precision"]) + run_scoped: dict[str, str] = { + "RUNNER_TYPE": runner_type, + "RESULT_FILENAME": "inferencex_result", + "RESULT_DIR": str(workspace), + } + # Scriptable scripts (e.g. xDiT) gate tracing on PROFILE=1 and read the + # trace dir from VLLM/SGLANG_TORCH_PROFILER_DIR. if profile: - env["PROFILE"] = "1" + run_scoped["PROFILE"] = "1" if profile_dir: - env["VLLM_TORCH_PROFILER_DIR"] = profile_dir - env["SGLANG_TORCH_PROFILER_DIR"] = profile_dir - return scrub_benchmark_process_env(env) + run_scoped["VLLM_TORCH_PROFILER_DIR"] = profile_dir + run_scoped["SGLANG_TORCH_PROFILER_DIR"] = profile_dir + return build_benchmark_env(defaults, bench.get("envs"), run_scoped) def run_scriptable( diff --git a/src/hyperloom/orchestrator/actions/executors/integrate_patch.py b/src/hyperloom/orchestrator/actions/executors/integrate_patch.py index eea1ed9441..d5428f5b55 100644 --- a/src/hyperloom/orchestrator/actions/executors/integrate_patch.py +++ b/src/hyperloom/orchestrator/actions/executors/integrate_patch.py @@ -25,6 +25,7 @@ from ...framework.paths import resolve_session_framework_root, resolve_source_file_allowlist from ...specialists.patch_safety import patch_file_targets, patch_targets_missing from ...state.shared_state import inject_stack_base_params, resolve_grading_anchor_tput +from ..stop_attribution import stopped_by_the_run_class from ._accuracy_gate import ( DEFAULT_ENABLEMENT_ACCURACY_FLOOR, accuracy_keep_block, @@ -1781,7 +1782,7 @@ async def _stage_resolve( "patches_reverted": [], "config_changes_applied": {}, } - task_id = str(getattr(ctx.task, "task_id", "") or "build_launch_probe") + task_id = ctx.task.task_id scratch = runs_dir(self.session_dir, "integrate_patch", task_id) scratch.mkdir(parents=True, exist_ok=True) ctx._ip_specialist_task_id = task_id # type: ignore[attr-defined] @@ -2038,7 +2039,7 @@ def _base_reverted(error_class: str, reason: str) -> dict[str, Any]: f"localization touches path(s) outside the allowlist: {outside[:8]}", ) - loc_dir = runs_dir(self.session_dir, "integrate_patch", str(getattr(ctx.task, "task_id", "") or "localize")) + loc_dir = runs_dir(self.session_dir, "integrate_patch", ctx.task.task_id) loc_dir = loc_dir / "localization" loc_dir.mkdir(parents=True, exist_ok=True) gap_slug = re.sub(r"[^A-Za-z0-9_.-]+", "_", action.gap_id or "localization") @@ -2077,9 +2078,7 @@ async def _stage_apply( setup_result = _run_setup_commands( setup_cmds, cwd=self.session_dir, - log_dir=runs_dir( - self.session_dir, "integrate_patch", str(getattr(ctx.task, "task_id", "") or "setup") - ), + log_dir=runs_dir(self.session_dir, "integrate_patch", ctx.task.task_id), ) specialist_workspace: Path = ctx._ip_specialist_workspace # type: ignore[attr-defined] @@ -2998,7 +2997,7 @@ async def _gate_perf( gate_evidence: dict[str, Any], ctx: Any, ) -> dict[str, Any]: - """Throughput KEEP / REVERT decision with optional stack rebench.""" + """Throughput KEEP / REVERT decision, or no verdict when the run stopped it.""" base_tput = float(params.get("base_tput") or 0.0) # Grade against the current live anchor, not a stale task snapshot. live_anchor = resolve_grading_anchor_tput(shared_state) @@ -3012,6 +3011,29 @@ async def _gate_perf( base_tput = live_anchor keep_threshold_pct = float(params.get("keep_threshold_pct", self.keep_threshold_pct)) + + stopped = stopped_by_the_run_class(bench_result.get("error_class")) + if stopped is not None: + artifacts_reverted = self._revert_artifacts(applied_artifacts) + reverted = self._revert_patches(framework_root, applied) + return _with_stash_restore( + framework_root, + stash_state, + stash_note, + { + "status": "failed", + "error_class": stopped.error_class, + "error": stopped.interrupted, + "specialist_task_id": specialist_task_id, + "patches_applied": [], + "patches_reverted": [str(p) for p in reverted], + "artifacts_reverted": artifacts_reverted, + "config_changes_applied": {}, + "bench_result": bench_result, + "workspace": str(output_root), + }, + ) + new_tput = bench_result.get("output_throughput") delta_pct = None if isinstance(new_tput, (int, float)) and new_tput > 0 and base_tput > 0: @@ -4127,6 +4149,7 @@ async def _bench_patch( # Benchmark dir; ``_grade_accuracy`` locates accuracy artifacts here. "workspace": str(getattr(r, "workspace", "") or ""), "error": getattr(r, "error", "") or "", + "error_class": getattr(r, "error_class", "") or "", "nonfatal_warnings": list(getattr(r, "nonfatal_warnings", []) or []), # Materialized config used for this bench; needed by revalidation. "materialized_config": str(config_path), diff --git a/src/hyperloom/orchestrator/actions/executors/report.py b/src/hyperloom/orchestrator/actions/executors/report.py index 5888d56492..2d476d454b 100644 --- a/src/hyperloom/orchestrator/actions/executors/report.py +++ b/src/hyperloom/orchestrator/actions/executors/report.py @@ -6,8 +6,8 @@ Reads SharedState + the bus event log and writes ``$SESSION_DIR/reports/final.json`` (machine-readable, dashboard shape) and ``final.md`` (human-readable). The returned dict surfaces both paths. -``final.json`` carries the run identity, stop reason, baseline/best, per-round -and validated cumulative gain, completeness annotations, event counts and +``final.json`` carries the run identity, stop reason, baseline/best, the +validated cumulative gain, completeness annotations, event counts and highlights, plus optional blocks (failure summary, roofline comparison, external baseline, concurrency-sweep and kernel-optimization pointers) when the corresponding data exists; ``final.md`` renders the same content as sections. @@ -408,10 +408,9 @@ def _build_failure_summary( "sweep_done": "SWEEP finished the configured concurrency / shape grid.", "conc_sweep_done": "Post-sweep concurrency sweep finished.", "conc_sweep_failed": "Post-sweep concurrency sweep reached a failed terminal result.", - "explore_force_exit_low_budget": "EXPLORE force-exited: the remaining wall-clock budget was too low to start new work.", + "explore_force_exit_low_budget": "EXPLORE force-exited: it had spent its own phase budget down to the force-exit threshold.", "framework_agent_phase_done": "The framework-enablement agent completed its phase.", "framework_agent_plateau": "The framework-enablement agent plateaued with no further progress.", - "framework_agent_force_exit_low_budget": "The framework-enablement agent force-exited on a low remaining budget.", "global_converged": "Cyclic phases converged: repeated macro-cycles stopped yielding new validated gain.", # Pre-flight gates (fail fast before booting a server). "model_context_window_too_small": "Preflight gate: the model's max context window cannot hold the requested ISL + OSL.", @@ -544,7 +543,6 @@ def _build_summary_dict( "baseline_tput": state.baseline_tput, "baseline_accuracy": state.baseline_accuracy, "current_best": state.current_best, - "cumulative_gain": state.cumulative_gain, # Validated gain (what the run actually delivered). "cumulative_gain_validated": state.cumulative_gain_validated, "cumulative_gain_validated_ts": state.cumulative_gain_validated_ts, @@ -635,9 +633,7 @@ def _format_md(summary: dict[str, Any]) -> str: f"- current_best : `{framework_registry.format_primary_metric(_fw, cb_tput)}` " f"(action=`{cb.get('action', '?')}`)" ) - # Per-round sum — informational, not end-to-end deliverable. - lines.append(f"- cumulative_gain : `{summary['cumulative_gain']:.2f}%` *(per-round sum — informational only)*") - # Validated gain — always printed so the report never quotes only the raw sum. + # Printed even when never validated, so a missing rebench is stated, not implied. val_gain = summary.get("cumulative_gain_validated", 0.0) or 0.0 val_ts = summary.get("cumulative_gain_validated_ts") or "" val_len = summary.get("cumulative_gain_validated_stack_len", 0) or 0 @@ -1174,7 +1170,7 @@ def _write_kernel_opt_summary( """Build + write ``reports/kernel_optimization_summary.json``. Best-effort (failure logged, returns ``None`` so the final.json write - still happens). Aggregates ``kernel_opt_attempts`` with per-kernel + still happens). Aggregates ``kernel_opt_task_attempts`` with per-kernel ``results/.json`` for the "why no optimized kernel?" view. Args: @@ -1436,10 +1432,9 @@ async def __call__(self, ctx) -> dict[str, Any]: md_path.write_text(_format_md(summary), encoding="utf-8") log.info( - "report_executor: wrote %s and %s (cumulative_gain=%.2f%% per_round_sum / %.2f%% validated)", + "report_executor: wrote %s and %s (cumulative_gain_validated=%.2f%%)", md_path, json_path, - state.cumulative_gain, state.cumulative_gain_validated, ) publish_result = self._maybe_publish_results(session_dir, state) diff --git a/src/hyperloom/orchestrator/framework/artifacts.py b/src/hyperloom/orchestrator/framework/artifacts.py index a277601a30..b28d32a474 100644 --- a/src/hyperloom/orchestrator/framework/artifacts.py +++ b/src/hyperloom/orchestrator/framework/artifacts.py @@ -8,36 +8,18 @@ - :func:`candidate_key` is the canonical candidate identity (precedence ``candidate_id or pr_url or ref``) used for candidate selection, dedup, progress-row keying, and task idempotency across the whole pump. -- :func:`candidate_slug` is the shared path-slug helper the writers use. -- :func:`write_decision_json` drops a uniform ``decision.json`` under - ``runs/framework_agent//`` for every candidate terminal event - (critic-denied, executor KEEP/REVERT/apply_failed/..., authored-patch - KEEP/REVERT). This is the single per-candidate fate record an operator - or downstream tool can read without parsing the whole event log. -- :func:`write_semantic_audit` writes ``semantic_audit.json`` + a readable - ``semantic_audit.md`` alongside ``decision.json``. - :func:`summarize_candidate_outcomes` classifies a batch's progress rows into ``empty_discovery`` / ``tested_no_keep`` / ``tested_with_keep`` so the phase-done summary, report, and robustness advisory can tell "discovered nothing" apart from "tested candidates but none cleared the gate". -The key/slug helpers are pure; the writers are best-effort — a write failure -logs at debug and returns ``None`` rather than raising into the pump. +All helpers here are pure. """ from __future__ import annotations -import json -import logging -from datetime import datetime, timezone -from pathlib import Path from typing import Any -from hyperloom.inference_optimizer.session.session_paths import runs_dir - - -log = logging.getLogger(__name__) - # Per-candidate terminal statuses that mean the candidate reached the apply/bench # stage (as opposed to being filtered before any source change). @@ -69,139 +51,6 @@ def candidate_key(row: dict[str, Any] | None) -> str: return str(row.get("candidate_id") or row.get("pr_url") or row.get("ref") or "") -def candidate_slug(candidate_id: str) -> str: - """Filesystem-safe slug for a candidate id (PR url / ref / synthetic id). - - Args: - candidate_id: The candidate identifier (may contain ``/``, ``:`` …). - - Returns: - A lowercased slug with non-``[a-z0-9._-]`` runs collapsed to ``-``, - capped at 96 chars, defaulting to ``"candidate"`` when empty. - """ - out: list[str] = [] - for ch in str(candidate_id).lower(): - out.append(ch if (ch.isalnum() or ch in ".-_") else "-") - slug = "".join(out).strip("-") - return (slug or "candidate")[:96] - - -def write_decision_json( - session_dir: Path | str, - *, - candidate_id: str, - batch_id: str = "", - status: str, - kept: bool = False, - reason: str = "", - provenance: str = "", - gain_pct: float | None = None, - accuracy_pass: bool | None = None, - extra: dict[str, Any] | None = None, -) -> str | None: - """Write ``runs/framework_agent//decision.json`` for one candidate. - - Best-effort: returns the written path, or ``None`` on any failure (never - raises — observability must not wedge the pump). - - Args: - session_dir: The session root directory. - candidate_id: The candidate identifier (used for the slug + payload). - batch_id: The discovery batch this candidate belonged to. - status: The terminal status (e.g. ``kept`` / ``reverted`` / - ``critic_denied`` / ``apply_failed`` / ``already_present``). - kept: Whether the candidate was promoted into the stack. - reason: Human-readable rationale (critic rationale, failure text, …). - provenance: ``raw_diff`` / ``authored`` / ``critic`` / ``audit`` … - gain_pct: Measured throughput delta vs baseline, when benched. - accuracy_pass: Accuracy-gate verdict, when evaluated. - extra: Optional additional fields merged into the payload. - - Returns: - The absolute path to the written ``decision.json``, or ``None``. - """ - try: - slug = candidate_slug(candidate_id) - out_dir = runs_dir(Path(session_dir), "framework_agent", slug) - out_dir.mkdir(parents=True, exist_ok=True) - payload: dict[str, Any] = { - "candidate_id": str(candidate_id), - "batch_id": str(batch_id or ""), - "status": str(status or ""), - "kept": bool(kept), - "provenance": str(provenance or ""), - "reason": str(reason or ""), - "gain_pct": (float(gain_pct) if isinstance(gain_pct, (int, float)) else None), - "accuracy_pass": (bool(accuracy_pass) if isinstance(accuracy_pass, bool) else None), - "ts": datetime.now(timezone.utc).isoformat(), - } - if isinstance(extra, dict): - for k, v in extra.items(): - payload.setdefault(str(k), v) - dest = out_dir / "decision.json" - dest.write_text(json.dumps(payload, indent=2, sort_keys=True), encoding="utf-8") - return str(dest) - except Exception: # noqa: BLE001 — observability is best-effort - log.debug("framework_agent_artifacts: write_decision_json failed", exc_info=True) - return None - - -def write_semantic_audit( - session_dir: Path | str, - *, - candidate_id: str, - verdict: dict[str, Any], -) -> str | None: - """Persist a candidate's semantic-audit verdict next to its decision.json. - - Writes ``semantic_audit.json`` + a readable ``semantic_audit.md`` under - ``runs/framework_agent//``, alongside ``decision.json``. - Best-effort: returns the JSON path, or ``None`` on failure. - - Args: - session_dir: The session root directory. - candidate_id: The candidate identifier (slug source). - verdict: The ``fa phase-audit`` verdict dict. - - Returns: - The absolute path to ``semantic_audit.json``, or ``None``. - """ - if not isinstance(verdict, dict) or not verdict: - return None - try: - slug = candidate_slug(candidate_id) - out_dir = runs_dir(Path(session_dir), "framework_agent", slug) - out_dir.mkdir(parents=True, exist_ok=True) - json_path = out_dir / "semantic_audit.json" - json_path.write_text(json.dumps(verdict, indent=2, sort_keys=True), encoding="utf-8") - lines = [ - f"# Semantic audit — {candidate_id}", - "", - f"- semantic_status: {verdict.get('semantic_status')}", - f"- applicability: {verdict.get('applicability')}", - f"- recommended_next_step: {verdict.get('recommended_next_step')}", - f"- confidence: {verdict.get('confidence')}", - f"- layer: {verdict.get('layer')}", - "", - "## Evidence", - ] - for ev in verdict.get("evidence") or []: - if isinstance(ev, dict): - lines.append( - f"- {ev.get('local_file') or '(file?)'}" - + (f" [{ev.get('symbol')}]" if ev.get("symbol") else "") - + (f": {ev.get('reason')}" if ev.get("reason") else "") - ) - risks = verdict.get("risks") or [] - if risks: - lines += ["", "## Risks", *[f"- {r}" for r in risks]] - (out_dir / "semantic_audit.md").write_text("\n".join(lines) + "\n", encoding="utf-8") - return str(json_path) - except Exception: # noqa: BLE001 — observability is best-effort - log.debug("framework_agent_artifacts: write_semantic_audit failed", exc_info=True) - return None - - def summarize_candidate_outcomes( progress: list[dict[str, Any]] | None, *, @@ -250,8 +99,5 @@ def summarize_candidate_outcomes( __all__ = [ "candidate_key", - "candidate_slug", "summarize_candidate_outcomes", - "write_decision_json", - "write_semantic_audit", ] diff --git a/src/hyperloom/orchestrator/kernel/_kernel_decisions.py b/src/hyperloom/orchestrator/kernel/_kernel_decisions.py index cbd0d0a1a9..bce2078665 100644 --- a/src/hyperloom/orchestrator/kernel/_kernel_decisions.py +++ b/src/hyperloom/orchestrator/kernel/_kernel_decisions.py @@ -251,26 +251,11 @@ def _queue_kernel_keep( def _ensure_kernel_task_state(state) -> None: - """Lazily migrate ordinal attempts and KEEP patches into stable ledgers.""" + """Initialise the stable ledger and re-queue the KEEPs recorded in it.""" if not isinstance(getattr(state, "kernel_opt_task_attempts", None), dict): state.kernel_opt_task_attempts = {} if not isinstance(getattr(state, "pending_kernel_integrations", None), dict): state.pending_kernel_integrations = {} - for ledger_id, raw_entry in (state.kernel_opt_attempts or {}).items(): - if not isinstance(raw_entry, dict): - continue - entry = dict(raw_entry) - task_key = _stable_kernel_task_key( - task_group_key=str(entry.get("task_group_key") or ""), - kernel_id=str(entry.get("kernel_id") or ledger_id), - source_file=str(entry.get("last_source_file") or ""), - ) - entry.setdefault("stable_task_key", task_key) - entry.setdefault( - "current_kernel_id", - str(entry.get("kernel_id") or ledger_id), - ) - state.kernel_opt_task_attempts.setdefault(task_key, entry) for task_key, stable_entry in state.kernel_opt_task_attempts.items(): if not isinstance(stable_entry, dict): continue @@ -493,7 +478,7 @@ def _stamp_integration_validation( if task_key: entries.append((state.kernel_opt_task_attempts or {}).get(task_key)) if kernel_id: - entries.append((state.kernel_opt_attempts or {}).get(kernel_id)) + entries.append(_entry_by_kernel_id(state, kernel_id)) for attempt in entries: if not isinstance(attempt, dict): continue @@ -555,9 +540,7 @@ def record_kernel_integrate_result( kernel_id, patch_path, target_file, extra_args = _resolve_kernel_patch_identity(state, result) task_group_key = str( result.get("task_group_key") - or ((state.kernel_opt_attempts or {}).get(kernel_id) or {}).get( - "task_group_key" - ) + or (_entry_by_kernel_id(state, kernel_id) or {}).get("task_group_key") or "" ) integration_id = str(result.get("integration_id") or "") @@ -796,14 +779,6 @@ def record_kernel_integrate_result( stable_attempt["integration_status"] = "rejected" stable_attempt["integration_rejected_reason"] = reason stable_attempt["integration_rejected_at"] = _now_iso() - ordinal_attempt = (state.kernel_opt_attempts or {}).get(kernel_id) - if ( - isinstance(ordinal_attempt, dict) - and str(ordinal_attempt.get("stable_task_key") or "") == task_key - ): - ordinal_attempt["integration_status"] = "rejected" - ordinal_attempt["integration_rejected_reason"] = reason - ordinal_attempt["integration_rejected_at"] = _now_iso() return entry @@ -944,41 +919,7 @@ def record_kernel_opt(state, result: dict[str, Any]) -> None: ) ts = _now_iso() - prior_ledger_id = kernel_id - prior_entry = dict(state.kernel_opt_attempts.get(kernel_id) or {}) - if task_group_key: - matching_ledger = next( - ( - (ledger_id, ledger_entry) - for ledger_id, ledger_entry in ( - state.kernel_opt_attempts or {} - ).items() - if isinstance(ledger_entry, dict) - and str(ledger_entry.get("task_group_key") or "") - == task_group_key - ), - None, - ) - if matching_ledger is not None: - prior_ledger_id, matching_entry = matching_ledger - prior_entry = dict(matching_entry) - if prior_ledger_id != kernel_id: - displaced_entry = state.kernel_opt_attempts.get(kernel_id) - state.kernel_opt_attempts.pop(prior_ledger_id, None) - if ( - isinstance(displaced_entry, dict) - and str(displaced_entry.get("task_group_key") or "") - != task_group_key - ): - # Preserve a task currently occupying the new ordinal slot. - # Reranking commonly swaps two IDs; the vacated prior slot is - # collision-free and remains discoverable by task_group_key. - state.kernel_opt_attempts[prior_ledger_id] = displaced_entry - state.rejected_kernel_ids = [ - rejected_id - for rejected_id in (state.rejected_kernel_ids or []) - if rejected_id != prior_ledger_id - ] + prior_entry = dict(_entry_by_kernel_id(state, kernel_id) or {}) legacy_task_keys = { str(item) for item in (result.get("legacy_task_group_keys") or []) @@ -1023,21 +964,6 @@ def record_kernel_opt(state, result: dict[str, Any]) -> None: prior_entry = dict(stable_prior_entry) if migrated_stable_key and migrated_stable_key != stable_task_key: state.kernel_opt_task_attempts.pop(migrated_stable_key, None) - for ledger_id, legacy_entry in list( - state.kernel_opt_attempts.items() - ): - if ledger_id == kernel_id or not isinstance( - legacy_entry, - dict, - ): - continue - if ( - str(legacy_entry.get("stable_task_key") or "") - == migrated_stable_key - or str(legacy_entry.get("task_group_key") or "") - == migrated_stable_key - ): - state.kernel_opt_attempts.pop(ledger_id, None) prior_task_group_key = ( task_group_key if migrated_stable_key @@ -1053,11 +979,10 @@ def record_kernel_opt(state, result: dict[str, Any]) -> None: member_id for member_id in [ kernel_id, - prior_ledger_id, *task_group_kernel_ids, ] if str( - (state.kernel_opt_attempts.get(member_id) or {}).get( + (_entry_by_kernel_id(state, member_id) or {}).get( "task_group_key" ) or "" @@ -1262,7 +1187,6 @@ def record_kernel_opt(state, result: dict[str, Any]) -> None: entry["task_group_shape_case_count"] = int( result.get("task_group_shape_case_count") or 0 ) - state.kernel_opt_attempts[kernel_id] = entry state.kernel_opt_task_attempts[stable_task_key] = dict(entry) _queue_kernel_keep( state, @@ -1547,6 +1471,37 @@ def has_keep_pending_integrate(state) -> bool: return bool(next_pending_keep_kernel_id(state)) +def index_attempts_by_kernel_id(attempts: Any) -> dict[str, dict]: + """Re-index a stable-keyed attempt ledger by trace-local ``current_kernel_id``. + + The ordinal id is not an identity — reranking moves it between operators, so + two stable entries can claim the same one. The latest-stamped entry wins, + which is the one currently occupying the ordinal slot. + + Args: + attempts: A ``kernel_opt_task_attempts`` mapping, or anything falsy. + + Returns: + ``{current_kernel_id: attempt}``, holding the ledger's own entry dicts. + """ + latest: dict[str, tuple[str, dict]] = {} + for entry in (attempts or {}).values(): + if not isinstance(entry, dict): + continue + kernel_id = str(entry.get("current_kernel_id") or "") + if not kernel_id: + continue + ts = str(entry.get("last_ts") or entry.get("ts") or "") + if ts >= latest.get(kernel_id, ("", {}))[0]: + latest[kernel_id] = (ts, entry) + return {kernel_id: entry for kernel_id, (_ts, entry) in latest.items()} + + +def _entry_by_kernel_id(state, kernel_id: str) -> dict | None: + """The stable-ledger entry currently holding ``kernel_id``, or ``None``.""" + return index_attempts_by_kernel_id(state.kernel_opt_task_attempts).get(kernel_id) + + def kernel_opt_attempts_count(state) -> int: """Number of distinct kernel tasks with recorded kernel_opt attempts. diff --git a/src/hyperloom/orchestrator/kernel/attempt_summary.py b/src/hyperloom/orchestrator/kernel/attempt_summary.py index 4a79873d33..a2d88142dd 100644 --- a/src/hyperloom/orchestrator/kernel/attempt_summary.py +++ b/src/hyperloom/orchestrator/kernel/attempt_summary.py @@ -3,11 +3,12 @@ """Aggregate kernel-optimization attempts into a single forensic report. -Combines the per-kernel ledger (:attr:`SharedState.kernel_opt_attempts`) and -the collective campaign history (:attr:`SharedState.collective_attempts`) with -the kernel-agent run results to explain why the kernel-agent did not produce an -optimized kernel. All public helpers are pure functions over ``SharedState`` + -``session_dir`` returning JSON-ready dicts; never raise on missing files. +Combines the per-kernel ledger (:attr:`SharedState.kernel_opt_task_attempts`) +and the collective campaign history (:attr:`SharedState.collective_attempts`) +with the kernel-agent run results to explain why the kernel-agent did not +produce an optimized kernel. All public helpers are pure functions over +``SharedState`` + ``session_dir`` returning JSON-ready dicts; never raise on +missing files. """ from __future__ import annotations @@ -56,7 +57,7 @@ UNATTEMPTED_NEVER_DISPATCHED = "never_dispatched" UNATTEMPTED_UNKNOWN = "unknown" -#: ``kernel_opt_attempts`` rejection reasons we surface verbatim into +#: ``kernel_opt_task_attempts`` rejection reasons we surface verbatim into #: ``rejection_breakdown`` totals (anything else falls into ``other``). KNOWN_REJECTION_REASONS = ( "revert_decision", @@ -909,9 +910,7 @@ def build_kernel_optimization_summary( ) raw_attempts: dict[str, dict[str, Any]] = dict( - getattr(state, "kernel_opt_task_attempts", {}) - or getattr(state, "kernel_opt_attempts", {}) - or {} + getattr(state, "kernel_opt_task_attempts", {}) or {} ) attempts_map: dict[str, dict[str, Any]] = {} for ledger_id, attempt in raw_attempts.items(): diff --git a/src/hyperloom/orchestrator/kernel/request_handlers.py b/src/hyperloom/orchestrator/kernel/request_handlers.py index cfb6ecb6c9..13227856b6 100644 --- a/src/hyperloom/orchestrator/kernel/request_handlers.py +++ b/src/hyperloom/orchestrator/kernel/request_handlers.py @@ -42,6 +42,7 @@ async def handler(payload: dict, *, session_dir: Path) -> dict: DEFAULT_CODEX_MODEL, ) +from ..actions.stop_attribution import stopped_by_the_run_class from ..trace.llm_trace import LLMCallRecord, append_llm_call from ..trace.task_progress import heartbeat_while_output_flows from ..trace.parse_usage import ( @@ -54,6 +55,8 @@ async def handler(payload: dict, *, session_dir: Path) -> dict: # Re-exported: callers patch these at ``request_handlers.``. from ._kernel_decisions import ( _honest_flag as _honest_flag, + _entry_by_kernel_id as _entry_by_kernel_id, + index_attempts_by_kernel_id as index_attempts_by_kernel_id, _resolve_kernel_patch_identity as _resolve_kernel_patch_identity, kernel_patch_key as kernel_patch_key, find_rejected_kernel_patch as find_rejected_kernel_patch, @@ -1395,7 +1398,7 @@ def _fill_integrate_defaults_from_state( kernel_id = str(resolved.get("kernel_id") or "") if kernel_id: - attempt = (state.kernel_opt_attempts or {}).get(kernel_id) or {} + attempt = _entry_by_kernel_id(state, kernel_id) or {} if not resolved.get("task_group_key"): task_group_key = str(attempt.get("task_group_key") or "") if task_group_key: @@ -1549,7 +1552,7 @@ def _resolve_integrate_payload(payload: dict, *, session_dir: Path) -> tuple[dic # Multi-KEEP queue fallback: pull patch_path/source_file from the per-kernel # ledger for KEEPs other than the strongest pending one. if kernel_id: - attempt = (state.kernel_opt_attempts or {}).get(kernel_id) or {} + attempt = _entry_by_kernel_id(state, kernel_id) or {} _fill_integrate_snapshot_from_bundle(resolved, attempt.get("last_artifact_bundle")) if not resolved.get("snapshot_dir") and attempt.get("last_snapshot_dir"): resolved["snapshot_dir"] = str(attempt["last_snapshot_dir"]) @@ -4084,8 +4087,6 @@ def _active_forge_fusion_env_flags(state: Any) -> dict[str, str]: return {} if str(current_best.get("action") or "") != "fusion": return {} - if str(current_best.get("engine") or "") != "forge_fusion": - return {} envs = current_best.get("extra_envs") if isinstance(current_best, dict) else {} if not isinstance(envs, dict): return {} @@ -5891,8 +5892,8 @@ def _batch_kernel_candidates( # Build the "live" exclusion sets up front (empty without session_dir). rejected_kernel_ids: set[str] = set() - attempts_by_kid: dict[str, dict] = {} attempts_by_task: dict[str, dict] = {} + attempts_by_kid: dict[str, dict] = {} in_flight: set[str] = set() from ..state.shared_state import ( resolve_hot_kernel_min_gpu_pct, @@ -5907,8 +5908,8 @@ def _batch_kernel_candidates( state = SharedState.load_or_init(session_dir) rejected_kernel_ids = set(state.rejected_kernel_ids or []) - attempts_by_kid = dict(state.kernel_opt_attempts or {}) attempts_by_task = dict(state.kernel_opt_task_attempts or {}) + attempts_by_kid = index_attempts_by_kernel_id(attempts_by_task) in_flight = _in_flight_kernel_ids(session_dir) except Exception: log.exception( @@ -6003,10 +6004,7 @@ def _is_live( recorded_ledger = next( ( (ledger_id, entry) - for ledger_id, entry in { - **attempts_by_kid, - **attempts_by_task, - }.items() + for ledger_id, entry in attempts_by_task.items() if isinstance(entry, dict) and ( ( @@ -6020,9 +6018,9 @@ def _is_live( ) or ( not group_key - and ledger_id in member_ids and group_id and str(entry.get("task_group_id") or "") == group_id + and str(entry.get("current_kernel_id") or "") in member_ids ) ) ), @@ -7562,6 +7560,21 @@ def _restore_aiter_rebuild_env() -> None: rebaseline_error_class = ( str((bench_result or {}).get("error_class") or "").strip() if isinstance(bench_result, dict) else "" ) or "bench_exception" + stopped = stopped_by_the_run_class(rebaseline_error_class) + if stopped is not None: + # Nothing was measured, so the patch has no verdict to answer for. + return { + "status": "failed", + "error_class": stopped.error_class, + "error": stopped.interrupted, + "decision": "NEEDS_REVIEW", + "rebaseline_detail": bench_result, + "kernel_id": kernel_id, + "patch_path": patch_path, + "target_file": payload.get("target_file") or payload.get("source_file"), + "apply_result": apply_result, + "revert_result": revert_result, + } return { "status": "failed", "error_class": rebaseline_error_class, diff --git a/src/hyperloom/orchestrator/kernel/roofline_ceiling.py b/src/hyperloom/orchestrator/kernel/roofline_ceiling.py index e017d74419..49b7531812 100644 --- a/src/hyperloom/orchestrator/kernel/roofline_ceiling.py +++ b/src/hyperloom/orchestrator/kernel/roofline_ceiling.py @@ -1038,23 +1038,6 @@ def _read_baseline_yaml_conc(state: Any) -> int: ) -def _resolve_effective_concurrency(state: Any) -> int: - """Resolve the concurrency the actual benchmark ran with (returns int >= 1; on-disk baseline yaml CONC wins, since ``state.conc`` can stay stale and under-count the ceiling 8x). - - Args: - state: Shared run state (baseline yaml CONC preferred over ``conc``). - - Returns: - The effective concurrency, always ``>= 1``. - """ - yaml_conc = _read_baseline_yaml_conc(state) - if yaml_conc > 0: - return yaml_conc - conc = int(getattr(state, "conc", 0) or 0) - if conc > 0: - return conc - return 1 - @dataclass(frozen=True) class RooflineBreakdown: diff --git a/src/hyperloom/orchestrator/knowledge/remote_recipe/values.py b/src/hyperloom/orchestrator/knowledge/remote_recipe/values.py index b53bfd38e3..2d9c15798e 100644 --- a/src/hyperloom/orchestrator/knowledge/remote_recipe/values.py +++ b/src/hyperloom/orchestrator/knowledge/remote_recipe/values.py @@ -152,40 +152,6 @@ def add(self, source: Any, *, category: str, kind: str, name: str = "") -> str: self._sources[source_key] = rel return rel - def add_tree(self, source: Any, *, category: str, kind: str) -> list[str]: - """Copy a required artifact directory while preserving its relative tree.""" - raw = str(source or "").strip() - if not raw: - return [] - root = Path(raw) - if root.is_symlink() or not root.is_dir(): - raise RemoteRecipeValidationError( - f"accepted {category} artifact tree cannot be materialized: {root}" - ) - refs: list[str] = [] - for src in sorted(root.rglob("*")): - if src.is_symlink(): - raise RemoteRecipeValidationError( - f"accepted {category} artifact tree contains a symlink: {src}" - ) - if not src.is_file(): - continue - if src.stat().st_size > MAX_FILE_BYTES: - raise RemoteRecipeValidationError( - f"artifact {src} exceeds the {MAX_FILE_BYTES}-byte KB Store limit" - ) - relative = src.relative_to(root).as_posix() - rel = f"{category}/{kind}/{relative}" - destination = self.root / rel - destination.parent.mkdir(parents=True, exist_ok=True) - shutil.copy2(src, destination) - self.artifacts.append( - Artifact(path=rel, source=destination, kind=kind, meta={"origin": category}) - ) - self.refs.add(rel) - refs.append(rel) - return refs - def validate_adoption(self, source: Path, rel: str) -> None: """Fail before merge when a staged file cannot safely own ``rel``.""" if source.is_symlink(): @@ -1173,10 +1139,7 @@ def build_remote_knowledge( framework_entries = [item for item in stack if _entry_origin(item) == "framework"] current_best = _mapping(getattr(state, "current_best", {})) optimized_throughput = _number(current_best.get("tput")) - validated_gain = _number( - getattr(state, "cumulative_gain_validated", 0.0) - or getattr(state, "cumulative_gain", 0.0) - ) + validated_gain = _number(getattr(state, "cumulative_gain_validated", 0.0)) gains = list(getattr(state, "gain_per_stack_entry", []) or []) worked = _experience(state, "what_worked") or _worked_from_stack(stack, gains) if sections is None: diff --git a/src/hyperloom/orchestrator/loop/conversation.py b/src/hyperloom/orchestrator/loop/conversation.py index bce606daf5..e509471345 100644 --- a/src/hyperloom/orchestrator/loop/conversation.py +++ b/src/hyperloom/orchestrator/loop/conversation.py @@ -1044,9 +1044,7 @@ def _acceptance_threshold_advisory_block(self) -> str: The rendered advisory text, or ``""`` when not applicable. """ state = self.shared_state - keep = self._decaying_keep_threshold_pct() - if keep is None: - return "" + keep = _phase_state.resolve_keep_threshold(state) cycle = int(getattr(state, "macro_cycle", 0) or 0) if cycle < 1: return "" diff --git a/src/hyperloom/orchestrator/loop/coordinator.py b/src/hyperloom/orchestrator/loop/coordinator.py index aa7676a439..db198d2286 100644 --- a/src/hyperloom/orchestrator/loop/coordinator.py +++ b/src/hyperloom/orchestrator/loop/coordinator.py @@ -999,9 +999,9 @@ def router(self) -> IntentRouter: "_handle_gemm_tuning_result": "phase_kernel", "_sync_profile_state_after_gemm_roofline": "phase_kernel", "_journal_gemm_tuning_keep": "phase_kernel", - "_promote_gemm_tuning_keep": "phase_kernel", "_replace_latest_gemm_tuning_attempt": "phase_kernel", - "_validate_forge_gemm_tuning_e2e": "phase_kernel", + "_gemm_e2e_candidates": "phase_kernel", + "_validate_gemm_tuning_e2e": "phase_kernel", "_should_continue_kernel_after_gemm": "phase_kernel", "_run_kernel_opt_after_gemm": "phase_kernel", "_current_tput_from_validated_gain": "phase_kernel", @@ -1148,7 +1148,6 @@ def router(self) -> IntentRouter: "_kb_best_config_overrides_for_keep": "proposals", "_kb_amend_recipe": "proposals", "_inject_explore_runtime_params": "proposals", - "_decaying_keep_threshold_pct": "proposals", "_materialize_approved_proposal": "proposals", "_record_proposal_task_map": "proposals", "_registry_lanes_ttl": "dispatcher", @@ -1175,7 +1174,6 @@ def router(self) -> IntentRouter: "_record_policy_denied": "writeback", "_record_observation": "writeback", "_record_kernel_opt_partial": "writeback", - "_update_cumulative_gain_validated": "writeback", "_record_integrate_keep": "writeback", "_is_promotable_result": "writeback", "_record_intervention_for_task": "writeback", @@ -1196,11 +1194,12 @@ def router(self) -> IntentRouter: "ensure_recipe_finalized": "writeback", "finalize_recipe_and_journal": "writeback", "_lift_to_current_best": "writeback", + "_update_cumulative_gain_validated": "writeback", "_promote_to_shared_state": "writeback", "_should_run_prelude_bootstrap": "writeback", "_detect_resume_state": "writeback", "replay_for_resume": "writeback", - "_materialize_stack_config_for_resume": "writeback", + "_current_best_launch_config": "writeback", "build_env_spec": "writeback", "_resume_consistency_pass": "writeback", "_resume_reenter_kernel_if_needed": "writeback", @@ -1864,11 +1863,12 @@ async def run( # attempt, including stop-check exits that never enter PHASE_CLOSE. await self._recipe_kb_t4_hook() log.info( - "Coordinator.run: stopped tick=%d reason=%s baseline_tput=%.1f cumulative_gain=%.2f%% max_minutes=%.0f", + "Coordinator.run: stopped tick=%d reason=%s baseline_tput=%.1f " + "cumulative_gain_validated=%.2f%% max_minutes=%.0f", tick_n, stop_reason or "unknown", self.shared_state.baseline_tput, - self.shared_state.cumulative_gain, + self.shared_state.cumulative_gain_validated, max_minutes_value, ) # Best-effort cleanup of installed signal handlers. diff --git a/src/hyperloom/orchestrator/loop/maintenance.py b/src/hyperloom/orchestrator/loop/maintenance.py index b31269bdec..0e88360bb8 100644 --- a/src/hyperloom/orchestrator/loop/maintenance.py +++ b/src/hyperloom/orchestrator/loop/maintenance.py @@ -13,6 +13,49 @@ log = _logging.getLogger(__name__) +async def run_lease_and_db_reclaim( + host: Any, + summary: dict[str, Any], + *, + reason: str, +) -> None: + """Reap expired serving/GPU leases, reclaim orphaned running tasks, prune the DB. + + Shared by the periodic maintenance tick and the cycle soft-restart. The + task reclaim is the R6 watchdog: a running task whose execution lease + expired is failed so a dead worker never wedges a lane indefinitely. Every + step is individually best-effort — maintenance never aborts the run loop. + + Args: + host: Anything exposing the Coordinator's ``locks``, + ``gpu_specialist_pool``, ``tasks``, ``db`` and ``cursors``. + summary: Mutated in place with the per-step counts. + reason: Reclaim reason recorded on the tasks and used as the log prefix. + """ + try: + reaped = await host.locks.reap_expired() + summary["leases_reaped"] = len(reaped or []) + except Exception: # noqa: BLE001 + log.exception("%s: serving-lease reap failed", reason) + try: + summary["gpu_leases_reaped"] = await host.gpu_specialist_pool.reap_expired() + except Exception: # noqa: BLE001 + log.exception("%s: gpu-lease reap failed", reason) + try: + reclaimed = await host.tasks.reclaim_expired_running(reason=reason) + summary["running_tasks_reclaimed"] = len(reclaimed) + except Exception: # noqa: BLE001 + log.exception("%s: running-task reclaim failed", reason) + try: + from ..bus import db_maintenance as _db_maint + + res = await _db_maint.run_db_retention(host.db, host.cursors) + summary["events_pruned"] = res.events_deleted + summary["tasks_pruned"] = res.tasks_deleted + except Exception: # noqa: BLE001 + log.exception("%s: DB retention failed", reason) + + class MaintenanceCollaborator: """Extracted collaborator; delegates unknown attrs to its Coordinator.""" @@ -48,32 +91,7 @@ async def _maybe_run_maintenance_tick( if every <= 0 or tick <= 0 or (tick % every) != 0: return None summary: dict[str, Any] = {"tick": tick} - try: - reaped = await self.locks.reap_expired() - summary["leases_reaped"] = len(reaped or []) - except Exception: # noqa: BLE001 — maintenance never aborts the run loop - log.exception("maintenance: serving-lease reap failed") - try: - summary["gpu_leases_reaped"] = await self.gpu_specialist_pool.reap_expired() - except Exception: # noqa: BLE001 - log.exception("maintenance: gpu-lease reap failed") - # R6 watchdog/self-heal: reclaim orphaned running tasks whose execution - # lease has expired so a dead worker never wedges a lane indefinitely. - try: - reclaimed = await self.tasks.reclaim_expired_running( - reason="maintenance_watchdog", - ) - summary["running_tasks_reclaimed"] = len(reclaimed) - except Exception: # noqa: BLE001 - log.exception("maintenance: running-task reclaim failed") - try: - from ..bus import db_maintenance as _db_maint - - res = await _db_maint.run_db_retention(self.db, self.cursors) - summary["events_pruned"] = res.events_deleted - summary["tasks_pruned"] = res.tasks_deleted - except Exception: # noqa: BLE001 - log.exception("maintenance: DB retention failed") + await run_lease_and_db_reclaim(self, summary, reason="maintenance_watchdog") try: disk = self._maybe_prune_runs_for_disk() if disk is not None: diff --git a/src/hyperloom/orchestrator/loop/proposals.py b/src/hyperloom/orchestrator/loop/proposals.py index 8c603dbac8..241bb3cb4e 100644 --- a/src/hyperloom/orchestrator/loop/proposals.py +++ b/src/hyperloom/orchestrator/loop/proposals.py @@ -12,6 +12,7 @@ from ..bus.message_bus import Message from .coordinator_helpers import approved_proposal_idempotency_key from ..state.shared_state import inject_stack_base_params +from ..state.task_registry import TERMINAL_STATES if TYPE_CHECKING: from ..state.task_registry import Task @@ -23,8 +24,6 @@ log = _logging.getLogger(__name__) -# Task states past which the same idempotency key may be reused for a retry. -_TERMINAL_TASK_STATES: frozenset[str] = frozenset({"succeeded", "failed", "cancelled"}) _MAX_IDEMPOTENCY_ATTEMPTS: int = 6 @@ -408,26 +407,9 @@ def _inject_explore_runtime_params(self, params: dict) -> None: es = getattr(self.shared_state, "explore_search", None) if isinstance(es, dict) and es.get("tested"): params.setdefault("explore_search", es) - keep = self._decaying_keep_threshold_pct() - if keep is not None: - params.setdefault("keep_threshold_pct", keep) - params.setdefault("stack_stable_threshold_pct", keep / 2.0) - - def _decaying_keep_threshold_pct(self) -> float | None: - """Per-cycle KEEP threshold to inject, or ``None`` to keep executor defaults. - - The bar shrinks along the shared decaying curve as macro-cycles accrue. - - Returns: - The decayed per-cycle KEEP threshold percentage. - """ - from ..actions.executors._multi_node_env import is_multi_node - - cycle = int(getattr(self.shared_state, "macro_cycle", 0) or 0) - return _phase_state.decaying_keep_threshold_pct( - cycle, - multi_node=is_multi_node(), - ) + keep = _phase_state.resolve_keep_threshold(self.shared_state) + params.setdefault("keep_threshold_pct", keep) + params.setdefault("stack_stable_threshold_pct", keep / 2.0) async def _materialize_approved_proposal( self, @@ -484,9 +466,7 @@ async def _materialize_approved_proposal( self._inject_explore_runtime_params(params) inject_stack_base_params(params, self.shared_state, anchor=True) if pending.action_name == "integrate_patch": - keep = self._decaying_keep_threshold_pct() - if keep is not None: - params.setdefault("keep_threshold_pct", keep) + params.setdefault("keep_threshold_pct", _phase_state.resolve_keep_threshold(self.shared_state)) # Seed the patched-eval server with the same base args/config every # other eval server uses, else it launches on bare framework defaults # and crashes at startup regardless of the patch. @@ -511,7 +491,7 @@ async def _materialize_approved_proposal( ) if not was_existing: break - if task.state not in _TERMINAL_TASK_STATES: + if task.state not in TERMINAL_STATES: await self._record_observation( "coordinator", "observation", diff --git a/src/hyperloom/orchestrator/loop/writeback.py b/src/hyperloom/orchestrator/loop/writeback.py index 084db34623..1af99e0710 100644 --- a/src/hyperloom/orchestrator/loop/writeback.py +++ b/src/hyperloom/orchestrator/loop/writeback.py @@ -20,6 +20,7 @@ OUTCOME_KEEP, OUTCOME_NO_PROMOTE, OUTCOME_REVERT, + PROMOTION_REFUSED_KEY, classify_change_kind, derive_journal_outcome, operation_kind_for, @@ -467,7 +468,7 @@ def _drain_agent_keep_outbox(self) -> None: self.shared_state.save(self.session_dir) def _record_kernel_opt_partial(self, result: dict[str, Any]) -> None: - """Streaming callback for ``_run_optimization_batch`` sub-attempts: write each per-kernel entry to kernel_opt_attempts immediately so the next-tick prompt is accurate mid-batch. + """Streaming callback for ``_run_optimization_batch`` sub-attempts: write each per-kernel entry to kernel_opt_task_attempts immediately so the next-tick prompt is accurate mid-batch. Args: result: One sub-attempt's per-kernel result dict. @@ -543,11 +544,9 @@ def _update_cumulative_gain_validated( async def _record_integrate_keep(self, result: dict[str, Any]) -> None: """Promote a kernel integrate KEEP into the optimization stack. - Appends a deduped ``integrate`` entry to the optimization stack, mirrors - the gain into the per-entry gain ledger, updates ``current_best`` and - ``cumulative_gain`` / ``cumulative_gain_validated``, and fires a - watermark roofline when the gain crosses the threshold. No-op when the - result lacks a positive ``new_tput``. + Stamps ``cumulative_gain_validated`` and fires a watermark roofline once + the lift is accepted. No-op without a positive ``new_tput`` or when the + lift refuses the winner. Args: result (dict[str, Any]): The integrate-patch executor result. @@ -555,92 +554,32 @@ async def _record_integrate_keep(self, result: dict[str, Any]) -> None: new_tput = result.get("new_tput") if not isinstance(new_tput, (int, float)) or new_tput <= 0: return - cb = self.shared_state.current_best or {} - extra_args = ( - str(result.get("extra_server_args") or "") - or (str(cb.get("extra_server_args") or "") if isinstance(cb, dict) else "") - ).strip() - # An integrate carries no env delta of its own except a forge-GEMM tuning - # KEEP, so inherit the stack's env layer and let that delta win. Dropping - # it published a current_best whose args and envs came from different - # configs, which every dispatch site then seeded from. - extra_envs = dict((cb.get("extra_envs") or {}) if isinstance(cb, dict) else {}) - if isinstance(result.get("extra_envs"), dict): - extra_envs.update({str(k): str(v) for k, v in result["extra_envs"].items()}) - apply_result = result.get("apply_result") or {} - backup_manifest = apply_result.get("manifest_path") if isinstance(apply_result, dict) else None - if not backup_manifest and isinstance(apply_result, dict): - stack_applies = apply_result.get("stack_apply_results") - if isinstance(stack_applies, list): - for applied in stack_applies: - if isinstance(applied, dict) and applied.get("manifest_path"): - backup_manifest = applied.get("manifest_path") - break - entry = { - "action": "integrate", - "source_phase": str(getattr(self.shared_state, "phase", "") or "KERNEL_AGENT"), - "integration_id": result.get("integration_id"), - "kernel_id": result.get("kernel_id"), - "task_group_key": result.get("task_group_key"), - "identity_route": result.get("identity_route"), - "patch_path": result.get("patch_path"), - "target_file": result.get("target_file"), - "backup_manifest": backup_manifest, - "gain_pct": result.get("gain_pct"), - "tput": float(new_tput), - "workspace": result.get("workspace"), - "ts": datetime.now(timezone.utc).isoformat(), - } - stack_kernel_ids = result.get("stack_kernel_ids") - if isinstance(stack_kernel_ids, list) and stack_kernel_ids: - entry["stack_kernel_ids"] = [str(kid) for kid in stack_kernel_ids if str(kid)] - integrate_gap_cid = str(result.get("gap_canonical_id") or "").strip() - if integrate_gap_cid: - entry["gap_canonical_id"] = integrate_gap_cid - key = ( - entry.get("integration_id"), - entry["kernel_id"], - entry["patch_path"], - entry["target_file"], + lifted = self._lift_to_current_best( + "integrate", + float(new_tput), + { + "name": result.get("kernel_id"), + "candidate_extra_server_args": result.get("extra_server_args"), + "extra_envs": {str(k): str(v) for k, v in (result.get("extra_envs") or {}).items()}, + "source_phase": str(getattr(self.shared_state, "phase", "") or "KERNEL_AGENT"), + "ttft_mean_ms": result.get("ttft_mean_ms"), + "e2el_mean_ms": result.get("e2el_mean_ms"), + "tpot_mean_ms": result.get("tpot_mean_ms"), + "workspace": result.get("workspace"), + }, + gap_canonical_id=str(result.get("gap_canonical_id") or "").strip(), + entry_extra={ + "integration_id": result.get("integration_id"), + "kernel_id": result.get("kernel_id"), + "task_group_key": result.get("task_group_key"), + "identity_route": result.get("identity_route"), + "patch_path": result.get("patch_path"), + "target_file": result.get("target_file"), + "gain_pct": result.get("gain_pct"), + "stack_kernel_ids": [str(k) for k in (result.get("stack_kernel_ids") or []) if str(k)], + }, ) - existing = { - ( - item.get("integration_id"), - item.get("kernel_id"), - item.get("patch_path"), - item.get("target_file"), - ) - for item in self.shared_state.optimization_stack - if isinstance(item, dict) and item.get("action") == "integrate" - } - if key not in existing: - self.shared_state.optimization_stack.append(entry) - # Mirror into gain_per_stack_entry so breakdown attribution works without re-walking the event log. - self.shared_state.append_stack_gain_entry( - action="integrate", - variant_name=entry.get("kernel_id"), - new_tput=new_tput, - extra_server_args=extra_args, - ts=entry["ts"], - ) - - self.shared_state.current_best = { - "action": "integrate", - "tput": float(new_tput), - "integration_id": result.get("integration_id"), - "kernel_id": result.get("kernel_id"), - "extra_server_args": extra_args, - "extra_envs": extra_envs, - "optimization_stack": list(self.shared_state.optimization_stack), - "ttft_mean_ms": result.get("ttft_mean_ms"), - "e2el_mean_ms": result.get("e2el_mean_ms"), - "tpot_mean_ms": result.get("tpot_mean_ms"), - "workspace": result.get("workspace"), - } - if self.shared_state.baseline_tput > 0: - self.shared_state.cumulative_gain = ( - (float(new_tput) - self.shared_state.baseline_tput) / self.shared_state.baseline_tput * 100.0 - ) + if lifted and self.shared_state.baseline_tput > 0: # Integrate KEEP is already rebench-validated: promote into cumulative_gain_validated + watermark. self._update_cumulative_gain_validated(new_tput) await self._maybe_enqueue_watermark_roofline( @@ -1701,18 +1640,14 @@ def _collect_workload_tags(self) -> dict[str, Any]: return out def _build_kernel_optimizations_from_state(self) -> list[dict[str, Any]]: - """Collect KEEP'd kernel optimizations + their E2E verdict by joining kernel_opt_attempts (micro) and kernel_integrate_attempts (E2E) on kernel_id; non-integrated KEEPs surface integrated=False. Returns KernelOptimization-shaped dicts. + """Collect KEEP'd kernel optimizations + their E2E verdict by joining kernel_opt_task_attempts (micro) and kernel_integrate_attempts (E2E) on kernel_id; non-integrated KEEPs surface integrated=False. Returns KernelOptimization-shaped dicts. Returns: A list of KernelOptimization-shaped dicts for each KEEP'd kernel, joined with its E2E integrate verdict where available. """ ss = self.shared_state - opt_attempts = ( - getattr(ss, "kernel_opt_task_attempts", {}) - or getattr(ss, "kernel_opt_attempts", {}) - or {} - ) + opt_attempts = getattr(ss, "kernel_opt_task_attempts", {}) or {} integ_attempts = getattr(ss, "kernel_integrate_attempts", {}) or {} if not isinstance(opt_attempts, dict): return [] @@ -1901,9 +1836,7 @@ def _build_recipe_attrs_from_state(self) -> dict[str, Any]: what_failed.append(rev) kernel_optimizations = self._coord._build_kernel_optimizations_from_state() cumulative_validated = float(getattr(ss, "cumulative_gain_validated", 0.0) or 0.0) - cumulative_total = float(getattr(ss, "cumulative_gain", 0.0) or 0.0) validated_stack_len = int(getattr(ss, "cumulative_gain_validated_stack_len", 0) or 0) - stack_fingerprint = getattr(ss, "stack_fingerprint", "") or "" # Workload-shape tags for shape-filtered warm-start queries (shared via _collect_workload_tags). workload_tags = self._coord._collect_workload_tags() # framework_version left unset here (manifest-derived); the T0 backfill writes it. @@ -1913,13 +1846,12 @@ def _build_recipe_attrs_from_state(self) -> dict[str, Any]: "what_worked": what_worked, "what_failed": what_failed, "kernel_optimizations": kernel_optimizations, - "stack_fingerprint": {"sha": str(stack_fingerprint)} if stack_fingerprint else {}, "last_profiled": str(getattr(ss, "cumulative_gain_validated_ts", "") or ""), "workload": workload_tags, "sessions": [ { "session_id": str(getattr(ss, "recipe_kb_session_id", "") or self.session_dir.name), - "gain_pct": cumulative_validated or cumulative_total, + "gain_pct": cumulative_validated, "stack_len": validated_stack_len or len(opt_stack), # arbor-shape provenance so the session row is self-describing (before/after tput + knobs). "throughput_before": float(getattr(ss, "baseline_tput", 0.0) or 0.0), @@ -2060,9 +1992,7 @@ def finalize_recipe_and_journal( ss = self.shared_state cb = getattr(ss, "current_best", {}) or {} final_tput = float(cb.get("tput", 0.0)) if isinstance(cb, dict) else 0.0 - total_gain = float( - getattr(ss, "cumulative_gain_validated", 0.0) or getattr(ss, "cumulative_gain", 0.0) or 0.0, - ) + total_gain = float(getattr(ss, "cumulative_gain_validated", 0.0) or 0.0) journal.finalize( final_throughput=final_tput if final_tput > 0 else None, total_gain_pct=total_gain, @@ -2246,14 +2176,6 @@ def finalize_recipe_and_journal( if has_validated_win and my_tput > live_tput: overrides["best_config"] = attrs["best_config"] overrides["best_throughput"] = my_tput - # Merge stack_fingerprint rather than replace (CLOSE only has the sha; T0 stamps version keys). - merged_fp = dict(existing_row.get("stack_fingerprint") or {}) - for fp_key, fp_val in (attrs.get("stack_fingerprint") or {}).items(): - if fp_val not in (None, "", {}): - merged_fp[fp_key] = fp_val - if merged_fp: - overrides["stack_fingerprint"] = merged_fp - self._kb_amend_recipe( recipe_overrides=overrides, provenance_details={ @@ -2408,20 +2330,6 @@ async def _record_specialist_result( task.task_id, ) - # ``_route_steward_verdict`` has no definition anywhere, so this branch - # cannot succeed — the except below swallows the AttributeError. - if domain == "session_steward_specialist": - try: - await self._route_steward_verdict( - task=task, - done_payload=done_payload, - ) - except Exception: # noqa: BLE001 — defensive - log.exception( - "steward routing failed for task=%s; no phase-routing change applied", - task.task_id, - ) - # Harvest research-scout output (hints, competitor target, gap seeds, PR dedup). Fail-soft. if domain == "research_scout_specialist": try: @@ -2622,12 +2530,17 @@ def _lift_to_current_best( bv: dict[str, Any], *, gap_canonical_id: str = "", + entry_extra: Mapping[str, Any] | None = None, ) -> bool: """Lift a winner only when it improves the current throughput anchor. - The stack append is skipped when the winner is already applied, keyed by - ``(action, variant_name)`` or by ``fingerprint``, so a rerun of an - already-stacked config cannot double-apply it. + The only writer of ``optimization_stack``, and the only config-KEEP + writer of ``current_best`` (the baseline anchor is the other one). Also + the only place a winner is merged onto the previous config instead of + replacing it, so ``unset_envs`` and cumulative args stay correct across + the whole stack. The stack append is skipped when the winner is already + applied, keyed by ``(action, variant_name)`` or by ``fingerprint``, so a + rerun of an already-stacked config cannot double-apply it. Args: task_kind: The action kind that produced the winner (stamped on the @@ -2636,6 +2549,8 @@ def _lift_to_current_best( bv: The winning variant dict (args, envs, metrics, provenance). gap_canonical_id: When known, stamped onto the stack entry so provenance resolves by gap id rather than name. + entry_extra: Per-action metadata for the stack entry only; + ``current_best`` stays a pure config record. Returns: ``True`` when the winner was lifted, ``False`` when it was refused @@ -2654,6 +2569,10 @@ def _lift_to_current_best( base_args = "" if isinstance(previous, dict): base_args = str(previous.get("extra_server_args") or "").strip() + # An authored-kernel overlay stays active until another KEEP replaces it. + _overlay = str((bv.get("final_overlay") if isinstance(bv, dict) else "") or "").strip() + if not _overlay and isinstance(previous, dict): + _overlay = str(previous.get("final_overlay") or "").strip() candidate_args = "" if isinstance(bv, dict): candidate_args = str(bv.get("candidate_extra_server_args") or bv.get("extra_server_args") or "").strip() @@ -2796,6 +2715,11 @@ def _lift_to_current_best( for _origin_key in ("domain", "gap_layer"): if bv.get(_origin_key): stack_entry[_origin_key] = str(bv.get(_origin_key)) + if _overlay: + stack_entry["final_overlay"] = _overlay + for _extra_key, _extra_val in (entry_extra or {}).items(): + if _extra_val not in (None, "", [], {}): + stack_entry[str(_extra_key)] = _extra_val self.shared_state.optimization_stack.append(stack_entry) # Mirror append into gain_per_stack_entry so the two lists stay index-aligned. self.shared_state.append_stack_gain_entry( @@ -2819,6 +2743,7 @@ def _lift_to_current_best( "variant_name": variant_name, "extra_server_args": full_args, "extra_envs": _merged_envs, + "final_overlay": _overlay, "optimization_stack": list(self.shared_state.optimization_stack), "ttft_mean_ms": bv.get("ttft_mean_ms") if isinstance(bv, dict) else None, "e2el_mean_ms": bv.get("e2el_mean_ms") if isinstance(bv, dict) else None, @@ -2836,10 +2761,6 @@ def _lift_to_current_best( if (bv.get("remove_args") or bv.get("unset_envs")) and not current_best.get("args_mode"): current_best["args_mode"] = "replace" self.shared_state.current_best = current_best - if self.shared_state.baseline_tput > 0: - self.shared_state.cumulative_gain = ( - (float(best_tput) - self.shared_state.baseline_tput) / self.shared_state.baseline_tput * 100.0 - ) return True def _should_run_prelude_bootstrap(self, tput: Any) -> bool: @@ -3376,38 +3297,6 @@ async def _promote_profile( audit_extras["framework_rewrite_evidence"] = evidence_path audit_extras["framework_rewrite_candidate_count"] = result.get("framework_rewrite_candidate_count") changed = True - # profile result may include a tput; promote into current_best on the +1% rule. - tput = result.get("output_throughput") - cb = self.shared_state.current_best or {} - cb_tput = cb.get("tput") if isinstance(cb, dict) else None - cur_best = ( - float(cb_tput) - if isinstance(cb_tput, (int, float)) and cb_tput > 0 - else float(self.shared_state.baseline_tput or 0.0) - ) - if ( - isinstance(tput, (int, float)) - and tput > 0 - and cur_best > 0 - and (tput - cur_best) / cur_best * 100.0 >= 1.0 - ): - promoted = dict(cb) if isinstance(cb, dict) else {} - promoted.update( - { - "action": "profile", - "tput": float(tput), - "ttft_mean_ms": result.get("ttft_mean_ms"), - "e2el_mean_ms": result.get("e2el_mean_ms"), - "tpot_mean_ms": result.get("tpot_mean_ms"), - "workspace": result.get("workspace"), - } - ) - self.shared_state.current_best = promoted - if self.shared_state.baseline_tput > 0: - self.shared_state.cumulative_gain = ( - (float(tput) - self.shared_state.baseline_tput) / self.shared_state.baseline_tput * 100.0 - ) - changed = True # On a successful profile, re-anchor last_roofline_tput and clear the pending field. if profile_status == "succeeded": anchor_tput = self._current_tput_from_validated_gain() @@ -3522,9 +3411,10 @@ async def _promote_explore( changed = False audit_decision: str | None = None audit_extras: dict[str, Any] = {} - # The executor already did per-variant KEEP/REVERT + rebench, so winners - # are authoritative; Coordinator is single-writer for explore_search.accepted + - # current_best + optimization_stack and does not re-threshold. + # Winners arrive already graded by the executor (per-variant KEEP/REVERT + + # rebench); Coordinator is single-writer for explore_search.accepted + + # current_best + optimization_stack. The lift still refuses a winner that + # no longer beats the live anchor. # 1. Apply the executor's ledger increment. update = result.get("explore_search_update") if isinstance(update, dict): @@ -3640,7 +3530,6 @@ async def _promote_explore( self._promote_geak_from_candidate( ps, measured_tput=float(measured), - provenance="geak_orch_harness_validated", ) elif decision == "no_material": # No material GEAK product; the rebench beating current_best @@ -4037,6 +3926,7 @@ async def _promote_integrate_patch( audit_decision = "promoted" elif kept_flag: audit_decision = "no_promote" + result[PROMOTION_REFUSED_KEY] = True elif status == "kept_inert": # Applied but every switch off: nothing was promoted, yet the patch # stays on disk as registered levers, so it is not a discard either. @@ -4125,23 +4015,6 @@ async def _promote_framework_agent( if not isinstance(self.shared_state.framework_agent_phase_progress, list): self.shared_state.framework_agent_phase_progress = [] self.shared_state.framework_agent_phase_progress.append(progress_entry) - try: - from ..framework.artifacts import write_decision_json - - write_decision_json( - self.session_dir, - candidate_id=cand_id, - batch_id=batch_id, - status=status, - kept=kept_flag, - provenance="raw_diff", - reason=str(result.get("reason") or ""), - gain_pct=(float(delta_pct) if isinstance(delta_pct, (int, float)) else None), - accuracy_pass=result.get("accuracy_pass"), - extra={"workspace": str(result.get("workspace") or "")}, - ) - except Exception: # noqa: BLE001 - log.debug("FRAMEWORK: executor decision.json write failed", exc_info=True) # Update batch max-gain rolling stat (for the plateau judge). batches = getattr(self.shared_state, "framework_agent_batches", None) or [] if isinstance(batches, list) and batches: @@ -4156,12 +4029,9 @@ async def _promote_framework_agent( lifted = False if kept_flag and isinstance(new_tput, (int, float)) and new_tput > 0: if not cand_id: - # The name used to be prefixed, which made an empty key look - # non-empty to the stack's guard and stacked a nameless entry. - # The bare key is falsy, so the append is skipped — current_best - # and cumulative_gain are set regardless, further down. The win - # therefore counts without leaving a step anything can reconcile, - # dedupe or replay, which is worth saying out loud. + # A falsy key skips the stack append but still lifts current_best, + # so the win counts without leaving a step anything can reconcile, + # dedupe or replay. log.warning( "FRAMEWORK: KEEP carries no candidate key (candidate_id / pr_url / ref all " "empty, and task params had none either). current_best still advances, but " @@ -4192,6 +4062,7 @@ async def _promote_framework_agent( audit_decision = "promoted" elif kept_flag: audit_decision = "no_promote" + result[PROMOTION_REFUSED_KEY] = True else: audit_decision = "discarded" audit_extras = { @@ -4283,7 +4154,7 @@ async def _promote_conc_sweep( # Three semantic boundaries live below: the live-promote / replay path # (``_replay_keep_from_result``), the resume-reconcile path # (``_resume_consistency_pass`` + its recover helpers), and the - # current_best lift path (``_materialize_stack_config_for_resume`` / + # current_best lift path (``_current_best_launch_config`` / # ``build_env_spec``). Methods keep bare ``self.`` access; tests # monkeypatch them via ``coord.writeback.`` (or bare-name # ``_DELEGATED`` on the coordinator). @@ -4354,56 +4225,30 @@ async def replay_for_resume(self) -> dict[str, Any]: "verdicts_seen": len(verdicts), } - def _materialize_stack_config_for_resume(self) -> dict[str, Any]: - """Rebuild cumulative launch args/envs from ``optimization_stack``.""" - stack = [e for e in (getattr(self.shared_state, "optimization_stack", []) or []) if isinstance(e, dict)] - args = "" + def _current_best_launch_config(self) -> dict[str, Any]: + """The launch config ``current_best`` was measured on. + + Returns: + ``extra_server_args`` / ``extra_envs`` / ``final_overlay``. + """ + cb = self.shared_state.current_best if isinstance(self.shared_state.current_best, Mapping) else {} + args = str(cb.get("extra_server_args") or "").strip() envs: dict[str, str] = {} - overlay = "" - tput: float | None = None - variant_name = "" - action = "resume_reconstructed" - workspace = None - for entry in stack: - candidate = str(entry.get("candidate_extra_server_args") or "").strip() - full = str(entry.get("extra_server_args") or "").strip() - args = _merge_cumulative_extra_server_args(args, candidate, full) - raw_envs = entry.get("extra_envs") or {} - if isinstance(raw_envs, Mapping): - for k, v in raw_envs.items(): - ks = str(k) - # A flag mis-stored under extra_envs (e.g. a ``--compilation-config`` - # key from an integrate_patch entry) is a SERVER ARG, not an env - # var — the grid runner would otherwise inject it verbatim as an - # env the backend ignores, silently dropping it from the rebuilt - # config. Route any ``-``-prefixed key back into extra_server_args - # so the materialized stack reproduces the real launch. General: - # keyed on the ``-`` prefix, never on a specific flag name. - if ks.startswith("-"): - tok = ks if v in ("", None) else f"{ks}={v}" - args = _merge_cumulative_extra_server_args(args, "", tok) - else: - envs[ks] = str(v) - # Carry the authored-kernel overlay (PYTHONPATH prefix) so a native - # rebuild of an overlay winner actually loads the built kernels - # instead of measuring the un-optimized stack. Last non-empty wins. - entry_overlay = str(entry.get("final_overlay") or "").strip() - if entry_overlay: - overlay = entry_overlay - if isinstance(entry.get("tput"), (int, float)) and float(entry["tput"]) > 0: - tput = float(entry["tput"]) - variant_name = str(entry.get("variant_name") or variant_name or "") - action = str(entry.get("action") or action) - workspace = entry.get("workspace") or workspace + raw_envs = cb.get("extra_envs") + if isinstance(raw_envs, Mapping): + for key, value in raw_envs.items(): + name = str(key) + # A ``-``-prefixed key is a server arg; exported as an env the + # backend ignores it and the flag is silently lost. + if name.startswith("-"): + token = name if value in ("", None) else f"{name}={value}" + args = _merge_cumulative_extra_server_args(args, token, "") + else: + envs[name] = str(value) return { - "action": action, - "variant_name": variant_name, "extra_server_args": args, "extra_envs": envs, - "final_overlay": overlay, - "tput": tput, - "workspace": workspace, - "optimization_stack": stack, + "final_overlay": str(cb.get("final_overlay") or "").strip(), } def build_env_spec(self) -> dict[str, Any]: @@ -4424,7 +4269,7 @@ def build_env_spec(self) -> dict[str, Any]: baseline ref is materialized from the SAME layers as ``current_best`` (not just its flags/env), closing the cross-harness baseline gap. """ - materialized = self._materialize_stack_config_for_resume() + materialized = self._current_best_launch_config() stack = [e for e in (getattr(self.shared_state, "optimization_stack", []) or []) if isinstance(e, dict)] source_snapshots: list[dict[str, Any]] = [] for entry in stack: @@ -4489,11 +4334,11 @@ def build_env_spec(self) -> dict[str, Any]: async def _resume_consistency_pass(self) -> dict[str, Any]: """One-shot resume audit + recovery for stack/current_best consistency. - Order matters: recover half-applied / orphaned KEEPs FIRST (they mutate - the stack), then reconcile ``current_best`` against the resulting stack, - then compensate the validation watermark by enqueuing a single - full-stack end-to-end rebench. Idempotent — only runs on a resumed - session and every recovery step dedupes, so a second pass is a no-op. + Recovers half-applied / orphaned KEEPs through the same lift the live + path uses, then compensates the validation watermark by enqueuing a + single full-stack end-to-end rebench. Idempotent — only runs on a + resumed session and every recovery step dedupes, so a second pass is a + no-op. """ if not self._resumed_from.get("is_resume"): return {"skipped": True, "reason": "not_resume"} @@ -4538,46 +4383,8 @@ async def _resume_consistency_pass(self) -> dict[str, Any]: # that crashed before the append landed; surface ambiguous ones loudly. await self._resume_recover_orphaned_keeps(report) - # (3) current_best <-> stack reconcile (after 1/2 may have grown stack). - stack = [e for e in (getattr(state, "optimization_stack", []) or []) if isinstance(e, dict)] - cb = state.current_best if isinstance(state.current_best, dict) else {} - if stack: - rebuilt = self._materialize_stack_config_for_resume() - cb_args = str(cb.get("extra_server_args") or "") - cb_envs = ( - {str(k): str(v) for k, v in (cb.get("extra_envs") or {}).items()} - if isinstance(cb.get("extra_envs"), Mapping) - else {} - ) - if cb_args != rebuilt["extra_server_args"] or cb_envs != rebuilt["extra_envs"]: - # The append-only stack is authoritative; a disagreeing - # current_best is the inconsistency, recorded distinctly from the - # rebuild fix so operators can see a stale best was detected. - report["warnings"].append( - { - "kind": "resume_inconsistent_current_best", - "current_best_args": cb_args, - "stack_args": rebuilt["extra_server_args"], - } - ) - new_cb = dict(cb) - new_cb.update( - { - "action": rebuilt["action"], - "variant_name": rebuilt["variant_name"], - "extra_server_args": rebuilt["extra_server_args"], - "extra_envs": rebuilt["extra_envs"], - "optimization_stack": list(stack), - "source": "resume_consistency_rebuild_from_stack", - } - ) - if rebuilt["tput"] is not None and not isinstance(new_cb.get("tput"), (int, float)): - new_cb["tput"] = rebuilt["tput"] - if rebuilt["workspace"] and not new_cb.get("workspace"): - new_cb["workspace"] = rebuilt["workspace"] - state.current_best = new_cb - report["fixes"].append("rebuilt_current_best_config_from_stack") - elif cb: + # (3) A config with no stack behind it cannot be reproduced. + if state.current_best and not state.optimization_stack: report["warnings"].append({"kind": "current_best_without_stack"}) # Persist recovered stack/current_best before materializing their KB @@ -4786,12 +4593,13 @@ def _gc_attempt_runtime(attempt_dir: str) -> bool: return False async def _resume_recover_pending_integrate(self, report: dict[str, Any]) -> None: - """Recover a crashed integrate_patch window from the sentinel (Gap C). + """Recover a crashed integrate_patch window from the sentinel. Three-way decision keyed on whether a ``kept`` delegated-result exists for the sentinel's task: replay the missing append (crashed after KEEP), roll back the half-applied patch (crashed after apply, before KEEP), or - clear a stale sentinel. The sentinel is always cleared afterwards. + clear a stale sentinel. A scan that could not read the event log reaches + none of the three, so it must neither roll back nor clear the sentinel. Args: report: The resume report dict to append fixes/warnings to. @@ -4802,6 +4610,7 @@ async def _resume_recover_pending_integrate(self, report: dict[str, Any]) -> Non return task_id = str(pending.get("task_id") or "") kept_res: dict[str, Any] | None = None + scanned = False try: for msg in await self.bus.tail(topic="delegated_result", n=10_000): payload = msg.payload or {} @@ -4819,8 +4628,12 @@ async def _resume_recover_pending_integrate(self, report: dict[str, Any]) -> Non ): kept_res = res break + scanned = True except Exception: # noqa: BLE001 log.exception("Coordinator: pending_integrate kept-result scan failed") + if not scanned: + report["warnings"].append({"kind": "pending_integrate_scan_failed", "task_id": task_id}) + return if kept_res is not None: appended = self._replay_keep_from_result("integrate_patch", kept_res) report["fixes"].append( @@ -5145,10 +4958,10 @@ async def _resume_recover_orphaned_keeps(self, report: dict[str, Any]) -> None: log.exception("Coordinator: orphaned KEEP resume recovery failed") async def _enqueue_internal_stack_rebench(self, *, reason: str) -> dict[str, Any]: - """Enqueue one full-stack end-to-end rebench of the cumulative config (Gap A). + """Enqueue one full-stack end-to-end rebench of the cumulative config. - Builds a single-variant ``explore`` task from the stack-materialized - launch args/envs, benched against ``baseline_tput`` so the measured + Builds a single-variant ``explore`` task from ``current_best``'s launch + args/envs, benched against ``baseline_tput`` so the measured delta becomes the validated cumulative gain. Tagged ``source=resume_stack_revalidate`` so ``_promote_to_shared_state`` reconciles ``cumulative_gain_validated_stack_len`` + clears @@ -5240,16 +5053,16 @@ async def _enqueue_internal_stack_rebench(self, *, reason: str) -> dict[str, Any "mode": "geak_2b", } - rebuilt = self._materialize_stack_config_for_resume() - args = str(rebuilt.get("extra_server_args") or "").strip() - envs = rebuilt.get("extra_envs") or {} - overlay = str(rebuilt.get("final_overlay") or "").strip() + launch = self._current_best_launch_config() + args = launch["extra_server_args"] + envs = launch["extra_envs"] + overlay = launch["final_overlay"] cb_now = self.shared_state.current_best if isinstance(self.shared_state.current_best, dict) else {} cb_remove = cb_now.get("remove_args") cb_unset = cb_now.get("unset_envs") cb_replace = str(cb_now.get("args_mode") or "").strip().lower() == "replace" if not (args or envs or cb_remove or cb_unset or cb_replace): - return {"skipped": True, "reason": "empty_stack"} + return {"skipped": True, "reason": "empty_config"} params: dict[str, Any] = { "source": "resume_stack_revalidate", "reason": reason, @@ -5391,7 +5204,6 @@ async def _validate_geak_via_geak_harness(self, *, reason: str) -> dict[str, Any self._promote_geak_from_candidate( ps, measured_tput=measured, - provenance="geak_same_harness_geak", ) base = float(self.shared_state.baseline_tput or 0.0) gain_out = ((measured - base) / base * 100.0) if base > 0 else 0.0 diff --git a/src/hyperloom/orchestrator/phases/explore.py b/src/hyperloom/orchestrator/phases/explore.py index 4c7eb34227..9223d7adf5 100644 --- a/src/hyperloom/orchestrator/phases/explore.py +++ b/src/hyperloom/orchestrator/phases/explore.py @@ -18,6 +18,7 @@ from ..policy.gate import ( SPECIALIST_FROM_AGENT_PREFIX, ) +from ..loop.maintenance import run_lease_and_db_reclaim from ..loop.sub_agent_runner import SubAgentResult from ..prompts import write_prompt_snapshot as _write_prompt_snapshot from ..specialists.runner import SpecialistFailureType @@ -446,33 +447,8 @@ async def _run_cycle_soft_restart( summary["conversation_reset"] = True except Exception: # noqa: BLE001 — soft restart never aborts the run loop log.exception("cycle soft-restart: conversation reset failed") - # 2) Reap TTL-expired serving + GPU leases immediately. - try: - reaped = await self.locks.reap_expired() - summary["leases_reaped"] = len(reaped or []) - except Exception: # noqa: BLE001 - log.exception("cycle soft-restart: serving-lease reap failed") - try: - summary["gpu_leases_reaped"] = await self.gpu_specialist_pool.reap_expired() - except Exception: # noqa: BLE001 - log.exception("cycle soft-restart: gpu-lease reap failed") - # 2b) Reclaim orphaned running tasks (lease expired) → failed. Idempotent. - try: - reclaimed = await self.tasks.reclaim_expired_running( - reason="cycle_soft_restart", - ) - summary["running_tasks_reclaimed"] = len(reclaimed) - except Exception: # noqa: BLE001 - log.exception("cycle soft-restart: running-task reclaim failed") - # 3) Prune the events/tasks DB (strictly below the resume anchor). - try: - from ..bus import db_maintenance as _db_maint - - res = await _db_maint.run_db_retention(self.db, self.cursors) - summary["events_pruned"] = res.events_deleted - summary["tasks_pruned"] = res.tasks_deleted - except Exception: # noqa: BLE001 - log.exception("cycle soft-restart: DB retention failed") + # 2-3) Reap leases, reclaim orphaned running tasks, prune DB. + await run_lease_and_db_reclaim(self, summary, reason="cycle_soft_restart") # 4) Deep-clean any lingering inference-server processes. if getattr(self, "_cycle_restart_servers", False): try: @@ -1090,17 +1066,7 @@ async def _warm_specialist_params(self, params: dict[str, Any]) -> None: if _cgv != 0: params["cumulative_gain_validated"] = _cgv if "keep_threshold_pct" not in params: - try: - from ..phases.machine_state import decaying_keep_threshold_pct - from ..actions.executors._multi_node_env import is_multi_node - - _kth = decaying_keep_threshold_pct( - int(getattr(state, "macro_cycle", 0) or 0), - multi_node=is_multi_node(), - ) - params["keep_threshold_pct"] = _kth - except Exception: # noqa: BLE001 - pass + params["keep_threshold_pct"] = _phase_state.resolve_keep_threshold(state) if "applied_stack" not in params: _stack = list(getattr(state, "optimization_stack", None) or []) if _stack: diff --git a/src/hyperloom/orchestrator/phases/framework.py b/src/hyperloom/orchestrator/phases/framework.py index 762886c5ed..62625572d9 100644 --- a/src/hyperloom/orchestrator/phases/framework.py +++ b/src/hyperloom/orchestrator/phases/framework.py @@ -644,17 +644,6 @@ async def _audit_framework_agent_candidate(self, candidate: dict[str, Any]) -> d candidate["_audit"] = audit except Exception: # noqa: BLE001 — caching is best-effort pass - # Persist the verdict next to decision.json. - try: - from ..framework.artifacts import write_semantic_audit - - write_semantic_audit( - self.session_dir, - candidate_id=self._framework_candidate_key(candidate), - verdict=audit, - ) - except Exception: # noqa: BLE001 — observability is best-effort - log.debug("FRAMEWORK: write_semantic_audit failed", exc_info=True) log.info( "FRAMEWORK: audit candidate=%s status=%s appl=%s next=%s", self._framework_candidate_key(candidate), @@ -733,26 +722,6 @@ async def _record_framework_agent_audit_skip( "cycle": int(getattr(state, "macro_cycle", 0) or 0), } ) - try: - from ..framework.artifacts import write_decision_json - - write_decision_json( - self.session_dir, - candidate_id=cand_id, - batch_id=batch_id, - status=status, - kept=False, - provenance="audit", - reason="; ".join(str(r) for r in ((audit or {}).get("risks") or [])) or semantic, - extra={ - "semantic_status": semantic, - "applicability": (audit or {}).get("applicability"), - "confidence": (audit or {}).get("confidence"), - "evidence": (audit or {}).get("evidence") or [], - }, - ) - except Exception: # noqa: BLE001 — observability is best-effort - log.debug("FRAMEWORK: audit-skip decision.json write failed", exc_info=True) if status == "already_present": try: from ..knowledge.kb_writeback import OUTCOME_ALREADY_PRESENT, write_framework_record @@ -2716,7 +2685,7 @@ async def _rank_framework_agent_candidates_llm( if not model: return None state = self.shared_state - best = getattr(state, "best_throughput", None) or getattr(state, "baseline_throughput", None) + best = resolve_grading_anchor_tput(state) cap = 60 listed = candidates[:cap] candidate_rows: list[str] = [] @@ -3526,9 +3495,9 @@ async def _discover_next_framework_batch(self) -> bool: async def _enqueue_framework_agent_task(self, candidate: dict[str, Any]) -> None: """Enqueue a single ``framework_agent`` task for ``candidate``. - Builds the task params (candidate, batch id, baseline throughput, - framework) and creates an idempotent ``framework_agent`` task whose - lanes and lease TTL come from the action catalogue. On enqueue failure, + Builds the task params (candidate, batch id, baseline throughput, KEEP + threshold, framework) and creates an idempotent ``framework_agent`` task + whose lanes and lease TTL come from the action catalogue. On enqueue failure, records an ``enqueue_failed`` progress row so the pump skips the candidate next tick instead of spinning. @@ -3541,6 +3510,8 @@ async def _enqueue_framework_agent_task(self, candidate: dict[str, Any]) -> None "candidate": candidate, "batch_id": candidate.get("batch_id") or "", "base_tput": resolve_grading_anchor_tput(state), + # Same decaying bar the explore and integrate_patch dispatch paths inject. + "keep_threshold_pct": _phase_state.resolve_keep_threshold(state), "framework": str(candidate.get("framework") or getattr(state, "framework", "") or "").strip().lower(), # Source patches require the accuracy gate for KEEP. "require_accuracy_for_keep": True, @@ -3890,22 +3861,6 @@ def _stamp_framework_progress( for k, v in extra.items(): row.setdefault(str(k), v) progress.append(row) - try: - from ..framework.artifacts import write_decision_json - - write_decision_json( - self.session_dir, - candidate_id=cand_id, - batch_id=str(batch_id or ""), - status=str(status or ""), - kept=bool(kept), - provenance=str(provenance or ""), - reason=str(rationale or ""), - gain_pct=gain_pct, - extra=extra if isinstance(extra, dict) else None, - ) - except Exception: # noqa: BLE001 — observability is best-effort - log.debug("FRAMEWORK: stamp decision.json write failed", exc_info=True) try: state.save(self.session_dir) except Exception: # noqa: BLE001 — defensive diff --git a/src/hyperloom/orchestrator/phases/kernel.py b/src/hyperloom/orchestrator/phases/kernel.py index 630f7417ea..3b5532d362 100644 --- a/src/hyperloom/orchestrator/phases/kernel.py +++ b/src/hyperloom/orchestrator/phases/kernel.py @@ -20,6 +20,7 @@ from typing import Any, Callable from . import machine_state as _phase_state from ..kernel import collective_recovery as _collective_recovery +from ..actions.stop_attribution import stopped_by_the_run_class from ..kernel._recorder_trace import trace_recording_skipped from ..state.optimization_journal import ( KIND_GEMM_TUNING, @@ -117,7 +118,7 @@ def _serving_config_signature(serving_config: Any) -> str: Reuses the exact ``serving_config`` shape built by ``SharedState.profile_workload_context`` so the reprofile gate and the recorded-trace workload are normalized identically (no second, drifting - copy of the engine/args/env rules). + copy of the args/env rules). """ if not isinstance(serving_config, Mapping) or not serving_config: return "" @@ -132,13 +133,12 @@ def _serving_config_signature(serving_config: Any) -> str: else {} ) payload = { - "engine": str(serving_config.get("engine") or "").strip().lower(), "extra_server_args": str( serving_config.get("extra_server_args") or "" ).strip(), "extra_envs": envs, } - if not any((payload["engine"], payload["extra_server_args"], envs)): + if not any((payload["extra_server_args"], envs)): return "" return "hyperloom-profile-config:" + json.dumps( payload, @@ -1142,7 +1142,7 @@ def _record_geak_candidate(self, result: dict[str, Any]) -> None: Stores the accepted config + the optimizer's own (audit-only) throughput/speedup under ``geak_pending`` without touching - ``current_best`` / ``optimization_stack`` / ``cumulative_gain*``. The + ``current_best`` / ``optimization_stack`` / ``cumulative_gain_validated*``. The headline is written later from a measured rebench by ``_promote_geak_from_candidate``; the config is captured verbatim as the source the rebench launches from. @@ -1202,15 +1202,13 @@ def _promote_geak_from_candidate( result: dict[str, Any], *, measured_tput: float, - provenance: str, ) -> None: """Write the GEAK headline from a MEASURED main-flow rebench. - The single headline writer: lifts ``current_best`` (config/overlay/scripts - + the measured tput), appends the ``geak_e2e`` optimization_stack entry + - gain ledger, and stamps ``cumulative_gain`` / ``cumulative_gain_validated`` - as the same-harness total ``(measured - baseline)/baseline``. Clears - ``geak_pending`` and the revalidation flag. + Lifts the measured config/overlay onto ``current_best`` and stamps + ``cumulative_gain_validated`` as the same-harness total + ``(measured - baseline)/baseline``. Clears ``geak_pending`` and the + revalidation flag. """ if not isinstance(result, dict): return @@ -1236,7 +1234,7 @@ def _promote_geak_from_candidate( result, measured_tput=measured, current_best_tput=float(cb_tput), - provenance=provenance, + provenance="geak_promote_rejected", ) try: from hyperloom.inference_optimizer.breakdown.recorder import instrument @@ -1258,7 +1256,7 @@ def _promote_geak_from_candidate( status="failed", validated=False, measured_tput=measured, - validation_source=provenance, + validation_source="geak_promote_rejected", ) except Exception: # noqa: BLE001 log.debug( @@ -1270,71 +1268,32 @@ def _promote_geak_from_candidate( return accepted_flags, parsed_envs = self._parse_geak_accepted_config(result) - cb = dict(self.shared_state.current_best or {}) - cb_envs = dict(cb.get("extra_envs") or {}) if isinstance(cb.get("extra_envs"), Mapping) else {} - cb_envs.update(parsed_envs) - cb.update( + self._lift_to_current_best( + "geak_e2e", + measured, { - "action": "geak_e2e", - "tput": measured, - "ttft_mean_ms": result.get("ttft_ms"), - "tpot_mean_ms": result.get("tpot_ms"), - "extra_server_args": accepted_flags, - "extra_envs": cb_envs, - "geak_launch_script": result.get("final_launch_script"), - "geak_bench_script": result.get("bench_script"), - "geak_eval_dir": result.get("eval_dir"), - "final_overlay": result.get("final_overlay") or "", - "workspace": result.get("eval_dir"), - } - ) - # Audit cross-check: GEAK's own within-harness speedups (not the headline). - am = result.get("alignment_metrics") or {} - cb["geak_alignment"] = { - "hot_geak_speedup": am.get("hot_geak_speedup"), - "cold_geak_speedup": am.get("cold_geak_speedup"), - "hot_speedup": am.get("hot_speedup"), - "cold_speedup": am.get("cold_speedup"), - "final_basis": am.get("final_basis") or result.get("final_throughput_basis"), - "geak_throughput_speedup": result.get("throughput_speedup"), - } - self.shared_state.current_best = cb - - ts = datetime.now(timezone.utc).isoformat() - if not self._geak_win_already_recorded(): - entry = { - "action": "geak_e2e", - "source_phase": "KERNEL_AGENT", - "variant_name": "geak_e2e", - "tput": measured, + "name": "geak_e2e", "candidate_extra_server_args": accepted_flags, "extra_envs": dict(parsed_envs), "final_overlay": result.get("final_overlay") or "", + "source_phase": "KERNEL_AGENT", + "ttft_mean_ms": result.get("ttft_ms"), + "tpot_mean_ms": result.get("tpot_ms"), "workspace": result.get("eval_dir"), + }, + entry_extra={ "accepted_kernels": result.get("accepted_kernels") or [], "accepted_heads": result.get("accepted_heads") or [], "report_path": result.get("report_path"), "source": "geak_e2e", - "ts": ts, - } - self.shared_state.optimization_stack.append(entry) - self.shared_state.append_stack_gain_entry( - action="geak_e2e", - variant_name="geak_e2e", - new_tput=measured, - extra_server_args=accepted_flags, - ts=ts, - ) + }, + ) - base = float(self.shared_state.baseline_tput or 0.0) - if base > 0: - self.shared_state.cumulative_gain = (measured - base) / base * 100.0 + if self.shared_state.baseline_tput > 0: self._update_cumulative_gain_validated( measured, source="geak_e2e_promote", - ts=ts, ) - self.shared_state.cumulative_gain_provenance = provenance self.shared_state.resume_pending_revalidation = False self.shared_state.geak_pending = {} try: @@ -1348,7 +1307,7 @@ def _promote_geak_from_candidate( status="succeeded", validated=True, measured_tput=measured, - validation_source=provenance, + validation_source="geak_orch_harness", ) except Exception: # noqa: BLE001 log.debug("geak v4 final validation recording failed", exc_info=True) @@ -1734,7 +1693,7 @@ def _read_header(path: Path) -> list[str]: break if not source_paths: log.warning( - "forge gemm E2E: no complete aiter config found for %s; " + "gemm E2E: no complete aiter config found for %s; " "candidate-only validation would be unsafe", env_var, ) @@ -1804,7 +1763,7 @@ def _normalize(rows: list[dict[str, str]]) -> list[dict[str, str]]: key_cols.append(column) if not key_cols: log.warning( - "forge gemm E2E: cannot derive dispatch keys for %s", + "gemm E2E: cannot derive dispatch keys for %s", env_var, ) return None @@ -1823,7 +1782,7 @@ def _deduplicate_dispatch_rows( return rows if "us" not in all_columns: log.warning( - "forge gemm E2E: %s has duplicate dispatch keys for %s " + "gemm E2E: %s has duplicate dispatch keys for %s " "but no 'us' column to select the fastest row", label, env_var, @@ -1849,7 +1808,7 @@ def _us(row: dict[str, str]) -> float: best[key] = row deduplicated = [best[key] for key in order] log.info( - "forge gemm E2E: removed %d duplicate %s row(s) for %s", + "gemm E2E: removed %d duplicate %s row(s) for %s", len(rows) - len(deduplicated), label, env_var, @@ -1874,7 +1833,7 @@ def _us(row: dict[str, str]) -> float: writer.writeheader() writer.writerows(merged_rows) log.info( - "forge gemm E2E: merged %d candidate rows into %d rows from %d " + "gemm E2E: merged %d candidate rows into %d rows from %d " "aiter config file(s) -> %d total (%s)", len(candidate_rows), len(runtime_rows), @@ -1884,7 +1843,7 @@ def _us(row: dict[str, str]) -> float: ) return str(merged_path) except Exception as exc: # noqa: BLE001 - log.warning("forge gemm E2E: merge failed (%s); rejecting candidate", exc) + log.warning("gemm E2E: merge failed (%s); rejecting candidate", exc) return None def _ck_blockscale_switch_eligible(self, result: dict[str, Any]) -> bool: @@ -1996,7 +1955,6 @@ def _sync_profile_state_after_gemm_roofline(self, result: dict[str, Any]) -> Non "last_profile_workload", "last_profile_workload_action", "last_trace_analyze", - "roofline_snapshot_id", "roofline_snapshots", "baseline_eager_fallback", ): @@ -2013,19 +1971,12 @@ async def _handle_gemm_tuning_result(self, result: dict[str, Any]) -> None: """Record and post-process a run_gemm_tuning result from any entrypoint. Both the KERNEL-entry auto hook and orchestration-issued - ``run_gemm_tuning`` requests converge here so forge results never bypass - per-tuner E2E validation. + ``run_gemm_tuning`` requests converge here so no backend bypasses + per-candidate E2E validation. """ self._sync_profile_state_after_gemm_roofline(result) self.shared_state.record_gemm_tuning(result) - # Forge results route to the per-tuner E2E validator when table tuning - # asked for it OR when the CK block-scale backend switch is eligible. - if result.get("backend") == "forge" and ( - result.get("requires_e2e_validation") or self._ck_blockscale_switch_eligible(result) - ): - await self._validate_forge_gemm_tuning_e2e(result) - else: - self._promote_gemm_tuning_keep(result) + await self._validate_gemm_tuning_e2e(result) try: from hyperloom.inference_optimizer.breakdown.recorder import instrument @@ -2095,141 +2046,6 @@ def _journal_gemm_tuning_keep( except Exception: # noqa: BLE001 — journaling is best-effort log.exception("gemm_tuning journal append failed") - def _promote_gemm_tuning_keep(self, result: dict[str, Any]) -> None: - """Promote a successful GEMM tuning run into the main gain ledger. - - Only acts on a successful, ``KEEP``-decision result with a speedup - greater than 1.0 and a known baseline. Appends an entry to the - optimization stack (deduped on tuned file), updates ``current_best``, - and stamps ``cumulative_gain`` / ``cumulative_gain_validated`` since - the GEMM benchmark is itself an end-to-end serving measurement. - - Forge results that requested per-tuner E2E validation - (``requires_e2e_validation``), or that are eligible for the CK - block-scale switch (``_ck_blockscale_switch_eligible``), are routed to - ``_validate_forge_gemm_tuning_e2e`` by ``_handle_gemm_tuning_result`` - and normally never reach this promoter. Anything that does reach it — - including a forge result whose validation already completed — has its - gain stamped as validated. - - Args: - result (dict[str, Any]): The GEMM tuning handler result; ignored if - not a successful KEEP. - """ - if not isinstance(result, dict): - return - status = str(result.get("status") or "").strip().lower() - decision = str(result.get("decision") or "").strip().upper() - if status not in {"ok", "complete", "completed", "succeeded", "success"}: - return - if decision != "KEEP": - return - try: - speedup = float(result.get("best_speedup") or 0.0) - baseline = float(self.shared_state.baseline_tput or 0.0) - except (TypeError, ValueError): - return - if speedup <= 1.0 or baseline <= 0: - return - - backend = str(result.get("backend") or "geak").strip().lower() - ts = datetime.now(timezone.utc).isoformat() - - # Resolve extra_envs: forge provides them; GEAK infers from tuned_file. - if backend == "forge": - extra_envs = dict(result.get("extra_envs") or result.get("recommended_env") or {}) - tuned_file = "" - artifacts = result.get("artifacts") or {} - if isinstance(artifacts, dict) and artifacts: - tuned_file = str(next(iter(artifacts.values()), "")) - if not tuned_file: - tuned_file = str(next(iter(extra_envs.values()), "")) if extra_envs else "" - variant_name = "forge_gemm_tuned" - else: - tuned_file = str(result.get("tuned_file") or "") - extra_envs = {"AITER_CONFIG_GEMM_A8W8_BLOCKSCALE": tuned_file} if tuned_file else {} - variant_name = "a8w8_blockscale_tuned_gemm" - - # fp8 block-scale CK backend switch safety net (an operator-set value - # wins via setdefault); the primary forge path validates it standalone. - if self._ck_blockscale_switch_eligible(result): - extra_envs.setdefault("SGLANG_FP8_BLOCKSCALE_CK_MAX_M", "256") - - final_report = str(result.get("final_report_path") or "") - - # GEAK path: E2E already validated internally. - tuned_tput = baseline * speedup - existing = { - str(item.get("tuned_file") or "") - for item in (self.shared_state.optimization_stack or []) - if isinstance(item, dict) and item.get("action") == "gemm_tuning" - } - - entry = { - "action": "gemm_tuning", - "source_phase": "KERNEL_AGENT", - "variant_name": variant_name, - "tuned_file": tuned_file, - "final_report_path": final_report, - "gain_pct": (speedup - 1.0) * 100.0, - "tput": tuned_tput, - "workspace": result.get("workspace"), - "extra_envs": extra_envs, - "backend": backend, - "source": "kernel_entry_auto", - "ts": ts, - } - if tuned_file not in existing: - self.shared_state.optimization_stack.append(entry) - self.shared_state.append_stack_gain_entry( - action="gemm_tuning", - variant_name=variant_name, - new_tput=tuned_tput, - ts=ts, - ) - self._journal_gemm_tuning_keep( - entry, - task_id=str(result.get("task_id") or ""), - ) - self.shared_state.current_best = { - "action": "gemm_tuning", - "engine": backend, - "tput": tuned_tput, - "variant_name": variant_name, - "tuned_file": tuned_file, - "final_report_path": final_report, - "workspace": result.get("workspace"), - "extra_envs": extra_envs, - } - # Unlike every other promotion this figure is inferred from a - # micro-benchmark's speedup, never measured end to end, so it is - # recorded as such rather than through the e2e path. - self.shared_state.cumulative_gain = (speedup - 1.0) * 100.0 - self.shared_state.cumulative_gain_validated = self.shared_state.cumulative_gain - self.shared_state.cumulative_gain_validated_ts = ts - self.shared_state.cumulative_gain_validated_stack_len = len(self.shared_state.optimization_stack or []) - try: - from hyperloom.inference_optimizer.breakdown.recorder import instrument - - instrument.record_session_validation( - self.session_dir, - baseline_tput=float(self.shared_state.baseline_tput or 0.0) or None, - validated_tput=float(tuned_tput) if tuned_tput else None, - validated_gain_pct=float(self.shared_state.cumulative_gain), - stack_len=self.shared_state.cumulative_gain_validated_stack_len, - source="gemm_tuning_promote", - measurement_basis="derived_speedup", - ts=ts, - ) - except Exception as exc: # noqa: BLE001 - log.debug("record_session_validation failed", exc_info=True) - trace_recording_skipped( - "session_validation", - reason="caller raised before the recorder", - entity="gemm_tuning_promote", - error=exc, - ) - def _replace_latest_gemm_tuning_attempt(self, result: dict[str, Any]) -> None: """Sync the latest GEMM history row after forge E2E rewrites ``result``.""" if not isinstance(result, dict): @@ -2245,19 +2061,23 @@ def _replace_latest_gemm_tuning_attempt(self, result: dict[str, Any]) -> None: self.shared_state.gemm_tuning_attempts = attempts self.shared_state.last_gemm_tuning = entry - async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: - """Sequentially E2E-validate each forge tuner's env independently. + def _gemm_e2e_candidates(self, result: dict[str, Any]) -> list[dict[str, Any]]: + """Reduce a GEMM tuning result to the env sets worth E2E-validating. - Like kernel_opt's per-kernel integrate: try each tuner's env one by - one. KEEPs accumulate (stacked envs); REVERTs are discarded. This - prevents one bad tuner from dragging down the whole set. - """ - from ..kernel.request_handlers import integrate_handler + Selection is by result shape: ``tuners_run`` entries name their own env + vars, whereas a bare ``tuned_file`` is only meaningful under the GEAK + a8w8 tuner's env var. - tuners_run = result.get("tuners_run") or [] + Args: + result (dict[str, Any]): The GEMM tuning handler result. + + Returns: + list[dict[str, Any]]: Candidates with ``tuner`` / ``env_var`` / + ``env_value`` / ``envs`` / ``micro_speedup``. + """ + candidates: list[dict[str, Any]] = [] # The list is already priority-sorted by forge CLI (fmoe_ck first). - candidates = [] - for t in tuners_run: + for t in result.get("tuners_run") or []: if not isinstance(t, dict): continue if t.get("status") != "ok": @@ -2289,6 +2109,27 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: } ) + if not candidates and str(result.get("backend") or "").strip().lower() != "forge": + tuned_file = str(result.get("tuned_file") or "").strip() + try: + micro_speedup = float(result.get("best_speedup") or 0.0) + except (TypeError, ValueError): + micro_speedup = 0.0 + keeps = str(result.get("decision") or "").strip().upper() == "KEEP" and str( + result.get("status") or "" + ).strip().lower() in {"ok", "complete", "completed", "succeeded", "success"} + if tuned_file and micro_speedup > 1.0 and keeps: + env_var = "AITER_CONFIG_GEMM_A8W8_BLOCKSCALE" + candidates.append( + { + "tuner": "a8w8_blockscale_tuned_gemm", + "env_var": env_var, + "env_value": tuned_file, + "envs": {env_var: tuned_file}, + "micro_speedup": micro_speedup, + } + ) + # Standalone fp8 block-scale CK backend switch: inject as its own # candidate so the loop E2E-validates baseline Triton vs CK. if self._ck_blockscale_switch_eligible(result): @@ -2302,9 +2143,22 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: "micro_speedup": 1.0, } ) + return candidates + + async def _validate_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: + """Sequentially E2E-validate each tuning candidate's env independently. + Like kernel_opt's per-kernel integrate: try each candidate's env one by + one, measured against ``current_best``. KEEPs accumulate (stacked envs); + REVERTs are discarded, so one bad candidate cannot drag down the set. + A round the run stopped ends the sweep with its tuners unrecorded. + """ + from ..kernel.request_handlers import integrate_handler + + backend = str(result.get("backend") or "geak").strip().lower() + candidates = self._gemm_e2e_candidates(result) if not candidates: - log.info("forge gemm tuning: no candidates to E2E validate") + log.info("gemm tuning: no candidates to E2E validate") return baseline_tput = float(self.shared_state.baseline_tput or 0.0) @@ -2312,7 +2166,6 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: stacked_envs: dict[str, str] = {} kept: list[dict[str, Any]] = [] reverted: list[dict[str, Any]] = [] - ts = datetime.now(timezone.utc).isoformat() try: from ..actions.executors.explore import _compute_explore_variant_timeout @@ -2341,7 +2194,7 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: tuner_name = cand["tuner"] if tuner_name == "vllm_moe_triton" and triton_moe_inert: log.warning( - "forge gemm E2E: skipping %s — the server dispatches MoE through " + "gemm E2E: skipping %s — the server dispatches MoE through " "aiter, which never reads VLLM_TUNED_CONFIG_FOLDER, so the tuned " "Triton config cannot take effect", tuner_name, @@ -2353,7 +2206,7 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: int(getattr(self.shared_state, "tp", 0) or 0), ): log.info( - "forge gemm E2E: skipping %s — aiter CK fused-MoE cannot serve " + "gemm E2E: skipping %s — aiter CK fused-MoE cannot serve " "this model at tp=%s (intermediate size is not 128-aligned)", tuner_name, getattr(self.shared_state, "tp", 0), @@ -2386,7 +2239,7 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: break if merge_failure_reason: log.error( - "forge gemm E2E: refusing aiter candidate for %s (%s: %s)", + "gemm E2E: refusing aiter candidate for %s (%s: %s)", tuner_name, merge_failure_env, merge_failure_reason, @@ -2410,7 +2263,7 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: test_envs.update(env) log.info( - "forge gemm E2E: validating tuner=%s env=%s (base_tput=%.1f)", + "gemm E2E: validating tuner=%s env=%s (base_tput=%.1f)", tuner_name, cand["env_var"], running_tput, @@ -2432,19 +2285,29 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: ) except Exception as exc: # noqa: BLE001 log.warning( - "forge gemm E2E: integrate failed for %s: %s", + "gemm E2E: integrate failed for %s: %s", tuner_name, exc, ) reverted.append({**cand, "reason": repr(exc)}) continue + stopped = stopped_by_the_run_class(integrate_result.get("error_class")) + if stopped is not None: + # Recording the rest would report a clock as a verdict on them. + log.info( + "gemm E2E: %s left unmeasured — %s", + tuner_name, + stopped.interrupted, + ) + break + decision = str(integrate_result.get("decision") or "").upper() new_tput = float(integrate_result.get("new_tput") or 0.0) gain_pct = float(integrate_result.get("gain_pct") or 0.0) log.info( - "forge gemm E2E: tuner=%s decision=%s new_tput=%.1f gain=%.2f%%", + "gemm E2E: tuner=%s decision=%s new_tput=%.1f gain=%.2f%%", tuner_name, decision, new_tput, @@ -2456,7 +2319,7 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: cand = {**cand, "tuned_config_coverage": coverage} if not coverage.get("artifact_applied"): log.error( - "forge gemm E2E: tuner=%s produced an artifact the runtime never " + "gemm E2E: tuner=%s produced an artifact the runtime never " "applied — 0 of %d requested shape(s) resolve to a tuned row; " "the %.2f%% e2e delta measures nothing about the tuning", tuner_name, @@ -2465,7 +2328,7 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: ) else: log.info( - "forge gemm E2E: tuner=%s tuned-config coverage %.2f%% (%d/%d shapes)", + "gemm E2E: tuner=%s tuned-config coverage %.2f%% (%d/%d shapes)", tuner_name, coverage.get("coverage_pct") or 0.0, coverage.get("covered") or 0, @@ -2484,34 +2347,31 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: } ) - entry = { - "action": "gemm_tuning", - "source_phase": "KERNEL_AGENT", - "variant_name": f"forge_{tuner_name}", - "tuned_file": ( - env.get(cand["env_var"]) - or next(iter(env.values()), "") - ), - "gain_pct": gain_pct, - "tput": new_tput, - "workspace": result.get("workspace"), - "extra_server_args": extra_server_args, - "extra_envs": dict(stacked_envs), - "backend": "forge", - "source": "kernel_entry_auto", - "ts": ts, - } - self.shared_state.optimization_stack.append(entry) - self.shared_state.append_stack_gain_entry( - action="gemm_tuning", - variant_name=f"forge_{tuner_name}", - new_tput=new_tput, - ts=ts, - ) - self._journal_gemm_tuning_keep( - entry, - task_id=f"gemm_tune_e2e_{tuner_name}", + lifted = self._lift_to_current_best( + "gemm_tuning", + new_tput, + { + "name": f"{backend}_{tuner_name}", + "candidate_extra_server_args": extra_server_args, + "extra_envs": dict(env), + "source_phase": "KERNEL_AGENT", + "workspace": result.get("workspace"), + }, + entry_extra={ + "tuned_file": ( + env.get(cand["env_var"]) + or next(iter(env.values()), "") + ), + "gain_pct": gain_pct, + "backend": backend, + "source": "kernel_entry_auto", + }, ) + if lifted: + self._journal_gemm_tuning_keep( + self.shared_state.optimization_stack[-1], + task_id=f"gemm_tune_e2e_{tuner_name}", + ) else: reason = f"decision={decision}, gain={gain_pct:.2f}%" if coverage is not None and not coverage.get("artifact_applied"): @@ -2520,33 +2380,16 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: reason = f"tuned_config_never_applied ({reason})" reverted.append({**cand, "reason": reason}) - # Update current_best and cumulative_gain with final stacked result. + # The watermark covers the whole run, so it waits for the last KEEP. if kept: - self.shared_state.current_best = { - "action": "gemm_tuning", - "engine": "forge", - "tput": running_tput, - "variant_name": "forge_gemm_tuned", - "extra_server_args": "--moe-runner-backend aiter" if "AITER_CONFIG_FMOE" in stacked_envs else "", - "extra_envs": stacked_envs, - "workspace": result.get("workspace"), - } total_gain = (running_tput - baseline_tput) / baseline_tput * 100.0 if baseline_tput > 0 else 0.0 - self.shared_state.cumulative_gain = total_gain if baseline_tput > 0: self._update_cumulative_gain_validated( running_tput, source="forge_gemm_tuning_e2e", - ts=ts, - ) - else: - self.shared_state.cumulative_gain_validated = total_gain - self.shared_state.cumulative_gain_validated_ts = ts - self.shared_state.cumulative_gain_validated_stack_len = len( - self.shared_state.optimization_stack or [] ) log.info( - "forge gemm E2E: %d tuners KEEP (total gain=+%.2f%%), %d REVERT", + "gemm E2E: %d tuners KEEP (total gain=+%.2f%%), %d REVERT", len(kept), total_gain, len(reverted), @@ -2555,7 +2398,7 @@ async def _validate_forge_gemm_tuning_e2e(self, result: dict[str, Any]) -> None: stacked_envs = {} total_gain = 0.0 log.info( - "forge gemm E2E: all %d tuners REVERT, no E2E gain", + "gemm E2E: all %d tuners REVERT, no E2E gain", len(reverted), ) @@ -3060,7 +2903,6 @@ async def _integrate_collective(self, result: dict) -> None: self.shared_state.gain_per_stack_entry or [] ), "current_best": dict(self.shared_state.current_best or {}), - "cumulative_gain": self.shared_state.cumulative_gain, "cumulative_gain_validated": ( self.shared_state.cumulative_gain_validated ), @@ -3181,7 +3023,11 @@ def _promote_collective_integrate_keep( *, extra_envs: dict[str, str] | None = None, ) -> None: - """Promote an E2E-validated Collective KEEP into the optimization stack.""" + """Promote an E2E-validated Collective KEEP through the current_best lift. + + A no-op when the patch is already stacked, or when the lift refuses a + winner that does not beat the live throughput anchor. + """ if not isinstance(collective_result, dict) or not isinstance( integrate_result, dict ): @@ -3243,8 +3089,6 @@ def _promote_collective_integrate_keep( raise ValueError("Collective KEEP is missing integration_id") if not isinstance(self.shared_state.optimization_stack, list): raise ValueError("optimization_stack must be a list") - if not isinstance(self.shared_state.gain_per_stack_entry, list): - raise ValueError("gain_per_stack_entry must be a list") existing = { str(item.get("patch_path") or "") for item in (self.shared_state.optimization_stack or []) @@ -3252,66 +3096,45 @@ def _promote_collective_integrate_keep( } if patch in existing: return - ts = datetime.now(timezone.utc).isoformat() envs = dict(extra_envs or integrate_result.get("extra_envs") or {}) extra_args = str(integrate_result.get("extra_server_args") or "") - entry = { - "action": "collective", - "source_phase": "KERNEL_AGENT", - "variant_name": "forge_collective", - "backend": "forge", - "engine": "forge_collective", - "provenance": "forge_collective", - "source": "kernel_entry_auto", - "integration_id": integration_id, - "kernel_id": str(collective_result.get("kernel_id") or ""), - "kernel_name": str(collective_result.get("kernel_name") or ""), - "tput": new_tput, - "gain_pct": incremental_gain, - "workspace": integrate_result.get("workspace"), - "patch_path": patch, - "target_file": collective_result.get("source_file") - or integrate_result.get("target_file"), - "extra_envs": envs, - "extra_server_args": extra_args, - "kernel_speedup": collective_result.get("kernel_speedup"), - "gpu_pct": collective_result.get("gpu_pct"), - "collective_op": collective_result.get("collective_op"), - "world_size": collective_result.get("world_size"), - "ts": ts, - } - self.shared_state.optimization_stack.append(entry) - self.shared_state.append_stack_gain_entry( - action="collective", - variant_name="forge_collective", - new_tput=new_tput, - extra_server_args=extra_args, + lifted = self._lift_to_current_best( + "collective", + new_tput, + { + "name": "forge_collective", + "candidate_extra_server_args": extra_args, + "extra_envs": envs, + "source_phase": "KERNEL_AGENT", + "provenance": "forge_collective", + "workspace": integrate_result.get("workspace"), + }, + entry_extra={ + "backend": "forge", + "engine": "forge_collective", + "source": "kernel_entry_auto", + "integration_id": integration_id, + "kernel_id": str(collective_result.get("kernel_id") or ""), + "kernel_name": str(collective_result.get("kernel_name") or ""), + "gain_pct": incremental_gain, + "patch_path": patch, + "target_file": collective_result.get("source_file") + or integrate_result.get("target_file"), + "kernel_speedup": collective_result.get("kernel_speedup"), + "gpu_pct": collective_result.get("gpu_pct"), + "collective_op": collective_result.get("collective_op"), + "world_size": collective_result.get("world_size"), + }, + ) + if not lifted: + return + ts = datetime.now(timezone.utc).isoformat() + self._update_cumulative_gain_validated( + new_tput, + source="collective_promote", ts=ts, ) - self.shared_state.current_best = { - "action": "collective", - "backend": "forge", - "engine": "forge_collective", - "tput": new_tput, - "variant_name": "forge_collective", - "workspace": integrate_result.get("workspace"), - "patch_path": patch, - "target_file": entry["target_file"], - "extra_envs": envs, - "extra_server_args": extra_args, - } total_gain = (new_tput - baseline_tput) / baseline_tput * 100.0 - self.shared_state.cumulative_gain = total_gain - self.shared_state.cumulative_gain_validated = total_gain - self.shared_state.cumulative_gain_validated_ts = ts - self.shared_state.cumulative_gain_validated_stack_len = len( - self.shared_state.optimization_stack - ) - # This lane settles its own verdict instead of going through the kernel - # integrate queue, so no kernel recorder fires for it. Without these two - # the change is invisible to the read model: the patch lands, the - # workload moves, and every point it earned reports as belonging to no - # step at all. try: from hyperloom.inference_optimizer.breakdown.recorder import instrument @@ -3323,7 +3146,11 @@ def _promote_collective_integrate_keep( new_tput=new_tput, gain_pct=incremental_gain, patch_path=patch, - target_file=str(entry["target_file"] or ""), + target_file=str( + collective_result.get("source_file") + or integrate_result.get("target_file") + or "" + ), collective_op=str(collective_result.get("collective_op") or ""), world_size=collective_result.get("world_size"), kernel_speedup=collective_result.get("kernel_speedup"), @@ -3521,65 +3348,34 @@ def _promote_fusion_integrate_keep( return patch = str(fusion_result.get("patch") or integrate_result.get("patch_path") or "") - existing = { - str(item.get("patch_path") or "") - for item in (self.shared_state.optimization_stack or []) - if isinstance(item, dict) and item.get("action") == "fusion" - } - ts = datetime.now(timezone.utc).isoformat() envs = dict(extra_envs or integrate_result.get("extra_envs") or fusion_result.get("env_flags") or {}) extra_args = str(integrate_result.get("extra_server_args") or "") - entry = { - "action": "fusion", - "source_phase": "KERNEL_AGENT", - "variant_name": "forge_fusion", - "backend": "forge", - "engine": "forge_fusion", - "provenance": "forge_fusion", - "source": "kernel_entry_auto", - "tput": new_tput, - # integrate reports the increment against its own base_tput (the - # currently active stack). Keep that local meaning on the entry; - # the session headline below must use the original baseline. - "gain_pct": incremental_gain, - "workspace": integrate_result.get("workspace"), - "patch_path": patch, - "extra_envs": envs, - "extra_server_args": extra_args, - "kernel_speedup": fusion_result.get("kernel_speedup"), - "best_pattern": fusion_result.get("best_pattern"), - "ts": ts, - } - if patch not in existing: - self.shared_state.optimization_stack.append(entry) - self.shared_state.append_stack_gain_entry( - action="fusion", - variant_name="forge_fusion", - new_tput=new_tput, - extra_server_args=extra_args, - ts=ts, - ) - self.shared_state.current_best = { - "action": "fusion", - "backend": "forge", - "engine": "forge_fusion", - "tput": new_tput, - "variant_name": "forge_fusion", - "workspace": integrate_result.get("workspace"), - "patch_path": patch, - "extra_envs": envs, - "extra_server_args": extra_args, - } - try: - baseline_tput = float(self.shared_state.baseline_tput or 0.0) - except (TypeError, ValueError): - baseline_tput = 0.0 - if baseline_tput > 0: - self.shared_state.cumulative_gain = (new_tput - baseline_tput) / baseline_tput * 100.0 + lifted = self._lift_to_current_best( + "fusion", + new_tput, + { + # The patch is the identity; the engine that produced it is not. + "name": f"forge_fusion:{Path(patch).name}" if patch else "forge_fusion", + "candidate_extra_server_args": extra_args, + "extra_envs": envs, + "source_phase": "KERNEL_AGENT", + "provenance": "forge_fusion", + "workspace": integrate_result.get("workspace"), + }, + entry_extra={ + "backend": "forge", + "engine": "forge_fusion", + "source": "kernel_entry_auto", + # integrate's increment is against the active stack, not the + # session baseline the headline uses. + "gain_pct": incremental_gain, + "patch_path": patch, + }, + ) + if lifted and float(self.shared_state.baseline_tput or 0.0) > 0: self._update_cumulative_gain_validated( new_tput, source="fusion_promote", - ts=ts, ) def _current_tput_from_validated_gain(self) -> float: diff --git a/src/hyperloom/orchestrator/phases/kernel_stack.py b/src/hyperloom/orchestrator/phases/kernel_stack.py index 75dc6dc7d0..d018c0cd42 100644 --- a/src/hyperloom/orchestrator/phases/kernel_stack.py +++ b/src/hyperloom/orchestrator/phases/kernel_stack.py @@ -9,6 +9,7 @@ from datetime import datetime, timezone from typing import Any from ..bus.message_bus import Message +from ..kernel._kernel_decisions import _entry_by_kernel_id from ..state.shared_state import resolve_grading_anchor_tput from ..state.task_registry import Task from .base import PhaseHandler @@ -82,7 +83,7 @@ async def _drain_pending_keep_integrates(self) -> None: and kid not in state.rejected_kernel_ids ): state.rejected_kernel_ids.append(kid) - attempt = (state.kernel_opt_attempts or {}).get(kid) + attempt = _entry_by_kernel_id(state, kid) if isinstance(attempt, dict): attempt["rejected_reason"] = "integrate_dispatch_exception" stable_attempt = (state.kernel_opt_task_attempts or {}).get( diff --git a/src/hyperloom/orchestrator/phases/machine.py b/src/hyperloom/orchestrator/phases/machine.py index 7a7661bb96..3e017d4052 100644 --- a/src/hyperloom/orchestrator/phases/machine.py +++ b/src/hyperloom/orchestrator/phases/machine.py @@ -233,21 +233,16 @@ async def _track_kernel_idle_streak(self) -> None: async def _advance_phase_if_needed(self) -> None: """Scan exit conditions and transition phase at most once per tick. - Priority order (Inv-8.2): abort > exit_terminal > exit_normal, per phase_state.compute_next_phase. + Priority order (Inv-8.2): global terminal > exit_terminal > exit_normal, per phase_state.compute_next_phase. """ state = self.shared_state await self._track_kernel_idle_streak() - max_hours_arg: float | None = None - mm = float(getattr(state, "max_minutes", 0) or 0.0) - if mm > 0: - max_hours_arg = mm / 60.0 next_phase = _phase_state.compute_next_phase( state, kernel_enabled=self._kernel_enabled(), budget_pct=self._phase_budget_pct, framework_agent_phase_enabled=bool(state.framework_agent_phase_enabled), explore_enabled=self._explore_enabled(), - max_hours=max_hours_arg, ) if str(state.phase or "").upper() == "EXPLORE": await self._maybe_enqueue_explore_research_scout() diff --git a/src/hyperloom/orchestrator/phases/machine_state.py b/src/hyperloom/orchestrator/phases/machine_state.py index 39a511e6f6..35fe18c621 100644 --- a/src/hyperloom/orchestrator/phases/machine_state.py +++ b/src/hyperloom/orchestrator/phases/machine_state.py @@ -220,7 +220,6 @@ def render_phase_proposable_bullets( # FRAMEWORK_AGENT phase transitions. "framework_agent_phase_done", # FRAMEWORK_AGENT → EXPLORE normal completion (no more candidates) "framework_agent_plateau", # FRAMEWORK_AGENT → EXPLORE; N consecutive resolved candidates with no KEEP (benchmarked or not) - "framework_agent_force_exit_low_budget", # FRAMEWORK_AGENT → EXPLORE; remaining wall-clock dropped below configured fraction of max_hours "framework_agent_budget_cap", # FRAMEWORK_AGENT → EXPLORE; per-phase wall-clock budget fraction reached # Cyclic phase machine back-edge reasons (transitions that reopen a macro-cycle). "cycle_reloop", # SWEEP → FRAMEWORK/EXPLORE; opens a new macro-cycle while budget + leverage remain @@ -285,7 +284,6 @@ def render_phase_proposable_bullets( "explore_force_exit_low_budget", "framework_agent_phase_done", "framework_agent_plateau", - "framework_agent_force_exit_low_budget", # R7: cyclic phase machine exhausted leverage across macro-cycles. "global_converged", # Context-window preflight: max_position_embeddings can't hold ISL+OSL. @@ -400,40 +398,16 @@ def is_valid_phase_exit_reason(value: str) -> bool: DEFAULT_PLATEAU_KERNEL_KEEP_GAIN_PCT: float = 0.5 DEFAULT_PLATEAU_KERNEL_LOOKBACK: int = 5 -# EXPLORE hard force-exit thresholds (IR-6 HARD time gate; overrides plateau). -# Fires when remaining wall-clock < HOURS_REMAINING OR EXPLORE budget fraction < BUDGET_PCT. -# HOURS_REMAINING is a leave-behind for later phases on long runs. When it is -# not strictly smaller than the session, the hours gate is ignored so a 3h -# smoke does not skip EXPLORE the moment the phase starts. -DEFAULT_EXPLORE_FORCE_EXIT_HOURS_REMAINING: float = 3.0 +# EXPLORE hard force-exit (overrides plateau). Fires on the unspent fraction of +# EXPLORE's own charge-back budget, which already reserves the later phases' +# share, so no session-remaining floor belongs beside it. DEFAULT_EXPLORE_FORCE_EXIT_BUDGET_PCT: float = 0.20 -# FRAMEWORK plateau/force-exit knobs: plateau when each LOOKBACK batch < KEEP_GAIN_PCT; force-exit when remaining < RATIO * max_hours. +# FRAMEWORK plateau knobs: plateau when each LOOKBACK batch < KEEP_GAIN_PCT. DEFAULT_FRAMEWORK_PLATEAU_LOOKBACK: int = 5 DEFAULT_FRAMEWORK_PLATEAU_KEEP_GAIN_PCT: float = 1.0 -import os as _os_fw_ratio # noqa: E402 +import os as _os_env # noqa: E402 - -def _default_framework_force_exit_ratio() -> float: - """FRAMEWORK force-exit ratio; env-overridable via - ``INFERENCE_OPTIMIZER_FRAMEWORK_FORCE_EXIT_HOURS_REMAINING_RATIO``. - - Default 0.6 reserves the last 40% of budget for later phases; lower it when - the FRAMEWORK pipeline is the primary objective so it can process forced - candidates instead of force-exiting with a full pending queue. - """ - raw = (_os_fw_ratio.environ.get("INFERENCE_OPTIMIZER_FRAMEWORK_FORCE_EXIT_HOURS_REMAINING_RATIO", "") or "").strip() - if raw: - try: - v = float(raw) - if 0.0 <= v <= 1.0: - return v - except (TypeError, ValueError): - pass # malformed env override; fall through to the 0.6 default - return 0.6 - - -DEFAULT_FRAMEWORK_FORCE_EXIT_HOURS_REMAINING_RATIO: float = _default_framework_force_exit_ratio() # FRAMEWORK per-candidate plateau: after this many consecutive resolved # candidates without a KEEP (including non-benchmarked terminal outcomes), the # phase exits to EXPLORE. A KEEP — or a macro-cycle boundary — resets it. @@ -458,7 +432,7 @@ def _default_cycle_reloop_min_remaining_sec() -> float: smaller of it and :data:`_CYCLE_RELOOP_BUDGET_RATIO` of their total budget, so a short session is not blocked by a threshold it can never satisfy. """ - raw = (_os_fw_ratio.environ.get("INFERENCE_OPTIMIZER_CYCLE_RELOOP_MIN_REMAINING_SEC", "") or "").strip() + raw = (_os_env.environ.get("INFERENCE_OPTIMIZER_CYCLE_RELOOP_MIN_REMAINING_SEC", "") or "").strip() if raw: try: v = float(raw) @@ -482,15 +456,34 @@ def _default_cycle_reloop_min_remaining_sec() -> float: # (percentage points); guards against float noise being read as progress. DEFAULT_CYCLE_MIN_GAIN_PCT: float = 1e-6 -# Decaying acceptance curve: the marginal-gain bar shrinks each macro-cycle. The -# KEEP threshold, stack-stable threshold (=keep/2) and convergence gain bar all -# ride this single curve. +# Decaying acceptance curve: the marginal-gain bar shrinks each macro-cycle. It +# is injected at dispatch for explore, integrate_patch and framework_agent, and +# also sets the stack-stable threshold (=keep/2) and the convergence gain bar. +# The kernel-owned families hold their own fixed thresholds instead. KEEP_THRESHOLD_FLOOR_PCT: float = 0.1 KEEP_THRESHOLD_SPAN_PCT: float = 0.9 # Multi-node baseline noise floor is ~2x single-node; scale the curve to match. MULTI_NODE_KEEP_THRESHOLD_FACTOR: float = 2.0 +def resolve_keep_threshold(state: Any) -> float: + """Current-cycle KEEP threshold for every path that injects ``keep_threshold_pct``. + + Reads ``macro_cycle`` off ``state`` and the multi-node flag off the + environment so callers pass nothing but the state. + + Args: + state: The SharedState (or any object carrying ``macro_cycle``). + + Returns: + The gain percentage a variant must clear to be KEPT this cycle. + """ + from ..actions.executors._multi_node_env import is_multi_node + + cycle = int(getattr(state, "macro_cycle", 0) or 0) + return decaying_keep_threshold_pct(cycle, multi_node=is_multi_node()) + + def decaying_keep_threshold_pct(macro_cycle: int, *, multi_node: bool = False) -> float: """KEEP / convergence gain threshold for cycle N = ``macro_cycle`` + 1. @@ -666,7 +659,7 @@ def _kernel_idle_max_ticks() -> int: still winding down promptly once candidates are genuinely exhausted, instead of spinning until the KERNEL wall-clock cap. """ - raw = (_os_fw_ratio.environ.get("INFERENCE_OPTIMIZER_KERNEL_IDLE_MAX_TICKS", "") or "").strip() + raw = (_os_env.environ.get("INFERENCE_OPTIMIZER_KERNEL_IDLE_MAX_TICKS", "") or "").strip() try: val = int(raw) return val if val >= 1 else 3 @@ -689,7 +682,7 @@ def _kernel_idle_min_seconds() -> float: longer than any dispatch gap a healthy phase produces, while still cutting hours off a phase that has genuinely stopped moving. """ - raw = (_os_fw_ratio.environ.get("INFERENCE_OPTIMIZER_KERNEL_IDLE_MIN_SECONDS", "") or "").strip() + raw = (_os_env.environ.get("INFERENCE_OPTIMIZER_KERNEL_IDLE_MIN_SECONDS", "") or "").strip() try: val = float(raw) return val if val > 0.0 else 600.0 @@ -766,13 +759,18 @@ def normalize_budget_pct( ) -> dict[str, float]: """Return a sanitized ``phase -> pct`` mapping (budgets are upper bounds, not renormalized to 1.0). + ``0.0`` is kept, not dropped: it is the sentinel + :func:`redistribute_budget_pct` writes for a phase the run turned off, and + overlaying the default back on it would leak that phase's share into the + charge-back denominator of every earlier phase. + Args: budget (dict[str, float] | None): Raw ``phase -> pct`` overrides; unknown phases and out-of-range / unparseable values are dropped. Returns: dict[str, float]: The defaults overlaid with the valid overrides (each - in the ``(0.0, 1.0]`` range). + in the ``[0.0, 1.0]`` range). """ out = dict(DEFAULT_PHASE_BUDGET_PCT) if not budget: @@ -785,7 +783,7 @@ def normalize_budget_pct( f = float(val) except (TypeError, ValueError): continue - if not (0.0 < f <= 1.0): + if not (0.0 <= f <= 1.0): continue out[canon] = f return out @@ -1147,14 +1145,18 @@ def _phase_budget_total_seconds( ``session_remaining_seconds`` / ``phase_elapsed_seconds``). Returns: - float | None: Effective total budget in seconds, or ``None`` when no - finite budget applies (unbounded window or the phase has no fraction). + float | None: Effective total budget in seconds, ``0.0`` when the phase + is explicitly allocated no budget, or ``None`` when no finite budget + applies (unbounded window, or an unset/unrecognized phase). """ budget = normalize_budget_pct(budget_pct or getattr(state, "phase_budget_pct", None)) phase = (getattr(state, "phase", "") or "").strip().upper() - pct = float(budget.get(phase, 0.0)) - if pct <= 0.0: + if phase not in budget: return None + pct = float(budget[phase]) + if pct <= 0.0: + # Zero fraction = no time. ``None`` would read as "unbounded" to callers. + return 0.0 session_remaining = session_remaining_seconds(state, now_unix=now_unix) if session_remaining is not None: @@ -1253,12 +1255,17 @@ def phase_cap_seconds( :func:`is_long_run`.) Returns: - float | None: Cap in seconds, or ``None`` when no fraction applies. + float | None: Cap in seconds, ``0.0`` when the phase is explicitly + allocated no budget, or ``None`` when the phase is unset/unrecognized. """ budget = normalize_budget_pct(budget_pct or getattr(state, "phase_budget_pct", None)) - pct = budget.get((getattr(state, "phase", "") or "").upper(), 0.0) - if pct <= 0: + phase = (getattr(state, "phase", "") or "").upper() + if phase not in budget: return None + pct = float(budget[phase]) + if pct <= 0.0: + # Zero fraction = no wall-clock allowed, as opposed to no cap at all. + return 0.0 proportional = effective_max_minutes(state) * 60.0 * pct abs_cap = math.ceil(PHASE_ABSOLUTE_CAP_REFERENCE_MINUTES * pct) * 60.0 return float(min(proportional, abs_cap)) @@ -1294,30 +1301,6 @@ def phase_cap_exceeded( return phase_cumulative_seconds(state, now_unix=now_unix) >= cap -# EXPLORE hard force-exit (HARD time gate) -def _explore_hours_leavebehind_applies(state: Any, *, threshold_hours: float) -> bool: - """Whether IR-6's hours-remaining gate can fire for this session. - - Remaining starts at ``max_hours``, so a leave-behind that is not strictly - smaller than the session fires the moment EXPLORE starts. The 3h default - is for long runs; a 3h session (the CI smoke, the 3h example) must not - inherit it. - - Args: - state (Any): Frozen SharedState view. - threshold_hours (float): Configured hours-remaining leave-behind. - - Returns: - bool: ``True`` when the hours gate may fire. - """ - if float(threshold_hours) <= 0.0: - return False - session_hours = _max_minutes(state) / 60.0 - if session_hours <= 0.0: - return False - return float(threshold_hours) < session_hours - - def session_remaining_seconds( state: Any, *, @@ -1362,80 +1345,39 @@ def session_remaining_seconds( def should_force_exit_explore( state: Any, *, - hours_remaining_threshold: float = DEFAULT_EXPLORE_FORCE_EXIT_HOURS_REMAINING, budget_pct_threshold: float = DEFAULT_EXPLORE_FORCE_EXIT_BUDGET_PCT, budget_pct: dict[str, float] | None = None, now_unix: float | None = None, ) -> tuple[bool, dict[str, Any]]: - """Return ``(True, evidence)`` when HARD EXPLORE force-exit fires (IR-6). + """Return ``(True, evidence)`` when HARD EXPLORE force-exit fires. - Fires when session remaining ≤ hours_threshold*3600 OR phase remaining - pct ≤ budget_pct_threshold; ``evidence`` records which fired. The hours - gate is ignored when the leave-behind is not strictly smaller than the - session (it would otherwise fire the moment EXPLORE starts). + Fires when the unspent fraction of EXPLORE's own charge-back budget drops + to ``budget_pct_threshold`` or below. A freshly entered phase always reads + 1.0, however little of the session is left. Args: state (Any): Frozen SharedState view. - hours_remaining_threshold (float): Session-hours-remaining gate; - non-positive disables it. Also ignored when it covers the session. budget_pct_threshold (float): Phase-budget-fraction gate; non-positive - disables it. + disables the force-exit entirely. budget_pct (dict[str, float] | None): Phase-budget overrides; defaults to ``state.phase_budget_pct`` when None. now_unix (float | None): Override for the current time. Returns: tuple[bool, dict[str, Any]]: ``(fired, evidence)`` — whether HARD - force-exit fires, and the evidence map recording which gate(s) fired. + force-exit fires, and the measurements behind that verdict. """ - evidence: dict[str, Any] = { - "hours_remaining_threshold": float(hours_remaining_threshold), - "budget_pct_threshold": float(budget_pct_threshold), - } - fired = False - fired_reasons: list[str] = [] - - # Non-positive threshold = disabled; both disabled turns force-exit off. - # A leave-behind that covers the whole session is also disabled: remaining - # starts at max_hours, so it would fire the moment EXPLORE starts. - hours_threshold_enabled = _explore_hours_leavebehind_applies( - state, threshold_hours=hours_remaining_threshold - ) - pct_threshold_enabled = float(budget_pct_threshold) > 0.0 - if float(hours_remaining_threshold) > 0.0 and not hours_threshold_enabled: - session_hours = _max_minutes(state) / 60.0 - if session_hours > 0.0: - evidence["hours_remaining_gate"] = "disabled_leavebehind_covers_session" - - session_remaining = session_remaining_seconds(state, now_unix=now_unix) - if session_remaining is not None and hours_threshold_enabled: - evidence["session_remaining_seconds"] = round(session_remaining, 2) - threshold_sec = float(hours_remaining_threshold) * 3600.0 - if session_remaining <= threshold_sec: - fired = True - fired_reasons.append("session_remaining") - - phase_remaining = phase_budget_remaining_seconds( - state, - budget_pct=budget_pct, - now_unix=now_unix, - ) - if phase_remaining is not None: - # Fraction of the phase's EFFECTIVE total budget — same helper that - # produced phase_remaining, so numerator and denominator stay in the same - # units (charge-back against the session for short runs, against the - # cycle-window-capped base for long bounded runs, flat per-window for - # unbounded runs). - phase_total_sec = _phase_budget_total_seconds(state, budget_pct=budget_pct, now_unix=now_unix) - if phase_total_sec and phase_total_sec > 0: - remaining_pct = phase_remaining / phase_total_sec - evidence["phase_remaining_pct"] = round(remaining_pct, 4) - evidence["phase_remaining_seconds"] = round(phase_remaining, 2) - if pct_threshold_enabled and remaining_pct <= float(budget_pct_threshold): - fired = True - fired_reasons.append("phase_remaining_pct") - - evidence["fired_reasons"] = fired_reasons + evidence: dict[str, Any] = {"budget_pct_threshold": float(budget_pct_threshold)} + phase_total_sec = _phase_budget_total_seconds(state, budget_pct=budget_pct, now_unix=now_unix) + if not phase_total_sec: + evidence["fired_reasons"] = [] + return False, evidence + phase_remaining = max(0.0, phase_total_sec - phase_cumulative_seconds(state, now_unix=now_unix)) + remaining_pct = phase_remaining / phase_total_sec + evidence["phase_remaining_pct"] = round(remaining_pct, 4) + evidence["phase_remaining_seconds"] = round(phase_remaining, 2) + fired = float(budget_pct_threshold) > 0.0 and remaining_pct <= float(budget_pct_threshold) + evidence["fired_reasons"] = ["phase_remaining_pct"] if fired else [] return fired, evidence @@ -1817,12 +1759,8 @@ def compute_kernel_progress_fingerprint( import json attempts: list[list[str]] = [] - for ledger in ( - getattr(state, "kernel_opt_task_attempts", None), - getattr(state, "kernel_opt_attempts", None), - ): - if not isinstance(ledger, dict): - continue + ledger = getattr(state, "kernel_opt_task_attempts", None) + if isinstance(ledger, dict): for ledger_id, attempt in ledger.items(): if not isinstance(attempt, dict): continue @@ -1943,11 +1881,7 @@ def kernel_work_pending(state: Any) -> bool: if source_file: integrated_sources.add(source_file) - attempts = ( - getattr(state, "kernel_opt_task_attempts", None) - or getattr(state, "kernel_opt_attempts", None) - or {} - ) + attempts = getattr(state, "kernel_opt_task_attempts", None) or {} if not isinstance(attempts, dict): return False for ledger_id, attempt in attempts.items(): @@ -2497,27 +2431,6 @@ def exit_terminal_prelude(state: Any) -> tuple[str, dict[str, Any]] | None: return None -def abort_prelude(state: Any) -> tuple[str, dict[str, Any]] | None: - """Detect a PRELUDE-aborting stop reason on the state. - - Recognizes terminal stop reasons (e.g. ``recipe_kb_t0_failed``, - ``time_exhausted_during_prelude``) so phase history captures the - boundary. - - Args: - state: Object exposing a ``stop_reason`` attribute. - - Returns: - A ``(reason, metadata)`` tuple when an abort reason is present, - otherwise ``None``. - """ - # Treat these stop reasons as a PRELUDE abort so phase_history records it. - sr = (getattr(state, "stop_reason", "") or "").strip() - if sr in ("recipe_kb_t0_failed", "time_exhausted_during_prelude", "prelude_policy_loop", "user_stop_requested"): - return sr, {"reason_origin": "shared_state.stop_reason"} - return None - - def exit_normal_explore( state: Any, *, @@ -2526,7 +2439,6 @@ def exit_normal_explore( plateau_lookback: int = DEFAULT_PLATEAU_EXPLORE_LOOKBACK, plateau_keep_gain_threshold_pct: float = DEFAULT_PLATEAU_EXPLORE_KEEP_GAIN_PCT, plateau_empty_streak_threshold: int = DEFAULT_PLATEAU_EXPLORE_EMPTY_STREAK, - force_exit_hours_remaining: float = DEFAULT_EXPLORE_FORCE_EXIT_HOURS_REMAINING, force_exit_budget_pct: float = DEFAULT_EXPLORE_FORCE_EXIT_BUDGET_PCT, ) -> tuple[str, dict[str, Any]] | None: """EXPLORE normal exit. @@ -2546,8 +2458,6 @@ def exit_normal_explore( which the plateau gain arm is active. plateau_empty_streak_threshold: Consecutive empty specialist rounds required for the plateau streak arm. - force_exit_hours_remaining (float): Session-hours-remaining force-exit - threshold. force_exit_budget_pct (float): Phase-budget-fraction force-exit threshold. @@ -2557,7 +2467,6 @@ def exit_normal_explore( """ forced, force_ev = should_force_exit_explore( state, - hours_remaining_threshold=force_exit_hours_remaining, budget_pct_threshold=force_exit_budget_pct, budget_pct=budget_pct, now_unix=now_unix, @@ -2948,27 +2857,20 @@ def _framework_agent_plateau_streak_threshold() -> int: def exit_normal_framework_agent( state: Any, *, - max_hours: float | None = None, now_unix: float | None = None, - force_exit_hours_remaining_ratio: float = (DEFAULT_FRAMEWORK_FORCE_EXIT_HOURS_REMAINING_RATIO), budget_pct: dict[str, float] | None = None, ) -> tuple[str, dict[str, Any]] | None: """FRAMEWORK normal exit. - Priority: 0. HARD force-exit when remaining < ratio*max_hours → - ``framework_agent_force_exit_low_budget``; 0.5. per-phase budget cap reached → - ``framework_agent_budget_cap``; 1. plateau when - :func:`_framework_agent_consecutive_no_keep` ≥ threshold → - ``framework_agent_plateau``; 2. ``framework_agent_phase_done``; else ``None``. + Priority: 0. per-phase budget cap reached → ``framework_agent_budget_cap``; + 1. plateau when :func:`_framework_agent_consecutive_no_keep` ≥ threshold → + ``framework_agent_plateau``; 2. ``framework_agent_phase_done``; else + ``None``. Args: - state (Any): Frozen SharedState view; may expose a ``remaining_minutes`` - callable and ``framework_agent_phase_done`` flag. - max_hours (float | None): Session wall-clock budget in hours; enables - the force-exit gate when positive. + state (Any): Frozen SharedState view; may expose a + ``framework_agent_phase_done`` flag. now_unix (float | None): Override for the current time. - force_exit_hours_remaining_ratio (float): Fraction of ``max_hours`` - below which the force-exit gate fires. budget_pct (dict[str, float] | None): Phase-budget overrides; defaults to ``state.phase_budget_pct``. Enables the per-phase wall-clock cap. @@ -2976,28 +2878,6 @@ def exit_normal_framework_agent( tuple[str, dict[str, Any]] | None: ``(reason, evidence)`` for the FRAMEWORK exit, or ``None`` when the phase should continue. """ - if max_hours and max_hours > 0: - remaining_min_fn = getattr(state, "remaining_minutes", None) - if callable(remaining_min_fn): - try: - remaining_minutes = float(remaining_min_fn(now_unix=now_unix)) - except TypeError: - remaining_minutes = float(remaining_min_fn()) - except Exception: # noqa: BLE001 - remaining_minutes = float("inf") - else: - remaining_minutes = float("inf") - threshold_minutes = float(force_exit_hours_remaining_ratio) * float(max_hours) * 60.0 - if remaining_minutes < threshold_minutes: - return "framework_agent_force_exit_low_budget", { - "evidence": "force_exit", - "remaining_minutes": remaining_minutes, - "threshold_minutes": threshold_minutes, - "hours_remaining_ratio": float(force_exit_hours_remaining_ratio), - "max_hours": float(max_hours), - "pending_candidate_count": _framework_agent_pending_candidate_count(state), - } - # Per-phase wall-clock budget cap: rotate to EXPLORE once the phase has # burned its share so it cannot monopolise the run. if phase_cap_exceeded(state, budget_pct=budget_pct, now_unix=now_unix): @@ -3060,11 +2940,10 @@ def compute_next_phase( now_unix: float | None = None, framework_agent_phase_enabled: bool = False, explore_enabled: bool = True, - max_hours: float | None = None, ) -> tuple[str, str, dict[str, Any]] | None: """Return ``(next_phase, reason, evidence)`` or ``None``. - Priority (Inv-8.2): global terminal first, then abort > exit_terminal > exit_normal. + Priority (Inv-8.2): global terminal first, then exit_terminal > exit_normal. Args: state (Any): Frozen SharedState view exposing the current ``phase``. @@ -3075,8 +2954,6 @@ def compute_next_phase( framework_agent_phase_enabled (bool): Whether the FRAMEWORK_AGENT phase runs after PRELUDE. explore_enabled (bool): Whether the EXPLORE phase is enabled. - max_hours (float | None): Session wall-clock budget in hours (used by - the FRAMEWORK force-exit gate). Returns: tuple[str, str, dict[str, Any]] | None: ``(next_phase, reason, @@ -3092,9 +2969,6 @@ def compute_next_phase( return PHASE_CLOSE, reason, {"terminal": True, **evidence} if current == PHASE_PRELUDE: - ab = abort_prelude(state) - if ab is not None: - return PHASE_CLOSE, ab[0], {"terminal": True, **ab[1]} term = exit_terminal_prelude(state) if term is not None: return PHASE_CLOSE, term[0], {"terminal": True, **term[1]} @@ -3128,14 +3002,7 @@ def compute_next_phase( if current == PHASE_FRAMEWORK_AGENT: norm = exit_normal_framework_agent( state, - max_hours=max_hours, now_unix=now_unix, - force_exit_hours_remaining_ratio=float( - overrides.get( - "framework_agent_force_exit_hours_ratio", - DEFAULT_FRAMEWORK_FORCE_EXIT_HOURS_REMAINING_RATIO, - ) - ), budget_pct=budget_pct, ) if norm is not None: @@ -3173,12 +3040,6 @@ def compute_next_phase( DEFAULT_PLATEAU_EXPLORE_EMPTY_STREAK, ) ), - force_exit_hours_remaining=float( - overrides.get( - "force_exit_hours_remaining", - DEFAULT_EXPLORE_FORCE_EXIT_HOURS_REMAINING, - ) - ), force_exit_budget_pct=float( overrides.get( "force_exit_budget_pct", @@ -3617,7 +3478,6 @@ def record_lifecycle_event( __all__ = [ "DEFAULT_EXPLORE_FORCE_EXIT_BUDGET_PCT", - "DEFAULT_EXPLORE_FORCE_EXIT_HOURS_REMAINING", "DEFAULT_PHASE_BUDGET_PCT", "OPTIMIZATION_RESERVE_PCT", "DEFAULT_PLATEAU_EXPLORE_EMPTY_STREAK", @@ -3657,15 +3517,14 @@ def record_lifecycle_event( "make_lifecycle_event", "DEFAULT_FRAMEWORK_PLATEAU_LOOKBACK", "DEFAULT_FRAMEWORK_PLATEAU_KEEP_GAIN_PCT", - "DEFAULT_FRAMEWORK_FORCE_EXIT_HOURS_REMAINING_RATIO", "DEFAULT_MAX_MACRO_CYCLES", "DEFAULT_CYCLE_RELOOP_MIN_REMAINING_SEC", "DEFAULT_GLOBAL_CONVERGENCE_NO_GAIN_CYCLES", "DEFAULT_CYCLE_MIN_GAIN_PCT", "DEFAULT_LONGRUN_THRESHOLD_MINUTES", "is_long_run", + "resolve_keep_threshold", "should_reloop_to_explore", - "abort_prelude", "allowed_actions_for", "apply_escalate_budget_bump", "bank_phase_segment", diff --git a/src/hyperloom/orchestrator/phases/prelude.py b/src/hyperloom/orchestrator/phases/prelude.py index a11d620f7e..8a2ca25d5b 100644 --- a/src/hyperloom/orchestrator/phases/prelude.py +++ b/src/hyperloom/orchestrator/phases/prelude.py @@ -11,7 +11,6 @@ import logging as _logging import math import os -from datetime import datetime, timezone from pathlib import Path from collections.abc import Mapping from typing import Any @@ -115,28 +114,34 @@ def _merge_named_current_recipe_configs( return " ".join(token for key in order for token in pairs[key]), envs -def _warm_kernel_keep_threshold_pct(default: float = 1.0) -> float: - """Return the approved combined current-contract KEEP threshold.""" +def _warm_kernel_keep_threshold_pct(state: Any) -> float: + """Gain a replayed champion set must clear. + + Follows the shared decaying curve so the bar tracks the session's + macro-cycle; ``HYPERLOOM_WARM_KERNEL_KEEP_PCT`` overrides it. + + Args: + state: The SharedState the curve reads ``macro_cycle`` from. + + Returns: + The KEEP threshold percentage for the warm-kernel replay. + """ + default = _phase_state.resolve_keep_threshold(state) raw = str(os.environ.get("HYPERLOOM_WARM_KERNEL_KEEP_PCT", "") or "").strip() if not raw: return default try: value = float(raw) except ValueError: - log.warning( - "warm replay: invalid HYPERLOOM_WARM_KERNEL_KEEP_PCT=%r; using %.2f", - raw, - default, - ) - return default - if not math.isfinite(value): - log.warning( - "warm replay: non-finite HYPERLOOM_WARM_KERNEL_KEEP_PCT=%r; using %.2f", - raw, - default, - ) - return default - return value + value = math.nan + if math.isfinite(value): + return value + log.warning( + "warm replay: unusable HYPERLOOM_WARM_KERNEL_KEEP_PCT=%r; using %.2f", + raw, + default, + ) + return default class PreludePhase(PhaseHandler): @@ -1747,7 +1752,7 @@ async def _maybe_enqueue_warm_replay( "combined_current_contract": bool( current_remote or kernel_pending ), - "combined_keep_threshold_pct": _warm_kernel_keep_threshold_pct(), + "combined_keep_threshold_pct": _warm_kernel_keep_threshold_pct(self.shared_state), "workload_compatibility": workload_compatibility, } try: @@ -2079,12 +2084,12 @@ def _promote_warm_replay( keep_threshold = ( float(raw_threshold) if raw_threshold is not None - else _warm_kernel_keep_threshold_pct() + else _warm_kernel_keep_threshold_pct(self.shared_state) ) except (TypeError, ValueError): - keep_threshold = _warm_kernel_keep_threshold_pct() + keep_threshold = _warm_kernel_keep_threshold_pct(self.shared_state) if not math.isfinite(keep_threshold): - keep_threshold = _warm_kernel_keep_threshold_pct() + keep_threshold = _warm_kernel_keep_threshold_pct(self.shared_state) min_reproduce = float( getattr(self, "_warm_replay_min_reproduce_pct", 0.8) or 0.8, ) @@ -2142,6 +2147,9 @@ def _promote_warm_replay( ) if promoted_checkout: outcome["active_framework_root"] = promoted_checkout + # Resume re-points $INFERENCEX_PATH at this checkout, and stops + # the run when it has since vanished. + state.active_inferencex_path = promoted_checkout warm_args = str(params.get("extra_server_args") or "").strip() warm_envs = dict(params.get("extra_envs") or {}) replayed_patch_refs = [ @@ -2185,31 +2193,21 @@ def _promote_warm_replay( outcome.pop("replayed_patch_refs", None) if replayed_patch_refs: outcome["replayed_patch_refs"] = replayed_patch_refs - # Push warm best_config onto the stack (schema mirrors explore-KEEP). - stack_entry = { - "action": "replay_warm_recipe", - "source_phase": "PRELUDE", - "name": "warm_replay", - "variant_name": "warm_replay", - "task_id": str(getattr(task, "task_id", "") or ""), - "extra_server_args": warm_args, - "extra_envs": warm_envs, - "tput": float(single_round_tput), + # Stack-entry-only metadata; the lift keeps current_best pure config. + entry_extra: dict[str, Any] = { + "gain_pct": round(measured_gain, 3), "hot_tput": float(hot_tput), "cold_tput": float(cold_round_tput) if cold_round_tput > 0 else None, - "gain_pct": round(measured_gain, 3), - "workspace": str(result.get("workspace") or ""), - "ts": datetime.now(timezone.utc).isoformat(), # source_tier records the warm-recipe tier for breakdown attribution. "source_tier": outcome.get("warm_recipe_tier", ""), "source_confidence": outcome.get("warm_recipe_conf", 0.0), } if promoted_checkout: - stack_entry["framework_source_root"] = promoted_checkout + entry_extra["framework_source_root"] = promoted_checkout kernel_outcome = self._book_combined_kernel_keep(result, task) outcome["kernel"] = dict(kernel_outcome) if kernel_outcome.get("kept"): - stack_entry["kernel_replay"] = { + entry_extra["kernel_replay"] = { "validation": "combined_recipe_kernel", "count": kernel_outcome["kept"], "columns": sorted( @@ -2222,14 +2220,14 @@ def _promote_warm_replay( } patch_result = result.get("warm_patch_result") if isinstance(patch_result, dict): - stack_entry["recipe_patch_statuses"] = list( + entry_extra["recipe_patch_statuses"] = list( patch_result.get("patches") or [] ) if replayed_patch_refs: - stack_entry["replayed_patch_refs"] = replayed_patch_refs + entry_extra["replayed_patch_refs"] = replayed_patch_refs # Resume safety: do not clobber existing stack entries. state.optimization_stack = list(state.optimization_stack or []) - # Idempotency guard: skip push if a prior promote already pushed it. + # A prior promote owns the outcome; re-running would re-journal it. already_pushed = any( isinstance(e, dict) and e.get("action") == "replay_warm_recipe" for e in state.optimization_stack ) @@ -2242,32 +2240,21 @@ def _promote_warm_replay( state.warm_replay_outcome = outcome state.save(self.session_dir) return - state.optimization_stack.append(stack_entry) - # gain_per_stack_entry runs in lock-step with optimization_stack. - gp = list(getattr(state, "gain_per_stack_entry", []) or []) - gp.append(round(measured_gain, 3)) - state.gain_per_stack_entry = gp - # Cumulative gain is absolute tput vs baseline, not additive deltas. - total_gain = (single_round_tput / baseline_tput - 1.0) * 100.0 - state.cumulative_gain = round(total_gain, 3) - state.cumulative_gain_validated = round(total_gain, 3) - state.cumulative_gain_validated_ts = stack_entry["ts"] - state.cumulative_gain_validated_stack_len = len(state.optimization_stack) - state.current_best = { - "action": "warm_replay", - "name": "warm_replay", - "tput": single_round_tput, - "hot_tput": hot_tput, - "cold_tput": cold_round_tput if cold_round_tput > 0 else None, - "extra_server_args": warm_args, - "extra_envs": warm_envs, - } - if promoted_checkout: - state.current_best["framework_source_root"] = promoted_checkout - if stack_entry.get("kernel_replay"): - state.current_best["kernel_replay"] = dict( - stack_entry["kernel_replay"] - ) + self._lift_to_current_best( + "replay_warm_recipe", + float(single_round_tput), + { + "name": "warm_replay", + "candidate_extra_server_args": warm_args, + "extra_envs": warm_envs, + "source_phase": "PRELUDE", + "task_id": str(getattr(task, "task_id", "") or ""), + "workspace": str(result.get("workspace") or ""), + }, + entry_extra=entry_extra, + ) + if baseline_tput > 0: + self._update_cumulative_gain_validated(single_round_tput) log.info( "warm-replay REPRODUCED: measured=+%.2f%% (expected=+%.2f%%, " "min_required=+%.2f%%); pushed warm_replay onto stack", diff --git a/src/hyperloom/orchestrator/policy/gate.py b/src/hyperloom/orchestrator/policy/gate.py index db505c9250..3532bea756 100644 --- a/src/hyperloom/orchestrator/policy/gate.py +++ b/src/hyperloom/orchestrator/policy/gate.py @@ -539,7 +539,6 @@ def _source_file_candidates(value: str) -> tuple[str, ...]: # away from the reason it was stamped for. "stop_ts", "last_tick_exception", - "cumulative_gain", "cumulative_gain_validated", "cumulative_gain_validated_ts", "cumulative_gain_validated_stack_len", @@ -567,7 +566,6 @@ def _source_file_candidates(value: str) -> tuple[str, ...]: "schema_version", # Recipe KB integration fields (Coordinator-only writes). "recipe_kb_session_id", - "recipe_kb_session_summary", "warm_start_recipe", "warm_start_pitfalls", "warm_start_lessons", @@ -660,7 +658,6 @@ def _source_file_candidates(value: str) -> tuple[str, ...]: "last_trace_analyze", "last_kernel_opt", "last_kernel_opt_dispatch_skip", - "kernel_opt_attempts", "kernel_opt_task_attempts", "pending_kernel_integrations", "last_collective", diff --git a/src/hyperloom/orchestrator/prompts/prompt_builder.py b/src/hyperloom/orchestrator/prompts/prompt_builder.py index 8ca28a90b7..b5e47cd182 100644 --- a/src/hyperloom/orchestrator/prompts/prompt_builder.py +++ b/src/hyperloom/orchestrator/prompts/prompt_builder.py @@ -90,7 +90,7 @@ def _section_mission() -> list[str]: "## 1. MISSION", "", "You are the Orchestration agent of an autonomous inference-optimization loop.", - "Your single most important goal is to maximise the run's **cumulative_gain**", + "Your single most important goal is to maximise the run's **cumulative_gain_validated**", "(percent over baseline_tput) within the wall-clock budget.", "", "Every tick, ask yourself:", @@ -99,9 +99,8 @@ def _section_mission() -> list[str]: "", 'An optimization is only "real" once it has been validated as part of the', "full optimization_stack. ``explore`` inlines a per-KEEP stack rebench, so", - "the validated cumulative gain advances automatically — sums of per-round", - "gains still do NOT compose linearly, so drive the loop until ``explore``", - "has produced at least one KEEP that survived the stack rebench.", + "cumulative_gain_validated advances automatically — drive the loop until", + "``explore`` has produced at least one KEEP that survived the rebench.", ] @@ -432,7 +431,7 @@ def _format_emit_hint(meta: ActionMetadata) -> str: "specialist_task_id=, " "patches?=[], " "config_changes?={ENV_VAR: value}, " - "keep_threshold_pct?=1.0, " + "keep_threshold_pct?=, " "accuracy_baseline?=}}" ) return f"propose_action{{action_name='{meta.name}', predicted_gain_pct=}}" @@ -456,7 +455,7 @@ def _format_grid_injection_hint(name: str) -> str | None: "args_mode?: 'append'|'replace', provenance, kb_evidence?, " "pr_evidence?, source_evidence?}, ...], " "base_extra_args?, base_tput?, accuracy_baseline?, " - "keep_threshold_pct?: 1.0, stack_stable_threshold_pct?: 0.5}}`. " + "keep_threshold_pct?: , stack_stable_threshold_pct?: }}`. " "Variants run serially; each KEEP triggers an inlined stack " "rebench. Variant identity is content-based (args+envs+" "remove_args+unset_envs+args_mode); only exact duplicates within " @@ -555,7 +554,7 @@ def _section_decision_framework(*, kernel_enabled: bool, phase: str = "", transp "These are reference heuristics and objective facts, not a forced", "sequence. Read the dynamic SharedState section and decide:", "", - "1. **Stop**: if `stop_reason` is set OR `cumulative_gain >= target_gain_pct`,", + "1. **Stop**: if `stop_reason` is set OR `cumulative_gain_validated >= target_gain_pct`,", " propose `report` once (if not already done) then heartbeat 'goal-reached'.", "2. **Measure**: if `baseline_tput == 0`, propose `baseline`. Wait for", " delegated_result; do NOT re-baseline on a positive result with warnings.", @@ -771,7 +770,7 @@ def _idea_generation_lines() -> list[str]: Backend policy: DO NOT add a `backends` field. Current GEAK owns the KERNEL phase by default. Forge per-kernel mode is available only when the operator set exactly `KERNEL_OPT_BACKEND_ORDER=forge`. - Read `kernel_opt_attempts` + + Read `kernel_opt_task_attempts` + `pending_keep_kernels` to see what's still queueable; the batch handler filters rejected/in-flight/exhausted candidates. @@ -787,7 +786,7 @@ def _idea_generation_lines() -> list[str]: Omit `base_tput` / `patch_path` / `source_file` and the Coordinator fills them from `current_best.tput` and the per-kernel - `kernel_opt_attempts` ledger (this is what drains a multi-KEEP queue). + `kernel_opt_task_attempts` ledger (this is what drains a multi-KEEP queue). PARTIAL / REVERT → do NOT integrate; pick the next action normally. **Multi-KEEP queue:** `pending_keep_kernels` (sorted strongest-first) diff --git a/src/hyperloom/orchestrator/state/_shared_state/explore_state.py b/src/hyperloom/orchestrator/state/_shared_state/explore_state.py index 1c04a41b7b..078d2396d2 100644 --- a/src/hyperloom/orchestrator/state/_shared_state/explore_state.py +++ b/src/hyperloom/orchestrator/state/_shared_state/explore_state.py @@ -351,10 +351,6 @@ def bump_specialist_dispatched(self, n: int = 1) -> int: self.explore_specialist_dispatched_count = int(self.explore_specialist_dispatched_count or 0) + int(n) return self.explore_specialist_dispatched_count - def reset_specialist_dispatched(self) -> None: - """Zero the per-EXPLORE specialist dispatch counter (on fresh EXPLORE entry).""" - self.explore_specialist_dispatched_count = 0 - def bump_research_scout_runs(self, n: int = 1) -> int: """Increment the research-scout dispatch counter; return new total. @@ -764,22 +760,6 @@ def record_framework_lever_attribution( return False return False - def framework_levers_by_state(self, *, default_on: bool) -> list[dict[str, Any]]: - """Return registered levers filtered by whether they are currently on. - - Args: - default_on: Select levers that are on (``True``) or dormant - (``False``). - - Returns: - The matching lever rows, in registration order. - """ - return [ - row - for row in (getattr(self, "authored_framework_levers", None) or []) - if isinstance(row, dict) and bool(row.get("default_on")) is bool(default_on) - ] - def record_discovered_flags( self, *, diff --git a/src/hyperloom/orchestrator/state/_shared_state/render.py b/src/hyperloom/orchestrator/state/_shared_state/render.py index 3060bb7f26..3445954c66 100644 --- a/src/hyperloom/orchestrator/state/_shared_state/render.py +++ b/src/hyperloom/orchestrator/state/_shared_state/render.py @@ -108,8 +108,7 @@ def to_mission_summary(self, *, now: datetime | None = None) -> str: lines = [ f"baseline : {framework_registry.format_primary_metric(self.framework, self.baseline_tput)}", f"current : {self._format_current_best_for_mission()}", - f"gain : per-round-sum={self.cumulative_gain:.2f}% " - f"validated={self.cumulative_gain_validated:.2f}%{validated_age}", + f"gain : validated={self.cumulative_gain_validated:.2f}%{validated_age}", f"stack : {len(self.optimization_stack)} entries " f"(validated_at_len={self.cumulative_gain_validated_stack_len})" f"{unvalidated_tag}{resume_revalidation_tag}{geak_pending_tag}", @@ -150,7 +149,7 @@ def to_phase_status_summary( budget_pct: dict[str, float] | None = None, now_unix: float | None = None, ) -> str: - """Render the per-tick ``=== Phase ===`` block (≤7 lines). EXPLORE adds a ``force_exit`` line showing runway before the hard force-exit gate; the mid-chain phases add a ``cycle_reloop`` line showing whether another macro-cycle is still affordable. + """Render the per-tick ``=== Phase ===`` block (≤7 lines). EXPLORE adds a ``force_exit`` line showing the budget fraction left before the hard force-exit gate; the mid-chain phases add a ``cycle_reloop`` line showing whether another macro-cycle is still affordable. Args: budget_pct (dict[str, float] | None): Per-phase budget fractions; @@ -162,7 +161,6 @@ def to_phase_status_summary( """ from ...phases.machine_state import ( DEFAULT_EXPLORE_FORCE_EXIT_BUDGET_PCT, - DEFAULT_EXPLORE_FORCE_EXIT_HOURS_REMAINING, PHASE_EXPLORE, PHASE_FRAMEWORK_AGENT, PHASE_KERNEL_AGENT, @@ -210,23 +208,12 @@ def to_phase_status_summary( # EXPLORE-only: distance to hard force-exit alongside the soft budget. if phase == PHASE_EXPLORE: overrides = self.plateau_overrides or {} - hours_thresh = float( - overrides.get( - "force_exit_hours_remaining", - DEFAULT_EXPLORE_FORCE_EXIT_HOURS_REMAINING, - ) - ) pct_thresh = float( overrides.get( "force_exit_budget_pct", DEFAULT_EXPLORE_FORCE_EXIT_BUDGET_PCT, ) ) - session_remaining = session_remaining_seconds( - self, - now_unix=now_unix, - ) - session_buffer = int(session_remaining - hours_thresh * 3600.0) if session_remaining is not None else None # Same effective-total helper as `remaining` so the fraction stays in # one unit (charge-back against the session for short runs, against # the cycle-window-capped base for long bounded runs). @@ -235,9 +222,7 @@ def to_phase_status_summary( phase_remaining_pct = remaining / phase_total_sec else: phase_remaining_pct = None - force_line = f"force_exit: hours_thresh={hours_thresh:.1f}h pct_thresh={pct_thresh:.2f}" - if session_buffer is not None: - force_line += f" session_buffer_sec={session_buffer}" + force_line = f"force_exit: pct_thresh={pct_thresh:.2f}" if phase_remaining_pct is not None: force_line += f" phase_remaining_pct={phase_remaining_pct:.3f}" lines.append(force_line) @@ -580,7 +565,6 @@ def to_prompt_summary(self) -> str: f"baseline_failure_streak={self.baseline_failure_streak}", f"current_best={self.current_best or '(none)'}", f"optimization_stack={self._format_optimization_stack()}", - f"cumulative_gain={self.cumulative_gain}%", ( f"cumulative_gain_validated={self.cumulative_gain_validated}% " f"(stack_len_at_validation={self.cumulative_gain_validated_stack_len}, " diff --git a/src/hyperloom/orchestrator/state/objective.py b/src/hyperloom/orchestrator/state/objective.py index 904153c576..14a6fc5df2 100644 --- a/src/hyperloom/orchestrator/state/objective.py +++ b/src/hyperloom/orchestrator/state/objective.py @@ -15,6 +15,8 @@ from pathlib import Path from typing import TYPE_CHECKING, Any +from .shared_state import resolve_grading_anchor_tput + if TYPE_CHECKING: # pragma: no cover from .shared_state import SharedState @@ -23,15 +25,6 @@ class ObjectiveError(ValueError): """Raised by `build_objective` on bad/conflicting inputs.""" -def _resolve_current_tput(state: "SharedState") -> float: - """Resolve current throughput, preferring ``current_best['tput']`` over baseline (0.0 if none).""" - cb = state.current_best or {} - v = cb.get("tput") if isinstance(cb, dict) else None - if isinstance(v, (int, float)) and v > 0: - return float(v) - return float(state.baseline_tput or 0.0) - - @dataclass class Objective(ABC): """Goal optimized against (pure functions of SharedState).""" @@ -124,7 +117,7 @@ def gap_pct(self, state: "SharedState") -> float: @dataclass class TargetGainObjective(_RatioObjective): - """Reach ``target_gain_pct`` % over baseline_tput (progress = cumulative_gain / target, capped at 1.0).""" + """Reach ``target_gain_pct`` % over baseline_tput (progress = cumulative_gain_validated / target, capped at 1.0).""" target_gain_pct: float @@ -146,8 +139,8 @@ def kind(self) -> str: return "gain_pct" def _current(self, state: "SharedState") -> float: - """Return the per-round-sum cumulative gain percentage (``cumulative_gain``, not ``cumulative_gain_validated``).""" - return state.cumulative_gain + """Return the cumulative validated gain percentage.""" + return state.cumulative_gain_validated def _target(self) -> float: """Return the configured gain-percent target.""" @@ -197,7 +190,7 @@ def kind(self) -> str: def _current(self, state: "SharedState") -> float: """Resolve current throughput (best-so-far, else baseline).""" - return _resolve_current_tput(state) + return resolve_grading_anchor_tput(state) def _target(self) -> float: """Return the configured per-GPU throughput target.""" @@ -253,7 +246,7 @@ def kind(self) -> str: def _current(self, state: "SharedState") -> float: """Resolve current throughput (best-so-far, else baseline).""" - return _resolve_current_tput(state) + return resolve_grading_anchor_tput(state) def _target(self) -> float: """Return the reference throughput loaded from the baseline session.""" diff --git a/src/hyperloom/orchestrator/state/optimization_journal.py b/src/hyperloom/orchestrator/state/optimization_journal.py index 05c670bd7b..1165424de5 100644 --- a/src/hyperloom/orchestrator/state/optimization_journal.py +++ b/src/hyperloom/orchestrator/state/optimization_journal.py @@ -49,6 +49,9 @@ # The only status meaning the change was adopted into current_best. _JOURNAL_KEEP_STATUSES: frozenset[str] = frozenset({"kept"}) +# Stamped on the result when the anchor gate refused an executor-granted KEEP. +PROMOTION_REFUSED_KEY: str = "promotion_refused" + # Statuses meaning a real change was tested/applied then rolled back or rejected # on measured grounds → REVERT. Everything else is ``no_promote``. _JOURNAL_REVERT_STATUSES: frozenset[str] = frozenset({"reverted", "accuracy_unavailable_reject", "regression"}) @@ -366,7 +369,8 @@ def derive_journal_outcome( For source-patch kinds (``integrate_patch`` / ``framework_agent``) the outcome follows the executor's authoritative per-status verdict: - - ``status == "kept"`` → ``OUTCOME_KEEP`` + - ``status == "kept"`` → ``OUTCOME_KEEP``, unless the promote path stamped + :data:`PROMOTION_REFUSED_KEY` because the anchor gate declined to lift it - ``status in {reverted, accuracy_unavailable_reject, regression}`` → ``OUTCOME_REVERT`` - any other status → ``OUTCOME_NO_PROMOTE`` @@ -388,12 +392,15 @@ def derive_journal_outcome( One of :data:`OUTCOME_KEEP` / :data:`OUTCOME_REVERT` / :data:`OUTCOME_NO_PROMOTE` / :data:`OUTCOME_SKIP`. """ + result = result_dict or {} kind = (task_kind or "").lower() - if kind in _SKIPPABLE_JOURNAL_KINDS and (result_dict or {}).get("was_skipped"): + if kind in _SKIPPABLE_JOURNAL_KINDS and result.get("was_skipped"): return OUTCOME_SKIP if kind in _STATUS_DRIVEN_JOURNAL_KINDS: - status = str((result_dict or {}).get("status") or "").strip().lower() + status = str(result.get("status") or "").strip().lower() if status in _JOURNAL_KEEP_STATUSES: + if result.get(PROMOTION_REFUSED_KEY): + return OUTCOME_NO_PROMOTE return OUTCOME_KEEP if status in _JOURNAL_REVERT_STATUSES: return OUTCOME_REVERT @@ -542,6 +549,7 @@ def summarize_change( "OUTCOME_NO_PROMOTE", "OUTCOME_REVERT", "OUTCOME_SKIP", + "PROMOTION_REFUSED_KEY", "classify_change_kind", "derive_journal_outcome", "summarize_change", diff --git a/src/hyperloom/orchestrator/state/shared_state.py b/src/hyperloom/orchestrator/state/shared_state.py index ac0f022802..02627f339a 100644 --- a/src/hyperloom/orchestrator/state/shared_state.py +++ b/src/hyperloom/orchestrator/state/shared_state.py @@ -25,7 +25,7 @@ current_best dict — champion snapshot: ``action`` + ``tput`` plus per-writer detail (variant_name, extra_server_args, extra_envs, workspace, latency means) - cumulative_gain float — % over baseline + cumulative_gain_validated float — % over baseline at the last full-stack rebench stop_reason str — set when graceful stop fires stop_ts str — ISO timestamp of the first stop_reason write resumed_ts str — ISO timestamp of the most recent --resume @@ -587,7 +587,7 @@ class SharedState(_RenderMixin, _ExploreStateMixin): # One-shot guard for PRELUDE warm-kernel KB read/apply (resume can't re-fire). warm_kernel_kb_attempted: bool = False # Resolved prior-champion kernel columns (gemm/fusion/rewrite) loaded at - # PRELUDE from the Recipe ``value.kernel`` section, with file paths resolved. + # PRELUDE from the Recipe ``value.kernel``; read back by the combined promote. warm_kernel_kb_plan: list = field(default_factory=list) # Baseline COLD (warmup-round) full boot+bench wall-clock; the hard-cap # anchor from which ExploreExecutor derives the overtime-kill deadline. @@ -634,13 +634,10 @@ class SharedState(_RenderMixin, _ExploreStateMixin): optimization_stack: list[dict[str, Any]] = field(default_factory=list) # Index-aligned with ``optimization_stack``: per-entry incremental gain pct; missing => None. gain_per_stack_entry: list[float | None] = field(default_factory=list) - cumulative_gain: float = 0.0 - # Validated cumulative gain: re-baselined fresh server with every KEEP (per-round gains don't compose linearly); standalone validate_stack denied by PolicyGate. + # Total gain over ``baseline_tput``, stamped only from a measurement taken + # with the whole stack applied; standalone validate_stack denied by PolicyGate. cumulative_gain_validated: float = 0.0 cumulative_gain_validated_ts: str = "" - # Provenance/basis of the currently-recorded gain (provisional cross-harness - # vs same-harness-validated). Display/audit only; never gates scheduling. - cumulative_gain_provenance: str = "" # ``optimization_stack`` length at last successful inline rebench; longer => new KEEPs need validation. cumulative_gain_validated_stack_len: int = 0 # Resume sentinels. ``pending_integrate`` is written before a @@ -731,9 +728,9 @@ class SharedState(_RenderMixin, _ExploreStateMixin): # kept or reverted whole. authored_framework_levers: list[dict[str, Any]] = field(default_factory=list) - # Roofline-v2 trace-analyze cache written by record_trace_analyze; ``roofline_snapshot_id`` mirrors the nested value for hot-path access. + # Roofline-v2 trace-analyze cache written by record_trace_analyze. + # roofline_snapshot_id is a property derived from this dict. last_trace_analyze: dict[str, Any] = field(default_factory=dict) - roofline_snapshot_id: int = 0 # Append-only compact roofline snapshots for report.py; capped at ``_ROOFLINE_SNAPSHOTS_CAP`` (snapshot #1 always retained as the report's baseline anchor). roofline_snapshots: list[dict[str, Any]] = field(default_factory=list) # Outer roofline failure counter; bumped on fail, reset on success. @@ -843,7 +840,6 @@ class SharedState(_RenderMixin, _ExploreStateMixin): last_gemm_tuning: dict[str, Any] = field(default_factory=dict) # merged explore action snapshot (same schema as other ``last_`` mirrors). last_explore: dict[str, Any] = field(default_factory=dict) - # Composite roofline action audit snapshot plus capped history. last_roofline: dict[str, Any] = field(default_factory=dict) baseline_attempts: list[dict[str, Any]] = field(default_factory=list) profile_attempts: list[dict[str, Any]] = field(default_factory=list) @@ -857,10 +853,7 @@ class SharedState(_RenderMixin, _ExploreStateMixin): last_action_failures: list[dict[str, Any]] = field(default_factory=list) # Structured per-variant failure evidence, capped and keyed by failure_id (last-wins). failures: list[dict[str, Any]] = field(default_factory=list) - # Per-kernel run_optimization history by kernel_id; record_kernel_opt retires kernels stuck in PARTIAL (default 2; override via INFERENCE_OPTIMIZER_KERNEL_OPT_MAX_PARTIAL). - kernel_opt_attempts: dict[str, Any] = field(default_factory=dict) - # Authoritative optimization history keyed by stable operator identity. - # ``kernel_opt_attempts`` remains the current ordinal compatibility index. + # Authoritative per-kernel optimization history keyed by stable task identity. kernel_opt_task_attempts: dict[str, Any] = field(default_factory=dict) # Immutable KEEP snapshots awaiting E2E integration, keyed by integration_id. # Their lifecycle is independent of trace-local kernel ordinals. @@ -876,11 +869,6 @@ class SharedState(_RenderMixin, _ExploreStateMixin): # dispatch / KEEP. rounds_since_last_specialist: dict[str, int] = field(default_factory=dict) rounds_since_last_keep: dict[str, int] = field(default_factory=dict) - # Legacy session_steward slots (steward removed); kept only for resume back-compat, never written. - steward_continuation_used: bool = False - steward_infra_failures_by_round: dict[str, int] = field( - default_factory=dict, - ) # last specialist task snapshot (parity with other ``last_`` mirrors). last_specialist: dict[str, Any] = field(default_factory=dict) # Per-specialist patch verdict ledger by task_id; Critic must approve/advise before PolicyGate allows the integrate_patch delegate. @@ -1015,8 +1003,6 @@ class SharedState(_RenderMixin, _ExploreStateMixin): # Recipe KB integration fields — Coordinator-only writers. # ``recipe_kb_session_id`` — hyperloom-local id carried into KB fact-write attrs; defaults to session_dir.name. recipe_kb_session_id: str = "" - # Kept (always ``{}``) for resume back-compat. - recipe_kb_session_summary: dict[str, Any] = field(default_factory=dict) # Snapshot of ``recipe_kb_t0._cascade_warm_start_search`` output (parsed dict); empty on first session for a (workload, hw) pair. warm_start_recipe: dict[str, Any] = field(default_factory=dict) # Snapshot of ``pitfalls`` output (negative priors), list of KB point dicts; consumed by the specialist prompt. Resume tolerates older snapshots. @@ -1138,19 +1124,12 @@ def _normalized_list(base_name: str, direct_name: str) -> list[str]: else {} ) serving_config = { - "engine": str(current_best.get("engine") or "").strip().lower(), "extra_server_args": str( current_best.get("extra_server_args") or "" ).strip(), "extra_envs": extra_envs, } - if any( - ( - serving_config["engine"], - serving_config["extra_server_args"], - extra_envs, - ) - ): + if any((serving_config["extra_server_args"], extra_envs)): context["serving_config"] = serving_config return context @@ -1383,36 +1362,10 @@ def from_dict(cls, raw: dict[str, Any]) -> "SharedState": ) if not isinstance(filtered.get("specialist_patch_verdicts"), dict): filtered["specialist_patch_verdicts"] = {} - if not isinstance(filtered.get("kernel_opt_attempts"), dict): - filtered["kernel_opt_attempts"] = {} if not isinstance(filtered.get("kernel_opt_task_attempts"), dict): filtered["kernel_opt_task_attempts"] = {} if not isinstance(filtered.get("pending_kernel_integrations"), dict): filtered["pending_kernel_integrations"] = {} - if incoming_version < 3: - for ledger_id, entry in filtered["kernel_opt_attempts"].items(): - if not isinstance(entry, dict): - continue - task_key = str(entry.get("task_group_key") or "").strip() - if not task_key: - source = str(entry.get("last_source_file") or "") - task_key = json.dumps( - { - "version": 1, - "kind": "legacy-kernel", - "kernel_id": str(ledger_id), - "source_file": source, - }, - sort_keys=True, - separators=(",", ":"), - ) - migrated = dict(entry) - migrated.setdefault("current_kernel_id", str(ledger_id)) - migrated.setdefault("stable_task_key", task_key) - filtered["kernel_opt_task_attempts"].setdefault( - task_key, - migrated, - ) # Normalize the unified ``explore_search`` ledger at load. filtered["explore_search"] = cls._build_explore_search( existing=filtered.get("explore_search"), @@ -1688,7 +1641,6 @@ def _langfuse_status_summary(self) -> dict[str, Any]: "stop_reason": self.stop_reason or "", "closing_phase": bool(self.closing_phase), "degraded_mode": bool(self.degraded_mode), - "cumulative_gain": round(float(self.cumulative_gain or 0.0), 2), "cumulative_gain_validated": round( float(self.cumulative_gain_validated or 0.0), 2, @@ -2566,6 +2518,54 @@ def pending_kernel_integration_records(self) -> list[dict[str, Any]]: return _m.pending_kernel_integration_records(self) + @property + def kernel_opt_attempts(self) -> dict[str, Any]: + """``kernel_opt_task_attempts`` re-indexed by the trace-local kernel id. + + Returns: + A fresh dict; mutating it does not touch the ledger, but the entry + values are the ledger's own dicts. + """ + from ..kernel import _kernel_decisions as _m + + return _m.index_attempts_by_kernel_id(self.kernel_opt_task_attempts) + + @kernel_opt_attempts.setter + def kernel_opt_attempts(self, value: dict[str, Any]) -> None: + """Seed ``kernel_opt_task_attempts`` from an ordinal-keyed dict. + + Args: + value: ``{kernel_id: attempt}``. Each attempt is stamped with its + ``current_kernel_id`` / ``stable_task_key`` and filed under the + stable key; an entry already filed under that key wins. + """ + from ..kernel._kernel_decisions import _stable_kernel_task_key + + if not isinstance(self.kernel_opt_task_attempts, dict): + object.__setattr__(self, "kernel_opt_task_attempts", {}) + for kernel_id, entry in value.items(): + if not isinstance(entry, dict): + continue + stamped = dict(entry) + stamped.setdefault("current_kernel_id", str(kernel_id)) + task_key = stamped.get("stable_task_key") or _stable_kernel_task_key( + task_group_key=str(stamped.get("task_group_key") or ""), + kernel_id=str(kernel_id), + source_file=str(stamped.get("last_source_file") or ""), + ) + stamped.setdefault("stable_task_key", task_key) + self.kernel_opt_task_attempts.setdefault(task_key, stamped) + + @property + def roofline_snapshot_id(self) -> int: + """Counter of the newest roofline snapshot, or 0 before the first one. + + Lives inside ``last_trace_analyze`` so clearing that cache resets the + counter with it — ``record_trace_analyze`` then restarts from 1. + """ + raw = (self.last_trace_analyze or {}).get("roofline_snapshot_id") + return int(raw) if isinstance(raw, int) else 0 + @property def has_keep_pending_integrate(self) -> bool: """True when kernel KEEP results still await kernel ``integrate``. @@ -3044,7 +3044,7 @@ def record_trace_analyze( payload: dict[str, Any], result: dict[str, Any], ) -> None: - """Write ``last_trace_analyze`` (single writer); ``roofline_snapshot_id`` is previous + 1, resetting when the cache was cleared. + """Write ``last_trace_analyze`` (single writer); ``roofline_snapshot_id`` increments from the previous nested value, restarting from 1 when the cache was cleared. Args: payload (dict[str, Any]): The trace_analyze task payload (supplies @@ -3180,8 +3180,6 @@ def record_trace_analyze( ), "ts": ts_iso, } - # Top-level mirror so PolicyGate/Coordinator skip the nested-dict lookup. - self.roofline_snapshot_id = snapshot_id self._append_roofline_snapshot_history( payload=payload, diff --git a/src/hyperloom/orchestrator/state/task_registry.py b/src/hyperloom/orchestrator/state/task_registry.py index 0a1ebe44b5..74e987f4a1 100644 --- a/src/hyperloom/orchestrator/state/task_registry.py +++ b/src/hyperloom/orchestrator/state/task_registry.py @@ -44,7 +44,7 @@ "cancelled": frozenset(), } -TERMINAL_STATES = frozenset({"succeeded", "cancelled"}) +TERMINAL_STATES = frozenset(state for state, outgoing in _TRANSITIONS.items() if not outgoing) # Progress notes a task's ``history`` retains, oldest dropped first. # ``record_progress`` re-reads and rewrites the whole blob inside a