Fix three orchestrator bugs found during extended framework testing

[AMD-melliott]

Context

Found while stress-testing Hyperloom's orchestrator loop against --framework atom, pushing past the vLLM/SGLang-centric path Hyperloom appears to have been primarily validated against. A long (9h+) unattended run produced zero validated gain despite 95.8% roofline headroom; the orchestrator's own self-diagnosis in reports/final.md root-caused all three independently of framework choice, so they should reproduce under vLLM/SGLang too.

Fixes

1. PolicyGate false-rejects specialist delegates on a placeholder path string. A literal "Not found" in a source_file field isn't empty, so it reached the trusted-scope check and was denied, identically, 15 times in one session, zeroing out the entire framework-agent lane. Now treats known absent-value sentinels as an omitted field.
2. EXPLORE can exit with zero variants tested. A stale pending_escalate_hint left over from an unrelated phase or a prior macro-cycle was honored unconditionally, short-circuiting compute_plateau_explore() before a single round ran (explore_search.tested=0). The direct cause of cumulative_gain_validated=0.00% in that run. Now cleared on unrelated phase transitions and macro-cycle reload.
3. No recovery path when the local server becomes unreachable. local_server_unreachable's own suggestion text pointed at server_lifecycle, which isn't a real dispatchable action. Wired to the existing recover action, mirroring the gpu_memory_leaked branch.

This PR also threads PolicyGate rejection's specific rule through SubAgentResult as error_class instead of the generic unknown_error, so future policy denials are recognizable in the gap ledger.

Testing

test_policy_gate.py, test_machine_state.py, test_decision_action_ladder.py, test_delegate_denial_loop.py: 105 passed.
Also validated live: a same-day re-run against the same workload showed zero
PolicyDenied hits and EXPLORE completing 3 real variant attempts (previously 0).

[lishuoshuo-amd]

Thanks for chasing these down. The root-causing in the description holds up, and the local_server_unreachable one is genuinely hard to find without a long unattended run — a symptom whose own suggestion text names an action that can't be dispatched is the kind of thing that only surfaces when something actually needs it. A few things I ran into going through it.

1. The new unrelated-transition cleanup drops a legitimate skip_to_kernel. Orchestration can emit escalate_strategy_change{next_action_hint='skip_to_kernel'} from any phase, and intent_router defers it on purpose — "skip_to_kernel / skip_to_close are deferred; next compute_next_phase picks them up". The only place that actually consumes it is exit_normal_explore (machine_state.py:1731), so a hint emitted while in FRAMEWORK_AGENT has to survive that phase to be useful. It doesn't anymore: FRAMEWORK_AGENT exits on its own reason, that exit carries no hint in evidence, and the new elif at machine.py:163 claims it. Driving the machine directly with phase=FRAMEWORK_AGENT, pending_escalate_hint='skip_to_kernel', framework_agent_phase_done=True gives ('EXPLORE', 'framework_agent_phase_done', {'evidence': 'no_more_candidates', 'batch_count': 0}) — no hint key, so the hint is gone and the KERNEL_AGENT advance it was carrying never happens. Nothing logs that it was dropped.

2. The EXPLORE fix doesn't close the failure it's named for. exit_normal_explore still tests skip_to_kernel at priority 1, right after the hard force-exit and before compute_plateau_explore() is reached (machine_state.py:1731). That ordering is untouched here — what changed is only which hints can be sitting there when it runs. A skip_to_kernel emitted during EXPLORE still exits on the next compute with explore_search.tested=0, which is the cumulative_gain_validated=0.00% path from the description. The stale-hint vector is closed; the zero-round exit isn't.

3. The server_lifecycle string this PR calls non-dispatchable is still going to the model. The workaround at action_ladder.py:314 is fine, and the comment above it is right that PolicyGate would reject server_lifecycle as unknown_action — but the symptom still carries suggestion="delegate(server_lifecycle) to restart the inference server" at local_health.py:160. That field isn't inert: per the action_ladder module docstring, "Strategic suggestions ride the alert detail.suggestion field", and _detail() packs sym.suggestion into every alert, which lands in the orchestration prompt. So the ladder now does the right thing while the prompt keeps telling the model to reach for an action that doesn't exist. Same string again at local_health.py:206 on log_error_pattern.

4. When every probe target is down, the branch fires once per target. _server_unreachable emits one symptom per unreachable URL and only marks them HIGH when all of them fail — which is precisely the case that reaches the new branch. The idempotency key at action_ladder.py:324 only carries the tick, so two dead targets on one tick give two delegates keyed recover-server-unreachable-tick-42. The first creates the task and the rest come back as duplicate-idempotency denials, which land in the denial history and feed repeated_policy_denied — so the recovery works, but it books itself as a policy problem on the way.

5. error_class is threaded through one of the three failure exits. run_task has three return SubAgentResult(state="failed", result={}, ...) sites; only the PolicyDenied one sets it (sub_agent_runner.py:243). no_executor and the executor-exception path still return an empty result with no class, so they keep collapsing into unknown_error at the gap key (explore.py:1234) — the same generic bucket this was meant to get out of.

6. The sentinel pass-through is silent. gate.py:2095 returns on a match with no log. What used to be a loud PolicyDenied naming the offending value is now a quiet accept: the delegate runs on with an absent source_file and fails somewhere downstream, with nothing tying the two events together and nothing pointing back at whichever resolver emitted the placeholder in the first place.

7. The discarded hint is written to an audit field that means the opposite. machine.py:163 drops the hint through consume_pending_escalate_hint(), which records it in last_consumed_escalate_hint — documented at shared_state.py:1649 as recording consumption. After this change a hint that drove a transition and a hint that was thrown away look identical in the breakdown, with no log line to separate them.

8. policy_{rule} enters a field that has no vocabulary. error_class is matched exactly in several places (crash, oom, hang, detokenizer_stall), and no policy_* value will hit any of them. Nothing breaks, but it's a new prefix family in a field that already has no central definition.

9. Four comments now contradict the code. The action_ladder module docstring still lists delegate(recover) as (gpu_memory_leaked) only (action_ladder.py:12). shared_state.py:766 still says the hint is cleared "once acted on", which is no longer the only way it's cleared. consume_pending_escalate_hint() still calls itself "recording consumption" (shared_state.py:1649) while also serving discards. And reset_per_cycle_plateau_state() is described as resetting "transient plateau and dispatch state" (explore_state.py:400), which pending_escalate_hint (explore_state.py:417) isn't.

10. Two of the defensive reads can't fire. getattr(state, "pending_escalate_hint", "") (machine.py:163) defaults a dataclass field that always exists, one line under a direct state.consume_pending_escalate_hint(). getattr(denied, "rule", "") (sub_agent_runner.py:237) defaults an attribute PolicyDenied.__init__ always assigns (gate.py:119) — only the or "denied" does any work, for the None case.

11. None of the three behaviours has a test. The 105 in the description pass on this head, but they're all pre-existing; nothing exercises the sentinel pass-through, the hint cleanup, or the local_server_unreachable mapping.

On the rebase. dispatcher.py and gate.py both conflict with current main. Two things to watch in gate.py: main now routes source_file through _source_file_candidates() (0782edff8) to accept the trace-frame form path.py(124): fn the model cites verbatim out of roofline evidence, so that expansion needs to survive the merge. And fix 1 overlaps something already landed — tracelens_analysis.reject_non_path_source() (e4852a96c) zeroes the same placeholders at the producer, keyed on whether the value carries a source extension rather than on a list of literals, with common/kernel_source_contract.py making it a contract that source_file is empty when resolution failed. That guard's test pins the values TraceLens actually writes into the field: Not found, N/A, none, unknown, TBD, <unresolved>, AITER (vendor), Triton (vendor). _SOURCE_FILE_ABSENT_SENTINELS (gate.py:436) covers the first four, so a delegate carrying TBD or AITER (vendor) is still denied.

[AMD-melliott]

@lishuoshuo-amd Quick status update: pushed a commit (29a9468) addressing the two core issues from the review:

• The skip_to_kernel hint-drop regression: the cleanup in machine.py now only discards a pending hint when the transition target isn't EXPLORE, so a hint set during FRAMEWORK_AGENT survives through to EXPLORE, where exit_normal_explore actually consumes it. Added tests for both the survive and discard paths.
• The zero-round EXPLORE exit (cumulative_gain_validated=0.00%): exit_normal_explore now requires at least one specialist round to have run this macro-cycle before honoring skip_to_kernel. Added a test reproducing the original bug plus one confirming the fix fires once a round has run.

Also picked up a few smaller items from the review while I was in there: extended the source-file sentinel list for TBD/AITER (vendor)/Triton (vendor), threaded error_class from policy-gate rejections through to the gap ledger instead of it defaulting to unknown_error, and gave local_server_unreachable a real delegate(recover, force_gpu_cleanup=True) path instead of recommending the non-dispatchable server_lifecycle action.

I haven't yet touched local_health.py. If that's still constructing the server_lifecycle string independent of the ladder recommendation, let me know and I'll follow up there too. Remaining review items (design/investigation notes) are still in progress. I've lost access to my test server for a bit (unrelated logistics on my end) but will keep pushing commits as I work through the rest. Thanks for the thorough review.

[AMD-melliott]

@lishuoshuo-amd Pushed four more commits addressing the remaining review items. Rebased onto current main in the process (conflicts in gate.py/dispatcher.py, both resolved — _source_file_candidates() from 0782edff8 is untouched, and I kept _SOURCE_FILE_ABSENT_SENTINELS rather than dropping it: reject_non_path_source() only protects source_file values that flow through _finalize_candidates(), so the gate-level check is still the last line of defense against an LLM-authored placeholder or a resumed session replaying pre-fix kernel_candidates.json).

3. stale server_lifecycle suggestion strings: Fixed at the source in local_health.py: both local_server_unreachable and log_error_pattern's HIGH suggestions now say delegate(recover, force_gpu_cleanup=True), matching what action_ladder.py actually dispatches.

4. idempotency key collision across probe targets: action_ladder.py's local_server_unreachable key now includes a short hash of the target URL (sym.subject["url"]), so multiple dead targets in one tick no longer collide on recover-server-unreachable-tick-N and book themselves as repeated_policy_denied.

5. error_class only threaded through one of three failure exits: run_task has three SubAgentResult(state="failed", ...) sites; only PolicyDenied set error_class. no_executor and the executor-exception exit both still collapsed into unknown_error at the gap key. no_executor now sets error_class="no_executor"; the exception exit sets it to the raised exception's own class name (same convention already used elsewhere in this codebase for exception-derived classes).

6. silent sentinel pass-through: gate.py now logs (INFO) when a sentinel match admits a delegate, naming the field and the sentinel value.

7. discard vs. consume: Added discard_pending_escalate_hint() alongside the existing consume_pending_escalate_hint(), writing to new last_discarded_escalate_hint(_ts) fields instead of last_consumed_escalate_hint (which specifically means "drove a transition"). machine.py's unrelated-transition branch and reset_per_cycle_plateau_state()'s cross-cycle clear both route through the new method now. Registered the new fields in both LLM-write-blocked allowlists (gate.py, robustness envelope.py).

8. policy_{rule} vocabulary: Checked both consumers rather than adding new handling: writeback._pitfall_severity_for already excludes policy_* from crash-severity correctly (a policy denial isn't a runtime crash), and explore._extract_gaps_from_attempts groups it into its own gap key from the raw string with no special-casing needed. Documented the field's known producers/consumers on SubAgentResult.error_class instead, since there's no central vocabulary registry to update.

9. stale comments: All four fixed: action_ladder.py:12, shared_state.py's pending_escalate_hint field comment, consume_pending_escalate_hint()'s docstring, reset_per_cycle_plateau_state()'s docstring.

10. no-op getattr()s: Both simplified to direct attribute access (machine.py, sub_agent_runner.py); the or "denied" / and target !=PHASE_EXPLORE guards that did real work are unchanged.

11. Added direct tests for all four previously-uncovered behaviors: sentinel pass-through (including the new log line), the discard/consume distinction (by audit field, not just by log line), local_server_unreachable → recover (including the per-target idempotency key), and both error_class-bearing failure exits from feat: Add OOB Support #5.

Full suite: 13,786 passed / 21 failed / 36 skipped — the 21 failures are pre-existing and unrelated (credential/auth-probe tests failing on missing local fixtures), confirmed identical before and after this branch's changes; unchanged by anything here.

[AMD-melliott]

Update: rebased onto current main to clear a real conflict in local_health.py/test_signals_local_health.py — main independently added idle-server detection (a benchmark-client/process check to distinguish "no server up by design" from a real outage) since this PR opened. Auto-merged cleanly in local_health.py itself; the test file needed one added local_processes fixture so the new idle-detection logic doesn't suppress the symptom this PR's test checks for. No functional changes to this PR's own fixes from the rebase.

Also ran a fresh ~24h validation against a different model/framework than the original bug reports (Qwen3.8-27B on vLLM, vs. DeepSeek-V4-Flash on ATOM), specifically to stress-test these three fixes end to end under real, extended load.

All three held up:

• Escalate-hint survival/discard: multiple phase transitions fired cleanly across the run (framework_agent_budget_cap, explore_force_exit_low_budget, kernel_no_more_leverage, conc_sweep_done) with no dropped or misattributed hints.
• Sentinel pass-through: no bogus source_file placeholders reached PolicyGate; no regressions in that path.
• local_server_unreachable → recover: fired correctly in an earlier session on this same branch, with the fixed suggestion string and a properly disambiguated per-target idempotency key.

More importantly, the underlying zero-round EXPLORE bug this PR targets is confirmed closed under real load: this run reached explore_search.tested=10 (every previous run/resume against this branch, and the original bug report, stayed at a hard 0) and completed all six phases naturally (stop_reason=conc_sweep_done, crash_count=0), landing a genuine +9.37% validated throughput gain.

Caveat on that last result, for accuracy: reaching that full clean run also needed two fixes outside this PR's scope — a critic completion-token-budget fix (independently found and fixed better in #1214) and a run_action_now inline-action-whitelist gap (separate PR incoming). Without those, FRAMEWORK_AGENT can still starve for hours before ever reaching EXPLORE, regardless of this PR's fixes. This PR's three fixes are necessary — confirmed working as designed above — but not sufficient on their own for the full happy path.

Full suite: 13,807 passed / 7 failed / 38 skipped — the 7 failures are pre-existing and unrelated (confirmed identical on a clean upstream/main checkout, and all 7 pass individually in isolation).