Manage specialist token spend: bound the proposal set, stop discarding it, and name a stop condition - #1233
Open
ZhengGong-amd wants to merge 11 commits into
Open
Manage specialist token spend: bound the proposal set, stop discarding it, and name a stop condition#1233ZhengGong-amd wants to merge 11 commits into
ZhengGong-amd wants to merge 11 commits into
Conversation
… to 4 The ceiling was the only number the prompt gave, and it became the target: after the Section 1 reminder landed, no run exceeded 6 but a quarter of them returned exactly 6. "Fewer is better than padding" did not carry, because it named no alternative. State a target in both places the specialist reads before it starts working, and give it a test for going past the target rather than a preference. Every extra entry costs a Critic review and a slot on the serial benchmark queue, so say that too. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
A specialist's config proposals reached the grid author exactly once, inside a delegated_result the inbox renders as a single status line, and the inbox tail is lossy. Only the research scout's proposals were ever rendered in full, and they are the only ones that got benched at any rate: 45% against 4% for everything else, while converting at a lower KEEP rate once benched. The selection was following the rendering channel, not the evidence. Derive the queue instead of storing it. specialist_rounds already persists the proposals and explore_search already records what was benched, so the block is a projection of two ledgers that were both already there. Identity is the subtle half. The executor keys explore_search on the variant's args and envs folded together with the union of its own removal controls and the stack's, so a proposal carrying remove_args hashes differently once the stack contributes its own. Hashing the proposal alone would silently never match the ledger, and one in six ledger rows carries such a control. Both sides now compute it through one helper, with the executor's pre-existing values pinned in a test so the shared path cannot re-key a resumed session. The line carries remove_args, unset_envs, args_mode and atomic rather than just args and envs: a removal-only proposal would otherwise render as two empty fields and read as a no-op, and two in five proposals are atomic, which the grid author is required to dispatch without re-deriving. Rendered on every turn rather than only on a seed, since a queue seen once is the amnesia this replaces, but only in EXPLORE, the sole phase where explore is proposable. Scoped to the current macro-cycle. The research-scout block gives up its proposal half to avoid rendering them twice. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
… grid Both decision lists told the author that "the proposal_set drives the next explore grid" when an explore round had just finished. Neither half was true: the proposal_set was never rendered, and finishing a round has nothing to do with what a specialist produced. Name the block that now carries them. Give the grid a size while we are here. It had one until the per-round breadth caps were removed in favour of a resource-derived bound, but explore reserves server_lifecycle and benchmark_lane, never research_lane, so the bound that was supposed to take over does not reach it. What is left is the session deadline skipping whatever the round cannot finish, from the end of the grid rather than the bottom of the ranking. Rounds of twenty variants followed. State the target and the ceiling on both surfaces the author reads, with the serial lane cost that makes them what they are. The removed gate's own hint told the author to defer runners-up to a later round, which only means something now that a later round can still see them. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…way to stop The ceiling had already become the target once: after it was restated in Section 1, no run exceeded it and a quarter landed exactly on it. A target states the same invitation more directly, so say which direction it binds -- one real proposal is a better round than two padded, and an empty one is better than a single padded entry. Nothing told the specialist when to stop looking. Section 1 spent its budget encouraging depth, Section 2a said turns are not the stop signal, and the only remaining bound was the wall clock, which is how a round with nothing to show still spends its full hour. Name the other stop: rounds that stop yielding findings are done. The cold-start branch fires exactly when there are no priors, which is when having nothing to propose is most likely, and it opened by forbidding an empty set before walking it back a paragraph later. Lead with the rule the two paragraphs agreed on and drop the contradiction. Its fallback bullets live in Section 1 whether or not the dispatch has a domain focus block, so point there rather than at a block a free-form specialist never gets. Also corrects the numbers this commit's predecessor claimed but never applied. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
![github-advanced-security[bot]](https://avatars.githubusercontent.com/in/57789?s=60&v=4){kind=small}
Committed alongside a one-hunk baseline fix, and the file says what it is in its own first line: a scratch script that prints ok/PREDICTED while sweeping the phase-pricing helpers for disagreements. Nothing imports it, no workflow or config names it, and it is the only module at the root of a src-layout tree. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
Eleven unused imports, a missing trailing newline, and one E402, all present on main and none of them from this branch's changes. A gate that is already red cannot report on anything. Two needed more than the autofix. forge_gemm_tuning imported truthy behind a sys.path insert/pop pair whose only purpose was that import, so the whole block goes rather than leaving the path dance wrapped around nothing; sys and Path keep other callers. build_actions imported re beside the regexes it builds rather than at the top, which is not the cycle-safe late import the per-file E402 list is for, so it moves up instead of joining that list. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
CI E2E report — ❌ Failed
|
The parser space-joins a list-form extra_args because the LLM sometimes emits the flags as JSON and str(list) yields a repr the server rejects. The queue built its own identity with a bare str(), so the same proposal hashed two ways: it never matched the ledger it was meant to be filtered against, and the line it rendered carried the repr back to whoever copied it into a grid. Move the coercion next to the identity it feeds and let the parser build its variant from the same projection, which also drops the duplicate alias and list handling it kept alongside it. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…ger key The ledger key folds in whatever removal controls the stack carried when the round opened, and a KEEP mid-round rewrites those. Every candidate benched before that KEEP then hashes differently from what the queue recomputes off the finished current_best, so the queue offers them again as untried. Around one session in eight ends with a stack carrying such a control. The queue asks a narrower question than the executor's in-round dedup does -- whether these flags have been tried, not whether this variant collides with another under one base -- and the ledger rows already store the variant's own fields. Match on those and the stack drops out of the comparison entirely, along with the base plumbing the renderer needed to reproduce it. Two smaller ones on the same path: a proposal with no name was offered as executable although the grid parser skips unnamed entries, so it now gets the same derived name the multi-node materialiser gives it; and a non-numeric cycle raised out of prompt assembly, which the shared int coercion settles. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
…hey were The two decision lists called the queue "every executable proposal", which reads as exhaustive next to a block that truncates and prints the count of what it withheld. And two comments still describe the cold-start branch as the thing that stops a specialist returning an empty set, which is the rule that branch now states the other way round. Co-Authored-By: Claude Opus 5 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Specialist sub-agents are the largest line in a session's LLM bill. Three mechanisms were spending that budget without a matching return. Prompt and state layer only — no execution semantics, no new gate.
Numbers from 38 archived sessions (801 proposals, 1,487 benchmarked variants) plus the 101-session cost audit.
Where the spend goes
Most output is never consumed. 105 of 801 config proposals ever reached a grid — 13.1%. Not dropped by judgement: a
proposal_setlives only inside adelegated_result, which the inbox renders as one status line, and that tail is lossy by design. The one domain rendered in full (research_scout, via its own SEED block) converted at 45.2%; every other domain sat near 4%. Quality does not explain it — among proposals that were benchmarked, scout's KEEP rate is 7.0% against serving's 22.2%.The ceiling was read as a target.
proposal_sethad a maximum and no target. After the maximum was restated in Section 1: 0 of 188 runs exceeded it, 24.5% landed exactly on it (up from 13.5%). Each surplus entry costs generation tokens, a Critic review, and a slot on a serial lane.Nothing said when to stop. 170 runs (12.6%) returned
empty: trueat a median of 46 turns vs 52 for productive ones — same effort, no output, $2,089 (20.6% of specialist spend). The wall clock was the only bound, and the cold-start branch — which fires exactly when there are no priors — opened by forbidding an empty result, then walked it back a paragraph later.What changes
Proposal set → 2 as the norm, 4 as the hard cap, on both surfaces the specialist reads, with the direction stated: one real proposal beats two padded,
empty=truebeats one padded. An output-based stop is named alongside the depth encouragement that was previously unopposed. The cold-start contradiction collapses into the rule both its paragraphs already agreed on — a coin-flip proposal is worse than none.A ranked queue of untested proposals. New
=== Untested proposals (current cycle) ===block: every executable proposal this cycle that no explore round has benchmarked, ranked by gap severity, capped at 12. Derived fromspecialist_rounds+explore_search— no new state, no migration.Identity is the subtle part. The executor keys
explore_search["tested"]on the variant folded together with the stack's removal controls, so hashing a proposal alone never matches the ledger — and 16.9% of ledger rows carry such a control. Both sides now go through one helper, with the executor's existing fingerprints pinned in a golden test so a resumed session cannot be re-keyed. The rendered line carriesremove_args/unset_envs/args_mode/atomic: 38.6% of proposals are atomic and must be dispatched verbatim, and a removal-only proposal would otherwise render as two empty fields. Rendered every turn but EXPLORE-only; the scout block gives up its proposal half so nothing renders twice.Grid bounded to target 4, max 6. It had a cap until #486 removed it for a resource-derived bound, but
explorereservesbenchmark_lane, neverresearch_lane, so that bound never reached it. What remained was the session deadline truncating from the end of the grid rather than the bottom of the ranking; 20-variant rounds followed. Prompt-only — reversing #486 with a gate needs its own justification.What it does not do
integrate_patch(already closed-loop: 50 patch-bearing specialists → 97 tasks, 1 undecided).Side-fixes
Neither is fixed on
mainas ofdf82cb27a; both verified against a pristine checkout.selfcheck_gates.pyremoved — arrived via fix: hold the session to its wall-clock budget #1171 beside a one-hunkbaseline.pyfix, calls itself a throwaway in its first line, zero references, only module at the root of a src-layout tree.ruff check .reported 13 errors onmainand therefore here, none from this branch. Eleven autofixed. Two were not:forge_gemm_tuningimportedtruthybehind asys.pathinsert/pop pair that existed only for it (whole block goes;sys/Pathkeep other callers), andbuild_actionsimportedrebeside its regexes rather than at the top — not the cycle-safe late import the per-fileE402list is for, so it moves up instead of joining it.Notes
9364b45dehas a title describing numbers it did not apply; the laterfix(specialist)commit applies them and says so. Not force-rewritten, since it was already pushed.cyclefallback in the renderer —specialist_roundshas no backfill andsetdefault("cycle")only landed in [FEAT] Slim the orchestration system prompt: strip version tags, de-duplicate rules, wire macro_cycle #972, so a pre-[FEAT] Slim the orchestration system prompt: strip version tags, de-duplicate rules, wire macro_cycle #972 resumed session genuinely lacks it.BLOCKED_EXTERNAL_ENV_NAMES. Plausible after [Stepping] Correctness sweep across the optimization loop: measurement integrity, phase budgets, resume safety, and a large dead-code removal #1188, but 0 of 801 proposals name a refused env — it would be an unreachable branch.40 new tests; each commit independently green under bisect.
ruff check .clean.