wave-l: budgets that were spent by failure (v0.18.2)

v0.18.1 left one measured defect open: fix-a-bug hit its 20-minute ceiling in
one run of three with failed_tasks == 0. Chasing it found the cause, and the
cause turned out to be a pattern rather than a bug. Six mechanisms here bounded
themselves with a fixed count, charged that count for negative or failed
attempts, and so switched themselves off on precisely the runs that needed them.
Every one failed silently.

The objective probe went blind three waves into a run. maxObjectiveProbes was 2
and the budget was charged for RED answers, so between-waves probing spent it on
the earliest waves — the two moments when "not yet" is most certain and least
worth paying to learn. The budget is per RUN, not per board, so two probes was
two for an entire run, and the post-drain probe that corrective boards rely on
(they run with early-stop off) was starved by the same exhaustion. A count also
cannot be right for two projects at once. The bound is economic now: the
command's cost is measured and asking continues while runway >= 6 x cost.

A slow endpoint lost structured decoding for seven days. One deadline covers up
to six sequential capability probes; a cold local model could spend most of it
loading weights for the first, and attempt() returns false for a transport error
exactly as for a 400. An all-false record was then stamped Source="probe" and
honored for CapabilityTTL, with no path back because nothing re-probes a fresh
record. A cut-short negotiation now falls back to the family preset and leaves
Probed zero. Preferring the preset is the recoverable direction: an over-claimed
mechanism costs one 400 and demoteCapability records it; an under-claimed one
has nothing that can ever notice.

Retrieval went blind when the corpus was most on-topic. The "corpus median" was
measured over Search's already-truncated top-k, so with TopK=5 the threshold was
third-best + NoiseMargin: at most two hits could survive, and a tightly
clustered top five cleared nothing at all and returned "" with no error.

A canceled QA gate reported red. Every cancellation path returned "the gate
failed", which finalizeAfterExecute turns into a synthesized tester verdict
through applyTesterFeedback — a planner call. Ctrl-C bought an LLM round-trip on
the way out and annotated a done task about a gate never allowed to run.

The QA gate re-ran unchanged trees: qaDiagnoseAndFix never reported whether it
wrote anything, so a fix pass that produced nothing was followed by an identical
command run at full price. Measured on a stalled gate: 4 command runs down to 1.

A cold start pinned concurrency for a month. Profile.Current checked ID,
MaxParallel, version and age but not Partial, so the degenerate max_parallel=1 a
budget-exhausted probe produces was cached for DefaultTTL — for the slowest
models, the ones with most to gain from a real measurement.

The thrash detector blinded itself on thrashing runs. The signature map FROZE at
MaxSignatures, and the classic near-miss ws_edit retry mints a distinct signature
each attempt, so a thrashing run filled it faster and went blind sooner.
Measured: 5 identical calls after 256 distinct ones counted 0 repeats, not 4.

Measured, three runs per scenario, Qwen3.5-9B, same fixtures and ceiling:

                                        v0.18.1     v0.18.2
  implement-from-tests median tokens    123,652     75,535 (-39%)
  implement-from-tests pass rate        3 of 3      3 of 3
  fix-a-bug median tokens               ~178,000    ~182,000
  fix-a-bug pass rate                   2 of 3      2 of 3

One scenario improved substantially and the other did not move. The probe fix is
confirmed firing in a live run — the metrics record objective_met_early passed,
ending a run in 6 LLM calls and 375s — but it was not what fix-a-bug was failing
on. That scenario needs 8 tool calls and 69k tokens on a good run and 32 calls
and 192k on a bad one; in the run that hits the ceiling the harness still returns
a correct result, and only the suite's wall budget fails. Model variance, not a
harness that cannot tell it is done.

Wall-clock is absent from that table on purpose: the same suite showed fewer
tokens taking more seconds across the session, so throughput degraded under an
hour of continuous local GPU load. Token counts do not depend on machine speed.

Three further findings are documented in the changelog rather than shipped,
because their benefit cannot be proven without a controlled study: the 20ms
reviewerStrictDelay means two reviewer requests per review with the second
discarded almost always; readBudgetLines sizes ws_read from the legacy
prompt-byte budget rather than the model's real context window; and loop-guard
interventions are charged against the ReAct iteration budget.

Author: piotrlaczkowski

Makefile

@@ -1,6 +1,6 @@
 MODULE := github.com/UnicoLab/slmcode
 BIN    := slmcode
-VERSION ?= 0.18.1
+VERSION ?= 0.18.2
 PREFIX ?= $(HOME)/.local
 GIT_COMMIT := $(shell git rev-parse --short HEAD 2>/dev/null || echo unknown)
 BUILD_TIME := $(shell date -u +%Y-%m-%dT%H:%M:%SZ)

cmd/slmcode/version.go

@@ -4,7 +4,7 @@ package main
 //
 //	go build -ldflags "-X main.Version=0.5.0 -X main.SourceRoot=/path -X main.GitCommit=abc -X main.BuildTime=…"
 var (
-	Version    = "0.18.1"
+	Version    = "0.18.2"
 	SourceRoot = "" // absolute path to the slmcode checkout used to build this binary
 	GitCommit  = "unknown"
 	BuildTime  = "unknown"

docs/changelog.md

@@ -1,5 +1,177 @@
 # Changelog
 
+## v0.18.2 — 2026-08-25
+
+Budgets that were spent by failure.
+
+v0.18.1 left one measured defect unsolved: `fix-a-bug` hit its 20-minute ceiling
+in one run of three with `failed_tasks == 0` — nothing wrong except that nobody
+asked whether the work was done. Chasing it found the cause, and the cause
+turned out to be a **pattern** rather than a bug. Six mechanisms in this
+codebase bounded themselves with a fixed count, charged that count for negative
+or failed attempts, and so switched themselves off precisely on the runs that
+needed them. Every one of them failed silently.
+
+**Measured, three runs per scenario, Qwen3.5-9B on oMLX, same fixtures and the
+same 20-minute ceiling as the v0.18.1 measurement:**
+
+| | v0.18.1 | v0.18.2 |
+|---|---|---|
+| `implement-from-tests` median prompt tokens | 123,652 | **75,535 (−39%)** |
+| `implement-from-tests` pass rate | 3 of 3 | 3 of 3 |
+| `fix-a-bug` median prompt tokens | ~178,000 | ~182,000 |
+| `fix-a-bug` pass rate | 2 of 3 | 2 of 3 |
+| runs terminating within budget | 5 of 6 | 5 of 6 |
+
+**Read that table honestly: one scenario improved substantially and the other
+did not move.** The probe-starvation defect was real, is fixed, and is confirmed
+firing in a live run — the run metrics record
+`"gates":[{"name":"qa_gate","passed":true},{"name":"objective_met_early","passed":true}]`,
+which is the between-waves probe ending a run the moment the objective went
+green, in 6 LLM calls and 375s. But it was **not** what `fix-a-bug` was failing
+on, and that scenario's 1-in-3 ceiling miss is unchanged.
+
+What that remaining miss actually is, measured: the same 9B needs 8 tool calls
+and 69k tokens for this three-line boundary fix on a good run and **32 tool
+calls and 192k tokens** on a bad one. In the run that hits the ceiling the
+harness still returns a CORRECT result — `engine_success=true`, the fixture
+tests green, the protected test file byte-identical, `failed_tasks=0`. The only
+failing assertion is the suite's own wall-clock budget. So this is model
+variance, not a harness that cannot tell it is done.
+
+The identified next step, with evidence behind it and deliberately NOT taken
+here: the probe fires only between waves, so when a worker fixes the bug at tool
+call 10 of 32, nothing notices until the task ends. That worker runs the
+objective command itself through `ws_shell` during those calls, and the harness
+already sees the output — a green result there is free evidence the probe could
+harvest without spending anything. It is a real design and an unproven one, and
+this release does not ship unproven changes.
+
+Wall-clock is deliberately absent from the table above. Across these runs the
+same suite showed FEWER tokens taking MORE seconds (75k/792s here against
+124k/608s for v0.18.1), i.e. throughput degraded over an hour of continuous
+local GPU load. Wall times are not comparable between sessions on this hardware;
+token counts do not depend on machine speed, so they are what is reported.
+
+### Fixed
+
+- **The objective probe no longer goes blind three waves into a run.**
+  `maxObjectiveProbes` was 2, and the budget was charged for RED answers.
+  Between-waves probing spends it on the earliest waves — the two moments in a
+  run when "not yet" is most certain and least worth paying to learn — so from
+  wave three nothing ever asked again. If the implementation landed at wave
+  five, the run burned its whole ceiling with the work already complete. The
+  budget is also **per run, not per board**: a run drives `RunBoard` once per
+  corrective round, so two probes was two for the entire run, and the post-drain
+  probe that corrective boards deliberately rely on (they run with early-stop
+  off) was starved by the same exhaustion.
+  A count also cannot be right for two projects at once — two probes is miserly
+  for a 200ms unit suite and profligate for a 6-minute integration suite. The
+  bound is now economic: the probe's cost is **measured** (`SmokeResult.Duration`,
+  timed where the command actually runs) and asking continues while
+  `runway >= 6 × cost`. A cheap gate is asked on every wave that wrote
+  something; an expensive one only while there is runway for the answer to pay
+  for itself. Neither number is guessed per project.
+- **A slow endpoint no longer loses structured decoding for seven days.**
+  One deadline covers up to six sequential capability probes. A cold local model
+  — the exact case the budget was widened for — could spend most of it loading
+  weights for the first, after which every later probe died on the shared
+  deadline. `attempt` returns false for a transport error exactly as for a 400,
+  so the negotiation could not tell "the server refused this field" from "we
+  never got to ask", and stamped `Source: "probe"` on a wholesale-false record.
+  `capCache` then persisted and honored it for `CapabilityTTL` — seven days in
+  which every structured role on that endpoint silently degraded to prompt-only
+  + repair, with no path back, because nothing re-probes a record that is still
+  fresh. A cut-short negotiation now falls back to the family preset and leaves
+  `Probed` zero, which keeps it out of the on-disk cache. Preferring the preset
+  over all-false is the recoverable direction: an over-claimed mechanism costs
+  one 400 and is then recorded by `demoteCapability`; an under-claimed one has
+  nothing that can ever notice it.
+- **Retrieval no longer goes blind when the corpus is most on-topic.**
+  The relative noise floor is documented as "the corpus's own median", but it
+  was measured over the value `Search` returns — already truncated to `TopK`.
+  With the default `TopK=5` the "median" was the **third-best hit**, so the
+  threshold became third-best + `NoiseMargin`: arithmetically at most two hits
+  could ever survive, and a tightly clustered top five — a set of uniformly
+  strong matches — cleared nothing at all and `RetrieveForQuery` returned `""`
+  with no error and no warning. It also made `MinChunksForNoiseFloor`
+  unreachable for its stated purpose, since it compared against `len(top-k)` and
+  never the corpus size. `Retriever.SearchAll` now supplies the whole scored
+  distribution for the floor, and the top-k truncation happens after.
+- **A cancelled QA gate no longer reports a red verdict.** `runQAGate` returns
+  "the gate failed", and every cancellation path returned **true**.
+  `finalizeAfterExecute` reads that as `QAFailed`, sets `TesterRejected`, and
+  feeds the board a synthesized tester verdict
+  (`{"passed":false,...,"qa_gate red"}`) through `applyTesterFeedback` — a
+  planner call. So Ctrl-C, or a scenario budget expiring, bought an extra LLM
+  round-trip during shutdown and annotated a done task with "QA gate still
+  failing" about a gate never allowed to finish. A cancelled gate now records
+  nothing: no verdict, no annotation.
+- **The QA gate stops re-running an unchanged tree.** `qaDiagnoseAndFix`
+  discarded both role outputs and never reported whether anything was written,
+  so a fix pass that produced nothing (budget exhausted, a refusal, a prose-only
+  answer) was followed by a byte-identical command run against a byte-identical
+  tree, at full price, for every remaining round. The objective probe has
+  refused exactly this since it was written; the gate never learned it. Measured
+  on a stalled gate: 4 command runs down to 1. Gate rounds also now price the
+  command for the probe budget, since they run the same one.
+- **A cold start no longer pins concurrency for a month.** The calibration probe
+  runs inside a fixed wall-clock budget in which every unit of work is a model
+  call — the thing being measured. On a slow server the warm-up and solo
+  baseline can exhaust it before any concurrency level is measured, leaving only
+  the synthetic single-level entry, from which `SelectKnee` returns 1.
+  `Profile.Current` checked ID, `MaxParallel`, version and age but **not
+  `Partial`**, so that degenerate verdict was served from cache for
+  `DefaultTTL`; `Apply` only checks `MaxParallel > 0`, and the "partial" marker
+  appears solely in `Summary()`, which the auto path never prints. So the
+  slowest models — the ones with the most to gain from a real measurement — were
+  silently capped at `max_parallel=1` for thirty days. Partial profiles now
+  expire after `PartialTTL` (1 hour): a cold start is transient, so the retry
+  should be too.
+- **The thrash detector no longer blinds itself on thrashing runs.** The
+  signature map **froze** at `MaxSignatures` — once full, a signature it had not
+  already seen was never admitted, so its later repeats were never counted. The
+  asymmetry is what makes it bite: the classic small-model edit failure is a
+  near-miss `ws_edit` retried with a slightly different `old_str` each time, and
+  every one of those is a *distinct* signature, so a thrashing run fills the map
+  faster than a healthy one and goes blind sooner. Measured: 5 identical calls
+  after 256 distinct ones counted **0** repeats instead of 4.
+  `RunReport.RedundantCalls` and the redundant-call-rate KPI under-reported with
+  nothing marking the count as capped, and evolve learned from the truncated
+  signal. It now evicts oldest-first, like every other bound in that file.
+- **A reviewer that timed out is no longer recorded as one that judged the work
+  worthless.** The synthesized error attempt carries `NoVerdict`, and
+  `Attempt.Score` documents that a 0 under an `error` verdict is an absence, not
+  a judgement.
+- **`autoresearch` no longer claims a surface was exhausted when it was not.**
+  `StopExhausted` is reached on `ErrNoProposal` from the *deterministic*
+  proposer, which cannot touch a text knob at all — yet its sentence read "every
+  value of every knob was tried", the one message in that package built to be
+  trusted, while every sibling carefully says when the surface was *not*
+  exhausted.
+
+### Known, reported rather than changed
+
+Three further findings are real and measured but are **behaviour changes whose
+benefit cannot be proven without a controlled study**, so they are documented
+instead of shipped unmeasured:
+
+- `reviewerStrictDelay` (20ms) means the default `max_parallel=4` issues **two
+  reviewer LLM requests per review**, and `strictOut` is used only when the
+  primary reply is empty — so the second is discarded almost always. The code is
+  honest about the cost (`noteExtraRequests` reports it), but on a local server
+  that runs inference serially this roughly doubles review latency. A delay
+  derived from measured reviewer p50 would keep the insurance and drop the cost.
+- `readBudgetLines` sizes `ws_read` from `MaxContextKB`, the legacy prompt-byte
+  budget, rather than the model's real `ContextLimit` — the exact conflation
+  `compact.WindowTokensFromKB` is already marked Deprecated for. On typical
+  source that caps a read around 80–120 lines whatever the model's real window
+  is. Fixing it would raise read sizes several-fold on a large-context model,
+  which needs measuring before it ships.
+- `agents.factory` charges loop-guard interventions against the ReAct iteration
+  budget (both escalation paths return a successful tool result), and a model
+  profile's `max_turns` can only *lower* the default 8, never raise it.
+
 ## v0.18.1 — 2026-08-25
 
 The harness stops when the work is done.

pkg/autoresearch/ratchet.go

@@ -67,7 +67,21 @@ func (r StopReason) Sentence() string {
 	case StopTokens:
 		return "token budget spent — the surface was NOT exhausted, so more remains untried"
 	case StopExhausted:
-		return "surface exhausted — every value of every knob was tried"
+		// "every knob" was not true and this is the one message in the package
+		// built to be trusted — every sibling above goes out of its way to say
+		// when the surface was NOT exhausted.
+		//
+		// StopExhausted is reached on ErrNoProposal from the DETERMINISTIC
+		// proposer, which enumerates values and therefore cannot touch a text
+		// knob at all (see surface.go: a text domain has no enumeration and
+		// returns nil). A text knob such as system_prompt is reachable only
+		// through the LLM proposer, which is asked on a fixed cadence and gives
+		// up its slot on a model error, an empty reply, or an unchanged
+		// rewrite — so a small local model can burn every chance and leave the
+		// knob untried while this line claimed it had been swept.
+		return "enumerable surface exhausted — every value of every enumerable knob " +
+			"was tried; text knobs, if the surface has any, are reachable only " +
+			"through the LLM proposer and may remain untried"
 	case StopCanceled:
 		return "canceled — the surface was restored to its pre-run state"
 	case StopEvalFailed:

pkg/backends/capabilities.go

@@ -352,6 +352,10 @@ func runProbe(ctx context.Context, provider, endpoint, model, apiKey string) Cap
 		"stream":     false,
 	}
 	reachable := false
+	// cutShort records that the SHARED negotiation deadline expired part way
+	// through. Without it a timeout is indistinguishable from "the server
+	// rejected this field" — see the note above the return.
+	cutShort := false
 	attempt := func(extra map[string]any) bool {
 		body := make(map[string]any, len(base)+len(extra))
 		for k, v := range base {
@@ -362,6 +366,9 @@ func runProbe(ctx context.Context, provider, endpoint, model, apiKey string) Cap
 		}
 		status, err := probeOnce(pctx, client, url, apiKey, body)
 		if err != nil {
+			if pctx.Err() != nil {
+				cutShort = true
+			}
 			return false
 		}
 		reachable = true
@@ -415,6 +422,29 @@ func runProbe(ctx context.Context, provider, endpoint, model, apiKey string) Cap
 	if !reachable {
 		return Capabilities{Source: "unreachable"}
 	}
+	if cutShort {
+		// ONE deadline covers up to six sequential requests, and a cold local
+		// model — the exact case this budget was widened for — can spend most of
+		// it loading weights for the first. When it expires part way through,
+		// every field not yet asked about is false because it was NEVER ASKED,
+		// not because the server refused it. Stamping that as `Source: "probe"`
+		// used to publish an all-false record and, because Probed was set, cache
+		// it for CapabilityTTL — seven days in which every structured role on
+		// this endpoint silently degraded to prompt-only + repair, with no path
+		// back: nothing re-probes a record that is still fresh.
+		//
+		// So fall back to the family preset and leave Probed ZERO, which is what
+		// keeps this out of the on-disk cache (capCache.put skips zero-Probed
+		// records) so the next process negotiates again from scratch.
+		//
+		// Preferring the preset over all-false is deliberate, and the asymmetry
+		// is the argument: an over-claimed mechanism costs ONE 400 on first use
+		// and is then recorded by demoteCapability, while an under-claimed one
+		// costs a week of degraded decoding that nothing can detect. Guessing in
+		// the recoverable direction is the whole point.
+		prior.Source = "prior-probe-timeout"
+		return prior
+	}
 	out.Probed = time.Now()
 	out.Source = "probe"
 	return out

pkg/backends/probetimeout_test.go

@@ -0,0 +1,91 @@
+package backends
+
+import (
+	"context"
+	"net/http"
+	"net/http/httptest"
+	"sync/atomic"
+	"testing"
+	"time"
+)
+
+// TestSlowEndpointDoesNotPoisonCapabilitiesForAWeek is the regression guard for
+// a defect with the worst possible blast radius: a COLD LOCAL MODEL — the exact
+// case slmcode exists to serve — silently losing structured decoding for seven
+// days.
+//
+// THE DEFECT: one deadline covers up to six sequential probe requests. If
+// weight-loading ate most of it, the plain probe still succeeded (setting
+// `reachable`), and every later probe died on the shared deadline. `attempt`
+// returns false for a transport error exactly as it does for a 400, so the
+// negotiation could not tell "the server refused this field" from "we never got
+// to ask". It then stamped Source="probe" and Probed=now on a wholesale-false
+// record, which capCache persisted and honored for CapabilityTTL. Nothing
+// re-probes a record that is still fresh, so there was no path back: every
+// structured role on that endpoint degraded to prompt-only + repair for a week.
+func TestSlowEndpointDoesNotPoisonCapabilitiesForAWeek(t *testing.T) {
+	var calls int64
+	// Every request costs 100ms — a stand-in for a model still loading weights.
+	// The first one fits inside the budget and succeeds; the rest cannot.
+	srv := httptest.NewServer(http.HandlerFunc(func(w http.ResponseWriter, _ *http.Request) {
+		atomic.AddInt64(&calls, 1)
+		time.Sleep(100 * time.Millisecond)
+		w.Header().Set("Content-Type", "application/json")
+		_, _ = w.Write([]byte(`{"choices":[{"message":{"content":"ok"}}]}`))
+	}))
+	defer srv.Close()
+
+	restore := ProbeTimeout
+	ProbeTimeout = 150 * time.Millisecond
+	defer func() { ProbeTimeout = restore }()
+
+	// runProbe directly: Probe() memoises per key and would hide a second call.
+	got := runProbe(context.Background(), "omlx", srv.URL, "slow-model", "k")
+
+	if n := atomic.LoadInt64(&calls); n < 2 {
+		t.Fatalf("the server saw %d request(s); the test needs the first to "+
+			"succeed and a later one to hit the shared deadline", n)
+	}
+	// The load-bearing assertion. A zero Probed is what capCache.put checks
+	// before persisting, so this is precisely what keeps a half-finished
+	// negotiation out of the on-disk cache and out of the next process.
+	if !got.Probed.IsZero() {
+		t.Fatalf("a probe cut short by its own deadline was stamped as a "+
+			"completed one (Probed=%s, Source=%q) — capCache will now persist it "+
+			"and honor it for %s", got.Probed, got.Source, CapabilityTTL)
+	}
+	if got.Source == "probe" {
+		t.Fatalf("Source=%q claims a completed negotiation", got.Source)
+	}
+	// Structured decoding must survive. Falling back to the family preset is
+	// safe in a way that all-false is not: an over-claimed mechanism costs one
+	// 400 and is then recorded by demoteCapability, while an under-claimed one
+	// has nothing that can ever notice it.
+	if !got.JSONSchema || !got.JSONObject {
+		t.Fatalf("a slow endpoint lost structured decoding entirely: %+v — "+
+			"omlx's preset supports json_schema and json_object", got)
+	}
+}
+
+// TestCompletedProbeStillWins is the control: without it the test above could
+// pass by never trusting a probe at all.
+func TestCompletedProbeStillWins(t *testing.T) {
+	// json_schema is refused, everything else answered — a real negotiation
+	// with a real negative result in it.
+	f := newFakeServer(t, "json_object", "tools")
+	restore := ProbeTimeout
+	ProbeTimeout = 10 * time.Second
+	defer func() { ProbeTimeout = restore }()
+
+	got := runProbe(context.Background(), "omlx", f.URL, "fast-model", "k")
+
+	if got.Source != "probe" || got.Probed.IsZero() {
+		t.Fatalf("a negotiation that finished was not trusted: %+v", got)
+	}
+	if got.JSONSchema {
+		t.Fatal("json_schema was reported supported though the server answered 400")
+	}
+	if !got.JSONObject {
+		t.Fatal("json_object was refused though the server accepted it")
+	}
+}