Skip to content

Commit 41f0089

Browse files
authored
Merge pull request #6031 from loopx-project/codex/edgebench-official-feedback
feat(benchmark): return complete official feedback on best-only improvements
2 parents 0d1c9eb + 78051b7 commit 41f0089

15 files changed

Lines changed: 473 additions & 75 deletions

‎benchmark/edgebench/README.md‎

Lines changed: 49 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -171,14 +171,17 @@ but short probes are not a prerequisite for running the intended protocol.
171171
`--feedback best-only` is now the default for **new attempts across all five
172172
workers**. This is an explicit protocol change from native, not a demonstrated
173173
score improvement. Existing trials and pinned study manifests keep their modes.
174-
Use `--feedback native` to retain the previous default or `--feedback blind` as
174+
Selected improvements now include the complete official result API response,
175+
rather than the earlier notification-only payload. This changes the new-run
176+
best-only treatment; old pinned attempts must keep their original protocol.
177+
Use `--feedback native` to retain agent-requested evaluation or `--feedback blind` as
175178
an evaluator-feedback-free control. Harbor is unchanged.
176179

177180
| Mode | Agent-visible evaluator feedback | Evaluation access |
178181
| --- | --- | --- |
179182
| native | Exact score, pass rate, counts, summary, metrics and failed names; `--details` exposes per-check messages; `--list` shows active-submission history | Agent may submit within the native cooldown/budget; automatic samples are hidden |
180183
| blind | None; public task files, local tests and compiler feedback remain available | Host evaluates fixed automatic samples; agent has no judge route or credentials |
181-
| best-only | Latest strict improvement notification and the corresponding submitted-source checkpoint; no score, delta, diagnostics or negative-result status | Fixed capture cadence; one evaluator and latest pending capture per run; agent cannot request extra evaluations |
184+
| best-only | Latest strict improvement's complete official result API response and corresponding submitted-source checkpoint; baseline, ties, regressions and unsuccessful evaluations stay silent | Fixed capture cadence; one evaluator and latest pending capture per run; agent cannot request extra evaluations |
182185

183186
Best-only supports non-game, offline tasks with `score_first`,
184187
`valid_then_score` or `pass_rate_first` selection, including maximizing and minimizing scores. It
@@ -199,7 +202,17 @@ a lost response cannot create duplicate evaluations. An unexpected judge restart
199202
holds that attempt for reconciliation instead of silently replaying it.
200203

201204
The publisher accepts only that sampler's admitted submission/round identities
202-
and verifies the original source digest. Offline/history-only results cannot
205+
and verifies the original source digest. Before publication it fetches the
206+
selected submission's official result, verifies task/submission/score/pass-rate
207+
against native history, and checks the admission epoch and evaluator provenance.
208+
Provenance includes native Python source and task-specification digests, the judge
209+
image key, selection policy and score direction; the image key is not an immutable
210+
image-content attestation. Result-body digests bind hook delivery to that response.
211+
An unavailable or mismatched response preserves the incumbent for retry.
212+
The response is returned in full as provided by the native API, including any
213+
summary, metrics and diagnostics the task supplies. This does not expand access
214+
to raw judge output, hidden files, evaluator code or unrelated trials.
215+
Offline/history-only results cannot
203216
establish the baseline or change the online incumbent. The solver command's exit
204217
pauses capture/delivery; official outer resume reuses the same publisher and lane.
205218
After native worker cleanup, the host sends an epoch-bound release for the exact
@@ -249,17 +262,19 @@ archives retain their original permissions; evaluator and secret directories
249262
are not made public. The hook script and system hook configuration are also
250263
checked for ordinary-worker readability before handoff.
251264

252-
All five workers receive the allowlisted notification directly through managed
265+
All five workers receive the source-bound official result directly through managed
253266
Codex `PostToolUse`, `SessionStart` and `UserPromptSubmit` hooks. A short synchronous
254267
reader adds `additionalContext` before the next model request, preserving the
255268
original tool result. It does not force a wake, interrupt active reasoning or
256269
change Stop/continuation behavior. The official worker keeps its native Stop
257270
hook. Delivery is serialized and deduplicated per Codex session; a fresh session
258271
receives the latest checkpoint, while resume receives only a new one. The
259272
reader neither queries the evaluator nor interprets arbitrary packet prose.
273+
Official report text is evaluation data, not executable instructions or authority.
260274

261-
This transport is qualified with staged Codex 0.160.0, the actual system hook in
262-
a disposable container and a synthetic Responses endpoint: a new result arriving
275+
Qualification uses a real native judge and publisher, staged Codex 0.160.0, the
276+
actual system hook in a disposable container and a synthetic Responses endpoint:
277+
a complete official result arriving
263278
during a tool call enters the next model request; a subsequent result enters a
264279
resumed request. Earlier Codex versions must be qualified before admission.
265280
Inspect session input and subsequent checkpoint adoption separately: injection
@@ -270,12 +285,37 @@ A notification looks like this (digest abbreviated for illustration):
270285

271286
```json
272287
{
273-
"schema_version": "edgebench_best_feedback_v1",
288+
"schema_version": "edgebench_best_feedback_v2",
274289
"latest": {
275290
"kind": "new_best",
276291
"snapshot_id": "auto-7",
277292
"source_sha256": "<SHA-256>",
278293
"source_archive": "/opt/edgebench-feedback/auto-7-<SHA-256>.tar.gz",
294+
"run_id": "example-run",
295+
"task_id": "example-task",
296+
"online_epoch": "<judge-process-epoch>",
297+
"evaluator": {
298+
"native_source_sha256": "<SHA-256>",
299+
"task_spec_sha256": "<SHA-256>",
300+
"judge_image_key": "example.judge.task:version",
301+
"selection": "score_first",
302+
"score_direction": "maximize"
303+
},
304+
"result_sha256": "<SHA-256 of the canonical official_result JSON>",
305+
"official_result": {
306+
"submission_id": "example-submission",
307+
"status": "completed",
308+
"error": null,
309+
"report": {
310+
"task_id": "example-task",
311+
"submission_id": "example-submission",
312+
"valid": true,
313+
"score": 3,
314+
"pass_rate": 0.75,
315+
"summary": "Task-provided official summary",
316+
"metrics": {"coverage": 0.75}
317+
}
318+
},
279319
"message": "This evaluated snapshot strictly improved the task's native ranking among valid scored snapshots. It may differ from your current files; keep using local validation."
280320
}
281321
}
@@ -287,6 +327,8 @@ failures retry before committing an improvement. The host-only `best-only-host`
287327
artifacts record file publication and health; never mount or copy them into the
288328
worker. Worker-local session hook receipts under `/logs/agent/best-feedback-delivery`
289329
record emission, not model acknowledgement.
330+
Each hook reads the current packet while holding its session delivery lock, so a
331+
waiting reader cannot overwrite a newer delivery cursor and replay old feedback.
290332
Treat missing archive/delivery evidence as an unqualified treatment, not as a
291333
successful best-only trial. Public notifications do not include these errors.
292334

‎benchmark/edgebench/export_report.py‎

Lines changed: 16 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,15 @@
1818
from loopx.capabilities.benchmark_toolkit.experiment_board import (
1919
normalize_benchmark_experiment_board_row,
2020
)
21+
from .feedback_hook import FEEDBACK_PAYLOAD
22+
23+
24+
def _validate_feedback_payload(runner, profile):
25+
payload = runner.get("feedback_payload")
26+
if payload != profile.get("feedback_payload"):
27+
raise ValueError("Settings profile disagrees with feedback payload")
28+
if payload is not None and (payload != FEEDBACK_PAYLOAD or runner["feedback"] != "best-only"):
29+
raise ValueError("Unsupported feedback payload for this mode")
2130

2231

2332
def digest(path: Path) -> str:
@@ -89,6 +98,7 @@ def _validate_run_data(row, config, summary, points):
8998
if config[key] != row[row_key]:
9099
raise ValueError(f"Settings identity mismatch: {key}")
91100
runner, profile = config["runner"], config["worker_profile"]
101+
_validate_feedback_payload(runner, profile)
92102
if (
93103
runner["runner_commit"] != row["runner_revision"]
94104
or runner["model"] != row["model_id"]
@@ -204,6 +214,7 @@ def export(runs_root: Path, selection: list, output: Path, *, observed_at: str):
204214
raise ValueError(f"Profile disagrees with runtime: {key}")
205215
if row.get("runner_revision") != receipt["runner_commit"]:
206216
raise ValueError("Board and runner revision disagree")
217+
_validate_feedback_payload(receipt, profile)
207218
if number(final["best_score"]) != number(receipt["best_score"]):
208219
raise ValueError("Final and runtime scores disagree")
209220
score_entries = [
@@ -257,6 +268,11 @@ def export(runs_root: Path, selection: list, output: Path, *, observed_at: str):
257268
"feedback",
258269
)
259270
}
271+
# Old notification-only attempts retain absent/unknown payload metadata;
272+
# never relabel them using today's default or operator-supplied settings.
273+
if "feedback_payload" in receipt:
274+
config["runner"]["feedback_payload"] = receipt["feedback_payload"]
275+
config["worker_profile"]["feedback_payload"] = profile["feedback_payload"]
260276
row["status"], row["observed_at"] = "completed", observed_at
261277
row["metrics"]["best_score"] = {
262278
"value": final["best_score"],

‎benchmark/edgebench/feedback.py‎

Lines changed: 42 additions & 5 deletions
Original file line numberDiff line numberDiff line change
@@ -1,8 +1,8 @@
1-
"""Host-owned, positive-only projection of SForge's automatic evaluations.
1+
"""Host-owned, best-only delivery of SForge's official evaluation response.
22
33
Provider policy only: SForge still captures, evaluates and selects final scores.
4-
The worker receives an allowlisted notification and its own submitted source,
5-
never judge reports, credentials, scores or negative-result metadata.
4+
The worker receives the selected improvement's complete native result and its
5+
own submitted source, never credentials, other runs or non-improving reports.
66
"""
77
from __future__ import annotations
88

@@ -16,6 +16,7 @@
1616

1717
import requests
1818
from sforge.harness.selection import select_best
19+
from .feedback_hook import SCHEMA_VERSION
1920

2021
FEEDBACK_MODES = ("native", "blind", "best-only")
2122
FEEDBACK_ROOT = PurePosixPath("/opt/edgebench-feedback")
@@ -88,7 +89,7 @@ def start(self, backend, handle):
8889
return
8990
if self.token is None or self.stop_event.is_set():
9091
raise RuntimeError("Feedback registration is missing or already closed")
91-
self._publish_json(backend, handle, {"schema_version": "edgebench_best_feedback_v1", "latest": None})
92+
self._publish_json(backend, handle, {"schema_version": SCHEMA_VERSION, "latest": None})
9293
self._record()
9394
self.sampler.start(backend, handle, self.token)
9495
self.thread = threading.Thread(target=self._loop, args=(backend, handle), daemon=True)
@@ -161,6 +162,9 @@ def update(self, history, backend, handle):
161162
admission = self.sampler.admitted()[candidate["submission_id"]]
162163
if admission["source_sha256"] != digest:
163164
raise ValueError("Native archive differs from admitted online capture")
165+
result = self._official_result(candidate)
166+
result_sha256 = hashlib.sha256(json.dumps(result, sort_keys=True,
167+
ensure_ascii=False, allow_nan=False).encode()).hexdigest()
164168
remote = FEEDBACK_ROOT / f"{snapshot}-{digest}.tar.gz"
165169
prepare_feedback_root(backend, handle)
166170
backend.copy_to_container(handle, archive, remote)
@@ -174,9 +178,13 @@ def update(self, history, backend, handle):
174178
if readable.exit_code or readable.output.split()[:1] != [digest]:
175179
raise RuntimeError("Could not verify best-only source checkpoint for worker")
176180
packet = {
177-
"schema_version": "edgebench_best_feedback_v1",
181+
"schema_version": SCHEMA_VERSION,
178182
"latest": {"kind": "new_best", "snapshot_id": snapshot,
179183
"source_sha256": digest, "source_archive": str(remote),
184+
"run_id": self.run_id, "task_id": self.task_id,
185+
"online_epoch": self.sampler.epoch,
186+
"evaluator": self.sampler.evaluators[self.task_id],
187+
"official_result": result, "result_sha256": result_sha256,
180188
"message": _MESSAGE},
181189
}
182190
if self.stop_event.is_set():
@@ -187,6 +195,35 @@ def update(self, history, backend, handle):
187195
self.notifications += 1
188196
self._record(packet)
189197

198+
def _official_result(self, candidate):
199+
# Fetch only the sampler-admitted, selected native submission. Do not
200+
# read raw judge logs, verifier code or another run's result directory.
201+
response = self.session.get(f"{self.judge_url}/api/v1/result/{candidate['submission_id']}",
202+
timeout=(3, 5))
203+
response.raise_for_status()
204+
result = response.json()
205+
report = result.get("report")
206+
if (result.get("submission_id") != candidate["submission_id"]
207+
or result.get("status") != "completed" or result.get("error") is not None
208+
or not isinstance(report, dict) or report.get("valid", True) is not True
209+
or report.get("task_id") != self.task_id
210+
or report.get("submission_id") != candidate["submission_id"]
211+
or type(report.get("score")) not in (int, float)
212+
or not math.isfinite(report["score"]) or report["score"] != candidate["score"]
213+
or report.get("pass_rate") != candidate.get("pass_rate")):
214+
raise ValueError("Official result does not match the selected native history entry")
215+
# A restarted service can reuse in-memory submission IDs. Its epoch and
216+
# evaluator must still match the pre-solver admission, even on retry.
217+
response = self.session.get(f"{self.judge_url}/api/v1/best-only/admission",
218+
params={"admin_secret": self.admin_secret}, timeout=(3, 5))
219+
response.raise_for_status()
220+
admission = response.json()
221+
evaluator = admission.get("evaluators", {}).get(self.task_id)
222+
if (admission.get("epoch") != self.sampler.epoch or not evaluator
223+
or evaluator != self.sampler.evaluators.get(self.task_id)):
224+
raise ValueError("Official evaluator changed; reconcile the original trial")
225+
return result
226+
190227
def _record(self, packet=None):
191228
# Host-only health/evidence. Do not mount this directory into the worker.
192229
value = {"best_score": self.score, "notifications": self.notifications, "errors": self.errors}

0 commit comments

Comments
 (0)