You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit 41f0089
Browse filesBrowse the repository at this point in the historyBrowse files
| native | Exact score, pass rate, counts, summary, metrics and failed names; `--details` exposes per-check messages; `--list` shows active-submission history | Agent may submit within the native cooldown/budget; automatic samples are hidden |
180
183
| blind | None; public task files, local tests and compiler feedback remain available | Host evaluates fixed automatic samples; agent has no judge route or credentials |
181
-
| best-only | Latest strict improvement notification and the corresponding submitted-source checkpoint; no score, delta, diagnostics or negative-result status| Fixed capture cadence; one evaluator and latest pending capture per run; agent cannot request extra evaluations |
184
+
| best-only | Latest strict improvement's complete official result API response and corresponding submitted-source checkpoint; baseline, ties, regressions and unsuccessful evaluations stay silent| Fixed capture cadence; one evaluator and latest pending capture per run; agent cannot request extra evaluations |
182
185
183
186
Best-only supports non-game, offline tasks with `score_first`,
184
187
`valid_then_score` or `pass_rate_first` selection, including maximizing and minimizing scores. It
@@ -199,7 +202,17 @@ a lost response cannot create duplicate evaluations. An unexpected judge restart
199
202
holds that attempt for reconciliation instead of silently replaying it.
200
203
201
204
The publisher accepts only that sampler's admitted submission/round identities
202
-
and verifies the original source digest. Offline/history-only results cannot
205
+
and verifies the original source digest. Before publication it fetches the
206
+
selected submission's official result, verifies task/submission/score/pass-rate
207
+
against native history, and checks the admission epoch and evaluator provenance.
208
+
Provenance includes native Python source and task-specification digests, the judge
209
+
image key, selection policy and score direction; the image key is not an immutable
210
+
image-content attestation. Result-body digests bind hook delivery to that response.
211
+
An unavailable or mismatched response preserves the incumbent for retry.
212
+
The response is returned in full as provided by the native API, including any
213
+
summary, metrics and diagnostics the task supplies. This does not expand access
214
+
to raw judge output, hidden files, evaluator code or unrelated trials.
215
+
Offline/history-only results cannot
203
216
establish the baseline or change the online incumbent. The solver command's exit
204
217
pauses capture/delivery; official outer resume reuses the same publisher and lane.
205
218
After native worker cleanup, the host sends an epoch-bound release for the exact
@@ -249,17 +262,19 @@ archives retain their original permissions; evaluator and secret directories
249
262
are not made public. The hook script and system hook configuration are also
250
263
checked for ordinary-worker readability before handoff.
251
264
252
-
All five workers receive the allowlisted notification directly through managed
265
+
All five workers receive the source-bound official result directly through managed
253
266
Codex `PostToolUse`, `SessionStart` and `UserPromptSubmit` hooks. A short synchronous
254
267
reader adds `additionalContext` before the next model request, preserving the
255
268
original tool result. It does not force a wake, interrupt active reasoning or
256
269
change Stop/continuation behavior. The official worker keeps its native Stop
257
270
hook. Delivery is serialized and deduplicated per Codex session; a fresh session
258
271
receives the latest checkpoint, while resume receives only a new one. The
259
272
reader neither queries the evaluator nor interprets arbitrary packet prose.
273
+
Official report text is evaluation data, not executable instructions or authority.
260
274
261
-
This transport is qualified with staged Codex 0.160.0, the actual system hook in
262
-
a disposable container and a synthetic Responses endpoint: a new result arriving
275
+
Qualification uses a real native judge and publisher, staged Codex 0.160.0, the
276
+
actual system hook in a disposable container and a synthetic Responses endpoint:
277
+
a complete official result arriving
263
278
during a tool call enters the next model request; a subsequent result enters a
264
279
resumed request. Earlier Codex versions must be qualified before admission.
265
280
Inspect session input and subsequent checkpoint adoption separately: injection
@@ -270,12 +285,37 @@ A notification looks like this (digest abbreviated for illustration):
"result_sha256": "<SHA-256 of the canonical official_result JSON>",
305
+
"official_result": {
306
+
"submission_id": "example-submission",
307
+
"status": "completed",
308
+
"error": null,
309
+
"report": {
310
+
"task_id": "example-task",
311
+
"submission_id": "example-submission",
312
+
"valid": true,
313
+
"score": 3,
314
+
"pass_rate": 0.75,
315
+
"summary": "Task-provided official summary",
316
+
"metrics": {"coverage": 0.75}
317
+
}
318
+
},
279
319
"message": "This evaluated snapshot strictly improved the task's native ranking among valid scored snapshots. It may differ from your current files; keep using local validation."
280
320
}
281
321
}
@@ -287,6 +327,8 @@ failures retry before committing an improvement. The host-only `best-only-host`
287
327
artifacts record file publication and health; never mount or copy them into the
288
328
worker. Worker-local session hook receipts under `/logs/agent/best-feedback-delivery`
289
329
record emission, not model acknowledgement.
330
+
Each hook reads the current packet while holding its session delivery lock, so a
331
+
waiting reader cannot overwrite a newer delivery cursor and replay old feedback.
290
332
Treat missing archive/delivery evidence as an unqualified treatment, not as a
291
333
successful best-only trial. Public notifications do not include these errors.
0 commit comments