[TheGreatAxios]

Summary

• jev() keeps thresholds and escalation only. It no longer takes an endpoint, model, API key, or HTTP client.
• judgeModel() runs that judge as one workflow step. It is exported only from @corbits/evals/workflow.
• runEvaluation throws before cases when a string or ask scorer has no judge.invoke. Code-only scorers do not need a judge.

Verification

• bun run check exited 0 (typecheck, oxlint, oxfmt, unit tests, 56 e2e tests).
• Review of cc8e6b0 found no defects.

Stacked on #24.

Fixes CL-11499

[linear-code]

CL-11499

[corbits-triage]

Triage: Pass

No changes needed from you. A maintainer will review this pull request.

Skipped:

• makes one focused change: stacked branch, based on feat!: name the case ground truth expectedOutput #24

Waiting on a maintainer to approve: approve, request review.

[TheGreatAxios]

Review · Request changes

Runs the judge as one workflow step whose input is the case payload { state, questions }. A bare step() does that. A one-step defineWorkflow() is rejected before the step runs.

Findings

• src/workflow.ts:107 — assertJudgeStep throws judgeModel: step must not set input for a workflow defineWorkflow builds. Both the plural form (steps: { only: step({ agent }) }) and the singular form (defineWorkflow({ id, agent })) stamp the only step with input: { from: "trigger.payload" } even when the author omits input. judgeModel overwrites input to that same value only after this check, so the normalized default is a false rejection. README says model may be a one-step workflow, and that form never receives the case payload. Callers have to pass a bare step() or a hand-built definition that skipped normalization.

A bare step() input was exactly { state, questions } for the case. A two-step workflow, a map, and a childWorkflow were rejected. A null step output failed closed as a parse error, and the step timeout aborted request.signal. bun test tmp/critique-tests/judge-model.test.ts exited 1 on the two defineWorkflow forms. That probe was deleted.

[TheGreatAxios]

Review · Follow-up

Commit 17e5d79 is clean.

judgeModel accepts a one-step defineWorkflow, plural or singular, including the default { from: "trigger.payload" } stamp, and invokes once with the case payload. An input with any other shape, including { from: "trigger.payload", literal: "fixed" }, still throws judgeModel: step must not set input. A two-step workflow, a map, and a childWorkflow are still rejected.

bun test e2e/jev.e2e.test.ts plus a deleted stamp probe exited 0 (7 pass on the filtered run).

[TheGreatAxios]

Superseded by the 0.2.0 release: #32

Do not merge this pull request separately. The release branch cherry-picks the feature stack (except the 0.1.3 bump on #22) and sets @corbits/evals to 0.2.0 with @corbits/system-one ^0.4.0.

[TheGreatAxios]

Superseded by the 0.2.0 release: #32