lab: wfbench — Workflow Editor Agent Benchmark harness + core pack step (re-land on main)

mcp/src/lab/AGENTS.md

@@ -519,7 +519,7 @@ are thin plumbing over `ctx.services.gaia.*`.
 - **Committed harness** (`gaia/seed.ts`, seeded at boot like harvey's):
   steps `gaia/list-tasks`, `gaia/get-task` (stages a task's attached file
   into the run's artifacts dir), `gaia/evaluate` (HARNESS-ONLY), and the
-  combiners `gaia/pack-result` + `gaia/summarize-batch`; workflows
+  combiner `gaia/summarize-batch` (+ vein's core `pack`); workflows
   `gaia-produce` (agent step; the produce system prompt, model, maxSteps: 50
   and agentTools live in `params`; an `onError` fallback scores a blown-up
   agent as an empty wrong answer instead of killing the batch), `gaia-run`
@@ -554,7 +554,7 @@ are thin plumbing over `ctx.services.gaia.*`.
   unmeasured (`baselineSamples: 1`, the pre-Phase-1 behavior).
   Candidate contract: input `{ taskId }`, last step outputs `taskId`,
   `answer` (bare string), `cost`, `steps`; candidates may use
-  `gaia/get-task` / `gaia/pack-result` as steps but NEVER `gaia/evaluate`
+  `gaia/get-task` / `pack` as steps but NEVER `gaia/evaluate`
   (produce-time oracle) and never gaia/*, eval/*, meta/* as agentTools.
   Scores are TRAIN scores — validate the best version on a held-out
   `gaia-batch` slice before promoting. Offline checks:
@@ -602,6 +602,53 @@ are thin plumbing over `ctx.services.gaia.*`.
   scorer-hash mismatch / LFS pointer stubs / idempotent re-entry / half-written
   checkout) with `exec` and `fetchText` faked (`npx tsx src/lab/gaia/smoke.ts`).
 
+### `wfbench/` — Workflow Editor Agent Benchmark (stakwork 58313's twin)
+
+The benchmark harness for the workflow-AUTHORING agent, rebuilt as a vein
+workflow (design + step map: `plans/wfbench-harness.md`). One task in
+(`{ task_slug, task_title?, instructions, criteria, workflow_input_json?,
+rerun_expected_output?, webhook_url? }` — Hive's payload shape), one callback
+out. `wfbench-run`: graph roster (EvalSet → EvalRequirement×N, EvalTrigger,
+HAS_BASELINE_TRIGGER on the EvalSet's first trigger else HAS_TRIGGER — 58313's
+ids, written with vein's `graph/*` steps under `params.namespace`) ‖ the
+author (core `agent` + `agentTools: ["meta/*"]`, editor tool only — 54419's
+twin) builds `wfbench-<slug>` → resolve the version it actually shipped
+(`vpin || vactive`, `published` vs `vbefore` — the gaia-evolve-gen guard) →
+input-key gate (`wfbench/check-input-keys`: the `input.<key>` references in
+the produced YAML vs the launch payload; a mismatch or empty body is a harness
+error, never launched) → rerun via `meta/run-workflow` (57425's twin; own runId
+= 58313's project_id) → `wfbench/classify-run` (a runtime-FAILED rerun is still
+judged; no runId is a harness error) → `wfbench/build-materials` (workflow YAML
++ custom step sources + launch payload + run output + expected output, inlined
+as one markdown block) → per-criterion judge (`wfbench-judge-criterion`: agent
+schema mode, nothing to read — a crash packs `{ error }` = honest FAIL) →
+`eval/aggregate-scores` → record `EvalTrigger -HAS_OUTPUT-> EvalTriggerOutput
+-HAS_CRITERION_RESULT-> CriterionResult <-HAS_CRITERION_RESULT- EvalRequirement`
+(`wfbench/build-eval-output`, 58312's ids: `<slug>-<runId>`,
+`<slug>-<runId>-<criterion_id>`) → `wfbench/webhook-body` (58313's 4-way
+`resolve_webhook_payload`: success `{ task_slug, task_title, n_passed, n_total,
+all_pass, pass_rate, judge_model, criteria_results }` or `{ harness_error:
+true, error_type, error }` — no fake 0/N) → POST `webhook_url` → the run's
+output IS that body (+ diagnostics), fixing 58313's set_output divergence.
+
+- Graph writes use only ontology-declared attributes (vein's backend rejects
+  the rest): 58313's `EvalSet.project_id` (an int there) and the `name` on
+  EvalTrigger / EvalTriggerOutput are omitted; 58312's `CriterionResult
+  -HAS_CAUSE-> Workflow_version` is not written (no such relationship in
+  the ontology, and a vein workflow is not a Workflow_version node).
+- Grant discipline: `wfbench/*` is never an agentTool. The author gets
+  `meta/*` (+ `meta/validate-workflow`) and the editor; the judge gets
+  nothing useful; the produced workflow is necessarily publisher `ai`.
+- The author's `params.authorSystem` carries a stakwork→vein translation
+  table, so PORTING a stakwork workflow is just a task whose instructions
+  are the stakwork body (plus a real project's input/output as
+  `workflow_input_json` / `rerun_expected_output`).
+- v1 is create-new only (58313 v1 too). Needs `ANTHROPIC_API_KEY` + the
+  graph; no stakwork credentials.
+- Smoke (offline — seeds, discovers, static-validates both workflows
+  through the authoring capability, drives every pure step, and checks the
+  graph payloads against `JARVIS_ONTOLOGY`): `npx tsx src/lab/wfbench/smoke.ts`.
+
 ### `eval/` — generic, reusable eval primitives (NOT an experiment)
 
 Domain-agnostic eval substrate, shared by every experiment. See

mcp/src/lab/createLabVein.ts

@@ -23,6 +23,7 @@ import { seedSheetsSteps } from "./sheets/seed.js";
 import { seedHarveySteps, seedHarveyWorkflows } from "./harvey/seed.js";
 import { seedGaiaSteps, seedGaiaWorkflows } from "./gaia/seed.js";
 import { seedArtifactSteps } from "./artifacts/seed.js";
+import { seedWfbenchSteps, seedWfbenchWorkflows } from "./wfbench/seed.js";
 import { buildHarveyServices, type HarveyServices } from "./harvey/service.js";
 import { buildGaiaServices, type GaiaServices } from "./gaia/service.js";
 import { buildGitseeServices, type GitseeServices } from "./gitsee/services/index.js";
@@ -184,6 +185,11 @@ export async function createLabVein(
   await seedGaiaWorkflows(workspace);
   // generic artifact plumbing (artifacts/dir — bridge runId → path for cwd).
   await seedArtifactSteps(workspace);
+  // wfbench — the Workflow Editor Agent Benchmark harness (stakwork 58313's
+  // twin; plans/wfbench-harness.md). Pure plumbing steps; never granted to
+  // an agent (the author gets meta/* only).
+  await seedWfbenchSteps(workspace);
+  await seedWfbenchWorkflows(workspace);
 
   const vein = await createVein<LabServices>({
     workspace,

mcp/src/lab/eval/seed.ts

@@ -28,7 +28,9 @@ import { SEED_OPTS } from "../seed-opts.js";
  *     — rubric-judged scoring plumbing (promoted from harvey/*): zip judge
  *     verdicts into scores_json, build the EvalSet→EvalTrigger→
  *     EvalTriggerOutput→CriterionResult batch-triplet payload, recover the
- *     persisted CriterionResult ref_ids. Used by harvey-score and wfbench.
+ *     persisted CriterionResult ref_ids. harvey-score uses all three;
+ *     wfbench uses aggregate-scores + criterion-refs (its record payload,
+ *     wfbench/build-eval-output, follows stakwork 58312's id conventions).
  */
 
 const SEED_STEPS: Array<{ file: string; type: string }> = [

mcp/src/lab/gaia/evolve-smoke.ts

@@ -37,7 +37,7 @@ async function main() {
     // step types referenced by the workflows all exist in the registry
     const { registry } = await buildRegistry(await workspace.materializeCustomSteps());
     const wanted = [
-      "gaia/list-tasks", "gaia/get-task", "gaia/evaluate", "gaia/pack-result",
+      "gaia/list-tasks", "gaia/get-task", "gaia/evaluate", "pack",
       "gaia/summarize-batch", "gaia/digest-results", "eval/evolve-loop",
       "eval/matrix",
       "artifacts/dir", "meta/run-workflow", "agent", "subflow", "foreach",

mcp/src/lab/gaia/seed.ts

@@ -18,7 +18,8 @@ import { SEED_OPTS } from "../seed-opts.js";
  *   artifacts dir so agent steps (cwd = artifacts dir) can read it.
  * - `gaia/evaluate` — the real leaderboard scorer. HARNESS-ONLY: grant only
  *   to harness workflows, never to a producing agent's `agentTools`.
- * - `gaia/pack-result`, `gaia/summarize-batch` — pure combiners.
+ * - `gaia/summarize-batch` — pure combiner (the echo combiner is vein's core
+ *   `pack` step now).
  * - `gaia/digest-results` — aggregate graded results into the evolve loop's
  *   propose digest (verdict channel only; accuracy as `fitness`).
  *
@@ -35,7 +36,6 @@ const SEED_STEPS: Array<{ file: string; type: string }> = [
   { file: "list-tasks.ts", type: "gaia/list-tasks" },
   { file: "get-task.ts", type: "gaia/get-task" },
   { file: "evaluate.ts", type: "gaia/evaluate" },
-  { file: "pack-result.ts", type: "gaia/pack-result" },
   { file: "summarize-batch.ts", type: "gaia/summarize-batch" },
   { file: "digest-results.ts", type: "gaia/digest-results" },
 ];

mcp/src/lab/gaia/workflows/gaia-candidate-run.yaml

@@ -54,7 +54,7 @@ steps:
       # deep must exist here (§5.3.5).
       onError:
         id: grade_failed
-        type: gaia/pack-result
+        type: pack
         config:
           taskId: "{{ input.taskId }}"
           isCorrect: false
@@ -68,7 +68,7 @@ steps:
         run: "{{ run }}"
 
   - id: result
-    type: gaia/pack-result
+    type: pack
     depends: grade
     config:
       taskId: "{{ input.taskId }}"

mcp/src/lab/gaia/workflows/gaia-evolve-gen.yaml

@@ -174,7 +174,7 @@ steps:
   # always runs. On the no-op branch `canddigest` is skipped, so `digest` is
   # undefined and `noop` tells the loop to score nothing.
   - id: result
-    type: gaia/pack-result
+    type: pack
     depends: [canddigest, published]
     config:
       candidate: "{{ input.candidateName }}"
@@ -310,7 +310,7 @@ params:
       the question/level and stage the task's attached file into the run's
       artifacts dir; give agent steps that dir as cwd via artifacts/dir,
       exactly as the base workflow does.
-    - the LAST step (use gaia/pack-result) must output: taskId, question,
+    - the LAST step (use pack) must output: taskId, question,
       level, answer (the BARE final-answer string), cost, steps.
     - NEVER use gaia/evaluate anywhere in the candidate — self-grading at
       produce time is oracle access, and a candidate that embeds the

mcp/src/lab/gaia/workflows/gaia-evolve.yaml

@@ -96,7 +96,7 @@ steps:
 
   # ── promote: the reviewable report ─────────────────────────────────────
   - id: report
-    type: gaia/pack-result
+    type: pack
     depends: evolve
     config:
       mission: "{{ input.mission }}"

mcp/src/lab/gaia/workflows/gaia-produce.yaml

@@ -18,7 +18,7 @@ steps:
       # task is simply scored incorrect instead of aborting every other task.
       onError:
         id: produce_failed
-        type: gaia/pack-result
+        type: pack
         config:
           object: { answer: "" }
           cost: 0
@@ -50,7 +50,7 @@ steps:
         additionalProperties: false
 
   - id: result
-    type: gaia/pack-result
+    type: pack
     config:
       taskId: "{{ task.taskId }}"
       question: "{{ task.question }}"

mcp/src/lab/gaia/workflows/gaia-run.yaml

@@ -24,7 +24,7 @@ steps:
           answer: "{{ produced.answer }}"
 
   - id: result
-    type: gaia/pack-result
+    type: pack
     config:
       accuracy: "{{ score.accuracy }}"
       correct: "{{ score.correct }}"

mcp/src/lab/harvey/evolve-smoke.ts

@@ -36,7 +36,7 @@ async function main() {
     // step types referenced by the workflows all exist in the registry
     const { registry } = await buildRegistry(await workspace.materializeCustomSteps());
     const wanted = [
-      "harvey/get-task", "harvey/evaluate", "harvey/pack-result", "harvey/digest-results",
+      "harvey/get-task", "harvey/evaluate", "pack", "harvey/digest-results",
       "eval/evolve-loop", "artifacts/dir", "meta/run-workflow", "agent", "subflow", "foreach",
     ];
     for (const t of wanted) assert.ok(registry[t], `registry has ${t}`);

mcp/src/lab/harvey/seed.ts

@@ -15,8 +15,7 @@ import { SEED_OPTS } from "../seed-opts.js";
  *   producing agents.
  * - `harvey/evaluate` — stage deliverables + run the real eval. Grant ONLY
  *   to harness workflows, never to the producing agent.
- * - `harvey/pack-result` — echo combiner (assemble workflow outputs; onError
- *   fallbacks).
+ * - (the echo combiner is vein's core `pack` step now — not seeded here)
  * - `harvey/digest-results` — aggregate graded results into the propose
  *   digest (verdict channel only; see the step header).
  *
@@ -27,7 +26,6 @@ import { SEED_OPTS } from "../seed-opts.js";
 const SEED_STEPS: Array<{ file: string; type: string }> = [
   { file: "get-task.ts", type: "harvey/get-task" },
   { file: "evaluate.ts", type: "harvey/evaluate" },
-  { file: "pack-result.ts", type: "harvey/pack-result" },
   { file: "digest-results.ts", type: "harvey/digest-results" },
   // harvey-deliver pipeline steps (standalone production-style pipeline —
   // rubric as input; NOT part of the benchmark harness): intake, drafting

mcp/src/lab/harvey/workflows/harvey-candidate-run.yaml

@@ -49,7 +49,7 @@ steps:
       retry: { max: 1, delayMs: 15000 }
       onError:
         id: grade_failed
-        type: harvey/pack-result
+        type: pack
         config:
           score: 0
           all_pass: false
@@ -61,7 +61,7 @@ steps:
       timeoutMs: "{{ params.evalTimeoutMs }}"
 
   - id: result
-    type: harvey/pack-result
+    type: pack
     depends: grade
     config:
       task: "{{ input.task }}"

mcp/src/lab/harvey/workflows/harvey-deliver.yaml

@@ -206,7 +206,7 @@ steps:
         webhookUrl: "{{ input.webhookUrl }}"
 
   - id: result
-    type: harvey/pack-result
+    type: pack
     depends: score
     config:
       task: "{{ input.task }}"

mcp/src/lab/harvey/workflows/harvey-dispute-criterion.yaml

@@ -28,7 +28,7 @@ steps:
       retry: { max: 1, delayMs: 10000 }
       onError:
         id: dispute_failed
-        type: harvey/pack-result
+        type: pack
         config:
           error: "{{ $error.cause ? $error.message + ' (cause: ' + $error.cause + ')' : $error.message }}"
     config:

mcp/src/lab/harvey/workflows/harvey-draft.yaml

@@ -82,7 +82,7 @@ steps:
     options:
       onError:
         id: verify_completeness_failed
-        type: harvey/pack-result
+        type: pack
         config: { error: "{{ $error.cause ? $error.message + ' (cause: ' + $error.cause + ')' : $error.message }}" }
     config:
       cwd: "{{ dir.path }}"
@@ -107,7 +107,7 @@ steps:
     options:
       onError:
         id: verify_correctness_failed
-        type: harvey/pack-result
+        type: pack
         config: { error: "{{ $error.cause ? $error.message + ' (cause: ' + $error.cause + ')' : $error.message }}" }
     config:
       cwd: "{{ dir.path }}"
@@ -132,7 +132,7 @@ steps:
     options:
       onError:
         id: verify_arithmetic_failed
-        type: harvey/pack-result
+        type: pack
         config: { error: "{{ $error.cause ? $error.message + ' (cause: ' + $error.cause + ')' : $error.message }}" }
     config:
       cwd: "{{ dir.path }}"
@@ -157,7 +157,7 @@ steps:
     options:
       onError:
         id: verify_doctrine_failed
-        type: harvey/pack-result
+        type: pack
         config: { error: "{{ $error.cause ? $error.message + ' (cause: ' + $error.cause + ')' : $error.message }}" }
     config:
       cwd: "{{ dir.path }}"
@@ -208,7 +208,7 @@ steps:
         why).
 
   - id: result
-    type: harvey/pack-result
+    type: pack
     depends: aggregate
     config:
       outputDir: "{{ dir.path }}/output"

mcp/src/lab/harvey/workflows/harvey-evolve-gen.yaml

@@ -168,7 +168,7 @@ steps:
   # always runs. On the no-op branch `canddigest` is skipped, so `digest` is
   # undefined and `noop` tells the loop to score nothing.
   - id: result
-    type: harvey/pack-result
+    type: pack
     depends: [canddigest, published]
     config:
       candidate: "{{ input.candidateName }}"
@@ -291,7 +291,7 @@ params:
     HARD CONTRACT for the candidate (the harness runs it as-is):
     - input { task, workdir? }; thread workdir into an artifacts/dir step
       (config sub) exactly as the base workflow does.
-    - the LAST step (use harvey/pack-result) must output: task, outputDir
+    - the LAST step (use pack) must output: task, outputDir
       (ABSOLUTE path containing the deliverable files), usage, cost, steps.
     - deliverables are written under outputDir; the grader stages exactly
       that directory.

mcp/src/lab/harvey/workflows/harvey-evolve.yaml

@@ -97,7 +97,7 @@ steps:
 
   # ── promote: the reviewable report ─────────────────────────────────────
   - id: report
-    type: harvey/pack-result
+    type: pack
     depends: evolve
     config:
       mission: "{{ input.mission }}"

mcp/src/lab/harvey/workflows/harvey-ingest-doc.yaml

@@ -64,7 +64,7 @@ steps:
       retry: { max: 1, delayMs: 15000 }
       onError:
         id: ingest_failed
-        type: harvey/pack-result
+        type: pack
         config:
           error: "{{ $error.cause ? $error.message + ' (cause: ' + $error.cause + ')' : $error.message }}"
     config:
@@ -121,7 +121,7 @@ steps:
   # skip-propagation never swallows this. Refs into a skipped branch resolve
   # undefined-safe in pack-result.
   - id: result
-    type: harvey/pack-result
+    type: pack
     depends: [mark, needed, docnode]
     config:
       file: "{{ input.file }}"