Skip to content

lab: wfbench — validation as judge evidence, namespace as run input - #1653

Merged
Evanfeenstra merged 1 commit into
mainfrom
wfbench-validation-namespace
Sep 4, 2026
Merged

lab: wfbench — validation as judge evidence, namespace as run input#1653
Evanfeenstra merged 1 commit into
mainfrom
wfbench-validation-namespace

Conversation

@Evanfeenstra

Copy link
Copy Markdown
Collaborator

Two small pieces toward the Hive "run in vein" toggle.

  • Validation as evidence. wfbench-run now runs meta/validate-workflow on the produced YAML and wfbench/build-materials inlines the result as a VALIDATION material (ok, errors, warnings, summary). Structural criteria ("workflow is structurally valid") get judged on the engine's own check plus the rerun status instead of on the judge's reading of the YAML. Not a gate: a candidate that validates poorly is still launched and judged.
  • Namespace as input. input.namespace || params.namespace on every roster/record write. Hive already upserts the EvalSet/EvalRequirement roster through jarvis before dispatch; passing that partition lets vein's writes merge onto the same nodes by id rather than creating a second roster under wfbench.

Verification: mcp tsc clean; npx tsx src/lab/wfbench/smoke.ts green (both workflows static-validate, build-materials asserts the VALIDATION section).

- wfbench-run runs meta/validate-workflow on the produced YAML and hands
  the result to build-materials as a VALIDATION material (ok / errors /
  warnings / summary) — evidence for structural criteria, never a gate.
- input.namespace overrides params.namespace for every roster/record
  write, so Hive can pass the partition its own roster upsert used and
  vein's EvalSet/EvalRequirement writes merge onto its nodes by id
  instead of creating a second roster.
@Evanfeenstra

Copy link
Copy Markdown
Collaborator Author

Live run 1788554443025 on this branch with input.namespace: wfbench-ns-test: 4/4, candidate v4. Namespace registered and the whole roster + record chain landed under wfbench-ns-test (1 EvalSet, 4 EvalRequirement, 1 EvalTrigger on HAS_BASELINE_TRIGGER since that partition had none, 1 EvalTriggerOutput, 4 CriterionResult) — the wfbench partition from earlier runs untouched. The judge materials carried ### VALIDATION: static validation: ok (0 error(s), 0 warning(s)) from meta/validate-workflow.

@Evanfeenstra
Evanfeenstra merged commit 0e7e3d8 into main Sep 4, 2026
6 checks passed
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant