vein: meta/validate-workflow; lab: promote eval scoring steps to eval/* - #1648
Merged
Conversation
Evanfeenstra
added a commit
that referenced
this pull request
Sep 4, 2026
Since the graph-backend workspace change, buildRegistry expects the directory returned by workspace.materializeCustomSteps(), not the workspace root; the harvey, sheets and jarvis smoke scripts still passed workspace.path and no longer discovered any custom steps. Same fix as deliver-smoke.ts in #1648.
…m harvey/* to eval/* - authoring capability gains validateWorkflow (the chat builder's validate_workflow) and a meta/validate-workflow lib step over it, so an in-run author can check YAML before meta/publish-workflow. - harvey/aggregate-scores, harvey/build-eval-chain, harvey/criterion-refs move to eval/* (domain-agnostic scoring plumbing; build-eval-chain's workflow input is now required). harvey-score, merge-disputes, the seeders and the deliver smoke are re-pointed; no aliases. - deliver-smoke: build the registry from materializeCustomSteps() (was the workspace root, broken since the graph-backend change). - plans/wfbench-harness.md: design for the 58313 benchmark harness port.
Evanfeenstra
force-pushed
the
meta-validate-and-eval-promote
branch
from
September 4, 2026 19:16
5f2d5f2 to
e678b63
Compare
This was referenced Sep 4, 2026
Merged
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Groundwork for the wfbench harness (the vein port of stakwork 58313, design in
mcp/src/lab/plans/wfbench-harness.md).vein
AuthoringCapability.validateWorkflow(yaml, name?)— the chat builder'svalidate_workflow, reachable in-run.meta/validate-workflowwrapping it, so an authoring agent can static-check YAML beforemeta/publish-workflow. Test covers unknown step type, name-less YAML with a publish name, no-publish side effect, and the step form inside a run.lab
harvey/aggregate-scores,harvey/build-eval-chain,harvey/criterion-refs→eval/*(they carry no harvey config;build-eval-chain.workflowis now required instead of defaulting toharvey-deliver).harvey-score,merge-disputes, both seeders, AGENTS.md and the deliver smoke re-pointed. No aliases kept: nothing else referenced the old names.deliver-smoke.tsnow builds the registry frommaterializeCustomSteps(); passing the workspace root had been failing at the first registry assertion since the graph-backend change.harvey/smoke.ts,sheets/smoke.ts,jarvis/smoke.tshave the same one-liner bug and are left for a follow-up.Verification
tscclean,npm test664/664.tsc --noEmitclean afteryarn refresh-vein;npx tsx src/lab/harvey/deliver-smoke.tsall checks pass.