harvey: deliver pipeline — production-style port with verbatim prompts - #1627
Merged
Conversation
The stakwork Harvey LAB workflow recreated as vein workflows: intake + namespace/EvalSet registration, per-doc agent ingestion (completion-marker dedupe), checklist/fact-base/case-law/tailor phase, drafter fanout + 4 fall-soft verifiers + aggregator, per-criterion LLM judge + dispute + unified eval-chain persistence + webhook + recursion gate. Standalone by design: rubric is a run input — never a harvey-run candidate. Prompts are params placeholders (PASTE-IN SURFACE markers) pending the production texts. New steps: 8 harvey plumbing/pure steps, the pinned read-only harvey/graph-sub-agent, jarvis/register-namespace. Offline smoke: npx tsx src/lab/harvey/deliver-smoke.ts. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Prompts live as markdown in harvey/prompts/ (14 files), spliced into step configs at seed time via @@include markers — stakwork [$(step).output.*] tokens translated to vein templates, tool names to lab step names, container paths to ./, and the ingestion prompt's docx-MCP/pre-parsed sections adapted to bash+pandoc. Raw exports kept in notes/harvey_prompts (unneeded ones deleted). Enabled by the real prompt contracts: allow_scratchpad passthrough on the jarvis triplet steps (ScratchpadEntry capture), harvey/generate-docx + generate-xlsx (pandoc/openpyxl agent tools), case-law secrets via secretsEnv, persona as drafter/verifier system prompt, and the production score ordering — record the eval chain BEFORE disputing so dispute agents get CriterionResult refs (harvey/criterion-refs) for Cause triplets, then write annotations back. EvalRequirement ids now <task_slug>-<criterion_id>. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tall preflight.ts runs the seeded jarvis steps against a REAL jarvis (JARVIS_URL/API_TOKEN from env, throwaway namespace) — the go/no-go check before a live pipeline run: namespace, ontology types, node create/read, ingest completion marker, search, HAS_REQUIREMENT triplet. Also: zod 4 typing fixes (z.record now needs a key schema; parse() returns unknown) in evolve-loop, both pack-results, and the gaia evolve smoke — latent since the #1626 zod bump, surfaced by the first fresh node_modules install; runtime was unaffected. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The secret store already holds SERPA_API_KEY + COURTLISTENER_API_KEY — align the case-law agent's secretsEnv and prompt text to those names (was GOOGLE_SERPA from the raw stakwork export). Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
The prompt body references 'the task goal below' but the exported text ends without it (production's runner appended it) — the agent would otherwise have to infer the engagement from checklist.md alone. Cross-check/tailor/ ingestion are record-first by design and carry no such dangling reference. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Promotes the harvey seed-side expandIncludes hack into a spec: a versioned files/ area in the workspace (publish/append-draft/get/list, content-hash semantics, ai-stamping) plus @@include expansion inside publishWorkflowByContent (publish-time, hash covers expanded text, raw .src.yaml kept for round-tripping, includes provenance in metadata). Kills the big-prompt problem for the authoring agent; implementation deferred. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Observed live: 'paste the service-account JSON into the vein secrets UI or mcp env' sent the checklist-writer agent credential-hunting — env greps and full-disk finds for service-account JSONs (killed mid-run). The throw now states sheets are UNAVAILABLE in this environment, forbids searching the filesystem/env for credentials or retrying sibling sheets tools, and says to proceed with local markdown files. Same note added to the knowledge-stage writer system prompt. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…tally) Observed live: the case-law agent did ~62min of real SerpAPI/CourtListener research, drafted in /tmp, never wrote its synthesis file, then ended the loop with a text-only 'now let's copy this' turn — finalAnswer mode ends on any no-tool turn, so nothing landed in the artifacts dir. The system prompt now pins the two rules: build ./case-law-research.md in the working dir incrementally (scratch may use /tmp, the deliverable may not), and never emit a tool-less message before final_answer. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
finalAnswer mode ends the AI SDK tool loop on ANY turn without a tool call —
including a mid-task narration ('now let's copy this…'), observed live
losing a 62-minute research session whose deliverable needed two more tool
calls. The existing salvage (forced no-tools answer turn) can rescue answer
TEXT but never unfinished file work.
classifyFinalAnswerStop distinguishes the cases: budget exhausted → the
existing forced turn; stopped tool-lessly with budget remaining → resume the
REAL tool loop once (remaining budget, min 4 turns, same tools/checkpoint/
step-finish plumbing, session prefixed with the original task prompt) with a
'continue or call final_answer' user message. A second tool-less stop falls
through to the forced turn / last-text fallback. Usage/cost/step counts fold
the continuation in.
Reaches mcp/lab on the next yarn refresh-vein (yarn dev restart).
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…sult persistence
The first full live run (60/60 pass) died at its FINAL echo step:
{{ post.error }} where post (the webhook step) was when-skipped — skipped
steps resolve undefined in scope and the expression language throws on
property access into undefined. Add ?. optional chaining to vein's template
language (one-hop guard, JS-style; absent idents still throw Undefined
reference to catch typos) and use it for every skippable ref in the deliver
workflows (post, ingest, cross_check, case_law).
Same run also silently dropped ALL 60 CriterionResult graph writes: the
criterion triplets' inline EvalTriggerOutput side carried a bare { id },
missing the batch step's (type + node_data)-keyed dedup cache and then
failing jarvis schema validation ('result' required). All EvalTriggerOutput
sides now share ONE full node_data object → single resolve, cache hits.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
A mid-stream connection death killed a whole agent step. Once response headers are in, the SDK's request-level retry is out of the picture: if the body then dies, undici raises a bare `TypeError: terminated` and EVERY result promise on the stream rejects (verified: steps/response/totalUsage/text all reject), so the old `if (streamError) throw` had nothing left to read and threw the session away. Seen live — a 34-tool-call case-law research step lost ~12 minutes in, taking its findings with it. Bank each step as it finishes (`onStepFinish` fires before the error, and `sf.response.messages` is exactly what a replay needs), then resume: replay the banked conversation plus a nudge saying the stream was cut and any pending tool call never ran. Bounded at 5 resumes, with the step budget decremented by work already done. `isTransientStreamError` gates it to connection faults only, walking the cause chain (cycle-safe). Auth/400/schema failures still throw on the first call. Cancels are excluded by identity via isCancelledError, not by message sniffing: a vein CancelledError is neither named AbortError nor worded like one, so run control was previously safe only by luck. Tested against a local server speaking the Anthropic SSE wire format that genuinely destroys the socket — the only way to reproduce a transport fault: work done before the cut survives, usage sums across attempts, the resume carries the banked calls + task + nudge; persistent severing stops at 5; a 401 fails on call 1. Also: `$error` gains `cause` (flattened, cycle-safe, depth-capped), and the harvey onError handlers surface it. The diagnosis for this failure lived only on the cause — handlers packed `$error.message`, i.e. the bare "terminated" — so the socket-level reason was discarded at exactly the moment it was needed. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Recreates the stakwork Harvey LAB production workflow as vein lab workflows (`mcp/src/lab/harvey`): normalize → register namespace → EvalSet/requirements → per-doc agent graph ingestion → checklist/fact-base/case-law/tailor → drafters + 4 parallel verifiers → aggregator → per-criterion LLM judge → dispute → unified eval-chain persistence → webhook → recursion gate.
Standalone by design: the rubric is a run input and the pipeline self-scores against it — never a harvey-run benchmark candidate (candidates may not see a rubric).
harvey/prompts/(14 files), spliced into step configs at seed time via@@includemarkers; stakwork[$(step).output.*]tokens translated to vein templates, tool names to lab step names, container paths to./.harvey/generate-docx/generate-xlsx(pandoc/openpyxl agent tools), the pinned read-onlyharvey/graph-sub-agent,jarvis/register-namespace,allow_scratchpadpassthrough on the jarvis triplet steps, and a livejarvis/preflight.ts.?.optional chaining in the template language (skipped-step refs), and a one-time nudge when an agent loop ends on a premature text-only turn withoutfinal_answer.employment-labor/draft-updated-anti(6 docs, 60 criteria): full pipeline pass — 60/60 on its own judge, complete eval chain persisted to the graph. Pause → restart → durable-resume exercised three times mid-run; the fixes in this branch (credential-hunting sheets errors, case-law task-goal footer, CriterionResult dedup,?.) all came from that run.deliver-smoke.ts(offline) + jarvis/harvey/sheets smokes + vein tests (556) green;tscclean. Also fixes latent zod-4 typing breaks surfaced by a fresh install.🤖 Generated with Claude Code