evaluation.md

Evaluation Harness And Suites

Roboclaws keeps four proof layers separate:

Layer	Command shape	Owns
Product run	just run::surface ...	One operator-facing run and its artifacts.
Eval harness	just agent::eval recommend|execute ...	Plan- or diff-aware selection of deterministic gates, product rows, suites, and opt-in live rows.
Eval suite	just agent::eval suite=<suite> ...	Versioned samples, trials, graders, aggregate metrics, and replay.
Package proof	python -m roboclaws.<owner>	Specialist mechanics owned by the package used by product or eval flows.

The maintained user-facing skill is @eval-harness. Current row, engine, provider, intent, and evidence-lane dimensions are summarized in Eval harness dimensions.

Eval-driven candidate improvement uses the separate maintainer workflow eval-evolution under the same just agent::eval facade:

just agent::eval evolve campaign=<campaign.json>
just agent::eval evolve campaign=<campaign.json> live_execution=run
just agent::eval evolve-promote report=<selection-report.json> \
  manifest=<maintainer-approved.json> live_execution=run

Both optimizer and robot roles use OpenAI Agents SDK and record distinct identities. Candidate generation cannot mutate the checkout. Quality and privacy gates precede efficiency ranking; one training winner may receive one sealed confirmation, and promotion always requires a digest-bound human approval. A failed authoritative training status rejects that candidate as ineligible for selection; it does not mean the optimizer or training infrastructure failed unless the attempt classification says so. Frozen campaigns carry both campaign-wide token/cost ceilings and explicit per-optimizer-run and per-robot-attempt reservations. The orchestrator must fit the complete reservation before a provider starts, then passes the reserved limit into the Agents SDK output cap; a positive but insufficient remainder is terminal inconclusive evidence rather than permission to overspend. Providers without catalog output pricing are blocked before execution because their cost cap cannot be derived safely.

MCP behavior candidates use a separate candidate worker boundary. The trusted runtime first projects an atomic baseline-public response, validates forbidden keys, and sends only that JSON to a content-addressed worker image. The worker runs without provider credentials, network, repository/eval mounts, writable root state, or host capabilities; only its ephemeral scratch space is writable. The trusted supervisor retains provider access and owns durable artifacts. Behavior execution remains fail-closed unless the campaign binds a passing local/remote isolation attestation and the candidate passes the static gate. Historical CloudML native-container attestation evidence is under output/eval-evolution/20260805-phase3-isolation/cloudml/.

An eval suite answers whether a capability is improving over time, not whether one demo happened to complete:

eval suite
  -> sample
  -> environment reset
  -> agent trial
  -> trace and artifacts
  -> deterministic and optional advisory graders
  -> aggregate metrics
  -> failure replay or regression sample

Deterministic privacy and safety failures are authoritative. Model or human rubric graders remain advisory until calibrated. Results record the relevant surface, intent, preset, world, backend, evidence lane, scenario, prompt hash, agent engine, provider/model, skill, MCP profile, limits, budgets, and artifact schema versions. Missing identity is explicit unavailable or not_applicable.

Versioned definitions live under evals/<capability>/; the first package is evals/household_world/. The Python schema and runner code lives under roboclaws.evals.

Evaluation is a Git-checkout-owned maintainer layer. The sdist and wheel omit roboclaws.evals, evals/**, and the eval-harness skill; installed product runtimes therefore do not expose roboclaws eval, top-level eval, or roboclaws agent eval. Run evals from a repository checkout through just agent::eval, which invokes the repo eval CLI directly.

agent::eval is the public maintainer facade, not an internal row executor. Selected eval-suite rows call .venv/bin/python -m roboclaws.evals.cli directly, and product rows call .venv/bin/python -m roboclaws.cli.main run surface directly. The harness keeps a subprocess per row for timeout and isolation, without re-entering Just. The eval CLI accepts an optional documented kebab-case tool name followed only by key=value arguments; underscore tool aliases, GNU-style flags, and hyphenated key aliases are unsupported.

Running Evals

Common commands:

just agent::eval recommend plan=docs/plans/example.md budget=focused
just agent::eval execute since=origin/main budget=focused
just agent::eval execute profile=baseline-core budget=focused
just agent::eval execute profile=baseline-live-default budget=focused
just agent::eval execute profile=baseline-refresh budget=focused
just agent::eval suite=smoke_regression budget=smoke
just agent::eval suite=map_build_quality budget=smoke
just agent::eval suite=map_consumer_no_prior budget=smoke
just agent::eval suite=cleanup_capability budget=smoke
just agent::eval suite=scene_sampler_stress budget=smoke
just agent::eval suite=long_horizon_tasks budget=smoke

When ROBOCLAWS_OPIK_ENDPOINT is configured, every completed local repo-native suite (roboclaws_eval_suite_v1) automatically projects its persisted eval_results.json to the local Opik service. The runner writes opik_projection.json beside the result bundle and includes its ready, disabled, or unavailable summary in CLI and Eval Harness evidence. Projection is fail-open: Opik availability never changes the eval outcome. Valid failed and blocked trials are included; rows without an eval result bundle are not. Specialist eval schemas such as operator_session_live keep their Trace telemetry but are not coerced into repo-suite Datasets.

For repair, backfill, or dataset-only projection, an existing suite and optional completed result bundle can still be projected without executing an eval:

just agent::eval opik-project \
  suite=smoke_regression \
  eval_results=output/evals/smoke_regression/<stamp>/eval_results.json \
  endpoint=http://127.0.0.1:5174 \
  deadline_s=60

suite and eval_results are required. The endpoint defaults to ROBOCLAWS_OPIK_ENDPOINT, must be a loopback HTTP base origin, and the bounded deadline is capped at 300 seconds. Automatic projection uses a 60-second global deadline. Unreachable or failing Opik services produce a sanitized unavailable receipt rather than changing local results.

opik-project is the read-only repair/backfill command. It does not launch an eval, provider, simulator, or hardware operation. Local suite and result artifacts remain canonical. Opik receives public sample and trial identity, digests, closed launch identity, and allowlisted grader status; it does not receive prompt bodies, private references, generated-mess truth, acceptable destinations, grader configuration, or sealed holdout identity. When results are supplied, annotation timestamps use the existing eval_results.json file modification time because the result bundle has no grader timestamps.

Opik uses four review levels:

• Dashboard summarizes canonical capability health, provider comparison, failures, and native-trace coverage without owning those calculations.
• Project answers where the trace ran: EvalTrial traces use roboclaws-eval.
• Dataset contains the complete public review population under immutable, content-addressed identity.
• Experiment and trace detail cover only rows backed by native sanitized spans. Experiment-only rows remain Dataset-only; no trace is invented.

Names are for review only. Exact Dataset, item, Experiment, trace, span, score, source-bundle, and evaluated-configuration identity remains in metadata and the local receipt. Reprojecting unchanged evidence reuses those identities, while changed canonical content creates a new content-addressed object or fails on a closed-content mismatch.

baseline-core is the normal broad local refresh without live providers. baseline-live-default adds the normal MiniMax live rows, with prior required for its open-task and cleanup consumers. baseline-refresh includes the four-cell fixed-prior comparison. Prior-consuming baseline rows resolve the canonical prior from assets/eval-priors/ by default, or accept an explicit prior for a controlled refresh; missing priors stay selected and are reported as blocked. baseline-ci is the deterministic PR subset and never executes providers or provider egress. Rows whose live preflight is not ready record blocked evidence instead of being silently skipped.

The just agent::eval execute worker runs locally. CloudML batch evaluation, including staging, submission, monitoring, and collection guidance, is retired. Historical remote results remain evidence and are not rerunnable deployment instructions.

max_parallel=1 preserves serial behavior; raising it runs independent rows concurrently while dependency chains and shared visual-backend groups remain ordered. Provider rows retain explicit network requirements and local placement. Grounding DINO runs alongside its corresponding local MuJoCo runtime. A report with required blocked rows is not an accepted complete baseline.

Scene expansion resolves an execution-neutral case from the catalog row, suite, provider profile, seed, and optional (scene_source, scene_index) identity. Only rows declared scene-portable may be expanded with scene=<source>/<index>,....

Provider Evals

OpenAI Agents SDK runs always select one profile explicitly:

Profile	Wire API	Required configuration
kimi-openai-chat	Chat Completions	KIMI_OPENAI_BASE_URL, KIMI_API_KEY
mimo-responses	Responses	MIMO_RESPONSES_BASE_URL, MIMO_RESPONSES_API_KEY, MIMO_RESPONSES_MODEL
minimax-responses	Responses	MM_BASE_URL, MM_API_KEY

Non-direct suite selections preserve live-agent identity and produce blocked provider/runtime evidence unless live_execution=run is explicit:

just agent::eval suite=cleanup_capability budget=smoke \
  agent_engine=openai-agents-sdk provider_profile=kimi-openai-chat

just agent::eval suite=open_ended_goals budget=smoke \
  agent_engine=openai-agents-sdk provider_profile=kimi-openai-chat \
  live_execution=run

Live eval rows use a row-owned decision_call_budget as the primary capability budget. It counts logical model decision requests, while provider retries share the same decision slot and remain visible in provider-attempt telemetry. The row's wall-clock timeout and no-progress timeout remain watchdogs for hung processes and startup failures; they are not the main cross-environment comparison metric. Provider availability, missing credentials, port conflicts, and runtime readiness are classified separately from agent behavior. Performance packets expose decision_call_count and decision_call_budget; missing telemetry stays unavailable rather than being reported as zero.

The session-live route verifies an Operator Session parent run, active-run steering, terminal artifacts, and a linked child goal. The parent and child are separate agent runs, so each receives a fresh 1500-second wall-clock budget. It is also opt-in:

just agent::eval session-live budget=smoke \
  agent_engine=openai-agents-sdk provider_profile=kimi-openai-chat \
  live_execution=run

Map Evaluation

map_build_quality compares builders and produces candidate artifacts. map_consumer_no_prior runs controls from the Base Metric Map and is kept out of baseline consumer results. map_consumer_fixed_prior consumes one explicit read-only canonical prior. The legacy map_build_consumer suite is an explicit same-provider end-to-end research profile, not the normal comparison baseline.

baseline-live-default proves prior consumption with a same-run direct chain: direct-map-build-world-public produces runtime_metric_map.json, then direct-cleanup-runtime-prior-consumer consumes that exact artifact. The fixed-prior provider matrix is not part of that default. Run baseline-refresh with an explicit runtime_map_prior=<path> when every provider must consume one identical immutable prior.

After maintainer approval, promote an accepted selector report to the content-addressed catalog. Keep selector output under output/ while it is being reviewed; publish only the small approved catalog and prior snapshot to the Git LFS-backed assets/eval-priors/ namespace:

just agent::eval runtime-prior-promote report=<selection-report.json> \
  manifest=<promotion-manifest.json> \
  output_dir=assets/eval-priors

The promotion manifest records approval, scene/source-map, backend, builder provider/model, prompt or skill version, evidence lane, camera labeler, seed, and map schema identity. Any identity change produces a different digest. The generated catalog paths must be repository-relative before they are committed; do not commit absolute paths from a local staging checkout. See assets/eval-priors/README.md for the update cadence, LFS checkout behavior, and artifact retention boundary.

Completed provider artifacts can be regraded without another live call:

just agent::eval suite=map_build_consumer budget=focused \
  agent_engine=openai-agents-sdk provider_profile=kimi-openai-chat \
  regrade_source=output/evals/<suite>/<stamp>

Regrade requires the exact source suite and sample release, preserves the source trial's engine, provider, model, Skill, delivery cell, runtime, and budget identity, and rejects any explicitly supplied execution override that conflicts with that source evidence.

Cross-run MapBuild reports are artifact-only:

just agent::eval map-build-report \
  eval_results=output/evals/<suite>/<stamp>/eval_results.json,output/evals/<suite>/<other-stamp>/eval_results.json \
  output_dir=output/evals/map-build-matrix-review

Observability Decision Report

Every authoritative terminal Eval Harness run adds a decision section to its existing eval_harness.json and eval_harness.md files, publishes a v2 completion marker that hashes those two canonical reports, and then performs a bounded fail-open Opik projection. Recommendation, nonterminal, and worker-shard manifests explicitly report the section as not applicable. The Opik Dashboard is the maintained browser review surface; it is a one-way projection rather than a second metric or evaluation owner.

Read the canonical report views in order:

1. Capability health and regression leads with selected-row outcomes and EvalTrial/sample quality. This is the gate for any performance claim.
2. Sample stability reports trial counts, passed/failed/blocked counts, pass rate, and explicit coverage for each sample. It intentionally does not treat repetition index as an analysis dimension.
3. Execution health groups diagnostic outcomes by execution target and failure class. It must not be used to compare quality or latency between local and CloudML execution.
4. Baseline regression summaries retain comparison row IDs and counts while also grouping regressions by stable capability dimensions such as suite and intent.
5. Fair provider comparison groups only matching dataset, sample, prompt/Skill, runtime, scenario, and execution evidence while varying provider profile, model, and wire API. Quality, model work, and latency have independent eligibility. A single run is diagnostic; concurrent or different execution environments make latency incomparable rather than producing a ranking.
6. Failure and stall triage preserves the recorded harness, environment, provider, operator, and behavior ownership and links to local evidence plus Opik Experiment and trace detail when native span projection is ready.

Every metric records availability, source, coverage, claim eligibility, and limitations. Missing model duration or token usage remains null/unavailable, never zero. Cost is absent because v1 has no versioned price contract. Runtime timing comes only from local canonical artifacts; Opik Experiment duration is not runtime evidence.

Review the report after every terminal candidate and before baseline or promotion decisions. During an eval campaign, use triage and telemetry coverage for active debugging and review provider/trajectory detail weekly. Opik's Dashboard contains only native trace-volume navigation by provider and recorded outcome plus the reviewed Dataset link and fidelity note; failure class remains available in trace drilldown. Opik does not own cohorts, sample or capability analysis, quality policy, private evidence, or promotion.

Regression Promotion

Failed, blocked, or inconclusive results can become durable regression samples:

just agent::eval promote-regression \
  eval_results=output/evals/<suite>/<stamp>/eval_results.json \
  source_sample_id=<sample-id> \
  regression_sample_id=regression.<name>

Human labels are eval-regression:accepted, eval-regression:needs-human-review, and eval-regression:do-not-promote. The final label is a stop signal and writes nothing.

Keep grader truth private. Generated mess sets, acceptable destinations, hidden targets, and private manifests may feed graders and maintainer reports, but they must not appear in agent-facing MCP inputs, capability metadata, or public map artifacts.

Eval Evolution MCP Gates

MCP description evolution is limited to existing public ToolDescriptor summaries. The frozen target records the profile id/version, exact public tool set, immutable semantic metadata, and a content digest. Candidate validation allows only non-empty summary text changes; tool additions/removals, metadata changes, stale parent identity, and newly introduced private terminology are rejected before any live execution.

MCP behavior candidates require a campaign-bound isolation attestation that proves credential scrubbing, private-data isolation, and unauthorized write/network denial. Retained local and remote attestation evidence records the previous proof; a new campaign still requires a valid bound attestation. A candidate may then project existing-tool public responses inside the isolated worker, but cannot change the public tool set or bypass the trusted supervisor's output validation.