Model Build AI evaluation artifacts at the JSON boundary #316

Current behavior

The Build AI evaluation adapter creates run.json as a dict[str, object] at examples/build_ai_evaluation/evaluate.py:437-473. A saved file is loaded with json.loads() and immediately used through .get() and indexed keys at evaluate.py:482-494.

Per-sample outcomes use one dictionary shape controlled by a free-form status string at evaluate.py:641-672:

• ok requires predicted_value and no error.
• invalid requires a parse error and no prediction.
• error requires an execution error and no prediction.

The summarizer reconstructs those invariants through string comparisons, key lookups, and casts at evaluate.py:543-633. A valid JSON value with the wrong shape can currently escape as AttributeError, KeyError, or a misleading summary.

Verified against main at fbfd553.

Pattern to copy

The direct model boundary in examples/build_ai_evaluation/judgment.py:43-69 already uses explicit parsed and unparsed variants. model_output_check_result() consumes them with an exhaustive match.

What to build

Introduce focused internal domain types for run metadata and the three sample outcomes. Parse run.json into the metadata type at the file boundary. Make _sample_result() return an explicit union and make summary computation consume that union.

Keep dictionaries at JSON serialization boundaries. Preserve the current run.json and summary.json schemas so existing experiment directories and comparison commands continue to work.

Definition of done

1. Successful, invalid-response, and execution-error samples are exclusive variants.
2. Only the successful variant can carry predicted_value.
3. Invalid and error variants carry their respective error text.
4. Run metadata is parsed once from saved JSON and is a typed value downstream.
5. A non-object or malformed saved run.json raises ValueError naming the file and the bad field.
6. Matching fingerprints still resume an experiment. Different fingerprints still refuse the output directory.
7. The current summary values and serialized JSON keys are unchanged for the existing mixed-outcome fixture.
8. Summary logic no longer branches on free-form sample status strings or casts run metadata.

Non-goals

• Building a shared framework for the other evaluation examples
• Changing Inspect log formats
• Changing prompts, scoring, or summary formulas
• Adding Pydantic or another dependency

Validation

uv sync --locked --project examples/build_ai_evaluation
uv run --locked --project examples/build_ai_evaluation \
  ruff check --fix examples/build_ai_evaluation
uv run --locked --project examples/build_ai_evaluation \
  ruff format examples/build_ai_evaluation
uv run --locked --project examples/build_ai_evaluation \
  ty check --project examples/build_ai_evaluation --extra-search-path . \
  examples/build_ai_evaluation
uv run --locked --project examples/build_ai_evaluation \
  pytest -q examples/build_ai_evaluation/tests