Build AI evaluation: the hand-count schema does not state its value set, but its sibling does #257

Current behavior

The Build AI evaluation defines two response schemas next to each other (examples/build_ai_evaluation/judgment.py:54-63). One states its allowed values and the other does not:

HAND_COUNT_RESPONSE_SCHEMA: dict[str, object] = {
    "type": "object",
    "properties": {"hand_count": {"type": "integer"}},
    "required": ["hand_count"],
}
ACTIVE_MANIPULATION_RESPONSE_SCHEMA: dict[str, object] = {
    "type": "object",
    "properties": {"answer": {"type": "string", "enum": ["yes", "no"]}},
    "required": ["answer"],
}

Both tasks have a closed value set. The prompt says so for each: prompts/hand_count.txt ends "Return only one of: 0, 1, 2. No extra words", and prompts/active_manipulation.txt ends 'Respond only with: "yes" or "no."'. Both parsers enforce it. Only answer tells the model.

Measured on current main (cdcc8d2):

hand_count 2 (in range)          -> 2
hand_count 7 (schema-valid)      -> ValueError: hand count must be 0, 1, or 2
hand_count -1 (schema-valid)     -> ValueError: hand count must be 0, 1, or 2
answer "yes"                     -> 'yes'
answer "maybe" (schema REJECTS)  -> ValueError: active manipulation must be "yes" or "no"

{"hand_count": 7} satisfies the schema exactly. {"answer": "maybe"} does not.

Why the asymmetry costs something

_response_format_payload (judgment.py:160) sends the schema as response_format: json_schema, so a provider that supports constrained decoding will not emit a value the schema forbids. That is enforcement at the model, before the tokens exist. For answer it is on; for hand_count it is off.

When the count comes back out of range, evaluate_image_with_model catches the ValueError (judgment.py:280-283) and records the episode as unparsed: a {prefix}/parse_error measurement, a {prefix}/unparsed tag, and no {prefix}/prediction key at all (judgment.py:230-245). So the frame drops out of the evaluation, and a paid API call was spent to learn that. The schema could have prevented the response instead of the parser rejecting it afterwards.

What to build

Give hand_count the same treatment answer already has, so the two schemas say the same kind of thing about their value sets.

{"type": "integer", "enum": [0, 1, 2]} is the direct parallel. minimum/maximum also expresses it and is worth a moment's thought: enum matches the sibling schema and matches how the prompt words it ("only one of: 0, 1, 2"). Say which you picked and why.

Then pin that the two cannot drift apart again. The real problem is not the missing three characters, it is that a schema and its parser encode the same contract in two places and nothing compares them. A test that walks each task's schema and checks the parser accepts exactly the values the schema permits, and rejects the ones it forbids, catches the next divergence too.

Definition of done

1. HAND_COUNT_RESPONSE_SCHEMA constrains hand_count to 0, 1, and 2.
2. A test asserts, for both tasks, that the value set the schema allows and the value set the parser accepts are the same. Table-driven over the two tasks rather than two hand-written tests, so a third task added later is covered by construction or fails loudly.
3. Every existing parser test still passes untouched. The parsers are already correct and this issue does not change them.
4. A test pins that a schema-permitted value is parseable: pick the boundary values from the schema itself, not literals, so the test follows a future edit to the schema.

Validation

uv sync --locked --all-extras
uv run ruff check --fix
uv run ruff format
uv run ty check
uv run pytest -q

Run plain uv run pytest -q from the repo root: pytest tests alone misses
packages/hflow-server/tests and this example's own tests under
examples/build_ai_evaluation/tests/.

Notes

No network and no API key needed. Everything here is a pure function over strings and dicts, which is why it is a good first issue despite living in the largest example in the repo.

Out of scope, but worth seeing while you are in the file: both prompts ask for a bare scalar ("No extra words") while the schemas ask for a JSON object. The parsers accept both shapes on purpose (_parse_json_or_scalar, judgment.py:74), so nothing is broken. Changing the prompt wording is a judgment call about eval semantics and belongs in its own issue with its own reasoning; do not fold it in here.