openenv/tbench2: per-task Daytona cloud-sandbox execution backend

[nblintao]

Stacked on #1487 (base = shi/openenv-miles-adapter, parallel to #1662). Pure additive change: the existing shared-server path is behaviorally unchanged (asserted by a dispatch unit test), and every #1487 mechanism — _apply_workdir, canonical exec + reward-marker scoring, rm-hack, policy.close() — is preserved as-is.

What

A second execution backend for the OpenEnv TB2 adapter: set OPENENV_TB2_TASKS_DIR to a terminal-bench-2 checkout and every episode runs in its own Daytona cloud sandbox, built from the task's official image (task.toml docker_image) plus a tbench2_env server layer, deleted when the episode ends.

Same per-task image fidelity as docker mode, with zero resident infrastructure: no Docker socket, no shared env server to size or babysit (no MAX_CONCURRENT_ENVS / ulimit tuning), no cross-episode state. Cost profile: first episode of a task builds its image (~10 min, layer-cached by definition hash), repeats start in ~1 min; no named snapshots, so no org snapshot quota. Sandboxes carry an openenv-tbench2-task=<task_id> label for safe sweeping in a shared org, and creation is throttled process-wide with jittered backoff (Daytona rate-limits creates).

Why the two backends score differently (on purpose)

Each leg matches what its env server actually provides:

• Shared-server leg (unchanged): adapter-driven canonical exec + reward-marker parse — this compensates for an UNMODIFIED upstream tbench2_env server, exactly as openenv: TB2 agentic RL adapter + GLM-4.7-Flash launcher #1487 established.
• Per-task leg (new): standard evaluate action. The sandbox recipe lives upstream-side in tbench2_env.task_snapshots (tbench2_env = OpenEnv's Terminal-Bench-2 env package; fixes proposed in fix(tbench2_env): canonical test.sh scoring, real timeouts, task-image workdir huggingface/OpenEnv#965 + feat(tbench2_env): per-task sandbox image recipe + Daytona materialization huggingface/OpenEnv#966) and bakes a patched server that runs the same canonical tests/test.sh natively and resolves the task WORKDIR server-side. No _apply_workdir prefix on this leg either — task WORKDIRs are not uniformly /app (fix-git = /app/personal-site, prove-plus-comm = /workspace), which the hardcoded prefix silently breaks.

Once #965/#966 land upstream, a follow-up can retire the adapter-driven machinery and unify both legs on evaluate.

Tooling (ships with the backend; produced the validation evidence)

• scan_golden.py — infra baseline with no LLM: replays each task's OFFICIAL solution/solve.sh (oracle-faithful: staged at /solution, DEBIAN_FRONTEND=noninteractive, [solution].env exported, task-workdir cwd) through the same sandbox + evaluate scoring; --logs captures solve/test log tails for anything below 1.0.
• eval_tbench2_via_api.py — the EXACT training agent-env loop (_multi_turn) with any OpenAI-compatible API as the policy: no GPU, no Ray/Megatron. For end-to-end smokes and for measuring base-model solve rates when picking a variance-band training subset.
• tests/ — offline unit tests (pytest examples/experimental/openenv/tests/ -q, not collected by the repo-level suite): backend dispatch on both legs, and the create-throttling retry/backoff/give-up behavior that only triggers under production rate limits.

Validation

• Per-task leg, no LLM (scan_golden.py): golden sweep passes 82/89 of the TB2 suite (0 infra errors; the 7 remaining have upstream-broken solutions, each with solve.log evidence). Golden regression on this exact branch: fix-git, chess-best-move, circuit-fibsqrt all 1.0 — covering per-task WORKDIR dispatch, apt paths, and long canonical evals.
• Per-task leg, end-to-end with a live API policy (eval_tbench2_via_api.py): full 89-task sweep with DeepSeek-V4-Flash as the policy (single sample per task, 30-turn cap, up to 38 concurrent episodes) — 33/89 solved, 0 systematic infra failures (10 non-scoring episodes: 8 heavyweight-task timeouts, i.e. reward 0 under training semantics, 1 transient registry network error, 1 sandbox-resource overrun).
• Shared-server leg: dispatch unit test asserts behavior identical to before this change (exec prefixed with cd /app, canonical-exec scoring, rm-hack present, evaluate never used).
• Unit tests: 7 passed (dispatch on both legs; throttle retry-until-success / give-up-after-budget / non-throttle-passthrough; throttle-error classification incl. the typed DaytonaRateLimitError path).

🤖 Generated with Claude Code

[gemini-code-assist]

Code Review

This pull request introduces support for per-task Daytona cloud sandboxes as an alternative to a shared environment server in the OpenEnv Terminal-Bench-2 adapter. It includes documentation updates, new standalone evaluation and validation scripts (eval_tbench2_via_api.py and scan_golden.py), and offline unit tests. The reviewer's feedback focuses on improving code robustness, such as replacing assert with ValueError for argument validation, using isinstance checks instead of parsing exception strings, and explicitly checking for environment variables rather than relying on exception handling.

Important

The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.

[gemini-code-assist]

According to the general rules, ValueError should be used instead of assert for validating function or constructor arguments. Please raise a ValueError if daytona_api_key is missing.

Suggested change

	assert args.daytona_api_key, "DAYTONA_API_KEY required in per-task Daytona mode"
	if not args.daytona_api_key:
	raise ValueError("DAYTONA_API_KEY required in per-task Daytona mode")

References

1. Use ValueError instead of assert for validating function or constructor arguments.

[nblintao]

Adopted in b604dc9 — switched to raise ValueError. Also makes this file consistent with the adapter, which already raises ValueError for the missing-task_id case.

[gemini-code-assist]

When identifying or filtering specific exception types, prefer using isinstance checks rather than parsing the exception's string representation or checking for class names within the exception string. If ThrottlerException is a specific exception class, import and check for it directly, or check for an HTTP 429 status code attribute if applicable.

References

1. When identifying or filtering specific exception types, prefer using isinstance checks rather than parsing the exception's string representation or checking for class names within the exception string.

[nblintao]

Deliberate, and now documented in the function docstring (b604dc9): the daytona SDK is a lazy, per-task-mode-only dependency — shared-server users don't install it — so an isinstance check on its exception classes isn't available at module scope. Rate limits also surface in several shapes across layers (SDK ThrottlerException, HTTP 429 text, wrapped errors), which one class check wouldn't cover. Same approach as _is_retryable_env_error above, which matches websockets close exceptions by name for the same no-hard-dependency reason.

[nblintao]

Update (1f9ed50): partially adopting after all — the module-scope constraint stands, but by the time a create has FAILED the daytona SDK has necessarily been imported, so a lazy in-function import can do a typed check. _is_throttle_error now checks isinstance(exc, DaytonaRateLimitError) first (the SDK normalizes HTTP 429 to it) and keeps the text match only as a fallback for older SDKs / text-only server messages. Covered by a new unit test asserting the typed path matches even when the message carries no throttle keywords, and that sibling error classes do not.

[gemini-code-assist]

Instead of relying on a caught KeyError when OPENENV_TB2_TASKS_DIR is missing from os.environ, explicitly check if the environment variable is set. This makes the code more intentional and robust.

                tasks_dir = os.getenv("OPENENV_TB2_TASKS_DIR")
                if tasks_dir:
                    try:
                        import tomllib

                        cfg = tomllib.loads(
                            (Path(tasks_dir) / task_id / "task.toml").read_text()
                        )
                        for k, v in (cfg.get("solution", {}).get("env", {}) or {}).items():
                            sol_env += f" {k}={v!r}"
                    except Exception:
                        pass

[nblintao]

Adopted with a stronger fix in b604dc9: this tool only exists on the per-task backend at all, so main() now fails fast with a clear message when OPENENV_TB2_TASKS_DIR is unset. That makes golden_one's os.environ[...] read unconditionally valid, and the try/except is left tolerating only its real target — a task.toml without [solution].env.