fix(sandbox): harden shared denylist enforcement across harnesses - #1118
Merged
Conversation
Codex Review SummaryThis comment shows the latest Codex review activity on this pull request.
ℹ️ About Codex in GitHubYour team has set up Codex to review pull requests in this repo. Reviews are triggered when you
Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings. |
bingran-you
temporarily deployed
to
pypi-internal-preview
September 9, 2026 18:07 — with
GitHub Actions
Inactive
bingran-you
temporarily deployed
to
pypi-internal-preview
September 9, 2026 18:22 — with
GitHub Actions
Inactive
Collaborator
Author
|
@Benjamin-eecs Would you want to have a look and review? |
bingran-you
temporarily deployed
to
pypi-internal-preview
September 9, 2026 18:50 — with
GitHub Actions
Inactive
bingran-you
temporarily deployed
to
pypi-internal-preview
September 9, 2026 18:51 — with
GitHub Actions
Inactive
5 tasks
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #1113, driven by FrontierPhysics #366. Harden the shared sandbox denylist so URL enforcement and model-gateway access work independently of the task, model name and supported ACP harness.
Live Docker/Daytona runs exposed three defects:
NO_PROXYsend the local model gateway through the egress proxy and hit its IP-literal rejection. Register only the controller-created gateway's exact loopback port, allowing HTTP forwarding and CONNECT while other IP/private destinations stay blocked. Reconnects register the current port.INITIAL_AGENT_MODE=agent-full-accesswhen BenchFlow already selected a non-root sandbox user; preserve an explicitly requested mode. BenchFlow's UID, filesystem and network restrictions still apply.-cCLI flags, including the inherited hosted-search switch. Applyweb_search=disabledthrough its supportedCODEX_CONFIGpath, overriding conflicting caller settings and rejecting malformed policy configuration.Certificate generation now declares
cryptographyas a core dependency. The proxy fix lives in the shared sandbox layer; Codex configuration stays in its existing adapter helper. The user-requested maintainability review retained one shared proxy and avoided model-specific branches.Live validation
Each network canary executes as UID1000, retrieves allowed origin content, receives eight policy-marked denials, fails a direct-socket bypass, and preserves its denial log. Additional Claude native-tool runs on both backends produce a ninth denial through WebFetch, successfully fetch an allowed sibling, and capture the actual settings
deny=[WebSearch]. Their installed runtime is Claude Code 2.1.257, ACP 0.73.0 and SDK 0.3.257. OAuth is the native subscription route with no API-key fallback; upstream max is not independently visible on that route.All 13 actual FrontierPhysics policies were also exercised through real sandbox proxies: 298 denials per backend, with ordinary internet available. Run-bot plans preserve all 13 denylists. The unchanged Rydberg task ran through
fpinfra.bench_driver.run_evalwith Terra on both backends: both completed with no runtime/verifier errors and scored 0.0 on the numerical physics criterion; both original oracles scored 1.0 (7/7 checks). These are feature-validation runs, not a claim that every scientific task was solved.Earlier fixed Gemini runs also passed on both backends. The shared connection matrix covers all 10 registered ACP harnesses plus a custom registration × four opaque model identifiers, including later roles and changed gateway ports. Real socket regressions check exact gateway access, invalid endpoints, private destinations and denied sources. Nine Codex configuration regressions cover isolation defaults, explicit overrides and fail-closed web policy.
Checks and reproducibility
Exact-head CI at
1643338fpasses: 6,252 tests passed, 11 skipped, 11 integration cases deselected; Ruff, formatting, type checks, manifest parity and dependency audit also pass. Both integration-light lanes (rollout-smoke and fixture-scenarios) pass. Gated medium integration and release jobs are skipped, not counted as passes. Companion checks pass: FrontierPhysics 192 tests plus 113 subtests (3 skipped), Infra 325 tests (1 skipped).Run
uv run pytest -m integration tests/test_egress_denylist_integration.pywith Docker and the Daytona extra. SetBENCHFLOW_DENYLIST_MODEL, provider credentials and optionallyBENCHFLOW_DENYLIST_AGENT/BENCHFLOW_DENYLIST_EFFORT. Native settings use JSONBENCHFLOW_DENYLIST_AGENT_ENV; OpenHands needs{"LLM_REASONING_EFFORT":"xhigh","BENCHFLOW_REASONING_EFFORT":"xhigh"}rather than an ACP effort option. Claude OAuth uses a bare model andCLAUDE_CODE_OAUTH_TOKENwith API keys unset.Limits and adoption
Supported enforcement remains Docker/Daytona plus ACP harnesses. Session-factory harnesses and unsupported backends fail closed. Hosted-search controls remain adapter-specific; unhandled provider-side fetches and unlisted copies are not covered by a sandbox URL proxy. This is not universal live validation of every model/harness combination.
Structured Terra trajectories passed independent artifact audits. Native OAuth retains ACP/token/timing/reward evidence but correctly stays non-training-ready without provider-wire capture. Built-in skills remain exposed; no task-skill use or reward hacking was observed. Gemini's separate training-export defect remains tracked by #1106. No trajectories are published by this PR.
Companion PRs FrontierPhysics #371 and FrontierPhysics-Infra #10 remain drafts until their pins can use a published preview containing this repair. No main merge or release is performed here.