Skip to content

fix(sandbox): harden shared denylist enforcement across harnesses - #1118

Merged
bingran-you merged 5 commits into
mainfrom
bry/research-denylist-integration
Sep 9, 2026
Merged

fix(sandbox): harden shared denylist enforcement across harnesses#1118
bingran-you merged 5 commits into
mainfrom
bry/research-denylist-integration

Conversation

@bingran-you

@bingran-you bingran-you commented Sep 9, 2026

Copy link
Copy Markdown
Collaborator

Follow-up to #1113, driven by FrontierPhysics #366. Harden the shared sandbox denylist so URL enforcement and model-gateway access work independently of the task, model name and supported ACP harness.

Live Docker/Daytona runs exposed three defects:

  • Clients that ignore NO_PROXY send the local model gateway through the egress proxy and hit its IP-literal rejection. Register only the controller-created gateway's exact loopback port, allowing HTTP forwarding and CONNECT while other IP/private destinations stay blocked. Reconnects register the current port.
  • Codex's default inner sandbox fails before tools run because Docker/Daytona disallow its namespace operations. Use the adapter's INITIAL_AGENT_MODE=agent-full-access when BenchFlow already selected a non-root sandbox user; preserve an explicitly requested mode. BenchFlow's UID, filesystem and network restrictions still apply.
  • codex-acp 1.6.0 ignores -c CLI flags, including the inherited hosted-search switch. Apply web_search=disabled through its supported CODEX_CONFIG path, overriding conflicting caller settings and rejecting malformed policy configuration.

Certificate generation now declares cryptography as a core dependency. The proxy fix lives in the shared sandbox layer; Codex configuration stays in its existing adapter helper. The user-requested maintainability review retained one shared proxy and avoided model-specific branches.

Live validation

Authentication/model and harness Docker Daytona
Azure GPT-5.6 Terra, Codex ACP, observed xhigh reward 1.0 reward 1.0
Azure GPT-5.6 Terra, OpenHands, observed xhigh reward 1.0 reward 1.0
Claude OAuth/Fable 5.1, Claude Code ACP, requested max reward 1.0 reward 1.0

Each network canary executes as UID1000, retrieves allowed origin content, receives eight policy-marked denials, fails a direct-socket bypass, and preserves its denial log. Additional Claude native-tool runs on both backends produce a ninth denial through WebFetch, successfully fetch an allowed sibling, and capture the actual settings deny=[WebSearch]. Their installed runtime is Claude Code 2.1.257, ACP 0.73.0 and SDK 0.3.257. OAuth is the native subscription route with no API-key fallback; upstream max is not independently visible on that route.

All 13 actual FrontierPhysics policies were also exercised through real sandbox proxies: 298 denials per backend, with ordinary internet available. Run-bot plans preserve all 13 denylists. The unchanged Rydberg task ran through fpinfra.bench_driver.run_eval with Terra on both backends: both completed with no runtime/verifier errors and scored 0.0 on the numerical physics criterion; both original oracles scored 1.0 (7/7 checks). These are feature-validation runs, not a claim that every scientific task was solved.

Earlier fixed Gemini runs also passed on both backends. The shared connection matrix covers all 10 registered ACP harnesses plus a custom registration × four opaque model identifiers, including later roles and changed gateway ports. Real socket regressions check exact gateway access, invalid endpoints, private destinations and denied sources. Nine Codex configuration regressions cover isolation defaults, explicit overrides and fail-closed web policy.

Checks and reproducibility

Exact-head CI at 1643338f passes: 6,252 tests passed, 11 skipped, 11 integration cases deselected; Ruff, formatting, type checks, manifest parity and dependency audit also pass. Both integration-light lanes (rollout-smoke and fixture-scenarios) pass. Gated medium integration and release jobs are skipped, not counted as passes. Companion checks pass: FrontierPhysics 192 tests plus 113 subtests (3 skipped), Infra 325 tests (1 skipped).

Run uv run pytest -m integration tests/test_egress_denylist_integration.py with Docker and the Daytona extra. Set BENCHFLOW_DENYLIST_MODEL, provider credentials and optionally BENCHFLOW_DENYLIST_AGENT/BENCHFLOW_DENYLIST_EFFORT. Native settings use JSON BENCHFLOW_DENYLIST_AGENT_ENV; OpenHands needs {"LLM_REASONING_EFFORT":"xhigh","BENCHFLOW_REASONING_EFFORT":"xhigh"} rather than an ACP effort option. Claude OAuth uses a bare model and CLAUDE_CODE_OAUTH_TOKEN with API keys unset.

Limits and adoption

Supported enforcement remains Docker/Daytona plus ACP harnesses. Session-factory harnesses and unsupported backends fail closed. Hosted-search controls remain adapter-specific; unhandled provider-side fetches and unlisted copies are not covered by a sandbox URL proxy. This is not universal live validation of every model/harness combination.

Structured Terra trajectories passed independent artifact audits. Native OAuth retains ACP/token/timing/reward evidence but correctly stays non-training-ready without provider-wire capture. Built-in skills remain exposed; no task-skill use or reward hacking was observed. Gemini's separate training-export defect remains tracked by #1106. No trajectories are published by this PR.

Companion PRs FrontierPhysics #371 and FrontierPhysics-Infra #10 remain drafts until their pins can use a published preview containing this repair. No main merge or release is performed here.

@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 9, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-09T18:12:09.994694Z c9e2b42 PR opened
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview September 9, 2026 18:07 — with GitHub Actions Inactive

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

✅ Devin Review: No Issues Found

Devin Review analyzed this PR and found no bugs or issues to report.

Devin Review

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview September 9, 2026 18:22 — with GitHub Actions Inactive
@bingran-you bingran-you changed the title fix(sandbox): keep Gemini model calls working with research denylists fix(sandbox): harden shared denylist enforcement across harnesses Sep 9, 2026
@bingran-you

Copy link
Copy Markdown
Collaborator Author

@Benjamin-eecs Would you want to have a look and review?

@bingran-you
bingran-you temporarily deployed to pypi-internal-preview September 9, 2026 18:50 — with GitHub Actions Inactive
@bingran-you
bingran-you temporarily deployed to pypi-internal-preview September 9, 2026 18:51 — with GitHub Actions Inactive
@bingran-you
bingran-you merged commit 10ae40b into main Sep 9, 2026
11 checks passed
@bingran-you
bingran-you deleted the bry/research-denylist-integration branch September 9, 2026 22:30
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant