Skip to content

refactor(agents): extract OpenClaw from BenchFlow core - #1093

Open
kywch wants to merge 8 commits into
mainfrom
refactor/openclaw-extraction
Open

refactor(agents): extract OpenClaw from BenchFlow core#1093
kywch wants to merge 8 commits into
mainfrom
refactor/openclaw-extraction

Conversation

@kywch

@kywch kywch commented Sep 3, 2026

Copy link
Copy Markdown
Collaborator

Closes #1090: #1090

Summary

  • Moves OpenClaw shim, manifest, config, and tests to agents PR #72.
  • Deletes the 973-line built-in shim and OpenClaw-specific production policy.
  • Adds generic lazy manifest loading with local-checkout override support.
  • Makes unknown bare agent IDs fail closed; explicit commands remain supported.
  • Preserves catalog collisions, diagnostics, listing, and atomic publication.

Agents PR #72 must merge first.

Validation

  • Ruff and ty pass.
  • BenchFlow agent tests: 82 passed, 3 skipped.
  • Paired manifest parity: 12 passed.
  • Agents contract/OpenClaw tests: 98 passed.
  • Full suite: 5,952 passed; 3 host-sensitive failures reproduced on main.
  • Production Python: net -873 LOC.

@kywch
kywch deployed to pypi-internal-preview September 3, 2026 20:26 — with GitHub Actions Active
@bingran-you
bingran-you marked this pull request as ready for review September 3, 2026 20:59
@bingran-you
bingran-you requested a review from xdotli September 3, 2026 20:59
@chatgpt-codex-connector

chatgpt-codex-connector Bot commented Sep 3, 2026

Copy link
Copy Markdown

Codex Review Summary

This comment shows the latest Codex review activity on this pull request.

Review Status Commit Review trigger
📝 Code Review Completed 2026-09-03T21:03:36.845766Z c3e770f Draft marked ready
ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review" or "@codex security review".

Codex reacts with 👀 while any review is running, comments if it has suggestions, and reacts with 👍 once all reviews finish with no findings.

@bingran-you
bingran-you deployed to pypi-internal-preview September 3, 2026 21:00 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator

Thermo-nuclear review of exact head c3e770f.

The central ownership change is a net simplification: the 973-line OpenClaw implementation and its feature-specific registry policy leave core, while generic manifest selection/loading remains in focused modules. I found and repaired one cross-feature regression: the new fail-closed launch boundary also tried to resolve the non-agent task-runtime as an ACP launch during default setup. Setup now models agentless modes once and skips both web policy and launch lookup; a regression test exercises real default rollout planes.

Validation on the exact tree: 5,911 tests passed with 87 skipped and 7 deselected; ty and Ruff passed; the paired agents catalog gate passed 28 tests; local list/show resolved OpenClaw from the external manifest. The supplied DeepSeek key was first validated with a direct HTTP 200 call. A complete Docker hello-world run then installed the manifest-owned OpenClaw shim, produced two proxied HTTP 200 model exchanges with provider-reported usage, emitted one completed write tool call, scored reward 1, and recorded no agent or verifier error.

Merge gate: the behavior and structure are acceptable, but agents #71 and #72 must land first, this PR must be refreshed against their immutable merged source, exact-head CI must finish green, and independent human approval is still required before squash merge.

@devin-ai-integration devin-ai-integration Bot left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Devin Review found 2 potential issues.

2 flags not posted on this PR by your GitHub settings — view them in Devin Review. (Configure)

Devin Review

Comment thread src/benchflow/agents/registry.py
Comment thread src/benchflow/agents/remote_manifests.py

@chatgpt-codex-connector chatgpt-codex-connector Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: c3e770fa05

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

  • Open a pull request for review
  • Mark a draft as ready
  • Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

Comment thread src/benchflow/agents/registry.py
Comment thread src/benchflow/agents/remote_manifests.py Outdated
@kywch
kywch deployed to pypi-internal-preview September 3, 2026 21:54 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator

Exact-head follow-up for f7de9af: the local-catalog override now activates before a directly registered built-in can return, and expected remote/core parity overlaps no longer become permanent warnings. I reran the real default resolution/task-runtime focused gate: 21 passed with 3 skipped; ty and Ruff pass, and every exact-head GitHub test, parity, rollout-smoke, and fixture-scenario check is green. The separate reader/writer atomicity limitation is pre-existing on main and remains tracked by #1089; it should not be misrepresented as solved here. Merge still depends on agents #71, #72, and the new streaming fix #73 landing first, followed by immutable-source parity, exact-final-head real E2E, and independent human approval.

@kywch
kywch deployed to pypi-internal-preview September 6, 2026 02:23 — with GitHub Actions Active
@kywch
kywch deployed to pypi-internal-preview September 6, 2026 02:37 — with GitHub Actions Active
@bingran-you

Copy link
Copy Markdown
Collaborator

Exact-head follow-up for 24c5074.

The new generic discovery path correctly accepts an advertised ACP effort option when the registry does not declare one, while preserving registry override precedence. The generalized proxy-alias metadata path keeps existing mapped capabilities and otherwise derives context/output/vision/reasoning metadata from LiteLLM; focused static probes and the regression suite did not expose a contract regression. Exact-head validation passed 108 focused tests, ty check src/, changed-file Ruff check/format, and all current GitHub test, parity, rollout-smoke, and fixture-scenario checks.

A real Docker run then paired this head with agents #73 at 279aa61538a3d494bb6dfc1affa40c5696c001c7: BenchFlow discovered OpenClaw effort dynamically, delivered reasoning_effort=low, recorded a real DeepSeek deepseek-v4-flash HTTP 200 exchange with provider usage, completed one write tool call, scored reward 1, and recorded no agent or verifier error.

No new merge-blocking defect found. Merge still requires agents #71, #72, and #73 to land first so their immutable sources can be pinned, followed by final parity/CI and independent human approval.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Extract OpenClaw shim from BenchFlow core into benchflow-ai/agents

2 participants