Repository navigation
fix(benchmark): validate LHTB cadence against current CLI choices - #6235
Conversation
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
Signed-off-by: LoopX Agent <337587101+loopx-agent@users.noreply.github.com>
loopx-agent
left a comment
There was a problem hiding this comment.
Reviewer: model_agent; gpt-6.1-sol; OpenAI; runtime_reported; xhigh.
Approval conclusion (author-owned PR; GitHub blocks formal self-approval)
Exact head: f1745a2a6946679cecfa89c0b06578bc1204a11e.
动机
使用 LHTB 新默认配置运行预检的 benchmark 维护者。新配置解析为六个工作 Turn,但旧预检只匹配五值帮助文本,合法配置被阻断,维护者无法进入原运行流程。修复后预检读取当前 CLI 公布的选项范围,支持六个 Turn,同时保留旧 Todo 节奏并拒绝不受支持的值。范围仅为 LHTB 预检和回归测试;不改产品默认、任务含义、评分、运行预算或权限,也不启动 benchmark。完整 Docker/Harbor 预检与最终发布资格仍未完成;当前 PR 不将这些缺口标为通过。
改动思路
保留原调用顺序:配置解析出具体 option/count,由选定 checkout 的 configure-goal --help 提供合法范围,预检只观察它。删除复制的五值范围,避免修成另一份六值硬编码。既有 check/failures 和退出 owner 不变,配置、预算和任务不会自动改写。
具体改动
关键代码讲解
cli_supports_replan_count 在当前帮助文本中匹配完整选项名称及 choice braces,再验证已解析 count 是否属于该范围。main 仍要求帮助命令成功,只将旧字符串条件替换为 helper。一个名字相近的选项、另一 cadence 单位或未声明的数字不能通过。新测试覆盖六个 Turn、旧 Todo 单位、不同 namespace、近似选项名与换行格式;真实 checkout CLI 集成测试验证当前默认。
依据 Benchmark runtime settings 和 Compatibility Gate,spec_revision 43643a5e5dd25a05382c232824ebeb806ce45e25:CLI-owned supported cadence 已 implemented;Full real backend preflight:deferred,完整 Docker/Harbor 预检未运行,本 PR 不证明整个运行环境具备条件。
对主干的风险
主要风险是模糊匹配错误选项或接受未声明数字。head 以转义后的完整 option 与整数 membership 限定输入;原始 check report 保留。对同一真实 CLI 帮助,immutable base 的实际预检表达式拒绝默认6,head 接受;旧 Todo3 两边接受,未支持 Turn7 两边拒绝。最终精确 head 的 58 项 focused tests 通过;33 项 optional backend tests 跳过;Ruff、diff hygiene、修正后的语义 advisory 完成。风险验证执行六项 selected checks 与四项 direct checks,无失败;原 benchmark manual-review hold 保留,这份审阅核对了该 helper 的边界和反例,并不声称运行了 Docker/Harbor。未查询 CI,未启动 benchmark。文件不含凭据、私有上下文或生成日志。
我的整体评价
APPROVE,justified_increment。范围仅为 LHTB 预检和回归测试;不改产品默认、任务含义、评分、运行预算或权限,也不启动 benchmark。Long-horizon 行为 preserved: worker、scheduler、trial 预算和续跑 owner 没有改动;预检的 operator journey improved,通过已有 CLI 支持证据移除误阻断。边界与代码量相称;这次 future-facing pass 删除复制的合法范围,没有新增框架、状态、CLI 开关或产品 authority。已删除复制的范围,保留原预检观察和失败 owner,无需新增 runtime 状态或框架。维护者的明确 admin-bypass 授权与精确 head readiness 仍需在合并前成立。完整 Docker/Harbor 预检与最终发布资格仍未完成;当前 PR 不将这些缺口标为通过。
English verdict: APPROVE at f1745a2a6946679cecfa89c0b06578bc1204a11e. The same real CLI help demonstrates the stale base predicate rejecting the supported six-Turn default and the head accepting it, while legacy Todo3 and unsupported Turn7 retain their outcomes. 58 exact-head focused tests and six selected/four direct risk checks pass; 33 optional backend tests are skipped. No benchmark launch, scoring, product cadence, budget or authority changes; complete Docker/Harbor and release qualification are not claimed.
LHTB preflight rejects the current six-Turn default because it still matches the old five-value CLI range. Read the resolved cadence option’s advertised choices from the owning CLI instead, so supported defaults and legacy Todo cadence pass while unsupported values remain rejected.
Validation: 57 focused tests passed, 33 optional backend tests skipped; real checkout CLI help reproduces rejection by the old predicate and acceptance by the corrected predicate. Ruff and diff hygiene passed. Risk validation executed six selected checks and four direct checks without failures; its benchmark manual-review hold is retained for exact-head review. No benchmark jobs were launched; Docker/Harbor end-to-end preflight was not run.
The bounded refactor removes a duplicated choice range from the provider preflight. Product cadence authority and scoring remain with their existing owners.