Skip to content

Commit fc101a5

Browse files
authored
docs(benchmark): add engineering construction plan
* docs(benchmark): add engineering construction plan Signed-off-by: huangruiteng <huangrt01@163.com> * docs(benchmark): sync engineering plan in Chinese RFC Signed-off-by: huangruiteng <huangrt01@163.com> --------- Signed-off-by: huangruiteng <huangrt01@163.com>
1 parent 228701d commit fc101a5

3 files changed

Lines changed: 302 additions & 14 deletions

File tree

‎benchmark/README.md‎

Lines changed: 23 additions & 0 deletions
Original file line numberDiff line numberDiff line change
@@ -18,6 +18,29 @@ This directory may contain thin examples and practice notes, but it is not
1818
installed as a second LoopX Python package and does not grant execution or
1919
publication authority.
2020

21+
## Engineering build map
22+
23+
The canonical engineering plan is [Section 11 of the research
24+
RFC](../docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md#11-engineering-construction-plan).
25+
It separates repository readiness from benchmark claims:
26+
27+
- **E0--E1:** typed contracts, public/private boundaries, native runner
28+
preflight, and one conformance slice;
29+
- **E2:** experiment-board lifecycle, concurrency admission, exact runtime
30+
observation, continuity, reconciliation, and safe closeout;
31+
- **E3:** preregistered matched arms, treatment fidelity, integrity,
32+
countability, and uncertainty;
33+
- **E4:** cross-benchmark replication and non-benchmark product qualification.
34+
35+
The active workspace should keep benchmark-family launch, verifier, scoring, and
36+
failure semantics in adapters while reusable permission, integrity, lifecycle,
37+
and public-safe projection rules stay in
38+
[`benchmark-toolkit`](../loopx/capabilities/benchmark_toolkit/README.md).
39+
Contributors should start with a bounded synthetic fixture or conformance seam;
40+
live tasks, hidden evaluation, credentials, raw trajectories, submissions, and
41+
unpublished comparisons remain maintainer-owned. Engineering readiness does not
42+
by itself establish a C2 uplift claim.
43+
2144
## Current work
2245

2346
- [`deepswe/README.md`](deepswe/README.md) records the current public-safe

‎docs/architecture/rfcs/long-horizon-harness-benchmark-research-program-v0.md‎

Lines changed: 151 additions & 7 deletions
Original file line numberDiff line numberDiff line change
@@ -676,7 +676,150 @@ The goal is not to fork each benchmark into a LoopX edition. It is to make
676676
LoopX a well-behaved harness participant whose results can be reproduced and
677677
audited by benchmark maintainers.
678678

679-
## 11. Research Program Milestones
679+
## 11. Engineering Construction Plan
680+
681+
The research contract above says what a benchmark result may prove. This
682+
section says how the repository should build, validate, and operate the
683+
supporting engineering. The two ladders are intentionally orthogonal:
684+
685+
- **Engineering readiness** describes whether a reusable contract, runner seam,
686+
evidence boundary, and operational path are implemented and qualified.
687+
- **Claim level (C0--C4)** describes what the resulting experiment is allowed to
688+
say about LoopX. Reaching an engineering stage never upgrades a research
689+
claim by itself.
690+
691+
### 11.1 Ownership layers
692+
693+
Build each benchmark integration in these layers, from the most reusable to the
694+
most benchmark-specific:
695+
696+
1. **Provider-neutral toolkit**
697+
- owns typed permissions, source and artifact boundaries, integrity receipts,
698+
runtime observations, experiment-board projections, countability gates,
699+
and public-safe post-run contracts;
700+
- grants no runner, model, Docker, verifier, upload, submission, or
701+
publication authority.
702+
2. **Native runtime bridge**
703+
- owns the real host transaction used by a supported harness, such as the
704+
native Codex Goal transport and continuation lifecycle;
705+
- keeps process, environment, and task-bridge authority with the runner.
706+
3. **Benchmark adapter**
707+
- owns task provisioning, benchmark-native lifecycle, verifier invocation,
708+
score reduction, checkpoint semantics, and benchmark-specific failure
709+
attribution;
710+
- must import toolkit contracts instead of recreating generic state or
711+
integrity logic.
712+
4. **Study and analysis layer**
713+
- owns manifests, preregistered arm selection, paired comparisons,
714+
uncertainty, and offline interpretation;
715+
- may not grant runtime authority or rewrite benchmark-native results.
716+
717+
Do not create a second active benchmark package, a generic score authority, or a
718+
benchmark-specific capability merely to make a runner installable. A new
719+
module needs a real caller, a named owner, a focused contract, and a validation
720+
surface in the same change or an explicitly staged follow-up.
721+
722+
### 11.2 Engineering readiness stages
723+
724+
Use these stages in PRs, contributor tasks, and pilot updates. They are delivery
725+
gates, not release promises:
726+
727+
- **E0 -- Contract and boundary foundation**
728+
- typed schema and ownership decision recorded;
729+
- public/private and authority boundaries have negative coverage;
730+
- no live benchmark run is required.
731+
- **E1 -- Native runner conformance**
732+
- a pinned public-safe runner path can preflight, execute, verify, and reduce
733+
one synthetic or approved task slice;
734+
- source, runtime, treatment, and integrity receipts are emitted without raw
735+
task material;
736+
- setup, solver, verifier, and scoring failures remain distinguishable.
737+
- **E2 -- Campaign operations**
738+
- experiment-board identity, concurrency admission, exact-job observation,
739+
runtime continuity, terminal closeout, and safe reconciliation work across
740+
repeated transitions;
741+
- a provider can recover or quarantine an invalid run without changing the
742+
score authority.
743+
- **E3 -- Matched-study readiness**
744+
- baseline, passive, governed, and ablation arms have a declared manifest;
745+
- task/model/harness/budget identity, treatment delivery, integrity, and
746+
countability are independently checked;
747+
- paired analysis and stopping rules are registered before outcome inspection.
748+
- **E4 -- Replication and promotion readiness**
749+
- a named mechanism reproduces across a second benchmark family or has a
750+
documented null result;
751+
- protocol tax, uncertainty, authority risk, and public/private handling are
752+
reviewed;
753+
- non-benchmark product qualification and maintainer promotion remain
754+
separate required gates.
755+
756+
The current repository is strongest at E0--E2. E3 is the next engineering and
757+
research target; E4 remains a future promotion gate. A benchmark adapter may be
758+
technically E1-ready while its study is still C0, and a C2 result is not valid
759+
when its E2 runtime evidence is incomplete.
760+
761+
### 11.3 Required delivery slice
762+
763+
Every benchmark engineering PR or contributor task should identify a bounded
764+
slice containing:
765+
766+
- the caller outcome and owning layer;
767+
- the schema or transition contract, including illegal states;
768+
- one real or synthetic call site;
769+
- characterization and negative or mutation coverage;
770+
- the compact public-safe receipt or projection;
771+
- the private-input boundary and artifact classification;
772+
- the exact validation commands and expected disposition;
773+
- documentation that links to this RFC without copying benchmark-specific truth
774+
into generic toolkit docs.
775+
776+
Prefer one complete vertical seam over a broad inactive framework. For example,
777+
an adapter should first prove one native preflight and one official-result
778+
reduction before adding a general campaign scheduler, a large ledger, or a
779+
benchmark-family abstraction with no caller.
780+
781+
### 11.4 Community contribution routing
782+
783+
Community contributors can work safely on public synthetic slices and reusable
784+
contracts:
785+
786+
- adapter lifecycle and result-reduction fixtures;
787+
- integrity, source, artifact, and authority-parity negative tests;
788+
- experiment-board transitions, reconciliation, and concurrency reducers;
789+
- public-safe trajectory and case-insight projections;
790+
- manifest validation, matched-pair analysis, uncertainty, and protocol-tax
791+
tooling;
792+
- benchmark-maintainer-friendly trace or runner conformance fixes;
793+
- documentation and examples that state claim level and non-goals.
794+
795+
Live tasks, hidden verifiers, raw trajectories, credentials, official
796+
submissions, unpublished comparisons, and leaderboard operations remain
797+
maintainer-owned. A contributor task must state its target base branch, smallest
798+
useful slice, non-goals, and validation plan; an RFC or direction tracker alone
799+
does not authorize implementation.
800+
801+
### 11.5 Engineering readiness review
802+
803+
Before calling a slice ready, reviewers should answer:
804+
805+
1. Is the behavior owned by the toolkit, native runtime, benchmark adapter, or
806+
offline evaluator, and is that ownership visible in the code layout?
807+
2. Does a real caller exercise the new contract, or should the proposal remain
808+
an RFC/fixture instead of production structure?
809+
3. Can an invalid, incomplete, or authority-mismatched run fail closed without
810+
leaking private data or silently becoming countable?
811+
4. Are public receipts sufficient to explain classification without copying raw
812+
evidence?
813+
5. Are benchmark-native score and LoopX control observations still separate?
814+
6. Does the validation cover the negative path and the next likely mutation?
815+
7. What is the highest C-level claim this engineering can support, and what
816+
evidence is still missing?
817+
818+
This review is deliberately part of ordinary engineering hygiene. It does not
819+
authorize a live run, a leaderboard submission, a capability promotion, or a
820+
change to LoopX defaults.
821+
822+
## 12. Research Program Milestones
680823

681824
### M0: RFC and source registry
682825

@@ -722,7 +865,7 @@ audited by benchmark maintainers.
722865
- evaluate capability candidates on held-out tasks;
723866
- require maintainer review and non-benchmark canaries before promotion.
724867

725-
## 12. Acceptance Criteria
868+
## 13. Acceptance Criteria
726869

727870
This RFC is successful when:
728871

@@ -745,7 +888,7 @@ This RFC is successful when:
745888
14. production promotion requires product qualification beyond benchmark
746889
evidence.
747890

748-
## 13. Non-Goals
891+
## 14. Non-Goals
749892

750893
- One universal long-horizon score or leaderboard.
751894
- Replacing benchmark-native harnesses, graders, or submission rules.
@@ -757,7 +900,7 @@ This RFC is successful when:
757900
- Requiring ALE, LHTB, and DeepSWE to share one runner implementation.
758901
- Claiming model capability when the experiment changed the harness.
759902

760-
## 14. Risks and Mitigations
903+
## 15. Risks and Mitigations
761904

762905
| Risk | Mitigation |
763906
|---|---|
@@ -774,7 +917,7 @@ This RFC is successful when:
774917
| One mechanism bundles several behavior changes | Single-mechanism ablations before combined profiles. |
775918
| Research policy gains production authority | Offline evaluator and explicit maintainer promotion gate. |
776919

777-
## 15. Open Research Questions
920+
## 16. Open Research Questions
778921

779922
1. Which benchmark-native checkpoints are frequent enough for causal analysis
780923
without changing agent behavior?
@@ -793,7 +936,7 @@ This RFC is successful when:
793936
8. How should benchmark maintainers and harness researchers share trace schemas
794937
without standardizing away meaningful harness differences?
795938

796-
## 16. RFC Maintenance Protocol
939+
## 17. RFC Maintenance Protocol
797940

798941
This is a living research RFC, not a frozen benchmark snapshot.
799942

@@ -818,8 +961,9 @@ This is a living research RFC, not a frozen benchmark snapshot.
818961
|---|---|
819962
| 2026-08-16 | Adopt ALE, LHTB, and DeepSWE as a complementary initial portfolio; separate capability evidence from mechanism research; require native outcomes and typed treatment integrity. |
820963
| 2026-08-16 | Make the current DeepSWE pilot the design-driving practice for the LoopX benchmark capability; treat legacy code/research as candidate reuse and add structured anti-cheating and authority-parity qualification. |
964+
| 2026-08-23 | Add an engineering readiness ladder and bounded delivery-slice rules so repository construction is tracked separately from C0--C4 research claims. |
821965

822-
## 17. References
966+
## 18. References
823967

824968
- [Agents' Last Exam project](https://agents-last-exam.org/)
825969
- [Agents' Last Exam paper](https://arxiv.org/abs/2606.05405)

0 commit comments

Comments
 (0)