You signed in with another tab or window. Reload to refresh your session.You signed out in another tab or window. Reload to refresh your session.You switched accounts on another tab or window. Reload to refresh your session.Dismiss alert
{{ message }}
Repository navigation
Commit fc101a5
Browse filesBrowse the repository at this point in the historyBrowse files
docs(benchmark): add engineering construction plan
* docs(benchmark): add engineering construction plan
Signed-off-by: huangruiteng <huangrt01@163.com>
* docs(benchmark): sync engineering plan in Chinese RFC
Signed-off-by: huangruiteng <huangrt01@163.com>
---------
Signed-off-by: huangruiteng <huangrt01@163.com>
runtime continuity, terminal closeout, and safe reconciliation work across
740
+
repeated transitions;
741
+
- a provider can recover or quarantine an invalid run without changing the
742
+
score authority.
743
+
-**E3 -- Matched-study readiness**
744
+
- baseline, passive, governed, and ablation arms have a declared manifest;
745
+
- task/model/harness/budget identity, treatment delivery, integrity, and
746
+
countability are independently checked;
747
+
- paired analysis and stopping rules are registered before outcome inspection.
748
+
-**E4 -- Replication and promotion readiness**
749
+
- a named mechanism reproduces across a second benchmark family or has a
750
+
documented null result;
751
+
- protocol tax, uncertainty, authority risk, and public/private handling are
752
+
reviewed;
753
+
- non-benchmark product qualification and maintainer promotion remain
754
+
separate required gates.
755
+
756
+
The current repository is strongest at E0--E2. E3 is the next engineering and
757
+
research target; E4 remains a future promotion gate. A benchmark adapter may be
758
+
technically E1-ready while its study is still C0, and a C2 result is not valid
759
+
when its E2 runtime evidence is incomplete.
760
+
761
+
### 11.3 Required delivery slice
762
+
763
+
Every benchmark engineering PR or contributor task should identify a bounded
764
+
slice containing:
765
+
766
+
- the caller outcome and owning layer;
767
+
- the schema or transition contract, including illegal states;
768
+
- one real or synthetic call site;
769
+
- characterization and negative or mutation coverage;
770
+
- the compact public-safe receipt or projection;
771
+
- the private-input boundary and artifact classification;
772
+
- the exact validation commands and expected disposition;
773
+
- documentation that links to this RFC without copying benchmark-specific truth
774
+
into generic toolkit docs.
775
+
776
+
Prefer one complete vertical seam over a broad inactive framework. For example,
777
+
an adapter should first prove one native preflight and one official-result
778
+
reduction before adding a general campaign scheduler, a large ledger, or a
779
+
benchmark-family abstraction with no caller.
780
+
781
+
### 11.4 Community contribution routing
782
+
783
+
Community contributors can work safely on public synthetic slices and reusable
784
+
contracts:
785
+
786
+
- adapter lifecycle and result-reduction fixtures;
787
+
- integrity, source, artifact, and authority-parity negative tests;
788
+
- experiment-board transitions, reconciliation, and concurrency reducers;
789
+
- public-safe trajectory and case-insight projections;
790
+
- manifest validation, matched-pair analysis, uncertainty, and protocol-tax
791
+
tooling;
792
+
- benchmark-maintainer-friendly trace or runner conformance fixes;
793
+
- documentation and examples that state claim level and non-goals.
794
+
795
+
Live tasks, hidden verifiers, raw trajectories, credentials, official
796
+
submissions, unpublished comparisons, and leaderboard operations remain
797
+
maintainer-owned. A contributor task must state its target base branch, smallest
798
+
useful slice, non-goals, and validation plan; an RFC or direction tracker alone
799
+
does not authorize implementation.
800
+
801
+
### 11.5 Engineering readiness review
802
+
803
+
Before calling a slice ready, reviewers should answer:
804
+
805
+
1. Is the behavior owned by the toolkit, native runtime, benchmark adapter, or
806
+
offline evaluator, and is that ownership visible in the code layout?
807
+
2. Does a real caller exercise the new contract, or should the proposal remain
808
+
an RFC/fixture instead of production structure?
809
+
3. Can an invalid, incomplete, or authority-mismatched run fail closed without
810
+
leaking private data or silently becoming countable?
811
+
4. Are public receipts sufficient to explain classification without copying raw
812
+
evidence?
813
+
5. Are benchmark-native score and LoopX control observations still separate?
814
+
6. Does the validation cover the negative path and the next likely mutation?
815
+
7. What is the highest C-level claim this engineering can support, and what
816
+
evidence is still missing?
817
+
818
+
This review is deliberately part of ordinary engineering hygiene. It does not
819
+
authorize a live run, a leaderboard submission, a capability promotion, or a
820
+
change to LoopX defaults.
821
+
822
+
## 12. Research Program Milestones
680
823
681
824
### M0: RFC and source registry
682
825
@@ -722,7 +865,7 @@ audited by benchmark maintainers.
722
865
- evaluate capability candidates on held-out tasks;
723
866
- require maintainer review and non-benchmark canaries before promotion.
724
867
725
-
## 12. Acceptance Criteria
868
+
## 13. Acceptance Criteria
726
869
727
870
This RFC is successful when:
728
871
@@ -745,7 +888,7 @@ This RFC is successful when:
745
888
14. production promotion requires product qualification beyond benchmark
746
889
evidence.
747
890
748
-
## 13. Non-Goals
891
+
## 14. Non-Goals
749
892
750
893
- One universal long-horizon score or leaderboard.
751
894
- Replacing benchmark-native harnesses, graders, or submission rules.
@@ -757,7 +900,7 @@ This RFC is successful when:
757
900
- Requiring ALE, LHTB, and DeepSWE to share one runner implementation.
758
901
- Claiming model capability when the experiment changed the harness.
759
902
760
-
## 14. Risks and Mitigations
903
+
## 15. Risks and Mitigations
761
904
762
905
| Risk | Mitigation |
763
906
|---|---|
@@ -774,7 +917,7 @@ This RFC is successful when:
774
917
| One mechanism bundles several behavior changes | Single-mechanism ablations before combined profiles. |
775
918
| Research policy gains production authority | Offline evaluator and explicit maintainer promotion gate. |
776
919
777
-
## 15. Open Research Questions
920
+
## 16. Open Research Questions
778
921
779
922
1. Which benchmark-native checkpoints are frequent enough for causal analysis
780
923
without changing agent behavior?
@@ -793,7 +936,7 @@ This RFC is successful when:
793
936
8. How should benchmark maintainers and harness researchers share trace schemas
794
937
without standardizing away meaningful harness differences?
795
938
796
-
## 16. RFC Maintenance Protocol
939
+
## 17. RFC Maintenance Protocol
797
940
798
941
This is a living research RFC, not a frozen benchmark snapshot.
799
942
@@ -818,8 +961,9 @@ This is a living research RFC, not a frozen benchmark snapshot.
818
961
|---|---|
819
962
| 2026-08-16 | Adopt ALE, LHTB, and DeepSWE as a complementary initial portfolio; separate capability evidence from mechanism research; require native outcomes and typed treatment integrity. |
820
963
| 2026-08-16 | Make the current DeepSWE pilot the design-driving practice for the LoopX benchmark capability; treat legacy code/research as candidate reuse and add structured anti-cheating and authority-parity qualification. |
964
+
| 2026-08-23 | Add an engineering readiness ladder and bounded delivery-slice rules so repository construction is tracked separately from C0--C4 research claims. |
821
965
822
-
## 17. References
966
+
## 18. References
823
967
824
968
-[Agents' Last Exam project](https://agents-last-exam.org/)
825
969
-[Agents' Last Exam paper](https://arxiv.org/abs/2606.05405)
0 commit comments