Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
Show all changes
47 commits
Select commit Hold shift + click to select a range
1868ade
docs(branch): add composed snapshotting + rollout branching RFC
JeremyJC67 Aug 15, 2026
32f0d02
feat(branch): compose sandbox + environment checkpoints in the branch…
JeremyJC67 Aug 15, 2026
1a9575f
fix(branch): review fixes — legacy restore rejects StageSnapshot cleanly
JeremyJC67 Aug 15, 2026
8171df5
feat(branch): per-child deltas + branch lineage artifacts
JeremyJC67 Aug 15, 2026
2360fca
fix(branch): review fixes — branch-first result semantics, per-node d…
JeremyJC67 Aug 15, 2026
652623e
feat(continue): replay cut-point — replay first K exchanges, then go …
JeremyJC67 Aug 15, 2026
30bc4ff
test(branch): docker-backed zero-delta round-trip proofs for composed…
JeremyJC67 Aug 15, 2026
1f8335d
fix(continue): review fixes — parsed-prefix stitching, served-cut rec…
JeremyJC67 Aug 15, 2026
39416af
feat(branch): stage-boundary snapshot policy for rollout branching
JeremyJC67 Aug 18, 2026
a207d4a
feat(branch): execute skill_mode deltas as fresh rollouts from the en…
JeremyJC67 Aug 18, 2026
5a622d6
feat(cli): bench eval ablate — stage-level ablation over branch children
JeremyJC67 Aug 18, 2026
741056a
fix(ablate): deploy a branch child's skills by upload, not Dockerfile
JeremyJC67 Aug 18, 2026
b68e363
fix(branch): unscored branch children must not report a fabricated re…
JeremyJC67 Aug 18, 2026
feed07d
fix(sandbox): restore replays the original container bind mounts
JeremyJC67 Aug 18, 2026
6aeb846
fix(ablate): run every env-ready branch child as a fresh rollout
JeremyJC67 Aug 18, 2026
fe9a2bf
fix(branch): branch children no longer clobber the parent's artifacts
JeremyJC67 Aug 18, 2026
40d8341
fix(branch): gate skill deltas on the parent's own skill mode
JeremyJC67 Aug 20, 2026
3c7b2e0
fix(branch): clear a stale branch value when V is undefined
JeremyJC67 Aug 20, 2026
dc419ba
fix(sandbox): restore fails closed when the container cannot be inspe…
JeremyJC67 Aug 20, 2026
db6d6df
fix(branch): stage branches enforce the non-empty layer check
JeremyJC67 Aug 20, 2026
1c65f68
feat(ablate): per-test attribution so a scalar tie cannot hide a beha…
JeremyJC67 Aug 21, 2026
ec26b56
style: apply ruff format to the branch-and-ablation files
JeremyJC67 Aug 21, 2026
4c4936c
fix(ablate): read per-test outcomes from preserved in-place child art…
JeremyJC67 Aug 21, 2026
8fb94fc
fix(branch): restore result-bearing rollout state after in-place chil…
JeremyJC67 Aug 21, 2026
c348fac
fix(branch): persist partial lineage when a branch child raises
JeremyJC67 Aug 21, 2026
8d69976
fix(ablate): resolve a task-declared environment manifest for parent …
JeremyJC67 Aug 21, 2026
d17d01c
fix(continue): stitch and account for the exchanges actually served
JeremyJC67 Aug 21, 2026
3245e2d
feat(branch): execute config_override deltas as fresh rollouts from e…
JeremyJC67 Aug 21, 2026
5a17e93
feat(branch): execute service-level environment_ref deltas from env-r…
JeremyJC67 Aug 21, 2026
5b0584c
feat(branch): synthesize a full result.json for in-place branch children
JeremyJC67 Aug 21, 2026
9729fe5
feat(ablate): bind an explicit environment manifest and stamp the bou…
JeremyJC67 Aug 21, 2026
bccd797
fix(acp): skip doomed set_model when codex launch config owns an off-…
JeremyJC67 Aug 21, 2026
beac03e
fix(ablate): track JSON string state in the arm splitter
JeremyJC67 Aug 23, 2026
fe57a12
fix(branch): artifact custody fails closed and preserves the hold dir…
JeremyJC67 Aug 28, 2026
5edcb86
fix(continue): replay divergence detection compares actual content an…
JeremyJC67 Aug 28, 2026
4bde66a
feat(ablate): --keep-snapshots export; ephemeral handles recorded tru…
JeremyJC67 Aug 28, 2026
6761a60
fix(ablate): resolve the canonical eval plan and overlay only the abl…
JeremyJC67 Aug 28, 2026
d7ff4a1
refactor(branch): split policy, transaction, delta execution, and rep…
JeremyJC67 Aug 28, 2026
4ef704d
feat(branch): record stage markers with trajectory exchange indices
JeremyJC67 Aug 28, 2026
99502ff
feat(continue): stage-named cut points
JeremyJC67 Aug 28, 2026
4850761
feat(ablate): post-research capture via a research-end marker file
JeremyJC67 Aug 28, 2026
92ad5ab
feat(task): accept branch_execution forked-snapshot now the engine su…
JeremyJC67 Aug 28, 2026
b73afe1
fix(sandbox): workspace digests are null-safe and fail closed
JeremyJC67 Aug 29, 2026
af67ff4
fix(ablate): validate request-global settings before provisioning; ne…
JeremyJC67 Aug 29, 2026
a61e6d1
feat(rollout): stage snapshots record their lifetime; --keep-snapshot…
JeremyJC67 Aug 29, 2026
cfb699f
refactor(ablate): arm parsing and report assembly move out; run_ablat…
JeremyJC67 Aug 29, 2026
72cb8cd
docs(branch): architecture and RFC text match the implemented composi…
JeremyJC67 Aug 29, 2026
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 2 additions & 2 deletions docs/architecture.md
Original file line number Diff line number Diff line change
Expand Up @@ -99,13 +99,13 @@ Every lifecycle the framework owns, as ordered phases.

**Rollout lifecycle.** `setup` (resolve config, build the environment object) → `start` (sandbox up) → `provision environment` (Environment plane starts services) → `readiness gate` (framework-guaranteed; the agent never runs before the world is healthy) → `connect agent` (ACP) → `execute` (the tree grows: Steps and Branches) → `verify` (Reward plane scores) → `teardown`.

**Branch lifecycle.** `quiesce` (disconnect the active agent) → `checkpoint` (snapshot Environment state) → `fork` (N children) → for each child, `restore` the Environment and start a fresh agent session → `score / aggregate` (per-child return → `V(parent)`) → restore the parent's linear rollout state. Agent-session snapshotting is not implemented. Callers that set `require_sandbox_snapshot=True` also require a sandbox with snapshot capability, but the current branch engine still uses the Environment snapshot as its restore point.
**Branch lifecycle.** `quiesce` (disconnect the active agent) → `checkpoint` (snapshot the requested layers — Environment state by default; adding `"sandbox"` to `snapshot_layers` composes a container snapshot with it, environment first) → `fork` (N children) → for each child, `restore` the composed checkpoint (container first, then environment state) and start a fresh agent session (an `env-ready` child runs as a fresh rollout that re-installs the agent for itself) → `score / aggregate` (per-child return → `V(parent)`) → restore the parent's linear rollout state. Agent-session snapshotting is not implemented. `require_sandbox_snapshot=True` keeps its original check-only semantics — it gates on sandbox snapshot capability without composing the layer; requesting the layer through `snapshot_layers` gates the same way and then actually composes the container snapshot into the checkpoint.

**Environment lifecycle** (Han's roll-out / roll-back). `provision` → `readiness` → `query` (expose state to the verifier) → `snapshot` → `restore` → `reset` → `teardown`. `snapshot`/`restore` are definitional — the substrate every `Branch` runs on.

**Sandbox lifecycle.** configure `expose_ports` → `start` → `exec` / `upload_file` / `upload_dir` / `download_file` / `download_dir` → optional `snapshot` / `restore` → `stop`.

A Rollout branch currently rolls back Environment state and starts a fresh agent session for every child. Container and agent-session checkpoint composition remain future work. The one store that deliberately does **not** roll back with a `Branch` is the continual-learning learner store (capability 5).
A Rollout branch rolls back the composed checkpoint — Environment state plus, when the fork requested it (`snapshot_layers`), the container layer — and starts a fresh agent session for every child. Container + environment checkpoint composition is implemented; agent-session checkpoint composition remains future work. The one store that deliberately does **not** roll back with a `Branch` is the continual-learning learner store (capability 5).

## The four planes

Expand Down
79 changes: 73 additions & 6 deletions docs/continue-runs.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,10 +13,13 @@ environment and continues its own loop with **no injected prompt**.

## The problem it solves

A finished run keeps nothing of the container — cleanup tears the sandbox down.
What survives on disk is the run folder: `config.json`, `result.json`,
`prompts.json`, and `trajectory/llm_trajectory.jsonl`. So a historical timeout
has only its *trajectory* + the *task*; there is no saved container to restore.
A finished run keeps nothing of the container unless it opted into snapshot
retention (`bench eval run --keep-snapshots`, rollout-branching RFC §3.6) —
cleanup tears the sandbox down and marks any recorded stage refs `ephemeral`
in `stage_snapshots.json`. What survives on disk otherwise is the run folder:
`config.json`, `result.json`, `prompts.json`, and
`trajectory/llm_trajectory.jsonl`. So a historical timeout has only its
*trajectory* + the *task*; there is no saved container to restore.

`bench eval continue` reconstructs the missing state from the trajectory.

Expand Down Expand Up @@ -64,6 +67,8 @@ optional.
| `--require-timeout` | off | Refuse runs whose recorded status isn't a timeout. |
| `--strict-divergence` | off | Abort if replay leaves the original rails. |
| `--replay-only` | off | Rebuild via replay and stop at the cut-point (no live model needed). |
| `--max-exchanges K` | all recorded | Replay only the first K recorded exchanges, then go live ([Cut-points](#cut-points)). |
| `--cut-stage STAGE` | — | Cut at a recorded stage boundary by name, resolving K from the run's `stage_snapshots.json` ([Cut-points](#cut-points)). |

### Models and credentials

Expand All @@ -75,14 +80,76 @@ optional.
host needs that provider's credentials (e.g. `GEMINI_API_KEY`) in its
environment. `--replay-only` skips the live leg entirely.

## Cut-points

By default the proxy replays the **entire** recorded prefix before going live.
`--max-exchanges K` cuts the replay short: the first K recorded exchanges are
replayed, then the proxy switches to the live model exactly as if the recording
had ended there. This is the replay cut-point API from the
[rollout-branching RFC §3.5](./rollout-branching-rfc.md) — replay a trajectory
verbatim up to a stage boundary, then go live, to localize which stage a run
went wrong in.

- `K` must satisfy `1 <= K <= n_recorded`; anything else fails closed before a
sandbox boots.
- The continued run's `source_provenance` gains a `cut_point` block:
`n_replayed_exchanges` plus two request digests named by what they hash —
`served_request_digest` (the request the agent *actually* sent at the cut)
and `recorded_request_digest` (the recorded request it answered for), both
sha256 over the canonical JSON of the comparable projection
`{messages, tools}` (`request_digest_basis` states this in the artifact).
Divergence is checked per replayed exchange on the same basis — a
same-message-count prompt/content/tool change is detected, not only a count
mismatch — and every event (exchange index + both digests) is recorded in
the block's `divergences` list. A divergence annotates rather than aborts
(fidelity caveats are recorded, not hidden — RFC §3.5); `--strict-divergence`
remains the opt-in abort. The block also carries `workspace_digest`: a
deterministic digest of the continuation workspace (`/app` — file contents,
tree and modes) taken as the run crosses the cut into the live leg; when no
live sandbox is reachable at that moment (sandbox proxy mode, a run that
never crossed the cut, a digest failure) the field is `null` and
`workspace_digest_reason` says why — it is never fabricated. The block's
`accounting` field names its basis: in **host** proxy mode the orchestrator
reconciles the block after the run with what the live replay proxy
*actually served* (`accounting: "served"`, plus `configured_max_exchanges`
when a cut was requested — so a run that went live before reaching the
requested cut is visible in artifacts); in **sandbox** proxy mode the
uploaded recording is truncated to the configured prefix and the block
records that basis (`accounting: "configured"`, with the live-only fields
null). A natural-end continuation records the same block, documenting the
end of the recording.
- The stitched `llm_trajectory.jsonl` contains only the replayed prefix (the
first K *parsed* recorded exchanges — a malformed recorded line is never
replayed and never stitched) plus the live suffix.
- Cut-points can be named by **stage** instead of by number:
`--cut-stage <stage>` (e.g. `--cut-stage post-research`) resolves the
exchange index the original run recorded when that stage boundary closed.
A run that captures stage boundaries (`RolloutConfig.snapshot_stages`, or
`Rollout.mark_stage()` for `post-research`) records
`exchanges_completed` per stage in its `stage_snapshots.json`; a cut at
that stage replays exactly that many exchanges. The resolved stage is
recorded as `branch_stage` in the `cut_point` block. Every miss fails
closed with a typed `ReplayCutPointError`: a run with no recorded stages,
an unrecorded stage (the error lists the stages the run *did* record), a
stage recorded without an index (`exchanges_completed: null` — the usage
gateway could not count at capture time), or a stage that closed before
the first exchange. `--cut-stage` and `--max-exchanges` are mutually
exclusive. Through the Python API
(`benchflow.continue_run.orchestrator.continue_run`), an explicit
`stage_tags` mapping (`stage -> 1-based completed-exchange count`)
overrides the recorded registry.

## Limitations and caveats

- **`openhands` only** for now (the proxy seam relies on `LLM_BASE_URL`).
- **Replay fidelity is best-effort.** Replay re-runs the original shell
commands for real; if a command's output diverges from the original
(network, timestamps, nondeterminism), the agent may see a different
observation than recorded. A message-count check warns on divergence
(`--strict-divergence` aborts instead).
observation than recorded. Divergence warns rather than aborts
(`--strict-divergence` aborts instead): through the host replay proxy the
per-exchange check compares content digests of the comparable
`{messages, tools}` projection ([Cut-points](#cut-points)); the in-sandbox
proxy checks message counts only.
- **"Identical output" means a faithful continuation**, not a bit-identical
result — the model samples, and no "original full run" exists past the
timeout. The bar is: the stitched trajectory reads as one continuous run, as
Expand Down
Loading