diff --git a/CATALOG.md b/CATALOG.md index c1ed86d..4c832ed 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -1,7 +1,7 @@ # Current eval catalog and evidence -This is the sole current corpus surface: **46 included cells** (15 model-free, +This is the sole current corpus surface: **47 included cells** (16 model-free, 31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run history: a failed last run is visible without being advertised as accepted evidence, and cells with no @@ -16,6 +16,7 @@ harness-native loader plus canonical hook file. | Cell | Harness | Model(s) / effort | Model seats | Cost | Timeout | Held-out judges | Latest accepted PASS | Last recorded run | |---|---|---|---:|---|---|---:|---|---| +| `adopt-only-migration` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** | | `context-resource-continuity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** | | `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 | — | **NO STRUCTURED RUN** | | `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** | @@ -82,6 +83,7 @@ while the last-run column makes a recorded failure distinct from a cell with no ## Harness hook exclusions +- `adopt-only-migration` / `cell` — deterministic native adoption/replacement lifecycle probe; no harness seat - `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat - `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface - `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat @@ -99,7 +101,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent - `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat -These **16** rows are generated from structured exclusions. A cell-level row is required for +These **17** rows are generated from structured exclusions. A cell-level row is required for every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected agent: it has no st2 identity, DING sidecar, or lifecycle hooks. diff --git a/cells/adopt-only-migration/README.md b/cells/adopt-only-migration/README.md new file mode 100644 index 0000000..3354050 --- /dev/null +++ b/cells/adopt-only-migration/README.md @@ -0,0 +1,49 @@ +# adopt-only-migration + +**Type:** st2 / live-migration lifecycle · **Runtime dependency:** implemented +by merged [compoundingtech/st2#99](https://github.com/compoundingtech/st2/pull/99), +which closed [compoundingtech/st2#98](https://github.com/compoundingtech/st2/issues/98), +and tracked in evals by +[compoundingtech/evals#52](https://github.com/compoundingtech/evals/issues/52). + +**Capabilities required:** `st2,pty,jq`. No model and no bus. Every declaration, +process, PTY record, state file, log, and receipt is synthetic and rooted below +the eval-owned catalog. + +**Discriminates:** can an operator publish a declaration that adopts one +already-live process generation without granting authority to create, collect, +or replace it? An absent task and a later-dead adopted task must both remain +held until the declaration explicitly transitions to ordinary service +lifecycle. + +## What it proves + +- **Live adoption:** a pre-existing PTY is adopted with the same daemon and + child process generation. +- **Absent hold:** an absent adopt-only task never executes its declared launch + command. +- **Exited hold:** once the adopted generation exits, reconciliation retains + its backend record and does not cold-launch a successor. +- **Explicit replacement:** removing the adopt-only lifecycle changes the + desired contract; ordinary reconciliation may then collect and replace the + exited generation. +- **Mutation-valid control:** an ordinary absent service task launches on the + first pass, proving the launch substrate is real. +- **Isolation and cleanup:** the cell uses only its temporary catalog and PTY + root and leaves no live synthetic processes. + +## Run it + +```sh +st2 eval ./cells/adopt-only-migration/ +``` + +The current cell passes 7/7 against merged st2 +`c6846f6239329f0803142afc06c15a07b93937c1`. Historically, it was intentionally +RED before #98/#99 because the runtime treated `lifecycle` as inert metadata; +that pre-implementation result is no longer the current shipping state. + +Immutable pre-reap generation receipts remain the distinct design and +acceptance surface in +[compoundingtech/st2#40](https://github.com/compoundingtech/st2/issues/40). +This cell does not freeze a receipt path or schema ahead of that design. diff --git a/cells/adopt-only-migration/adopt-only-migration.kdl b/cells/adopt-only-migration/adopt-only-migration.kdl new file mode 100644 index 0000000..f9a6382 --- /dev/null +++ b/cells/adopt-only-migration/adopt-only-migration.kdl @@ -0,0 +1,30 @@ +// Adoption-only migration boundary over synthetic tasks in an eval-owned PTY root. +eval { + copy "./fixture" + max-timeout "90s" + + run "matrix" { + command "bash ./matrix.sh" + } + + judges { + judge "LIVE ADOPTION - an existing task keeps its original process generation" { + exec "grep -Fqx LIVE-ADOPTED-UNCHANGED-GREEN-a091 $RUNS_DIR/matrix.out" + } + judge "ABSENT HOLD - an absent adopt-only task is never cold-launched" { + exec "grep -Fqx ABSENT-ADOPT-ONLY-HELD-GREEN-a091 $RUNS_DIR/matrix.out" + } + judge "EXIT HOLD - an adopted task that exits is retained without replacement" { + exec "grep -Fqx EXITED-ADOPTED-GENERATION-HELD-GREEN-a091 $RUNS_DIR/matrix.out" + } + judge "EXPLICIT REPLACEMENT - ordinary lifecycle transition permits a new generation" { + exec "grep -Fqx EXPLICIT-ORDINARY-REPLACEMENT-GREEN-a091 $RUNS_DIR/matrix.out" + } + judge "ORDINARY CONTROL - an absent ordinary service task cold-launches" { + exec "grep -Fqx ORDINARY-MISSING-LAUNCH-CONTROL-GREEN-a091 $RUNS_DIR/matrix.out" + } + judge "CLEANUP - no synthetic task remains live in the eval root" { + exec "grep -Fqx SYNTHETIC-MIGRATION-ROOT-CLEAN-GREEN-a091 $RUNS_DIR/matrix.out" + } + } +} diff --git a/cells/adopt-only-migration/fixture/matrix.sh b/cells/adopt-only-migration/fixture/matrix.sh new file mode 100755 index 0000000..948d6b4 --- /dev/null +++ b/cells/adopt-only-migration/fixture/matrix.sh @@ -0,0 +1,157 @@ +#!/usr/bin/env bash +set -euo pipefail + +root="${CATALOG:?CATALOG must be set}" +net="$root/net" +export PTY_ROOT="$net/pty" +export XDG_STATE_HOME="$root/state" + +live_id="migrate.live.agent" +absent_id="migrate.absent.agent" +control_id="migrate.control.agent" +live_spec="$net/agents/migrate/live/agent.kdl" + +pty_at() { + env -u PTY_SESSION PTY_ROOT="$PTY_ROOT" pty "$@" +} + +session() { + pty_at list --json | jq -c --arg id "$1" '.[] | select(.name == $id)' +} + +status_of() { + session "$1" | jq -r '.status // empty' +} + +pid_of() { + session "$1" | jq -r '.pid // empty' +} + +run_count() { + file="$net/$1-count" + test -f "$file" || { + printf '0\n' + return + } + cat "$file" +} + +wait_status() { + id="$1" + expected="$2" + for _ in $(seq 1 100); do + status="$(status_of "$id")" + test "$status" = "$expected" && return 0 + sleep 0.05 + done + printf '%s did not reach status %s (last status: %s)\n' \ + "$id" "$expected" "${status:-missing}" >&2 + return 1 +} + +wait_count() { + label="$1" + expected="$2" + for _ in $(seq 1 100); do + test "$(run_count "$label")" -eq "$expected" && return 0 + sleep 0.05 + done + printf '%s did not reach generation %s\n' "$label" "$expected" >&2 + return 1 +} + +retire() { + spec="$1" + grep -Fq 'retired #true' "$spec" || + sed -i '/role "worker"/a\\ retired #true' "$spec" +} + +cleanup() { + for identity in live absent control; do + retire "$net/agents/migrate/$identity/agent.kdl" 2>/dev/null || true + done + st2 up --once --catalog "$net" --host migrate >/dev/null 2>&1 || true + sleep 0.1 + st2 up --once --catalog "$net" --host migrate >/dev/null 2>&1 || true + for id in "$live_id" "$absent_id" "$control_id"; do + pty_at kill "$id" >/dev/null 2>&1 || true + pty_at rm "$id" >/dev/null 2>&1 || true + done +} +trap cleanup EXIT + +# Seed exactly one pre-existing process generation before st2 sees the +# declaration. The declared launch command is therefore observable but must not +# execute during adoption. +mkdir -p "$PTY_ROOT" +CATALOG="$net" pty_at run -d --id "$live_id" --tag keep=true \ + --cwd "$net/workspace" -- bash "$net/task.sh" live >/dev/null +wait_status "$live_id" running +wait_count live 1 +live_pid_before="$(pid_of "$live_id")" +test -n "$live_pid_before" + +st2 up --once --catalog "$net" --host migrate >"$root/first-pass.out" + +if + test "$(status_of "$live_id")" = running && + test "$(pid_of "$live_id")" = "$live_pid_before" && + test "$(run_count live)" -eq 1 && + grep -Fq 'adopted (1): live' "$root/first-pass.out" +then + echo "LIVE-ADOPTED-UNCHANGED-GREEN-a091" +fi + +if + test "$(run_count absent)" -eq 0 && + test -z "$(session "$absent_id")" && + grep -Fq 'held (1): migrate.absent.agent' "$root/first-pass.out" +then + echo "ABSENT-ADOPT-ONLY-HELD-GREEN-a091" +fi + +if + wait_count control 1 && + test "$(status_of "$control_id")" = running && + grep -Fq "$control_id" "$root/first-pass.out" +then + echo "ORDINARY-MISSING-LAUNCH-CONTROL-GREEN-a091" +fi + +# The adopted process exits. Adopt-only must retain the dead record and refuse +# both collection and replacement. +pty_at kill "$live_id" >/dev/null +wait_status "$live_id" exited +live_record_before="$(session "$live_id")" +test "$(run_count live)" -eq 1 + +st2 up --once --catalog "$net" --host migrate >"$root/held-pass.out" + +if + test "$(run_count live)" -eq 1 && + test "$(status_of "$live_id")" = exited && + test "$(session "$live_id")" = "$live_record_before" && + grep -Fq 'held (2): migrate.absent.agent, migrate.live.agent' "$root/held-pass.out" +then + echo "EXITED-ADOPTED-GENERATION-HELD-GREEN-a091" +fi + +# This is the explicit authorization edge. The declaration becomes an ordinary +# service task; only now may st2 reap the exited record and cold-launch. +sed -i 's/lifecycle "adopt-only"/lifecycle "service"/' "$live_spec" +st2 up --once --catalog "$net" --host migrate >"$root/replacement-pass.out" + +if + wait_count live 2 && + test "$(status_of "$live_id")" = running && + test "$(pid_of "$live_id")" != "$live_pid_before" && + grep -Fq "$live_id" "$root/replacement-pass.out" && + test "$(run_count absent)" -eq 0 +then + echo "EXPLICIT-ORDINARY-REPLACEMENT-GREEN-a091" +fi + +cleanup +trap - EXIT +test "$(pty_at list --json | jq 'length')" -eq 0 +echo "SYNTHETIC-MIGRATION-ROOT-CLEAN-GREEN-a091" diff --git a/cells/adopt-only-migration/fixture/net/agents/migrate/absent/agent.kdl b/cells/adopt-only-migration/fixture/net/agents/migrate/absent/agent.kdl new file mode 100644 index 0000000..f2777fe --- /dev/null +++ b/cells/adopt-only-migration/fixture/net/agents/migrate/absent/agent.kdl @@ -0,0 +1,10 @@ +agent "absent" { + host "migrate" + role "worker" + workspace "$CATALOG/workspace" + pty "agent" { + id "migrate.absent.agent" + command "exec bash \"$CATALOG/task.sh\" absent" + lifecycle "adopt-only" + } +} diff --git a/cells/adopt-only-migration/fixture/net/agents/migrate/control/agent.kdl b/cells/adopt-only-migration/fixture/net/agents/migrate/control/agent.kdl new file mode 100644 index 0000000..94a2bc4 --- /dev/null +++ b/cells/adopt-only-migration/fixture/net/agents/migrate/control/agent.kdl @@ -0,0 +1,9 @@ +agent "control" { + host "migrate" + role "worker" + workspace "$CATALOG/workspace" + pty "agent" { + id "migrate.control.agent" + command "exec bash \"$CATALOG/task.sh\" control" + } +} diff --git a/cells/adopt-only-migration/fixture/net/agents/migrate/live/agent.kdl b/cells/adopt-only-migration/fixture/net/agents/migrate/live/agent.kdl new file mode 100644 index 0000000..6e27122 --- /dev/null +++ b/cells/adopt-only-migration/fixture/net/agents/migrate/live/agent.kdl @@ -0,0 +1,10 @@ +agent "live" { + host "migrate" + role "worker" + workspace "$CATALOG/workspace" + pty "agent" { + id "migrate.live.agent" + command "exec bash \"$CATALOG/task.sh\" live" + lifecycle "adopt-only" + } +} diff --git a/cells/adopt-only-migration/fixture/net/task.sh b/cells/adopt-only-migration/fixture/net/task.sh new file mode 100755 index 0000000..c415a5a --- /dev/null +++ b/cells/adopt-only-migration/fixture/net/task.sh @@ -0,0 +1,12 @@ +#!/usr/bin/env bash +set -euo pipefail + +label="${1:?label is required}" +count_file="${CATALOG:?CATALOG must be set}/$label-count" +generation=1 +if test -f "$count_file"; then + generation=$(( $(cat "$count_file") + 1 )) +fi +printf '%s\n' "$generation" >"$count_file" +printf '%s-GENERATION-%s-a091\n' "$label" "$generation" +sleep 100000 diff --git a/cells/adopt-only-migration/fixture/net/workspace/.keep b/cells/adopt-only-migration/fixture/net/workspace/.keep new file mode 100644 index 0000000..ee8c1ee --- /dev/null +++ b/cells/adopt-only-migration/fixture/net/workspace/.keep @@ -0,0 +1 @@ +fixture diff --git a/docs/vrs/spec.md b/docs/vrs/spec.md index d3a66d4..32edcc6 100644 --- a/docs/vrs/spec.md +++ b/docs/vrs/spec.md @@ -39,6 +39,10 @@ tracked cell - An LLM judge is not a defect. Its provider, model, prompt, inputs, and failure behavior must be explicit, and mutation checks must demonstrate useful discrimination. +- The model-free `adopt-only-migration` cell separates process-generation + adoption from replacement authority: live generations are adopted, dead or + absent migration tasks are held without mutation, and only an explicit + lifecycle transition permits ordinary replacement. ## Current execution diff --git a/evidence/harness-exclusions.tsv b/evidence/harness-exclusions.tsv index 6de1f5e..e976ad5 100644 --- a/evidence/harness-exclusions.tsv +++ b/evidence/harness-exclusions.tsv @@ -1,4 +1,5 @@ cell surface reason +adopt-only-migration cell deterministic native adoption/replacement lifecycle probe; no harness seat context-resource-continuity cell deterministic native context/resource restart probe; no harness seat docs judge:cold-reader one-shot offline Claude print grader; no bus identity, DING, or hook surface exec-lifecycle-logging cell deterministic native exec lifecycle probe; no harness seat