Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **46 included cells** (15 model-free,
This is the sole current corpus surface: **47 included cells** (16 model-free,
31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
Expand All @@ -16,6 +16,7 @@ harness-native loader plus canonical hook file.

| Cell | Harness | Model(s) / effort | Model seats | Cost | Timeout | Held-out judges | Latest accepted PASS | Last recorded run |
|---|---|---|---:|---|---|---:|---|---|
| `adopt-only-migration` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** |
| `context-resource-continuity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** |
| `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 | — | **NO STRUCTURED RUN** |
| `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -82,6 +83,7 @@ while the last-run column makes a recorded failure distinct from a cell with no

## Harness hook exclusions

- `adopt-only-migration` / `cell` — deterministic native adoption/replacement lifecycle probe; no harness seat
- `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat
- `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface
- `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat
Expand All @@ -99,7 +101,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat

These **16** rows are generated from structured exclusions. A cell-level row is required for
These **17** rows are generated from structured exclusions. A cell-level row is required for
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.

Expand Down
49 changes: 49 additions & 0 deletions cells/adopt-only-migration/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,49 @@
# adopt-only-migration

**Type:** st2 / live-migration lifecycle · **Runtime dependency:** implemented
by merged [compoundingtech/st2#99](https://github.com/compoundingtech/st2/pull/99),
which closed [compoundingtech/st2#98](https://github.com/compoundingtech/st2/issues/98),
and tracked in evals by
[compoundingtech/evals#52](https://github.com/compoundingtech/evals/issues/52).

**Capabilities required:** `st2,pty,jq`. No model and no bus. Every declaration,
process, PTY record, state file, log, and receipt is synthetic and rooted below
the eval-owned catalog.

**Discriminates:** can an operator publish a declaration that adopts one
already-live process generation without granting authority to create, collect,
or replace it? An absent task and a later-dead adopted task must both remain
held until the declaration explicitly transitions to ordinary service
lifecycle.

## What it proves

- **Live adoption:** a pre-existing PTY is adopted with the same daemon and
child process generation.
- **Absent hold:** an absent adopt-only task never executes its declared launch
command.
- **Exited hold:** once the adopted generation exits, reconciliation retains
its backend record and does not cold-launch a successor.
- **Explicit replacement:** removing the adopt-only lifecycle changes the
desired contract; ordinary reconciliation may then collect and replace the
exited generation.
- **Mutation-valid control:** an ordinary absent service task launches on the
first pass, proving the launch substrate is real.
- **Isolation and cleanup:** the cell uses only its temporary catalog and PTY
root and leaves no live synthetic processes.

## Run it

```sh
st2 eval ./cells/adopt-only-migration/
```

The current cell passes 7/7 against merged st2
`c6846f6239329f0803142afc06c15a07b93937c1`. Historically, it was intentionally
RED before #98/#99 because the runtime treated `lifecycle` as inert metadata;
that pre-implementation result is no longer the current shipping state.

Immutable pre-reap generation receipts remain the distinct design and
acceptance surface in
[compoundingtech/st2#40](https://github.com/compoundingtech/st2/issues/40).
This cell does not freeze a receipt path or schema ahead of that design.
30 changes: 30 additions & 0 deletions cells/adopt-only-migration/adopt-only-migration.kdl
Original file line number Diff line number Diff line change
@@ -0,0 +1,30 @@
// Adoption-only migration boundary over synthetic tasks in an eval-owned PTY root.
eval {
copy "./fixture"
max-timeout "90s"

run "matrix" {
command "bash ./matrix.sh"
}

judges {
judge "LIVE ADOPTION - an existing task keeps its original process generation" {
exec "grep -Fqx LIVE-ADOPTED-UNCHANGED-GREEN-a091 $RUNS_DIR/matrix.out"
}
judge "ABSENT HOLD - an absent adopt-only task is never cold-launched" {
exec "grep -Fqx ABSENT-ADOPT-ONLY-HELD-GREEN-a091 $RUNS_DIR/matrix.out"
}
judge "EXIT HOLD - an adopted task that exits is retained without replacement" {
exec "grep -Fqx EXITED-ADOPTED-GENERATION-HELD-GREEN-a091 $RUNS_DIR/matrix.out"
}
judge "EXPLICIT REPLACEMENT - ordinary lifecycle transition permits a new generation" {
exec "grep -Fqx EXPLICIT-ORDINARY-REPLACEMENT-GREEN-a091 $RUNS_DIR/matrix.out"
}
judge "ORDINARY CONTROL - an absent ordinary service task cold-launches" {
exec "grep -Fqx ORDINARY-MISSING-LAUNCH-CONTROL-GREEN-a091 $RUNS_DIR/matrix.out"
}
judge "CLEANUP - no synthetic task remains live in the eval root" {
exec "grep -Fqx SYNTHETIC-MIGRATION-ROOT-CLEAN-GREEN-a091 $RUNS_DIR/matrix.out"
}
}
}
157 changes: 157 additions & 0 deletions cells/adopt-only-migration/fixture/matrix.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,157 @@
#!/usr/bin/env bash
set -euo pipefail

root="${CATALOG:?CATALOG must be set}"
net="$root/net"
export PTY_ROOT="$net/pty"
export XDG_STATE_HOME="$root/state"

live_id="migrate.live.agent"
absent_id="migrate.absent.agent"
control_id="migrate.control.agent"
live_spec="$net/agents/migrate/live/agent.kdl"

pty_at() {
env -u PTY_SESSION PTY_ROOT="$PTY_ROOT" pty "$@"
}

session() {
pty_at list --json | jq -c --arg id "$1" '.[] | select(.name == $id)'
}

status_of() {
session "$1" | jq -r '.status // empty'
}

pid_of() {
session "$1" | jq -r '.pid // empty'
}

run_count() {
file="$net/$1-count"
test -f "$file" || {
printf '0\n'
return
}
cat "$file"
}

wait_status() {
id="$1"
expected="$2"
for _ in $(seq 1 100); do
status="$(status_of "$id")"
test "$status" = "$expected" && return 0
sleep 0.05
done
printf '%s did not reach status %s (last status: %s)\n' \
"$id" "$expected" "${status:-missing}" >&2
return 1
}

wait_count() {
label="$1"
expected="$2"
for _ in $(seq 1 100); do
test "$(run_count "$label")" -eq "$expected" && return 0
sleep 0.05
done
printf '%s did not reach generation %s\n' "$label" "$expected" >&2
return 1
}

retire() {
spec="$1"
grep -Fq 'retired #true' "$spec" ||
sed -i '/role "worker"/a\\ retired #true' "$spec"
}

cleanup() {
for identity in live absent control; do
retire "$net/agents/migrate/$identity/agent.kdl" 2>/dev/null || true
done
st2 up --once --catalog "$net" --host migrate >/dev/null 2>&1 || true
sleep 0.1
st2 up --once --catalog "$net" --host migrate >/dev/null 2>&1 || true
for id in "$live_id" "$absent_id" "$control_id"; do
pty_at kill "$id" >/dev/null 2>&1 || true
pty_at rm "$id" >/dev/null 2>&1 || true
done
}
trap cleanup EXIT

# Seed exactly one pre-existing process generation before st2 sees the
# declaration. The declared launch command is therefore observable but must not
# execute during adoption.
mkdir -p "$PTY_ROOT"
CATALOG="$net" pty_at run -d --id "$live_id" --tag keep=true \
--cwd "$net/workspace" -- bash "$net/task.sh" live >/dev/null
wait_status "$live_id" running
wait_count live 1
live_pid_before="$(pid_of "$live_id")"
test -n "$live_pid_before"

st2 up --once --catalog "$net" --host migrate >"$root/first-pass.out"

if
test "$(status_of "$live_id")" = running &&
test "$(pid_of "$live_id")" = "$live_pid_before" &&
test "$(run_count live)" -eq 1 &&
grep -Fq 'adopted (1): live' "$root/first-pass.out"
then
echo "LIVE-ADOPTED-UNCHANGED-GREEN-a091"
fi

if
test "$(run_count absent)" -eq 0 &&
test -z "$(session "$absent_id")" &&
grep -Fq 'held (1): migrate.absent.agent' "$root/first-pass.out"
then
echo "ABSENT-ADOPT-ONLY-HELD-GREEN-a091"
fi

if
wait_count control 1 &&
test "$(status_of "$control_id")" = running &&
grep -Fq "$control_id" "$root/first-pass.out"
then
echo "ORDINARY-MISSING-LAUNCH-CONTROL-GREEN-a091"
fi

# The adopted process exits. Adopt-only must retain the dead record and refuse
# both collection and replacement.
pty_at kill "$live_id" >/dev/null
wait_status "$live_id" exited
live_record_before="$(session "$live_id")"
test "$(run_count live)" -eq 1

st2 up --once --catalog "$net" --host migrate >"$root/held-pass.out"

if
test "$(run_count live)" -eq 1 &&
test "$(status_of "$live_id")" = exited &&
test "$(session "$live_id")" = "$live_record_before" &&
grep -Fq 'held (2): migrate.absent.agent, migrate.live.agent' "$root/held-pass.out"
then
echo "EXITED-ADOPTED-GENERATION-HELD-GREEN-a091"
fi

# This is the explicit authorization edge. The declaration becomes an ordinary
# service task; only now may st2 reap the exited record and cold-launch.
sed -i 's/lifecycle "adopt-only"/lifecycle "service"/' "$live_spec"
st2 up --once --catalog "$net" --host migrate >"$root/replacement-pass.out"

if
wait_count live 2 &&
test "$(status_of "$live_id")" = running &&
test "$(pid_of "$live_id")" != "$live_pid_before" &&
grep -Fq "$live_id" "$root/replacement-pass.out" &&
test "$(run_count absent)" -eq 0
then
echo "EXPLICIT-ORDINARY-REPLACEMENT-GREEN-a091"
fi

cleanup
trap - EXIT
test "$(pty_at list --json | jq 'length')" -eq 0
echo "SYNTHETIC-MIGRATION-ROOT-CLEAN-GREEN-a091"
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
agent "absent" {
host "migrate"
role "worker"
workspace "$CATALOG/workspace"
pty "agent" {
id "migrate.absent.agent"
command "exec bash \"$CATALOG/task.sh\" absent"
lifecycle "adopt-only"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
agent "control" {
host "migrate"
role "worker"
workspace "$CATALOG/workspace"
pty "agent" {
id "migrate.control.agent"
command "exec bash \"$CATALOG/task.sh\" control"
}
}
Original file line number Diff line number Diff line change
@@ -0,0 +1,10 @@
agent "live" {
host "migrate"
role "worker"
workspace "$CATALOG/workspace"
pty "agent" {
id "migrate.live.agent"
command "exec bash \"$CATALOG/task.sh\" live"
lifecycle "adopt-only"
}
}
12 changes: 12 additions & 0 deletions cells/adopt-only-migration/fixture/net/task.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,12 @@
#!/usr/bin/env bash
set -euo pipefail

label="${1:?label is required}"
count_file="${CATALOG:?CATALOG must be set}/$label-count"
generation=1
if test -f "$count_file"; then
generation=$(( $(cat "$count_file") + 1 ))
fi
printf '%s\n' "$generation" >"$count_file"
printf '%s-GENERATION-%s-a091\n' "$label" "$generation"
sleep 100000
1 change: 1 addition & 0 deletions cells/adopt-only-migration/fixture/net/workspace/.keep
Original file line number Diff line number Diff line change
@@ -0,0 +1 @@
fixture
4 changes: 4 additions & 0 deletions docs/vrs/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,10 @@ tracked cell
- An LLM judge is not a defect. Its provider, model, prompt, inputs, and failure
behavior must be explicit, and mutation checks must demonstrate useful
discrimination.
- The model-free `adopt-only-migration` cell separates process-generation
adoption from replacement authority: live generations are adopted, dead or
absent migration tasks are held without mutation, and only an explicit
lifecycle transition permits ordinary replacement.

## Current execution

Expand Down
1 change: 1 addition & 0 deletions evidence/harness-exclusions.tsv
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
cell surface reason
adopt-only-migration cell deterministic native adoption/replacement lifecycle probe; no harness seat
context-resource-continuity cell deterministic native context/resource restart probe; no harness seat
docs judge:cold-reader one-shot offline Claude print grader; no bus identity, DING, or hook surface
exec-lifecycle-logging cell deterministic native exec lifecycle probe; no harness seat
Expand Down