diff --git a/CATALOG.md b/CATALOG.md index ffad55a..18c8d98 100644 --- a/CATALOG.md +++ b/CATALOG.md @@ -1,7 +1,7 @@ # Current eval catalog and evidence -This is the sole current corpus surface: **43 included cells** (12 model-free, +This is the sole current corpus surface: **46 included cells** (15 model-free, 31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run history: a failed last run is visible without being advertised as accepted evidence, and cells with no @@ -16,6 +16,9 @@ harness-native loader plus canonical hook file. | Cell | Harness | Model(s) / effort | Model seats | Cost | Timeout | Held-out judges | Latest accepted PASS | Last recorded run | |---|---|---|---:|---|---|---:|---|---| +| `agent-spec-lifecycle-catalog-prepare` | model-free | — | 0 | none | `60s` | 4 | — | **NO STRUCTURED RUN** | +| `agent-spec-lifecycle-catalog-publish-cas` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** | +| `agent-spec-lifecycle-catalog-root-atomicity` | model-free | — | 0 | none | `120s` | 7 | — | **NO STRUCTURED RUN** | | `context-resource-continuity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** | | `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 | — | **NO STRUCTURED RUN** | | `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** | @@ -79,6 +82,9 @@ while the last-run column makes a recorded failure distinct from a cell with no ## Harness hook exclusions +- `agent-spec-lifecycle-catalog-prepare` / `cell` — deterministic content-addressed prepare probe; no harness seat +- `agent-spec-lifecycle-catalog-publish-cas` / `cell` — deterministic immutable staging and CAS probe; no harness seat +- `agent-spec-lifecycle-catalog-root-atomicity` / `cell` — deterministic multi-seat catalog-root transaction probe; no harness seat - `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat - `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface - `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat @@ -93,7 +99,7 @@ while the last-run column makes a recorded failure distinct from a cell with no - `st2-network` / `cell` — deterministic catalog/message round-trip probe; no harness seat - `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat -These **13** rows are generated from structured exclusions. A cell-level row is required for +These **16** rows are generated from structured exclusions. A cell-level row is required for every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected agent: it has no st2 identity, DING sidecar, or lifecycle hooks. diff --git a/cells/agent-spec-lifecycle-catalog-prepare/README.md b/cells/agent-spec-lifecycle-catalog-prepare/README.md new file mode 100644 index 0000000..26010ca --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-prepare/README.md @@ -0,0 +1,6 @@ +# agent-spec-lifecycle-catalog-prepare + +Product-conformance cell for the first content-addressed catalog boundary. It +invokes `st2 catalog prepare` directly and proves that prepare imports the exact +Agent Spec bytes idempotently without creating a catalog root or making the +seat visible. diff --git a/cells/agent-spec-lifecycle-catalog-prepare/agent-spec-lifecycle-catalog-prepare.kdl b/cells/agent-spec-lifecycle-catalog-prepare/agent-spec-lifecycle-catalog-prepare.kdl new file mode 100644 index 0000000..4179d11 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-prepare/agent-spec-lifecycle-catalog-prepare.kdl @@ -0,0 +1,23 @@ +eval { + copy "./fixture" + max-timeout "60s" + + run "exercise" { + command "bash ./exercise.sh" + } + + judges { + judge "PRODUCT SURFACE — st2 persists exact prepared Agent Spec bytes" { + exec "test \"$(cat $CATALOG/results/prepare.code)\" = 0" + } + judge "HEAD UNCHANGED — prepare imports an object without publishing the seat" { + exec "bash ./judges/head-unchanged.sh" + } + judge "SOURCE SNAPSHOT — later source mutation cannot change the prepared object" { + exec "bash ./judges/source-snapshot.sh" + } + judge "IDEMPOTENT IMPORT — identical byte preparation returns one object identity" { + exec "bash ./judges/idempotent.sh" + } + } +} diff --git a/cells/agent-spec-lifecycle-catalog-prepare/fixture/exercise.sh b/cells/agent-spec-lifecycle-catalog-prepare/fixture/exercise.sh new file mode 100644 index 0000000..a0c8bb6 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-prepare/fixture/exercise.sh @@ -0,0 +1,26 @@ +#!/usr/bin/env bash +set -euo pipefail + +results="$CATALOG/results" +network="$CATALOG/network" +spec="$CATALOG/source/agents/h/seat/agent.kdl" +mkdir -p "$results" "$CATALOG/expected" +cp "$spec" "$CATALOG/expected/agent.kdl" + +capture() { + local name=$1 + shift + set +e + "$@" >"$results/$name.json" 2>"$results/$name.err" + local code=$? + set -e + printf '%s\n' "$code" >"$results/$name.code" +} + +capture head-before st2 --catalog "$network" catalog head +capture prepare st2 --catalog "$network" catalog prepare "$spec" +capture prepare-again st2 --catalog "$network" catalog prepare "$spec" + +# Source bytes are mutable after the import. The catalog object is not. +printf '\n// later source mutation\n' >>"$spec" +capture head-after st2 --catalog "$network" catalog head diff --git a/cells/agent-spec-lifecycle-catalog-prepare/fixture/source/agents/h/seat/agent.kdl b/cells/agent-spec-lifecycle-catalog-prepare/fixture/source/agents/h/seat/agent.kdl new file mode 100644 index 0000000..ee35063 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-prepare/fixture/source/agents/h/seat/agent.kdl @@ -0,0 +1,5 @@ +agent "seat" { + identity "seat" + host "h" + command "exec sleep 3600" +} diff --git a/cells/agent-spec-lifecycle-catalog-prepare/judges/head-unchanged.sh b/cells/agent-spec-lifecycle-catalog-prepare/judges/head-unchanged.sh new file mode 100644 index 0000000..3c2bbf7 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-prepare/judges/head-unchanged.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -euo pipefail +test "$(cat "$CATALOG/results/head-before.code")" = 0 +test "$(cat "$CATALOG/results/head-after.code")" = 0 +jq -e '. == null' "$CATALOG/results/head-before.json" >/dev/null +jq -e '. == null' "$CATALOG/results/head-after.json" >/dev/null diff --git a/cells/agent-spec-lifecycle-catalog-prepare/judges/idempotent.sh b/cells/agent-spec-lifecycle-catalog-prepare/judges/idempotent.sh new file mode 100644 index 0000000..e35fa9b --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-prepare/judges/idempotent.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -euo pipefail +test "$(cat "$CATALOG/results/prepare-again.code")" = 0 +a=$(jq -er .object "$CATALOG/results/prepare.json") +b=$(jq -er .object "$CATALOG/results/prepare-again.json") +test "$a" = "$b" diff --git a/cells/agent-spec-lifecycle-catalog-prepare/judges/source-snapshot.sh b/cells/agent-spec-lifecycle-catalog-prepare/judges/source-snapshot.sh new file mode 100644 index 0000000..33312c7 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-prepare/judges/source-snapshot.sh @@ -0,0 +1,5 @@ +#!/usr/bin/env bash +set -euo pipefail +object_path=$(jq -er .objectPath "$CATALOG/results/prepare.json") +test -f "$object_path" +cmp "$CATALOG/expected/agent.kdl" "$object_path" diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/README.md b/cells/agent-spec-lifecycle-catalog-publish-cas/README.md new file mode 100644 index 0000000..ddfc1cd --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/README.md @@ -0,0 +1,6 @@ +# agent-spec-lifecycle-catalog-publish-cas + +Product-conformance cell for explicit immutable staging. Ref CAS expects a +parent commit identity, not only an object identity, so A-B-A cannot revive a +stale writer. Operation replay is idempotent, manager ownership is fenced, and +none of the staging operations changes the selected catalog root. diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/agent-spec-lifecycle-catalog-publish-cas.kdl b/cells/agent-spec-lifecycle-catalog-publish-cas/agent-spec-lifecycle-catalog-publish-cas.kdl new file mode 100644 index 0000000..3c7a69f --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/agent-spec-lifecycle-catalog-publish-cas.kdl @@ -0,0 +1,26 @@ +eval { + copy "./fixture" + max-timeout "90s" + + run "exercise" { + command "bash ./exercise.sh" + } + + judges { + judge "PRODUCT SURFACE — explicit immutable staging is supported" { + exec "test \"$(cat $CATALOG/results/setup.code)\" = 0" + } + judge "LOST ACK — replaying one request returns the original ref commit" { + exec "bash ./judges/lost-ack.sh" + } + judge "MANAGER FENCE — a foreign manager cannot move or adopt the head" { + exec "bash ./judges/manager-fence.sh" + } + judge "ABA FENCE — stale parent ref fails after A -> B -> A" { + exec "bash ./judges/aba-fence.sh" + } + judge "ROOT UNCHANGED — staging never changes catalog visibility" { + exec "bash ./judges/root-unchanged.sh" + } + } +} diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/exercise.sh b/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/exercise.sh new file mode 100644 index 0000000..422371b --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/exercise.sh @@ -0,0 +1,61 @@ +#!/usr/bin/env bash +set -euo pipefail + +results="$CATALOG/results" +network="$CATALOG/network" +spec_a="$CATALOG/source/a/agents/h/seat/agent.kdl" +spec_b="$CATALOG/source/b/agents/h/seat/agent.kdl" +mkdir -p "$results" "$network/agents/h/seat/resources" + +capture() { + local name=$1 + shift + set +e + "$@" >"$results/$name.json" 2>"$results/$name.err" + local code=$? + set -e + printf '%s\n' "$code" >"$results/$name.code" +} + +stage() { + local name=$1 spec=$2 manager=$3 operation=$4 + shift 4 + capture "$name" st2 --catalog "$network" catalog stage "$spec" \ + --manager "$manager" \ + --state-relative agents/h/seat \ + --operation-id "$operation" \ + "$@" +} + +capture head-before st2 --catalog "$network" catalog head +stage publish-a "$spec_a" nix stage-a +cp "$results/publish-a.code" "$results/setup.code" + +ref_a=$(jq -er .refCommit "$results/publish-a.json") +binding_a=$(jq -er .resourceBindingCommit "$results/publish-a.json") + +# Simulate an acknowledged result being lost, then replay the same operation. +stage replay-a "$spec_a" nix stage-a + +# A foreign manager must not advance a Nix-owned ref. +stage foreign "$spec_b" human foreign \ + --expected-ref "$ref_a" --binding-parent "$binding_a" + +# Advance A -> B -> A with immutable parent commits. +stage publish-b "$spec_b" nix stage-b \ + --expected-ref "$ref_a" --binding-parent "$binding_a" +ref_b=$(jq -er .refCommit "$results/publish-b.json") +binding_b=$(jq -er .resourceBindingCommit "$results/publish-b.json") + +stage publish-a2 "$spec_a" nix stage-a2 \ + --expected-ref "$ref_b" --binding-parent "$binding_b" +ref_a2=$(jq -er .refCommit "$results/publish-a2.json") +printf '%s\n' "$ref_a2" >"$results/expected-final-ref" + +# The ref points at A again, but the old A parent commit is stale. +stage stale "$spec_b" nix stale-a-parent \ + --expected-ref "$ref_a" --binding-parent "$binding_a" +stage replay-a2 "$spec_a" nix stage-a2 \ + --expected-ref "$ref_b" --binding-parent "$binding_b" + +capture head-after st2 --catalog "$network" catalog head diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/source/a/agents/h/seat/agent.kdl b/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/source/a/agents/h/seat/agent.kdl new file mode 100644 index 0000000..6c00994 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/source/a/agents/h/seat/agent.kdl @@ -0,0 +1,8 @@ +agent "seat" { + identity "seat" + host "h" + pty "agent" { + id "h.seat.agent" + command "exec echo a; sleep 3600" + } +} diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/source/b/agents/h/seat/agent.kdl b/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/source/b/agents/h/seat/agent.kdl new file mode 100644 index 0000000..3b8e106 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/fixture/source/b/agents/h/seat/agent.kdl @@ -0,0 +1,8 @@ +agent "seat" { + identity "seat" + host "h" + pty "agent" { + id "h.seat.agent" + command "exec echo b; sleep 3600" + } +} diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/judges/aba-fence.sh b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/aba-fence.sh new file mode 100644 index 0000000..51a1733 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/aba-fence.sh @@ -0,0 +1,8 @@ +#!/usr/bin/env bash +set -euo pipefail +test "$(cat "$CATALOG/results/publish-b.code")" = 0 +test "$(cat "$CATALOG/results/publish-a2.code")" = 0 +test "$(cat "$CATALOG/results/stale.code")" != 0 +test "$(cat "$CATALOG/results/replay-a2.code")" = 0 +test "$(jq -er .refCommit "$CATALOG/results/replay-a2.json")" = \ + "$(cat "$CATALOG/results/expected-final-ref")" diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/judges/lost-ack.sh b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/lost-ack.sh new file mode 100644 index 0000000..3176165 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/lost-ack.sh @@ -0,0 +1,7 @@ +#!/usr/bin/env bash +set -euo pipefail +test "$(cat "$CATALOG/results/replay-a.code")" = 0 +test "$(jq -er .refCommit "$CATALOG/results/publish-a.json")" = \ + "$(jq -er .refCommit "$CATALOG/results/replay-a.json")" +test "$(jq -er .resourceBindingCommit "$CATALOG/results/publish-a.json")" = \ + "$(jq -er .resourceBindingCommit "$CATALOG/results/replay-a.json")" diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/judges/manager-fence.sh b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/manager-fence.sh new file mode 100644 index 0000000..2ca0785 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/manager-fence.sh @@ -0,0 +1,6 @@ +#!/usr/bin/env bash +set -euo pipefail +test "$(cat "$CATALOG/results/foreign.code")" != 0 +# The next Nix stage still succeeds from the pre-foreign expected ref, proving +# the rejected manager did not move it. +test "$(cat "$CATALOG/results/publish-b.code")" = 0 diff --git a/cells/agent-spec-lifecycle-catalog-publish-cas/judges/root-unchanged.sh b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/root-unchanged.sh new file mode 100644 index 0000000..7d73df2 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-publish-cas/judges/root-unchanged.sh @@ -0,0 +1,11 @@ +#!/usr/bin/env bash +set -euo pipefail + +test "$(cat "$CATALOG/results/head-before.code")" = 0 +test "$(cat "$CATALOG/results/head-after.code")" = 0 +jq -e '. == null' "$CATALOG/results/head-before.json" >/dev/null +jq -e '. == null' "$CATALOG/results/head-after.json" >/dev/null + +for result in publish-a replay-a publish-b publish-a2 replay-a2; do + jq -e '.rootChanged == false' "$CATALOG/results/$result.json" >/dev/null +done diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/README.md b/cells/agent-spec-lifecycle-catalog-root-atomicity/README.md new file mode 100644 index 0000000..b60e29e --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/README.md @@ -0,0 +1,11 @@ +# agent-spec-lifecycle-catalog-root-atomicity + +Product-conformance cell for catalog-root transactions. + +Two seats are staged without changing visibility. Root CAS rejects a stale +writer, and graph validation rejects a cross-seat ref/binding join without +moving the root. A valid two-seat Nix admission then commits both changes under +one root identity; concurrent readers see only complete old/new roots. Finally, +a dynamic manager admits a distinct seat without changing either Nix admission, +while both managers remain fenced from the other's refs. Corrupting an admitted +immutable object then makes resolution fail closed. diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/agent-spec-lifecycle-catalog-root-atomicity.kdl b/cells/agent-spec-lifecycle-catalog-root-atomicity/agent-spec-lifecycle-catalog-root-atomicity.kdl new file mode 100644 index 0000000..e2bf11a --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/agent-spec-lifecycle-catalog-root-atomicity.kdl @@ -0,0 +1,32 @@ +eval { + copy "./fixture" + max-timeout "120s" + + run "exercise" { + command "bash ./exercise.sh" + } + + judges { + judge "PRODUCT SURFACE — immutable root commits are supported" { + json "results/outcomes.json" field "surface" is #true + } + judge "ROOT CAS — a separately-valid stale seat patch cannot cross a newer root" { + json "results/outcomes.json" field "staleRootRejected" is #true + } + judge "GRAPH VALIDATION — a cross-seat ref/binding join cannot change the root" { + json "results/outcomes.json" field "invalidGraphRejected" is #true + } + judge "ATOMIC NIX PATCH — two seat changes publish under exactly one new root commit" { + json "results/outcomes.json" field "nixPatchOneRoot" is #true + } + judge "NO MIXED SNAPSHOT — readers observe only the old or new complete root" { + json "results/outcomes.json" field "noMixedSnapshot" is #true + } + judge "MIXED MANAGERS — dynamic admission preserves Nix seats and ref ownership" { + json "results/outcomes.json" field "mixedManagersPreserved" is #true + } + judge "CORRUPTION CLOSED — an admitted object digest mismatch cannot be inspected" { + json "results/outcomes.json" field "corruptionRejected" is #true + } + } +} diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/exercise.sh b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/exercise.sh new file mode 100644 index 0000000..f283372 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/exercise.sh @@ -0,0 +1,257 @@ +#!/usr/bin/env bash +set -euo pipefail + +results="$CATALOG/results" +network="$CATALOG/network" +mkdir -p \ + "$results" \ + "$network/agents/h/alpha/resources" \ + "$network/agents/h/beta/resources" \ + "$network/agents/h/gamma/resources" + +capture() { + local name=$1 + shift + set +e + "$@" >"$results/$name.json" 2>"$results/$name.err" + local code=$? + set -e + printf '%s\n' "$code" >"$results/$name.code" +} + +stage() { + local name=$1 spec=$2 state=$3 operation=$4 + shift 4 + capture "$name" st2 --catalog "$network" catalog stage "$spec" \ + --manager nix \ + --state-relative "$state" \ + --operation-id "$operation" \ + "$@" +} + +write_request() { + local path=$1 expected=$2 operation=$3 + shift 3 + if test "$expected" = absent; then + jq -n --arg operation "$operation" --args '$ARGS.positional' "$@" \ + | jq --arg operation "$operation" '{ + manager: "nix", + operationId: $operation, + selections: [range(0; length / 2) as $i | { + refCommit: .[$i * 2], + resourceBindingCommit: .[$i * 2 + 1] + }] + }' >"$path" + else + jq -n --arg expected "$expected" --arg operation "$operation" --args '$ARGS.positional' "$@" \ + | jq --arg expected "$expected" --arg operation "$operation" '{ + expectedRoot: $expected, + manager: "nix", + operationId: $operation, + selections: [range(0; length / 2) as $i | { + refCommit: .[$i * 2], + resourceBindingCommit: .[$i * 2 + 1] + }] + }' >"$path" + fi +} + +alpha_base="$CATALOG/source/alpha-base/agents/h/alpha/agent.kdl" +beta_base="$CATALOG/source/beta-base/agents/h/beta/agent.kdl" +alpha_nix="$CATALOG/source/alpha-nix/agents/h/alpha/agent.kdl" +beta_nix="$CATALOG/source/beta-nix/agents/h/beta/agent.kdl" +gamma_dynamic="$CATALOG/source/gamma-dynamic/agents/h/gamma/agent.kdl" + +stage alpha-base "$alpha_base" agents/h/alpha alpha-base +stage beta-base "$beta_base" agents/h/beta beta-base +alpha_ref0=$(jq -er .refCommit "$results/alpha-base.json") +alpha_binding0=$(jq -er .resourceBindingCommit "$results/alpha-base.json") +beta_ref0=$(jq -er .refCommit "$results/beta-base.json") +beta_binding0=$(jq -er .resourceBindingCommit "$results/beta-base.json") + +write_request "$results/base-request.json" absent admit-base \ + "$alpha_ref0" "$alpha_binding0" "$beta_ref0" "$beta_binding0" +capture admit-base st2 --catalog "$network" catalog admit "$results/base-request.json" +root0=$(jq -er .rootCommit "$results/admit-base.json") +capture inspect-base st2 --catalog "$network" catalog inspect + +stage alpha-nix "$alpha_nix" agents/h/alpha alpha-nix \ + --expected-ref "$alpha_ref0" --binding-parent "$alpha_binding0" +stage beta-nix "$beta_nix" agents/h/beta beta-nix \ + --expected-ref "$beta_ref0" --binding-parent "$beta_binding0" +alpha_ref1=$(jq -er .refCommit "$results/alpha-nix.json") +alpha_binding1=$(jq -er .resourceBindingCommit "$results/alpha-nix.json") +beta_ref1=$(jq -er .refCommit "$results/beta-nix.json") +beta_binding1=$(jq -er .resourceBindingCommit "$results/beta-nix.json") +capture head-after-stage st2 --catalog "$network" catalog head + +# Both commits exist, but they belong to different seats and cannot be joined. +write_request "$results/invalid-request.json" "$root0" admit-invalid \ + "$alpha_ref1" "$beta_binding1" +capture invalid st2 --catalog "$network" catalog admit "$results/invalid-request.json" +capture head-after-invalid st2 --catalog "$network" catalog head + +write_request "$results/nix-request.json" "$root0" admit-nix \ + "$alpha_ref1" "$alpha_binding1" "$beta_ref1" "$beta_binding1" + +# Sample the public root while the whole two-seat admission is published. +: >"$results/observed-heads.jsonl" +( + while test ! -e "$results/poll-stop"; do + st2 --catalog "$network" catalog head \ + >>"$results/observed-heads.jsonl" 2>/dev/null || true + done +) & +poller=$! +capture admit-nix st2 --catalog "$network" catalog admit "$results/nix-request.json" +touch "$results/poll-stop" +wait "$poller" +capture head-final st2 --catalog "$network" catalog head +capture inspect-final st2 --catalog "$network" catalog inspect +cat "$results/head-final.json" >>"$results/observed-heads.jsonl" +root1=$(jq -er .rootCommit "$results/admit-nix.json") + +surface=false +if test "$(cat "$results/admit-base.code")" = 0 \ + && test "$(jq -r .seats "$results/admit-base.json")" = 2 \ + && test "$(jq -r '.seats | length' "$results/inspect-base.json")" = 2; then + surface=true +fi + +invalid_graph_rejected=false +if test "$(cat "$results/invalid.code")" != 0 \ + && test "$(jq -er .commit "$results/head-after-invalid.json")" = "$root0"; then + invalid_graph_rejected=true +fi + +nix_patch_one_root=false +if test "$(cat "$results/admit-nix.code")" = 0 \ + && test "$root1" != "$root0" \ + && test "$(jq -er .commit "$results/head-final.json")" = "$root1" \ + && test "$(jq -er .value.parent "$results/head-final.json")" = "$root0" \ + && test "$(jq -r '.value.admissions | length' "$results/head-final.json")" = 2 \ + && test "$(jq -r .seats "$results/admit-nix.json")" = 2; then + nix_patch_one_root=true +fi + +no_mixed_snapshot=false +if jq -s -e --arg old "$root0" --arg new "$root1" \ + 'length > 0 and all(.[]; .commit == $old or .commit == $new) + and any(.[]; .commit == $new)' \ + "$results/observed-heads.jsonl" >/dev/null; then + no_mixed_snapshot=true +fi + +# A root transaction records its actor but does not take custody of every seat. +# A dynamic manager can append its own seat while each ref remains manager-owned. +capture gamma-dynamic st2 --catalog "$network" catalog stage "$gamma_dynamic" \ + --manager dynamic \ + --state-relative agents/h/gamma \ + --operation-id gamma-dynamic +gamma_ref=$(jq -er .refCommit "$results/gamma-dynamic.json") +gamma_binding=$(jq -er .resourceBindingCommit "$results/gamma-dynamic.json") + +# The original base root is now genuinely stale. This otherwise-valid dynamic +# seat must not cross the newer Nix root. +jq -n \ + --arg expected "$root0" \ + --arg ref "$gamma_ref" \ + --arg binding "$gamma_binding" \ + '{ + expectedRoot: $expected, + manager: "dynamic", + operationId: "admit-dynamic-stale", + selections: [{ + refCommit: $ref, + resourceBindingCommit: $binding + }] + }' >"$results/stale-request.json" +capture stale st2 --catalog "$network" catalog admit "$results/stale-request.json" +capture head-after-stale st2 --catalog "$network" catalog head + +capture dynamic-moves-nix st2 --catalog "$network" catalog stage "$alpha_nix" \ + --manager dynamic \ + --state-relative agents/h/alpha \ + --operation-id dynamic-moves-nix \ + --expected-ref "$alpha_ref1" \ + --binding-parent "$alpha_binding1" +capture nix-moves-dynamic st2 --catalog "$network" catalog stage "$gamma_dynamic" \ + --manager nix \ + --state-relative agents/h/gamma \ + --operation-id nix-moves-dynamic \ + --expected-ref "$gamma_ref" \ + --binding-parent "$gamma_binding" + +jq -n \ + --arg expected "$root1" \ + --arg ref "$gamma_ref" \ + --arg binding "$gamma_binding" \ + '{ + expectedRoot: $expected, + manager: "dynamic", + operationId: "admit-dynamic", + selections: [{ + refCommit: $ref, + resourceBindingCommit: $binding + }] + }' >"$results/dynamic-request.json" +capture admit-dynamic st2 --catalog "$network" catalog admit "$results/dynamic-request.json" +capture head-dynamic st2 --catalog "$network" catalog head +capture inspect-dynamic st2 --catalog "$network" catalog inspect +root2=$(jq -er .rootCommit "$results/admit-dynamic.json") + +stale_root_rejected=false +if test "$(cat "$results/stale.code")" != 0 \ + && test "$(jq -er .commit "$results/head-after-stale.json")" = "$root1"; then + stale_root_rejected=true +fi + +mixed_managers_preserved=false +if test "$(cat "$results/gamma-dynamic.code")" = 0 \ + && test "$(cat "$results/dynamic-moves-nix.code")" != 0 \ + && test "$(cat "$results/nix-moves-dynamic.code")" != 0 \ + && test "$(cat "$results/admit-dynamic.code")" = 0 \ + && test "$(jq -er .value.parent "$results/head-dynamic.json")" = "$root1" \ + && test "$(jq -r '.value.admissions | length' "$results/head-dynamic.json")" = 3 \ + && test "$(jq -er '.value.admissions["h.alpha"]' "$results/head-dynamic.json")" = \ + "$(jq -er '.value.admissions["h.alpha"]' "$results/head-final.json")" \ + && test "$(jq -er '.value.admissions["h.beta"]' "$results/head-dynamic.json")" = \ + "$(jq -er '.value.admissions["h.beta"]' "$results/head-final.json")" \ + && test "$(jq -Sc '.seats[] | select(.busId == "h.alpha")' "$results/inspect-dynamic.json")" = \ + "$(jq -Sc '.seats[] | select(.busId == "h.alpha")' "$results/inspect-final.json")" \ + && test "$(jq -Sc '.seats[] | select(.busId == "h.beta")' "$results/inspect-dynamic.json")" = \ + "$(jq -Sc '.seats[] | select(.busId == "h.beta")' "$results/inspect-final.json")"; then + mixed_managers_preserved=true +fi + +# Corruption is observable through the supported resolver. Preserve the head, +# corrupt one admitted object, and require inspect to reject the graph. +object_path=$(jq -er '.seats[] | select(.busId == "h.alpha") | .sourcePath' \ + "$results/inspect-dynamic.json") +chmod u+w "$object_path" +printf 'corrupt\n' >"$object_path" +capture inspect-corrupt st2 --catalog "$network" catalog inspect +capture head-after-corrupt st2 --catalog "$network" catalog head +corruption_rejected=false +if test "$(cat "$results/inspect-corrupt.code")" != 0 \ + && test "$(jq -er .commit "$results/head-after-corrupt.json")" = "$root2"; then + corruption_rejected=true +fi + +jq -n \ + --argjson surface "$surface" \ + --argjson stale "$stale_root_rejected" \ + --argjson invalid "$invalid_graph_rejected" \ + --argjson patch "$nix_patch_one_root" \ + --argjson mixed "$no_mixed_snapshot" \ + --argjson managers "$mixed_managers_preserved" \ + --argjson corruption "$corruption_rejected" \ + '{ + surface: $surface, + staleRootRejected: $stale, + invalidGraphRejected: $invalid, + nixPatchOneRoot: $patch, + noMixedSnapshot: $mixed, + mixedManagersPreserved: $managers, + corruptionRejected: $corruption + }' >"$results/outcomes.json" diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/alpha-base/agents/h/alpha/agent.kdl b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/alpha-base/agents/h/alpha/agent.kdl new file mode 100644 index 0000000..3745344 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/alpha-base/agents/h/alpha/agent.kdl @@ -0,0 +1,8 @@ +agent "alpha" { + identity "alpha" + host "h" + pty "agent" { + id "task-alpha" + command "exec sleep 3600" + } +} diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/alpha-nix/agents/h/alpha/agent.kdl b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/alpha-nix/agents/h/alpha/agent.kdl new file mode 100644 index 0000000..250431c --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/alpha-nix/agents/h/alpha/agent.kdl @@ -0,0 +1,8 @@ +agent "alpha" { + identity "alpha" + host "h" + pty "agent" { + id "task-nix-alpha" + command "exec sleep 3600" + } +} diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/beta-base/agents/h/beta/agent.kdl b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/beta-base/agents/h/beta/agent.kdl new file mode 100644 index 0000000..73e5d07 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/beta-base/agents/h/beta/agent.kdl @@ -0,0 +1,8 @@ +agent "beta" { + identity "beta" + host "h" + pty "agent" { + id "task-beta" + command "exec sleep 3600" + } +} diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/beta-nix/agents/h/beta/agent.kdl b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/beta-nix/agents/h/beta/agent.kdl new file mode 100644 index 0000000..72716c2 --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/beta-nix/agents/h/beta/agent.kdl @@ -0,0 +1,8 @@ +agent "beta" { + identity "beta" + host "h" + pty "agent" { + id "task-nix-beta" + command "exec sleep 3600" + } +} diff --git a/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/gamma-dynamic/agents/h/gamma/agent.kdl b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/gamma-dynamic/agents/h/gamma/agent.kdl new file mode 100644 index 0000000..2216f2f --- /dev/null +++ b/cells/agent-spec-lifecycle-catalog-root-atomicity/fixture/source/gamma-dynamic/agents/h/gamma/agent.kdl @@ -0,0 +1,8 @@ +agent "gamma" { + identity "gamma" + host "h" + pty "agent" { + id "task-dynamic-gamma" + command "exec sleep 3600" + } +} diff --git a/docs/vrs/.experiments/2026-07-29-content-addressed-catalog-eval-family.md b/docs/vrs/.experiments/2026-07-29-content-addressed-catalog-eval-family.md new file mode 100644 index 0000000..b4801d4 --- /dev/null +++ b/docs/vrs/.experiments/2026-07-29-content-addressed-catalog-eval-family.md @@ -0,0 +1,115 @@ +# Content-addressed catalog eval family + +## Question + +Can the existing model-free folder-eval contract express sharp, portable +outcomes for immutable Agent Spec preparation and atomic catalog admission +without changing the eval runner or assigning runtime policy to evals? + +## Method + +The exploration used: + +- `compoundingtech/evals` + `78210568e47244d80de99c18d0eea2d6b641c18a`; +- `compoundingtech/st2` + `6f1fe484b517406b80d06f335997859b1a53f609` for initial RED exploration; +- st2 prototype + `16ee7eb953508801a4603edd056ab32f28986601` for retained-cell validation; and +- ordinary team-less folder evals with deterministic run steps and held-out + shell or typed JSON judges. + +The exploration designed five cells that divide the publication boundary into +independently falsifiable outcomes: + +1. `agent-spec-lifecycle-catalog-prepare`; +2. `agent-spec-lifecycle-catalog-publish-cas`; +3. `agent-spec-lifecycle-catalog-root-atomicity`; +4. `agent-spec-lifecycle-catalog-root-resource-admission`; and +5. `agent-spec-lifecycle-catalog-crash-barriers`. + +The maintained first slice retains only prepare/import, staged ref publication, +and catalog-root admission: + +1. `agent-spec-lifecycle-catalog-prepare`; +2. `agent-spec-lifecycle-catalog-publish-cas`; and +3. `agent-spec-lifecycle-catalog-root-atomicity`. + +Typed resource admission and public crash-boundary injection remain future +discriminators. The prototype also intentionally excluded resident runtime +generation and replacement. Those outcomes belong in later cells only when the +corresponding st2 implementation exists. + +A reference publisher model exercised deterministic content identity, immutable +object-first publication, ref CAS, and durable-boundary crashes. Four reference +cells passed with scores `7/0`, `4/0`, `7/0`, and `4/0`. Mutations for a wrong +digest, corrupt object, absent CAS, and ref-before-object failed the intended +judges with scores `6/1`, `2/2`, `5/2`, and `1/3`. + +Separate executable state models tested root-level semantics: + +- independent per-seat pointers admit a catalog-invalid union, while one root + CAS plus whole-catalog validation rejects it; +- object-only expectations admit ABA, while immutable parent-root expectations + reject it; and +- asynchronous resource binding admits a desired revision before its required + binding, while one admission-root transition exposes only complete old or new + states and preserves stable identity-owned resource bytes. + +Early product-conformance drafts were run directly against st2 rather than the +reference shim. They began as honest RED scenarios while st2 lacked a catalog +API. Once the first concrete CLI slice existed, the retained cells were lowered +to `catalog prepare`, `catalog stage`, `catalog admit`, `catalog head`, and +`catalog inspect`. All retained judges pass locally against st2 prototype +`16ee7eb953508801a4603edd056ab32f28986601`; that is implementation feedback, +not yet accepted corpus-pin evidence. + +The concrete st2 CLI is being incubated in +[`compoundingtech/st2#52`](https://github.com/compoundingtech/st2/issues/52). +Command lowering in these cells must follow that implementation rather than +freeze speculative verbs. + +## Result + +The existing eval runner is sufficient. No custom parser, lifecycle hook, +daemon, model, or paid judge is required. + +The three retained E2E cells prove these portable outcomes against the local +st2 prototype: + +- preparation imports exact Agent Spec bytes idempotently without + changing the selected catalog root; +- staging uses a parent ref identity, manager fence, and replay key without + changing catalog visibility; +- the catalog root, not independent seat pointers, is the atomic desired-state + boundary; +- each seat ref retains its manager ownership while a root transaction records + its actor rather than taking global custody of every admitted seat; +- a multi-seat admission crosses one root CAS without exposing a mixed + snapshot; and +- digest corruption of an admitted object fails closed during inspection. + +The separate reference models suggest two additional directions but the +retained E2E family does not claim to prove them: typed resource compatibility +and interruption behavior at durable publication boundaries. They remain +future discriminators pending real public st2 surfaces. + +The three-cell first slice remains proposed, not accepted evidence. Its command +scripts target the concrete #52 prototype and pass locally, including +mixed-manager preservation and corruption rejection. It becomes maintained +evidence only after every judge has mutation evidence and all three cells pass +against an st2 commit pinned by this repository. + +The typed resource-admission and crash-barrier prototypes remain recorded here +but are deliberately absent from the maintained corpus until st2 exposes a real +typed binding contract and a stable black-box crash oracle. + +## VRS Impact + +- Suggests extending the spec-extraction section with catalog publication as a + three-cell model-free, portable outcome family. +- Does not change `vision.md`, `requirements.md`, or `AGENT-SPEC.md`. +- Leaves storage layout, digest encoding, CLI spelling, locking, fsync, + failpoint names, and recovery machinery owned by st2. +- Leaves manager selection, Nix lowering, authorization, and deployment + acceptance owned by consumers such as dotfiles. diff --git a/docs/vrs/spec.md b/docs/vrs/spec.md index f7325d1..8ae15d9 100644 --- a/docs/vrs/spec.md +++ b/docs/vrs/spec.md @@ -76,6 +76,39 @@ and continues or stops. current runtime—st2 today—executes that unit; evals supplies scenarios and evidence about whether the shape works. +### Proposed content-addressed catalog evidence + +The proposed `agent-spec-lifecycle-catalog-*` family applies R10–R11 to Agent +Spec publication without making evals a catalog implementation: + +```text +prepare exact immutable Agent Spec bytes + -> stage refs with parent CAS + replay fence + -> admit one whole-catalog root +``` + +The cells own portable observable outcomes: idempotent import, immutable +content identity, stale-parent and ABA rejection, whole-catalog validity, +atomic multi-seat patches, staging-operation replay, per-seat manager fences, and +mixed-manager preservation across root transactions. A root commit records the +transaction actor; it does not transfer custody of every admitted seat. st2 +owns the concrete API, storage, transactions, durability, inspection, and +recovery mechanisms. + +Typed resource admission and public crash-boundary injection remain future +discriminators documented by the experiment. They are not maintained cells +until st2 exposes real product surfaces that can make them green without a +reference shim or eval-only runner path. + +This is a spec suggestion, not accepted behavior. The cells pass locally +against st2 prototype commit `16ee7eb953508801a4603edd056ab32f28986601`, +incubated in +[`compoundingtech/st2#52`](https://github.com/compoundingtech/st2/issues/52) +but do not become accepted evidence until the implementation is pinned and the +held-out judges have mutation evidence. Until then they do not update +`AGENT-SPEC.md` or its accepted st2 pin. The experiment record is +[content-addressed catalog eval family](./.experiments/2026-07-29-content-addressed-catalog-eval-family.md). + The owner updates this spec with the corpus. Changing [vision.md](./vision.md) or [requirements.md](./requirements.md) requires Nathan's explicit approval. diff --git a/evidence/harness-exclusions.tsv b/evidence/harness-exclusions.tsv index eca3ff2..23ece46 100644 --- a/evidence/harness-exclusions.tsv +++ b/evidence/harness-exclusions.tsv @@ -1,4 +1,7 @@ cell surface reason +agent-spec-lifecycle-catalog-prepare cell deterministic content-addressed prepare probe; no harness seat +agent-spec-lifecycle-catalog-publish-cas cell deterministic immutable staging and CAS probe; no harness seat +agent-spec-lifecycle-catalog-root-atomicity cell deterministic multi-seat catalog-root transaction probe; no harness seat context-resource-continuity cell deterministic native context/resource restart probe; no harness seat docs judge:cold-reader one-shot offline Claude print grader; no bus identity, DING, or hook surface exec-lifecycle-logging cell deterministic native exec lifecycle probe; no harness seat