Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension


Conversations
Failed to load comments.
Loading
Jump to
The table of contents is too big for display.
Diff view
Diff view
  •  
  •  
  •  
63 changes: 53 additions & 10 deletions AGENT-SPEC.md
Original file line number Diff line number Diff line change
Expand Up @@ -5,8 +5,8 @@ by evals. st2 is the current implementation, not the owner of the contract; a fu
implementation can target the same contract and proofs.

The current corpus proof is pinned to st2
[`9887b2842222def0838c2cd82e6c24c218f7efa6`](https://github.com/compoundingtech/st2/commit/9887b2842222def0838c2cd82e6c24c218f7efa6)
(`0.1.0`, source `9887b28`). The pin identifies the implementation and version the corpus currently proves; it
[`0fed14bb5653b67e1d64f1199e240c4c5c612bf7`](https://github.com/compoundingtech/st2/commit/0fed14bb5653b67e1d64f1199e240c4c5c612bf7)
(`0.1.0`, source `0fed14b`). The pin identifies the implementation and version the corpus currently proves; it
does not transfer ownership of the specification to st2. A proposed behavior change must update this contract
and its maintained proof cells before an implementation claims conformance. Do not infer additional fields or
commands from older corpus fixtures.
Expand Down Expand Up @@ -52,6 +52,8 @@ agent "<identity>" {
retired #false
keep #false

resource "work" _tag="github-issue" uri="github-issue://example/project/123"

restart {
attempts 3
interval "60s"
Expand Down Expand Up @@ -88,6 +90,7 @@ Supported agent children are:
| `supervisor "…"` | Optional bare identity or full bus id for crash-loop routing. |
| `retired #true` | Decommission the declaration on the next reconciliation. Edit this flag; do not delete a live declaration to retire it. |
| `keep #true` | Freeze dead evidence and suppress collection/restart for every task; retirement still stops live tasks. |
| `resource "name" _tag="type" uri="absolute-uri"` | Binds one uniquely named, externally identified Resource as declaration metadata. |
| `restart { … }` | Optional service restart policy. |
| `env { KEY "value" }` | Environment inherited by the compact agent task and sidecars. |
| `command "…"` | Compact interactive task named `agent`. |
Expand All @@ -106,6 +109,41 @@ the window reset; `mode "fail"` parks the task after attempts are exhausted and
crash-loop message to `supervisor`. Invalid restart subfields currently fall back to defaults; authors must not
rely on that permissiveness.

## Resource bindings

An agent may directly carry zero or more Resource bindings:

```kdl
resource "work" _tag="github-issue" uri="github-issue://example/project/123"
resource "source" _tag="worktree" uri="worktree://example/project/main"
```

The positional name is the Resource's agent-local semantic role. Names are non-empty and unique within one
agent. `_tag` is a non-empty, opaque discriminator owned by the Resource type's downstream contract. `uri` is
an RFC 3986 absolute URI and is the Resource identity. st2 preserves the URI's exact bytes; it does not
normalize or resolve it. Declaration order has no meaning. Canonical KDL and supported TOML/JSON parsing lower
bindings to deterministic name order.

The generic envelope is closed: each binding has exactly the positional name, `_tag`, and `uri`. Missing or
duplicate fields, duplicate names, child nodes, invalid URI syntax, and unsupported properties such as access
or readiness policy fail validation. This prevents an ignored property from appearing enforced.

Resource bindings are declaration metadata, not launch targets. They do not make an otherwise unrunnable
service runnable and are excluded from effective task launch definitions. Editing only Resource bindings
therefore updates catalog inspection while an already-live task is adopted without stop, replacement, or
relaunch. `st2 agents --json [--enrich]` exposes every binding as a name-ordered
`{"name","_tag","uri"}` descriptor and preserves unknown downstream tags.

The envelope does not define Resource schemas, resolution, access grants, required/optional status, readiness,
lifecycle, mutation, or rendering. A URI's presence grants no authority. Those semantics belong to the
concrete Resource type and its consumer, not st2.

Executable evidence:
[`agent-spec-resource-bindings`](cells/agent-spec-resource-bindings/) covers strict parser failures,
deterministic JSON inspection, exact URI and unknown-tag preservation, Resource-only live adoption, and
cleanup. The matched [`assignment-contract-*`](cells/) tournament covers direct Resource selection against
Focus and Assignment controls; direct bindings are the selected treatment.

## Compact and explicit tasks

The canonical compact pair:
Expand Down Expand Up @@ -398,7 +436,7 @@ Codex launch explicitly selects `gpt-5.6-sol` at medium reasoning effort.

The folder-eval grammar preserves its `team`, `eval`, kickoff, timeout, and held-out judge shape. Its
agent projection supports `workspace`, `supervisor`, `env`, `command`, explicit `exec`, and bare
`ding`; it does not accept a catalog `render` block. For that grammar, `eval { copy … }` and a
`ding`; it does not accept catalog `resource` or `render` nodes. For that grammar, `eval { copy … }` and a
deterministic pre-boot materializer must place the equivalent harness files in each declared
workspace:

Expand All @@ -421,9 +459,9 @@ agent { command "true" }
At the pinned source it validates as one agent with zero errors and warnings. Production declarations should
normally make identity, host, `ST_AGENT`, workspace, and the real harness command explicit.

The complete declaration, compact pair, explicit PTY/exec blocks, restart policy, and render block earlier in
this document collectively exercise every implemented authoring field. Before validating/materializing the
example, create every `$CATALOG`-rooted workspace it names.
The complete declaration, Resource bindings, compact pair, explicit PTY/exec blocks, restart policy, and
render block earlier in this document collectively exercise every implemented authoring field. Before
validating/materializing the example, create every `$CATALOG`-rooted workspace it names.

## Free authoring gate

Expand All @@ -438,8 +476,13 @@ Inspect the declaration, every referenced template, and every workspace destinat
materialization command. Materialization is byte-idempotent and does not imply hook installation. Starting
the network is a separate, explicitly authorized action.

For source `9887b28`, the accepted Linux executable has SHA256
`d49d44fd4f3f6f655455c212353a469fefa956082bedf22163deb767d8a36a0d`; its published archive has SHA256
`32ee103bd17ccb3e155ac63d816a3906c2470a3c98e3cc04b56e5a67138b9927`. `bin/check-corpus.sh` verifies
the variable-age version contract, exact installed binary, embedded full source commit, strict semantic
For source `0fed14b`, the accepted published Linux executable has SHA256
`d61d12b2b1189a391c196ca28f8f4ba69072d14fcbad2571fc29db1f250f4eed`; its published archive has SHA256
`d14404ae678bbe3f2a5ad8580cde1e4b8f6009067c46555f392c6e0957b8a2da`, and the downloaded `SHA256SUMS`
asset has SHA256 `50cfd8722e58d1c74fdc543f3e3bb3bac768decd04575fde2360ea838ec5e9d3`. The immutable
[`v0.2.0+0fed14b`](https://github.com/compoundingtech/st2/releases/tag/v0.2.0%2B0fed14b) release targets the
full source commit above; the terminal-green
[`release-portable` run](https://github.com/compoundingtech/st2/actions/runs/30550227417) verifies a fresh
download, checksum, extraction, and execution.
`bin/check-corpus.sh` verifies the variable-age version contract, exact installed binary, strict semantic
validation, fixture resets, and the rest of the model-free corpus gate before an eval may run.
17 changes: 14 additions & 3 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,8 +1,8 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **47 included cells** (16 model-free,
31 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
This is the sole current corpus surface: **57 included cells** (17 model-free,
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
structured run row say so explicitly.
Expand All @@ -17,6 +17,16 @@ harness-native loader plus canonical hook file.
| Cell | Harness | Model(s) / effort | Model seats | Cost | Timeout | Held-out judges | Latest accepted PASS | Last recorded run |
|---|---|---|---:|---|---|---:|---|---|
| `adopt-only-migration` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** |
| `agent-spec-resource-bindings` | model-free | — | 0 | none | `60s` | 4 | — | **NO STRUCTURED RUN** |
| `assignment-contract-cold-assignment` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
| `assignment-contract-cold-focus` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
| `assignment-contract-cold-resources` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 3 | — | **NO STRUCTURED RUN** |
| `assignment-contract-handoff-assignment` | Codex | `gpt-5.6-sol` / medium | 3 | high | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `assignment-contract-handoff-focus` | Codex | `gpt-5.6-sol` / medium | 3 | high | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `assignment-contract-handoff-resources` | Codex | `gpt-5.6-sol` / medium | 3 | high | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `assignment-contract-hot-assignment` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `assignment-contract-hot-focus` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `assignment-contract-hot-resources` | Codex | `gpt-5.6-sol` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** |
| `context-resource-continuity` | model-free | — | 0 | none | `90s` | 4 | — | **NO STRUCTURED RUN** |
| `crash-ding` | mixed | `claude-sonnet-5+gpt-5.6-sol` / medium | 4 | high | `180s` | 5 | — | **NO STRUCTURED RUN** |
| `ding-mode` | Claude | `claude-sonnet-5` / medium | 2 | medium | `1200s` | 4 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -84,6 +94,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
## Harness hook exclusions

- `adopt-only-migration` / `cell` — deterministic native adoption/replacement lifecycle probe; no harness seat
- `agent-spec-resource-bindings` / `cell` — deterministic native Agent Spec Resource-envelope probe; no harness seat
- `context-resource-continuity` / `cell` — deterministic native context/resource restart probe; no harness seat
- `docs` / `judge:cold-reader` — one-shot offline Claude print grader; no bus identity, DING, or hook surface
- `exec-lifecycle-logging` / `cell` — deterministic native exec lifecycle probe; no harness seat
Expand All @@ -101,7 +112,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat

These **17** rows are generated from structured exclusions. A cell-level row is required for
These **18** rows are generated from structured exclusions. A cell-level row is required for
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.

Expand Down
38 changes: 37 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -97,7 +97,7 @@ VERDICT: PASS
```

Requirements are `st2 0.1.0` from source
[`9887b28`](https://github.com/compoundingtech/st2/commit/9887b2842222def0838c2cd82e6c24c218f7efa6),
[`0fed14b`](https://github.com/compoundingtech/st2/commit/0fed14bb5653b67e1d64f1199e240c4c5c612bf7),
`pty`, Bash, Git, `jq`, Rust/Cargo for the pinned KDL parser gate, and Node for JavaScript fixtures. A paid cell
also needs every harness named by its dry-run row.

Expand All @@ -123,6 +123,42 @@ current bus-connected model agent.
Team-less cells use deterministic `run` steps and judges without a model. Current examples cover native hook
materialization, network health, catalog/pty isolation, and pty send/peek behavior.

## Native Resource-envelope acceptance

`agent-spec-resource-bindings` is the model-free companion to
[`compoundingtech/st2#86`](https://github.com/compoundingtech/st2/pull/86). It exercises native catalog
declarations rather than the tournament's synthetic Resource documents. The cell proves strict envelope
validation, deterministic `st2 agents --json` inspection, opaque downstream tags, exact URI preservation, and
adoption of a live task after a Resource-only declaration edit.

Folder-eval Resource projection, Resource resolution, access, readiness, and lifecycle policy are outside this
cell. The portable Agent Spec envelope does not imply any of them.

## Resource-binding tournament

The nine `assignment-contract-*` cells form one matched tournament over three lifecycle scenarios and three
Agent Spec treatments. The kickoff and product task are treatment-neutral; only the experimental durable
declaration and its resolver rules differ.

| Scenario | Direct named resources | Resources plus `focus` | Resources plus `assignment` |
| --- | --- | --- | --- |
| Cold discovery | One `work` binding selects the issue | `focus` selects an `intent` binding | An Assignment groups four bindings |
| Hot retarget | Rebind `work`, then remove it | Rebind focused `intent`, then remove `focus` | Replace the active Assignment, then make it idle |
| Handoff/restart | Remove A's `work`, then add B's | Remove A's focus, then focus B | Make A idle, then activate B |

The selected contract is the direct-resource treatment: the resource URI is identity, `_tag` discriminates
the resource type, and the KDL name is the binding's semantic role. An agent has zero or one direct `work`
binding; zero means idle. A handoff publishes `A -> no holder -> B`. The Focus and Assignment treatments remain
as controls, not proposed product layers. Required/optional and access semantics are intentionally outside
this tournament.

Exploratory Codex E2E runs on 2026-07-30 completed the intended behavior in all nine cells. Three original
graders produced false negatives: abbreviated commit evidence in cold Focus, message cardinality in hot Focus,
and terminal ordering in resource-only handoff. The checked-in graders accept the preserved correct behavior
and include model-free regression tests for those oracle boundaries. These pre-publication runs are design
evidence, not accepted corpus receipts; `CATALOG.md` remains authoritative and will show no accepted PASS until
a committed cell is rerun.

## Add or change a cell

1. Create exactly one `cells/<name>/<name>.kdl`.
Expand Down
13 changes: 4 additions & 9 deletions bin/check-corpus.sh
Original file line number Diff line number Diff line change
Expand Up @@ -5,10 +5,9 @@ set -euo pipefail
repo_root="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
cd "$repo_root"

expected_source="9887b28"
expected_source_full="9887b2842222def0838c2cd82e6c24c218f7efa6"
expected_binary_sha256="d49d44fd4f3f6f655455c212353a469fefa956082bedf22163deb767d8a36a0d"
expected_version_regex='^st2 0\.1\.0 — running from local source \(9887b28, .+ ago\)$'
expected_source="0fed14b"
expected_binary_sha256="d61d12b2b1189a391c196ca28f8f4ba69072d14fcbad2571fc29db1f250f4eed"
expected_version_regex='^st2 0\.1\.0 — running from local source \(0fed14b, .+ ago\)$'
st2_path="$(command -v st2)"
actual_version="$(st2 --version)"
[[ "$actual_version" =~ $expected_version_regex ]] || {
Expand All @@ -20,11 +19,7 @@ actual_binary_sha256="$(sha256sum "$st2_path" | awk '{ print $1 }')"
echo "FAIL: expected st2 binary sha256 $expected_binary_sha256, found $actual_binary_sha256 at $st2_path" >&2
exit 1
}
LC_ALL=C grep -aFq "$expected_source_full" "$st2_path" || {
echo "FAIL: st2 binary at $st2_path does not embed full pinned source $expected_source_full" >&2
exit 1
}
echo "PASS: pinned runner source $expected_source ($actual_version; sha256 $actual_binary_sha256)"
echo "PASS: pinned published runner source $expected_source ($actual_version; sha256 $actual_binary_sha256)"

mapfile -d '' shell_files < <(
find bin cells -type f -name '*.sh' -not -path '*/_git/*' -print0 | sort -z
Expand Down
16 changes: 16 additions & 0 deletions cells/agent-spec-resource-bindings/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,16 @@
# agent-spec-resource-bindings

Model-free acceptance coverage for the native Agent Spec Resource envelope introduced by
[`compoundingtech/st2#86`](https://github.com/compoundingtech/st2/pull/86).

The cell proves the catalog contract directly through st2:

- KDL, TOML, and JSON reject malformed URIs, duplicate binding names, and unsupported policy properties.
- Resource names are unique and `_tag` remains opaque.
- Canonical KDL and supported TOML/JSON forms project the same stable, name-ordered
`st2 agents --json` descriptors without normalizing URI bytes.
- Editing only Resource bindings updates the declared roster while the existing PTY keeps the same process
identity and is adopted rather than stopped or relaunched.

It does not exercise folder-eval Resource projection, resolution, access, readiness, or Resource lifecycle
policy. Those are outside the portable envelope.
Original file line number Diff line number Diff line change
@@ -0,0 +1,24 @@
// Native Agent Spec Resource envelope: parse failures, roster projection, and nondisruptive adoption.
eval {
copy "./fixture"
max-timeout "60s"

run "contract" {
command "bash ./contract.sh"
}

judges {
judge "STRICT ENVELOPE — malformed URI, duplicate names, and policy fields fail closed" {
exec "grep -Fqx RESOURCE-STRICT-FAILURES-GREEN-c214 $RUNS_DIR/contract.out"
}
judge "INSPECTION — agents JSON is deterministic and preserves opaque tags plus exact URI bytes" {
exec "grep -Fqx RESOURCE-INSPECTION-GREEN-c214 $RUNS_DIR/contract.out"
}
judge "ADOPTION — a Resource-only declaration edit updates inspection without replacing the live task" {
exec "grep -Fqx RESOURCE-NONDISRUPTIVE-ADOPTION-GREEN-c214 $RUNS_DIR/contract.out"
}
judge "CLEANUP — the catalog leaves no live or retained PTY state" {
exec "grep -Fqx RESOURCE-CLEANUP-GREEN-c214 $RUNS_DIR/contract.out"
}
}
}
Loading