refactor: extract the agent-spec crate

[schickling-assistant]

Why

A second reader of the same catalog — in our case a read-only TUI over the agent
tree — has to answer "what agents are declared, on which host, under which
supervisor?". Today its only options are to shell out to st2 agents --json or
to re-implement the declaration format.

Neither works:

• The roster JSON is deliberately narrower than the declaration. It is
  {identity, status, name, retired, lastActivity, inbox} — supervisor,
  role and workspace are absent, and host survives only fused into the
  <host>.<identity> bus id. That is the right shape for a roster, and
  INVARIANTS.md pins it as a stable wire contract, so "just add fields" is the
  wrong fix.
• Re-implementing the parse is worse. The interesting part is not the KDL —
  it is discovery's identity/host precedence (content wins, the path supplies
  defaults, a mismatch warns) and the compact-form lowering that derives the
  agent pty and the ding sidecar. A second hand-rolled copy of that drifts
  silently, and the drift shows up as an agent the TUI names differently than
  the runner does.

Extracting the parser makes st2 the reference implementation of the format
rather than one of two copies of it.

What

spec (the model), kdl_format (the KDL parser) and discovery (the catalog
walk) move to crates/agent-spec. st2 depends on the crate and re-exports it
under the original paths, so it is the same code running.

How

• src/spec.rs and src/kdl_format.rs move byte-for-byte (git records
  both as pure renames, 0 changed lines).
• src/lib.rs re-exports agent_spec::{spec, discovery} plus the same item
  list it exported before, so st2::spec::… / st2::discovery::… still
  resolve. src/main.rs is untouched, as is every tests/*.rs except the one
  that moves (below) — nothing in the suite had to learn a new path.
• The ~9 in-tree consumers change only the path prefix: crate::spec:: →
  agent_spec::spec::.
• tests/discovery.rs moves into the crate; its 4 st2:: references become
  agent_spec::.

Two things that are not pure moves, both called out because they are the only
places a reviewer needs to think:

validate needed the raw shape. It read parse_raw_file (pub(crate)) to
see the type string before normalization — that is how a typo'd
type = "srvice" gets caught, since lowering turns any type into
JobType::Service. Exporting that would have dragged RawSpec — the
deliberately-permissive on-disk shape — into the public API, which is a semver
liability for a struct whose whole job is to absorb unknown fields.

Instead the crate exposes a narrow view of what a declaration says before
lowering:

pub struct Declared { pub identity: Option<String>, pub job_type: Option<String> }
pub fn parse_declared(path: &Path) -> anyhow::Result<Vec<Declared>>

RawSpec stays private. Note identity here is the content-declared one
(None when it came from the path) — that distinction is load-bearing, because
validate reports it as the issue's agent and the resolved AgentSpec.identity
would have silently substituted the path-derived value.

flake.nix gains --workspace (1 line). With a root [package] plus
[workspace] members, cargo test --lib --bins selects only st2. Verified
empirically: the crate's 5 unit tests (spec's duration parser, discovery's 4
path_tests) would have silently stopped being gated. 149 lib tests before =
144 + 5 after.

The root deliberately stays a real package rather than becoming a virtual
manifest, because flake.nix reads
(fromTOML (readFile ./Cargo.toml)).package.version and a virtual root has no
[package] to read. build.rs and tests/ stay at the root too.

Explicit non-goals

Kept out on purpose, so this stays reviewable as a move:

• status.rs, message.rs, resource.rs, context.rs are not included.
  These are st2's own conventions, not agent-spec surface — DQ3 in
  docs/vrs/spec.md defers exactly this shape as not yet in AGENT-SPEC.md.
  Shipping them in a crate called agent-spec would assert a contract upstream
  has not made.
• No API neutralization. No anyhow → thiserror, no render_only on the
  model, no splitting delete-on-read. Those change behavior or the error
  contract and belong in their own PR.
• validate.rs, materialize.rs, run.rs, hooks.rs stay. materialize
  pulls run + hooks, so the cut stops before it.
• No spec-version constant. See the question below.

Verified

• nix flake check green — 149 lib/bin unit tests (144 st2 + 5
  agent-spec), the same 149 that run on main.
• cargo build --locked green; Cargo.lock regenerated for the new member and
  committed here.
• cargo test --test validate green (29 tests), run in isolation. This is
  the one place the PR is not a pure move, and --lib --bins does not reach it,
  so I ran it deliberately: tests/validate.rs covers the unknown-type case
  (type = "srvice") that the parse_declared swap has to keep working, both
  as a human issue and in the JSON output.
• --test reconcile (14), --test catalog_selection (5), --test message (5),
  --test message_cli (2), --test compile_agent (4) green — the rest of the
  suite that leans on spec/discovery.
• cargo test --test invariants green, also run deliberately since
  --lib --bins does not reach it. INVARIANTS.md needs no edit: none of its 15
  referenced proof files are moved by this PR, and tests/invariants.rs stays
  in the root package, so CARGO_MANIFEST_DIR still resolves every row and the
  checked >= 20 floor still holds.

What I did not verify: the remaining tests/*.rs — the survival, transport,
PTY and hook gates. As the cargoTestFlags comment already notes, they need
jq, /bin/bash, live PTY backends and a systemd --user manager; my machine
does not fully provide them and they fail on main too. They are not gated by
CI either, so I am not offering them as a signal in any direction. No file they
touch is modified by this PR.

One question for you

Should the crate declare a spec revision — and in which direction? I left the
constant out, and having now read AGENT-SPEC.md I think that is right, but for
the opposite reason to the one I first assumed.

That document already pins, in the other direction: it declares itself pinned to
st2 9887b2842222def0838c2cd82e6c24c218f7efa6 (0.1.0, source 9887b28) and
documents "the hand-authored KDL accepted at that commit". So the spec names an
st2 revision, not the reverse. A crate-side constant naming a spec rev would
either duplicate that relationship or contradict it, and this repo deliberately
avoids duplicating ledgers.

If you do want the crate to declare conformance, the question is which direction
you intend: the crate naming the st2 commit the spec is pinned to, or the spec
growing its own identifier. For reference, 9887b28 is an ancestor of main,
20 commits back as of this PR — so the two are already drifting, which is the
argument for deciding this deliberately rather than in a refactor.

Happy to drop or reshape either the Declared API or the --workspace line if
you would rather solve those differently.

---

Corrections, applied after this description was first posted. Two things in
this section were wrong when originally published, and I would rather flag them
than quietly edit them:

1. It claimed compoundingtech/agent-spec exists as a separate repository with a
   draft README, and built a second question about crate naming on top of that.
   No such repository exists. I had seen a local checkout with that name and a
   remote configured for it, and took that as proof the remote existed — it is an
   unpushed local artifact, and git remote -v only reports configured intent.
   Both the claim and the question it supported are removed.
2. The first correction then asserted that AGENT-SPEC.md "carries no version,
   rev or date to point at". That was also wrong — it opens by pinning itself to
   an explicit st2 commit. The question above is rewritten against what the
   document actually says.

Nothing in the code, the diff, or the "Verified" section depends on either claim;
both were confined to this question. Apologies for the noise.

Posted on behalf of @schickling

field	value
agent_name	cl2-peony
agent_session_id	8464a883-6285-4cd0-a9ec-3eba09830300
agent_tool	Claude Code
agent_tool_version	2.1.220
agent_runtime	Claude Code 2.1.220
agent_model	claude-opus-5
runtime_profile	/nix/store/i8y8b542cyqi385ywcjw5fvsq24f75v4-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest	/nix/store/i81qxhzlrzcxrrdwpp6i8hagka2gby8y-agent-skills-corpus/share/agent-skills/manifest.json
worktree	st2/schickling-assistant/2026-07-29-2026-07-29-agent-spec-crate
machine	dev3
tooling_profile	dotfiles@unknown-dirty

[schickling]

@codex

[chatgpt-codex-connector]

💡 Codex Review

Here are some automated review suggestions for this pull request.

Reviewed commit: baab666aaf

ℹ️ About Codex in GitHub

Your team has set up Codex to review pull requests in this repo. Reviews are triggered when you

• Open a pull request for review
• Mark a draft as ready
• Comment "@codex review".

If Codex has suggestions, it will comment; otherwise it will react with 👍.

Codex can also answer questions or update the PR. Try commenting "@codex address that feedback".

[myobie]

Blocking exact head baab666aafce5d7d00564358ff780ec07c93de5e on test-gate coverage.

Cargo.toml:4-5 creates a non-virtual workspace but does not set default-members. Because the root remains a package, cargo metadata reports only st2 in workspace_default_members. The documented complete gate at README.md:301-306 therefore skips the extracted crate entirely:

cargo test --all-targets --all-features -- --list       => 293 tests
cargo test --workspace --all-targets --all-features -- --list => 315 tests
parses_full_kdl_service_job present in the first command => 0
parses_full_kdl_service_job present with --workspace     => 1

That is 5 crate unit tests plus the 17 tests moved to crates/agent-spec/tests/discovery.rs. flake.nix:97-101 adds --workspace, but --lib --bins still deliberately excludes the 17 moved integration tests, so exact-head hosted Nix CI being green does not cover the parser/lowering regression net.

Please make the documented root gate include both workspace packages (for example with default-members or documented --workspace) and add a hosted/hermetic gate for the extracted discovery integration test target. Also format the changed assertion in crates/agent-spec/tests/discovery.rs:216; cargo fmt --all --check reports it (alongside inherited baseline diffs).

Independent behavior evidence is otherwise green: 149 workspace unit tests; 17 discovery; 29 validate including malformed type = "srvice"; invariants; reconcile/materialize/catalog selection/compile-agent/run; full Hetz native suite including PTY survival and both systemd isolation tests; strict Clippy for workspace lib/bin; agent-spec rustdoc. Hosted Nix check is green for this exact SHA. Full all-target Clippy remains red on the pre-existing unused import at tests/exec_backend.rs:5, not this PR.

API audit: Declared preserves the content identity (including None for path-derived identity) and raw type before lowering; st2::spec / st2::discovery remain re-exported; the crate matches the runner-normative subset of evals/AGENT-SPEC.md. PR #50 will need a small rebase resolution in two src/run.rs import hunks, but no semantic incompatibility was found.

[myobie]

Superseding CHANGES_REQUESTED review 4806561853: exact head 3cd5d9a8b84d8135233ab1d0d959a92897619cb6 resolves every blocker. The documented root default gate now selects both workspace packages and lists/runs all 315 tests; hosted Nix now gates 149 workspace unit tests plus the 17 moved discovery integration tests; the affected discovery file passes rustfmt. Local full native coverage is green, including validation, invariants, survival, PTY, and both Linux systemd isolation tests; strict applicable Clippy is green. Hosted Nix run 30441991215 / job 90543228907 is green for this exact SHA. Approved.

[myobie]

Fresh exact-head approval for 8c803e018dfa2e3db58df0babe54eff1eb575ca7 after incorporating current main 85d5cc2cce496e82f889d28bceece7627bae9b49 (PR #56). The sole flake.nix conflict was resolved by retaining PR #56 lifecycle-hook gates and adding PR #54 workspace/discovery coverage. Current-base root coverage is 319/319 (the prior 315 plus four PR #56 units); the exact hermetic selection is green for 153 workspace units, 17 discovery tests, and 14 hook integration tests; full native survival/systemd coverage and strict applicable Clippy are green. Hosted Nix run 30442506571 / job 90544894207 is green for this exact SHA. Approved.