Evaluate direct KDL authoring versus a guarded Agent Spec mutation CLI

[schickling-assistant]

Question

For agents that create, revise, retire, and inspect canonical Agent Specs, which
authoring surface produces better outcomes:

1. direct editing of the hand-authored agent.kdl, guided by AGENT-SPEC.md
   and followed by strict validation; or
2. a narrow mutation CLI that publishes the same canonical KDL while enforcing
   selected lifecycle and authorization constraints?

The KDL remains the desired-state authority in both arms. This issue does not
propose a second declaration model.

Why this needs an eval

Direct KDL editing is the smaller system and matches the repository's canonical
authoring contract. A CLI could prevent malformed, partial, stale, or
unauthorized mutations, but it also adds vocabulary, hides exact bytes behind an
interface, and can become a competing source of truth.

Those tradeoffs should be demonstrated with agent behavior rather than settled
from architectural taste. The default should remain direct KDL unless a guarded
CLI shows a material, repeatable advantage.

Proposed eval shape

Give otherwise equivalent agents the same frozen lifecycle tasks and the same
canonical Agent Spec:

• direct-KDL arm: normal file-editing tools plus st2 validate --strict;
• guarded-CLI arm: a fixture-local reference CLI that writes inspectable
  canonical KDL and exposes no capability absent from the direct arm;
• optional no-guidance control: direct editing with the Agent Spec but without
  the focused authoring procedure, to measure whether a skill/rule closes the gap
  without a product CLI.

The CLI should initially be an eval fixture, not production implementation. Its
purpose is to test the interface hypothesis cheaply.

Scenarios

Exercise at least:

1. create a valid agent with explicit host, identity, workspace, ST_AGENT,
   command, and DING;
2. modify one intended field while preserving unrelated declaration bytes and
   resources;
3. reject an invalid or incomplete candidate before it becomes discoverable;
4. retire an active declaration with retired #true rather than deleting it;
5. distinguish retirement completion from safe declaration removal;
6. resolve a stale/concurrent edit without silently overwriting the newer
   generation;
7. preserve catalog/path identity agreement and avoid phantom declarations; and
8. explain the resulting change and the runtime action still required.

Include both straightforward and adversarially ambiguous prompts. Held-out
judges should accept any semantically correct KDL, not one golden formatting.

Measurements and judges

Compare:

• end-state semantic correctness under the pinned Agent Spec;
• whether invalid or partial declarations ever become discoverable;
• retirement and resource-preservation correctness;
• unintended declaration changes;
• successful handling of stale/concurrent state;
• ability to show the exact accepted KDL and explain why it is valid;
• recovery after a deliberately planted invalid candidate;
• turns, tool calls, tokens, elapsed time, and repair attempts; and
• whether the author confuses publication with materialization, running, or
  readiness.

Every safety judge needs a negative control or mutation proving it goes red for
the fault it claims to detect.

Decision rule

Prefer direct KDL plus a focused Agent Spec skill/rule unless the guarded CLI
shows a significant and repeatable improvement in safety or task success across
multiple providers/models.

A CLI only earns adoption if it:

• leaves canonical KDL fully inspectable;
• introduces no hidden declaration state;
• composes with strict whole-catalog validation;
• fails closed on stale writes and invalid publication;
• improves more than convenience or formatting; and
• does not merely encode guidance that agents follow equally well from the
  canonical spec and a skill.

If the CLI wins only on transactional publication, keep that primitive narrow
and avoid turning it into a higher-level agent schema.

Relationship to existing work

• Specify and evaluate transactional publication of canonical Agent Specs #37 asks for transactional publication and lifecycle conformance for external
  publishers. This issue evaluates whether an agent-facing CLI is needed on top
  of those primitives or whether direct KDL authoring is sufficient.
• The corpus must stay pinned to one exact st2 commit; the cell should land only
  when both arms can be implemented against the same runtime contract.

Non-goals

• adding personas, account selection, permissions, or spawn authorization to the
  Agent Spec grammar;
• making the fixture CLI a production recommendation before evidence exists;
• replacing KDL with intent JSON, Nix, or another intermediate schema; or
• evaluating the separate human-oriented launcher that may select an account,
  create a spec, reconcile it, and attach to its PTY.

Posted on behalf of @schickling

field	value
agent_name	co2-shale
agent_session_id	591448c4-5967-4b12-ae43-1c6320dc1d25
agent_tool	Codex CLI
agent_tool_version	0.145.0
agent_runtime	Codex CLI 0.145.0
agent_model	unknown
runtime_profile	/nix/store/v2g542rika3abxn98x5yimz431j5br13-coding-agent-runtime-profile/share/coding-agents/profile.json
skills_manifest	/nix/store/3aq75z8arl44fsvn49i2k4yqlzb1kcwb-agent-skills-corpus/share/agent-skills/manifest.json
worktree	dotfiles/schickling/2026-07-20-cos-misc
machine	dev3
tooling_profile	dotfiles@unknown-dirty