Skip to content
Draft
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
55 changes: 55 additions & 0 deletions docs/vrs/03-host-local/requirements.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,55 @@
# Host-local supervision — Requirements

Host-local supervision is one machine's application of the root
[st2 vision](../vision.md). It refines host placement, root supervision, and
control-plane replacement in
[R03](../requirements.md#L38-L39),
[R04](../requirements.md#L43-L46), and
[R11](../requirements.md#L65-L70). Partition safety remains a fleet-level
contract in [R18/R22](../requirements.md); this sub-VRS does not redefine it.

## Requirements

- **HOST-R01 One local subject:** Each resident st2 control-plane instance
selects one catalog and one host and reconciles only that pair. One host may
run separate resident instances for other catalogs. The instance is the st2
process, not the host's root agent or an ordinary supervisor persona. Another
host's declarations and runtime records are outside the selected subject.
- **HOST-R02 Local desired-versus-actual convergence:** The deterministic
resident st2 control-plane instance compares declarations pinned to the
selected host with that host's observed task state, adopts matching live
work, and starts only genuinely missing work.
- **HOST-R03 One control-plane writer:** At most one resident st2 control-plane
instance may reconcile a given catalog and host at a time. A different host
running an instance against the same synced catalog is a different subject.
- **HOST-R04 Independent task lifetime:** Stopping, killing, or replacing the
resident st2 control-plane instance does not stop or replace running agent
tasks. A successor instance adopts surviving tasks without duplicating them.
- **HOST-R05 Explicit destructive lifecycle:** Resident st2 control-plane
instance absence, restart, or loss of a transport peer is not teardown
authority. Local tasks are stopped only by an explicit local lifecycle
decision, including a locally applied retirement declaration or teardown
command.
- **HOST-R06 Intelligent local escalation:** The selected host's root agent
observes local health, performs bounded recovery, and escalates unresolved
failures without turning unavailable peer state into a fleet-health verdict.
This agent role is distinct from the resident st2 control-plane instance and
from an ordinary supervisor persona.
- **HOST-R07 Catalog liveness is not control-plane-instance liveness:** A
catalog remains live while any canonical agent belonging to it is running,
including while its resident st2 control-plane instance is stopped,
restarting, or unavailable. Instance state is reported separately. Its
absence may delay convergence but does not make continuing agents dead,
authorize teardown, or erase the last-applied catalog. DING/sidecar survival
alone does not satisfy this agent-liveness predicate. Under incomplete or
partitioned observation, absence of evidence cannot prove the catalog
globally not live.
- **HOST-R08 Stable resolved state roots:** While a catalog is live, or while
its resident st2 control-plane instance is running, the resolved catalog root
and PTY root remain stable mounted state paths and must not be relocated.
Their contents have different semantics; ordinary catalog edit/sync remains
allowed. Relocation requires the explicit coordinated migration contract in
[issue #85](https://github.com/compoundingtech/st2/issues/85).

Current mechanisms, executable evidence, and the remaining partition questions
are mapped in [spec.md](spec.md).
89 changes: 89 additions & 0 deletions docs/vrs/03-host-local/spec.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,89 @@
# Host-local supervision — Spec

This is a concise map from the
[Host-local supervision requirements](requirements.md) to current mechanisms
and evidence. It does not define transport, remote attachment, deployment
policy, or a second Doctor health model.

## Subject selection and ownership

A resident st2 control-plane instance selects one catalog and an explicit or
locally detected host, then reconciles only that pair. A host may run separate
resident instances for other catalogs. This instance is the st2 process, not
the host's root agent or an ordinary supervisor persona. A
[`HostLock`](../../../src/host_lock.rs#L1-L67) tracks resident ownership of the
selected `(catalog, host)` subject independently from another host's slice of
the same synced catalog.

## Current mechanism and evidence

| Requirement | Current mechanism | Evidence |
| --- | --- | --- |
| HOST-R01 | Reconciliation filters every declaration through its resolved host; remote declarations are reported but not acted on. | [`reconcile`](../../../src/reconcile.rs#L106-L191); [host-placement tests](../../../tests/reconcile.rs#L135-L153) |
| HOST-R02 | One resident-instance pass discovers declarations, obtains an authoritative local session view, computes desired versus actual, and then adopts or executes the plan. A failed session listing skips the whole pass. | [reconcile pass](../../../src/run.rs#L737-L813); [reconcile-plan model](../../../src/reconcile.rs#L66-L81) |
| HOST-R03 | The resident st2 control-plane instance checks for a live owner of the same catalog and host, records its own PID, and reclaims stale ownership. | [control-plane entry](../../../src/main.rs#L1738-L1838); [`HostLock` tests](../../../src/host_lock.rs#L83-L143) |
| HOST-R04 | PTY and exec tasks survive normal or forced resident st2 control-plane instance termination and binary replacement; the successor instance preserves PID and creation identity while adopting them. | [replacement acceptance](../../../tests/nomad_survival.rs#L592-L701) |
| HOST-R05 | Normal resident st2 control-plane instance exit leaves tasks running. Teardown and retirement are separate paths that target only the selected host's declared task IDs. | [`down` and teardown](../../../src/run.rs#L987-L1056); [explicit-lifecycle acceptance](../../../tests/nomad_survival.rs#L703-L780) |
| HOST-R06 | The deterministic loop surfaces bounded crash-loop failure to the declared supervisor persona; the distinct root-agent responsibility is owned by root R04. | [crash-loop surfacing](../../../src/run.rs#L1090-L1189); [R04](../requirements.md#L43-L46) |
| HOST-R07 | Agent/task liveness and the resident st2 control-plane instance's `HostLock` record are separate observations. A replacement instance adopts matching work visible in its selected current state. Sidecar-only work does not make an otherwise unrunnable agent live. No global catalog-liveness classifier is implemented. | [replacement adoption](https://github.com/compoundingtech/st2/blob/661c88b6e50cddbdf85e8ffaca9245c46491a1e0/tests/nomad_survival.rs#L608-L717); [separate control-plane report](https://github.com/compoundingtech/st2/blob/661c88b6e50cddbdf85e8ffaca9245c46491a1e0/src/main.rs#L985-L1008); [DING-only boundary](https://github.com/compoundingtech/st2/blob/661c88b6e50cddbdf85e8ffaca9245c46491a1e0/tests/reconcile.rs#L533-L561) |
| HOST-R08 | Stable catalog-root and PTY-root path lifetime is an accepted constraint. The guided coordinated relocation operation is not implemented. | [migration contract](https://github.com/compoundingtech/st2/issues/85) |

## Catalog liveness and path lifetime

A live canonical catalog agent keeps the catalog live across resident st2
control-plane instance downtime. The absent instance is separate factual state:
it can delay convergence, but it cannot turn the continuing agent into dead
work, erase the last-applied catalog, or authorize teardown. On restart, st2
adopts matching work that is visible in the selected current state and launches
only genuinely missing work. A surviving generated DING sidecar without a
canonical agent does not satisfy the catalog-agent liveness predicate.

This is a host-local contract, not cross-root or global discovery. Under a
partition or otherwise incomplete observation, st2 cannot infer that a catalog
is globally not live merely because it cannot currently see a canonical agent.
Resident st2 control-plane instance state remains independently reportable.

While the catalog is live, or while its resident st2 control-plane instance is
running, its resolved catalog root and PTY root are stable mounted state paths.
Ordinary edits and file sync within the catalog remain allowed under the
separate complete-version and last-known-good rules; this is a path-lifetime
constraint, not an opaque-database contract. Relocating either resolved root
requires the explicit coordinated operation in
[issue #85](https://github.com/compoundingtech/st2/issues/85). Ordinary
`up`, `doctor`, and reconciliation remain scoped to the currently selected
paths and never scan arbitrary old roots.

## Partition boundary

The root [R18/R22](../requirements.md) contract requires a complete, validated,
locally applied catalog to remain authoritative through transport loss. A plain
synced catalog folder and direct KDL remain a complete operating path. Neither
R18/R22 nor last-known-good host operation requires catalog publication,
compare-and-swap, durable staging, or a content-addressed store. Optional
transactional authoring may be added, but it cannot become a prerequisite for
ordinary direct-KDL operation.

The current reconciler discovers the live catalog filesystem on each pass
([source](../../../src/run.rs#L737-L758)), while validation is a separate
read-only command ([source](../../../src/validate.rs#L1-L24)). st2 does not yet
identify and order complete candidate versions or retain a durable
last-known-good receipt. That implementation gap does not prescribe CAS or a
content-addressed activation mechanism.

Peer reachability is not currently a reconciler input. This is consistent with
peer absence being neutral, but the declaration shape for an explicit local
operation that depends on a peer or source is not yet specified.

## Open questions

- What identifies a complete candidate catalog and orders it after the locally
applied version?
- What durable receipt lets a replacement control plane recover the
last-known-good version after interruption without making a transactional
authoring path mandatory?
- How does a declaration express a local operation's dependency on a peer or
source without turning peer presence into general health?
- [`HostLock` acquisition](../../../src/host_lock.rs#L26-L48) is currently
check-then-write rather than an atomic create. What ownership primitive
closes simultaneous first-start races while retaining host-scoped
stale-owner recovery?
13 changes: 13 additions & 0 deletions docs/vrs/requirements.md
Original file line number Diff line number Diff line change
Expand Up @@ -105,6 +105,7 @@ accepted.
- **R17 Durable error propagation:** Lifecycle, harness/eval, provider-turn,
task/exec/PTY, hook, and delivery errors are durably reported to the
responsible supervisor with agent/task identity and actionable context.

- **R19 Targeted reconciliation:** An exact agent/task selector resolves its
identity and pinned host before mutation; unknown, ambiguous, and wrong-host
targets refuse before writes, listing, or actions. Materialization, hook
Expand All @@ -122,3 +123,15 @@ accepted.
inspection exposes every Resource binding without interpreting its type or URI.
Resource-only declaration changes do not alter a task's effective launch
definition and do not stop, replace, or relaunch healthy work.

### Must remain safe through host partitions

- **R18 Last-known-good local desired state:** During transport loss, each host
continues converging locally applied work from its last complete, validated
catalog. A missing, partial, or invalid incoming catalog cannot replace that
desired state or authorize teardown.
- **R22 Independent fleet convergence:** Temporary catalog-version skew between
hosts is acceptable. On reconnect, a host may apply only a complete,
validated, newer catalog and then converges its own work independently. Peer
or source reachability is factual state, not a health classification; its
absence affects local work only through an explicit local dependency.
43 changes: 43 additions & 0 deletions docs/vrs/spec.md
Original file line number Diff line number Diff line change
Expand Up @@ -167,6 +167,17 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
lifecycle is the explicit authority to resume ordinary replacement.
`retired #true` remains the separate explicit teardown path.

- **Catalog liveness:** The
[Host-local supervision contract](03-host-local/requirements.md) separates
continuing canonical-agent liveness from resident st2 control-plane instance
state. A running canonical agent keeps its catalog live across instance
downtime; DING/sidecar-only survival does not. Incomplete observation cannot
prove a catalog globally not live. While the catalog is live, or its resident
st2 control-plane instance is running, the resolved catalog root and PTY root
remain stable paths; ordinary catalog edit/sync remains allowed, while
relocation requires the coordinated operation in
[issue #85](https://github.com/compoundingtech/st2/issues/85).

- **Session registry:** A catalog owns the `pty` registry holding its tasks.
`<catalog>/pty` is the default; a catalog may declare another so that one host
can share a single registry across catalogs. Resolution is an exported
Expand All @@ -179,6 +190,38 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler
closed instead of hanging reconciliation. The deadline is containment, not
the mechanism for admitting a larger fleet.

## Partition and catalog activation

- **R18:** Transport loss does not invalidate a host's locally applied desired
state. Missing, partial, or invalid incoming catalog state cannot replace the
last complete validated version or cause teardown.
- **R22:** Hosts may temporarily apply different catalog versions and converge
independently. Peer or source absence is neutral unless a declared local
operation explicitly depends on it; reconnect is not itself evidence that a
candidate catalog is complete, valid, or newer.

Current host isolation is source-backed: reconciliation filters declarations
to one selected host
([source](../../src/reconcile.rs#L106-L191)), a host-scoped lock detects an
existing resident writer for the same local subject
([source](../../src/host_lock.rs#L1-L67)), and running tasks survive control
plane loss and adoption
([evidence](../../tests/nomad_survival.rs#L592-L701)).

A plain synced catalog folder and direct KDL remain a complete operating path
for R18/R22. Last-known-good host operation does not require catalog
publication, compare-and-swap, durable staging, or a content-addressed store.
An optional transactional authoring path may be added, but it cannot become a
prerequisite for direct-KDL operation.

The complete-version and durable last-known-good half of R18/R22 is not
implemented. Each pass currently discovers the live catalog directory directly
([source](../../src/run.rs#L737-L758)); `st2 validate` is separate and
read-only. There is no identified and ordered complete candidate version or
durable last-known-good receipt. The exact completeness, version, and recovery
contracts remain open design work; that gap does not prescribe CAS or a
content-addressed activation mechanism.

## Message lifecycle

```text
Expand Down
Loading