diff --git a/docs/vrs/03-host-local/requirements.md b/docs/vrs/03-host-local/requirements.md new file mode 100644 index 0000000..8e2fb0d --- /dev/null +++ b/docs/vrs/03-host-local/requirements.md @@ -0,0 +1,55 @@ +# Host-local supervision — Requirements + +Host-local supervision is one machine's application of the root +[st2 vision](../vision.md). It refines host placement, root supervision, and +control-plane replacement in +[R03](../requirements.md#L38-L39), +[R04](../requirements.md#L43-L46), and +[R11](../requirements.md#L65-L70). Partition safety remains a fleet-level +contract in [R18/R22](../requirements.md); this sub-VRS does not redefine it. + +## Requirements + +- **HOST-R01 One local subject:** Each resident st2 control-plane instance + selects one catalog and one host and reconciles only that pair. One host may + run separate resident instances for other catalogs. The instance is the st2 + process, not the host's root agent or an ordinary supervisor persona. Another + host's declarations and runtime records are outside the selected subject. +- **HOST-R02 Local desired-versus-actual convergence:** The deterministic + resident st2 control-plane instance compares declarations pinned to the + selected host with that host's observed task state, adopts matching live + work, and starts only genuinely missing work. +- **HOST-R03 One control-plane writer:** At most one resident st2 control-plane + instance may reconcile a given catalog and host at a time. A different host + running an instance against the same synced catalog is a different subject. +- **HOST-R04 Independent task lifetime:** Stopping, killing, or replacing the + resident st2 control-plane instance does not stop or replace running agent + tasks. A successor instance adopts surviving tasks without duplicating them. +- **HOST-R05 Explicit destructive lifecycle:** Resident st2 control-plane + instance absence, restart, or loss of a transport peer is not teardown + authority. Local tasks are stopped only by an explicit local lifecycle + decision, including a locally applied retirement declaration or teardown + command. +- **HOST-R06 Intelligent local escalation:** The selected host's root agent + observes local health, performs bounded recovery, and escalates unresolved + failures without turning unavailable peer state into a fleet-health verdict. + This agent role is distinct from the resident st2 control-plane instance and + from an ordinary supervisor persona. +- **HOST-R07 Catalog liveness is not control-plane-instance liveness:** A + catalog remains live while any canonical agent belonging to it is running, + including while its resident st2 control-plane instance is stopped, + restarting, or unavailable. Instance state is reported separately. Its + absence may delay convergence but does not make continuing agents dead, + authorize teardown, or erase the last-applied catalog. DING/sidecar survival + alone does not satisfy this agent-liveness predicate. Under incomplete or + partitioned observation, absence of evidence cannot prove the catalog + globally not live. +- **HOST-R08 Stable resolved state roots:** While a catalog is live, or while + its resident st2 control-plane instance is running, the resolved catalog root + and PTY root remain stable mounted state paths and must not be relocated. + Their contents have different semantics; ordinary catalog edit/sync remains + allowed. Relocation requires the explicit coordinated migration contract in + [issue #85](https://github.com/compoundingtech/st2/issues/85). + +Current mechanisms, executable evidence, and the remaining partition questions +are mapped in [spec.md](spec.md). diff --git a/docs/vrs/03-host-local/spec.md b/docs/vrs/03-host-local/spec.md new file mode 100644 index 0000000..daed941 --- /dev/null +++ b/docs/vrs/03-host-local/spec.md @@ -0,0 +1,89 @@ +# Host-local supervision — Spec + +This is a concise map from the +[Host-local supervision requirements](requirements.md) to current mechanisms +and evidence. It does not define transport, remote attachment, deployment +policy, or a second Doctor health model. + +## Subject selection and ownership + +A resident st2 control-plane instance selects one catalog and an explicit or +locally detected host, then reconciles only that pair. A host may run separate +resident instances for other catalogs. This instance is the st2 process, not +the host's root agent or an ordinary supervisor persona. A +[`HostLock`](../../../src/host_lock.rs#L1-L67) tracks resident ownership of the +selected `(catalog, host)` subject independently from another host's slice of +the same synced catalog. + +## Current mechanism and evidence + +| Requirement | Current mechanism | Evidence | +| --- | --- | --- | +| HOST-R01 | Reconciliation filters every declaration through its resolved host; remote declarations are reported but not acted on. | [`reconcile`](../../../src/reconcile.rs#L106-L191); [host-placement tests](../../../tests/reconcile.rs#L135-L153) | +| HOST-R02 | One resident-instance pass discovers declarations, obtains an authoritative local session view, computes desired versus actual, and then adopts or executes the plan. A failed session listing skips the whole pass. | [reconcile pass](../../../src/run.rs#L737-L813); [reconcile-plan model](../../../src/reconcile.rs#L66-L81) | +| HOST-R03 | The resident st2 control-plane instance checks for a live owner of the same catalog and host, records its own PID, and reclaims stale ownership. | [control-plane entry](../../../src/main.rs#L1738-L1838); [`HostLock` tests](../../../src/host_lock.rs#L83-L143) | +| HOST-R04 | PTY and exec tasks survive normal or forced resident st2 control-plane instance termination and binary replacement; the successor instance preserves PID and creation identity while adopting them. | [replacement acceptance](../../../tests/nomad_survival.rs#L592-L701) | +| HOST-R05 | Normal resident st2 control-plane instance exit leaves tasks running. Teardown and retirement are separate paths that target only the selected host's declared task IDs. | [`down` and teardown](../../../src/run.rs#L987-L1056); [explicit-lifecycle acceptance](../../../tests/nomad_survival.rs#L703-L780) | +| HOST-R06 | The deterministic loop surfaces bounded crash-loop failure to the declared supervisor persona; the distinct root-agent responsibility is owned by root R04. | [crash-loop surfacing](../../../src/run.rs#L1090-L1189); [R04](../requirements.md#L43-L46) | +| HOST-R07 | Agent/task liveness and the resident st2 control-plane instance's `HostLock` record are separate observations. A replacement instance adopts matching work visible in its selected current state. Sidecar-only work does not make an otherwise unrunnable agent live. No global catalog-liveness classifier is implemented. | [replacement adoption](https://github.com/compoundingtech/st2/blob/661c88b6e50cddbdf85e8ffaca9245c46491a1e0/tests/nomad_survival.rs#L608-L717); [separate control-plane report](https://github.com/compoundingtech/st2/blob/661c88b6e50cddbdf85e8ffaca9245c46491a1e0/src/main.rs#L985-L1008); [DING-only boundary](https://github.com/compoundingtech/st2/blob/661c88b6e50cddbdf85e8ffaca9245c46491a1e0/tests/reconcile.rs#L533-L561) | +| HOST-R08 | Stable catalog-root and PTY-root path lifetime is an accepted constraint. The guided coordinated relocation operation is not implemented. | [migration contract](https://github.com/compoundingtech/st2/issues/85) | + +## Catalog liveness and path lifetime + +A live canonical catalog agent keeps the catalog live across resident st2 +control-plane instance downtime. The absent instance is separate factual state: +it can delay convergence, but it cannot turn the continuing agent into dead +work, erase the last-applied catalog, or authorize teardown. On restart, st2 +adopts matching work that is visible in the selected current state and launches +only genuinely missing work. A surviving generated DING sidecar without a +canonical agent does not satisfy the catalog-agent liveness predicate. + +This is a host-local contract, not cross-root or global discovery. Under a +partition or otherwise incomplete observation, st2 cannot infer that a catalog +is globally not live merely because it cannot currently see a canonical agent. +Resident st2 control-plane instance state remains independently reportable. + +While the catalog is live, or while its resident st2 control-plane instance is +running, its resolved catalog root and PTY root are stable mounted state paths. +Ordinary edits and file sync within the catalog remain allowed under the +separate complete-version and last-known-good rules; this is a path-lifetime +constraint, not an opaque-database contract. Relocating either resolved root +requires the explicit coordinated operation in +[issue #85](https://github.com/compoundingtech/st2/issues/85). Ordinary +`up`, `doctor`, and reconciliation remain scoped to the currently selected +paths and never scan arbitrary old roots. + +## Partition boundary + +The root [R18/R22](../requirements.md) contract requires a complete, validated, +locally applied catalog to remain authoritative through transport loss. A plain +synced catalog folder and direct KDL remain a complete operating path. Neither +R18/R22 nor last-known-good host operation requires catalog publication, +compare-and-swap, durable staging, or a content-addressed store. Optional +transactional authoring may be added, but it cannot become a prerequisite for +ordinary direct-KDL operation. + +The current reconciler discovers the live catalog filesystem on each pass +([source](../../../src/run.rs#L737-L758)), while validation is a separate +read-only command ([source](../../../src/validate.rs#L1-L24)). st2 does not yet +identify and order complete candidate versions or retain a durable +last-known-good receipt. That implementation gap does not prescribe CAS or a +content-addressed activation mechanism. + +Peer reachability is not currently a reconciler input. This is consistent with +peer absence being neutral, but the declaration shape for an explicit local +operation that depends on a peer or source is not yet specified. + +## Open questions + +- What identifies a complete candidate catalog and orders it after the locally + applied version? +- What durable receipt lets a replacement control plane recover the + last-known-good version after interruption without making a transactional + authoring path mandatory? +- How does a declaration express a local operation's dependency on a peer or + source without turning peer presence into general health? +- [`HostLock` acquisition](../../../src/host_lock.rs#L26-L48) is currently + check-then-write rather than an atomic create. What ownership primitive + closes simultaneous first-start races while retaining host-scoped + stale-owner recovery? diff --git a/docs/vrs/requirements.md b/docs/vrs/requirements.md index c9ab5ba..3574fb7 100644 --- a/docs/vrs/requirements.md +++ b/docs/vrs/requirements.md @@ -105,6 +105,7 @@ accepted. - **R17 Durable error propagation:** Lifecycle, harness/eval, provider-turn, task/exec/PTY, hook, and delivery errors are durably reported to the responsible supervisor with agent/task identity and actionable context. + - **R19 Targeted reconciliation:** An exact agent/task selector resolves its identity and pinned host before mutation; unknown, ambiguous, and wrong-host targets refuse before writes, listing, or actions. Materialization, hook @@ -122,3 +123,15 @@ accepted. inspection exposes every Resource binding without interpreting its type or URI. Resource-only declaration changes do not alter a task's effective launch definition and do not stop, replace, or relaunch healthy work. + +### Must remain safe through host partitions + +- **R18 Last-known-good local desired state:** During transport loss, each host + continues converging locally applied work from its last complete, validated + catalog. A missing, partial, or invalid incoming catalog cannot replace that + desired state or authorize teardown. +- **R22 Independent fleet convergence:** Temporary catalog-version skew between + hosts is acceptable. On reconnect, a host may apply only a complete, + validated, newer catalog and then converges its own work independently. Peer + or source reachability is factual state, not a health classification; its + absence affects local work only through an explicit local dependency. diff --git a/docs/vrs/spec.md b/docs/vrs/spec.md index 50498cd..2b3e6f2 100644 --- a/docs/vrs/spec.md +++ b/docs/vrs/spec.md @@ -167,6 +167,17 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler lifecycle is the explicit authority to resume ordinary replacement. `retired #true` remains the separate explicit teardown path. +- **Catalog liveness:** The + [Host-local supervision contract](03-host-local/requirements.md) separates + continuing canonical-agent liveness from resident st2 control-plane instance + state. A running canonical agent keeps its catalog live across instance + downtime; DING/sidecar-only survival does not. Incomplete observation cannot + prove a catalog globally not live. While the catalog is live, or its resident + st2 control-plane instance is running, the resolved catalog root and PTY root + remain stable paths; ordinary catalog edit/sync remains allowed, while + relocation requires the coordinated operation in + [issue #85](https://github.com/compoundingtech/st2/issues/85). + - **Session registry:** A catalog owns the `pty` registry holding its tasks. `/pty` is the default; a catalog may declare another so that one host can share a single registry across catalogs. Resolution is an exported @@ -179,6 +190,38 @@ validate ──► materialize ──► host-local st2 scheduler/reconciler closed instead of hanging reconciliation. The deadline is containment, not the mechanism for admitting a larger fleet. +## Partition and catalog activation + +- **R18:** Transport loss does not invalidate a host's locally applied desired + state. Missing, partial, or invalid incoming catalog state cannot replace the + last complete validated version or cause teardown. +- **R22:** Hosts may temporarily apply different catalog versions and converge + independently. Peer or source absence is neutral unless a declared local + operation explicitly depends on it; reconnect is not itself evidence that a + candidate catalog is complete, valid, or newer. + +Current host isolation is source-backed: reconciliation filters declarations +to one selected host +([source](../../src/reconcile.rs#L106-L191)), a host-scoped lock detects an +existing resident writer for the same local subject +([source](../../src/host_lock.rs#L1-L67)), and running tasks survive control +plane loss and adoption +([evidence](../../tests/nomad_survival.rs#L592-L701)). + +A plain synced catalog folder and direct KDL remain a complete operating path +for R18/R22. Last-known-good host operation does not require catalog +publication, compare-and-swap, durable staging, or a content-addressed store. +An optional transactional authoring path may be added, but it cannot become a +prerequisite for direct-KDL operation. + +The complete-version and durable last-known-good half of R18/R22 is not +implemented. Each pass currently discovers the live catalog directory directly +([source](../../src/run.rs#L737-L758)); `st2 validate` is separate and +read-only. There is no identified and ordered complete candidate version or +durable last-known-good receipt. The exact completeness, version, and recovery +contracts remain open design work; that gap does not prescribe CAS or a +content-addressed activation mechanism. + ## Message lifecycle ```text