Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
6 changes: 4 additions & 2 deletions CATALOG.md
Original file line number Diff line number Diff line change
@@ -1,7 +1,7 @@
<!-- Generated by bin/generate-catalog.sh from cell KDL and evidence/*.tsv. Do not edit. -->
# Current eval catalog and evidence

This is the sole current corpus surface: **58 included cells** (18 model-free,
This is the sole current corpus surface: **59 included cells** (19 model-free,
40 model-backed) and **4 retired exclusions**. Inventory fields are derived from the active
KDL and executable judge scripts. Latest accepted PASS evidence remains distinct from the append-only run
history: a failed last run is visible without being advertised as accepted evidence, and cells with no
Expand Down Expand Up @@ -50,6 +50,7 @@ harness-native loader plus canonical hook file.
| `presence-ding-matrix` | model-free | — | 0 | none | `60s` | 6 | — | **NO STRUCTURED RUN** |
| `pty-attach-machine-stream` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** |
| `pty-attach-only` | model-free | — | 0 | none | `90s` | 7 | — | **NO STRUCTURED RUN** |
| `pty-live-recovery` | model-free | — | 0 | none | `120s` | 10 | — | **NO STRUCTURED RUN** |
| `pty-send-peek` | model-free | — | 0 | none | `120s` | 5 | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) | **PASS** 2026-07-28T19:53:59Z, 13/13, source [`b7da882`](https://github.com/compoundingtech/evals/commit/b7da8826b503e0408e309af8702fb684d28e3a19), [receipt](evidence/stable-main-phase4-runs-20260728.json) |
| `reconcile-retire-keep` | model-free | — | 0 | none | `90s` | 6 | — | **NO STRUCTURED RUN** |
| `render-target-safety` | model-free | — | 0 | none | `90s` | 5 | — | **NO STRUCTURED RUN** |
Expand Down Expand Up @@ -103,6 +104,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `host-lock-health-negatives` / `cell` — deterministic native doctor negative probe; no harness seat
- `pty-attach-machine-stream` / `cell` — deterministic installed PTY attach composition probe; no harness seat
- `pty-attach-only` / `cell` — deterministic PTY dead-attach policy probe; no harness seat
- `pty-live-recovery` / `cell` — deterministic PTY live-registry recovery probe; no harness seat
- `pty-send-peek` / `cell` — deterministic PTY transport probe; no harness seat
- `presence-ding-matrix` / `cell` — deterministic native presence and DING probe; no harness seat
- `reconcile-retire-keep` / `cell` — deterministic native reconcile lifecycle probe; no harness seat
Expand All @@ -114,7 +116,7 @@ while the last-run column makes a recorded failure distinct from a cell with no
- `targeted-reconcile-isolation` / `cell` — deterministic selected reconcile isolation probe; no bus-connected agent
- `two-networks-coexist` / `cell` — deterministic catalog-isolation probe; no harness seat

These **19** rows are generated from structured exclusions. A cell-level row is required for
These **20** rows are generated from structured exclusions. A cell-level row is required for
every derived model-free cell. The one-shot docs cold reader is a model-backed grader, but not a bus-connected
agent: it has no st2 identity, DING sidecar, or lifecycle hooks.

Expand Down
44 changes: 44 additions & 0 deletions cells/pty-live-recovery/README.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,44 @@
# pty-live-recovery

Held-out, model-free black-box coverage for the selected live-daemon recovery
protocol from [PTY PR #128](https://github.com/compoundingtech/pty/pull/128),
squash-merged at exact source
`9eb958c5aae026d5c05690ab72b528662c55708d`.

Set `EVALS_PTY_PR128_ROOT` to a clean checkout of that merged source after
`npm ci && npm run build`. The cell rejects any other Git head, package lock,
or built CLI/server artifacts. Its synthetic PTY roots and providers live only
inside the temporary eval catalog; it launches no model or provider harness.

## What it proves

- `pty recover <name> --snapshot <metadata.json>` is the only accepted surface.
- A private root advertises the complete protocol, secret, process-start,
launch-identity, root/recovery-directory identity, and metadata-revision
capability.
- Recovery preserves the daemon PID, generation, process-start token, launch
identity, existing client, and provider while rotating the secret and signed
metadata revision.
- A fresh client reaches the republished pathname, and every synthetic provider
still has exactly one launch and one `session_start`.
- A rotated snapshot cannot replay; a pre-tag snapshot cannot roll back current
metadata; and a secret-tampered snapshot cannot publish a registry. Each
refusal leaves the original daemon alive and a current valid snapshot usable.
- Group/world-accessible root or `.recovery` modes fail before request/result,
lock, socket, PID, or metadata publication.
- A daemon started without a recovery capability is refused without signal,
restart, relaunch, or a recoverability claim.
- Cleanup binds every captured PID to its process-start token, uses bounded
TERM then KILL only for those exact identities, fails if any survives, and
exercises the KILL fallback with a TERM-resistant synthetic sentinel.

The experiment is exact-source merged evidence, not a published PTY release artifact.
It deliberately replaces the incompatible PR #127 oracle rather than merely
retargeting its metadata.

## Run it

```sh
EVALS_PTY_PR128_ROOT=/path/to/clean/pty-pr128 \
st2 eval ./cells/pty-live-recovery/
```
9 changes: 9 additions & 0 deletions cells/pty-live-recovery/fixture/live.sh
Original file line number Diff line number Diff line change
@@ -0,0 +1,9 @@
#!/usr/bin/env bash
set -euo pipefail

marker="${1:?marker path required}"
printf 'run\n' >>"$marker"
printf 'LIVE-RECOVERY-READY\n'
while IFS= read -r line; do
printf 'LIVE-RECOVERY-ACK:%s\n' "$line"
done
Loading