feat: Custom Software/Agents docs + native options schema (zod) + sidecar logfmt observability

Combines:
- docs: Custom Software section + Custom Agents guide
- feat: native AgentOs options schema (zod/v4) + defineSoftware re-export
- feat(sidecar): configurable logfmt logging + ext stall-watchdog tracing

Author: NathanFlurry

CLAUDE.md

@@ -5,13 +5,40 @@ Agent OS is the agent-facing wrapper around secure-exec. It provides ACP session
 ## Boundaries
 
 - secure-exec dependency workflow. Manage the secure-exec dependency ONLY through `scripts/secure-exec-dep.mjs` (the `just secure-exec-*` recipes); never hand-edit the `path` / `version` / `catalog:` pins.
-  - Testing against local secure-exec changes: run `just secure-exec-local` to repoint npm (`link:`) and crates (`path = "../secure-exec/..."`) at the sibling checkout, then `node scripts/secure-exec-dep.mjs set-crate-version <sibling-version>` so the Cargo version requirement matches the sibling crate version (otherwise cargo cannot resolve the path deps). Use `just secure-exec-status` to inspect. This mode is for local builds/tests ONLY.
+  - Testing against local secure-exec changes: run `just secure-exec-local` to repoint npm (`link:`) and crates (`path = "../secure-exec/..."`) at the sibling checkout, then `node scripts/secure-exec-dep.mjs set-crate-version <sibling-version>` so the Cargo version requirement matches the sibling crate version (otherwise cargo cannot resolve the path deps). Also run `pnpm install` in `../secure-exec` first, or cargo panics in `v8-runtime/build.rs` with "missing Node dependencies at .../packages/build-tools/node_modules" (the V8 bridge assets are built from there). Use `just secure-exec-status` to inspect. This mode is for local builds/tests ONLY.
   - Pushing changes that depend on secure-exec changes: NEVER push with local (`path:` / `link:`) dependencies. First preview-publish the secure-exec changes to their own secure-exec branch (the `preview-publish-secure-exec` flow), then point agent-os back at that exact published version with `just secure-exec-pinned` + `just secure-exec-set-version <version>` (and `set-crate-version <version>` for the crates). Only commit/push the pinned-to-remote state.
 - Keep generic runtime, kernel, VFS, language execution, and registry software behavior in secure-exec.
 - Agent OS owns ACP, sessions, agent adapters, toolkit semantics, quickstarts, and the AgentOs facade.
 - Call OS instances VMs, never sandboxes.
 - The protocol has no backwards compatibility. Clients and the sidecar ship in same-version lockstep, so never add protocol or config versioning, runtime negotiation, fallbacks, or converters. Configs such as `CreateVmConfig` carry no `version` field; the single same-version wire handshake is the only version check. Change the protocol freely and update both sides together.
 
+## Development
+
+### secure-exec dependency versions (`just`)
+
+Two independent version tracks:
+- **secure-exec** — the `@secure-exec/*` npm packages and the `secure-exec-*` Cargo crates always share **one** version (npm and crates are kept in sync; pin both to the same `<v>`).
+- **`@agentos-software/*`** software packages (registry agents / WASM commands) are on a **separate** track and version independently of secure-exec.
+
+Manage them ONLY via these recipes (never hand-edit `path`/`version`/`catalog:` pins):
+- `just secure-exec-local` — point deps at the sibling `../secure-exec` checkout for local hacking.
+- `just secure-exec-set-version <v>` — pin secure-exec to a published version: sets the `@secure-exec/*` npm packages **and** the `secure-exec-*` crates (same `<v>`, they're in sync) and switches to pinned mode.
+- `just agentos-pkgs-set-version <v>` — pin the `@agentos-software/*` software packages (separate version track).
+
+### Depending on unreleased secure-exec changes
+
+agent-os builds against secure-exec crates + npm packages, so a secure-exec change must reach agent-os before it can be pushed. NEVER push with local (`path:`/`link:`) deps. Flow: preview-publish the secure-exec branch (the `preview-publish-secure-exec` skill), then `just secure-exec-set-version <published-version>` (pins npm + crates + switches to pinned mode), and push only that pinned state. Caveat: a preview publishes npm but the crates.io job is dry-run/skipped — a secure-exec *crate* change only flows locally (`secure-exec-local`) or via a real crates.io release.
+
+### Preview-publishing agent-os
+
+`just preview-publish <branch>` dispatches `.github/workflows/publish.yaml` to cut a **preview** (debug build, npm-only, dist-tag = sanitized branch name) — for handing a build to an external project. **Preview-publish is for previews ONLY; never cut a release with it.** Releases go through `just release` (the `scripts/publish` flow).
+
+### Testing a local build from an external project (same machine)
+
+To consume an unpublished agent-os build in another project on this machine:
+- **npm:** `pnpm -r build`, then either `pnpm pack` the package(s) and `npm install ./rivet-dev-agentos-*.tgz` in the external project, or add a `link:`/`file:` override (e.g. `"@rivet-dev/agentos": "link:/abs/path/agent-os/packages/agentos"`). The sidecar binary ships as `@rivet-dev/agentos-sidecar`.
+- **cargo:** point the external Cargo project at the local crate via a path dep or `[patch.crates-io]` override (e.g. `[patch.crates-io] agentos-sidecar = { path = "/abs/path/agent-os/crates/agentos-sidecar" }`).
+
 ## Security Model
 
 Trust model (decide which side of the boundary something is on before judging whether it is a security bug). Three components:
@@ -41,6 +68,7 @@ Trust model (decide which side of the boundary something is on before judging wh
 
 ## Website And Docs
 
+- External/consumer usage (installing `@rivet-dev/agentos` and using it in your own project) is documented in the website quickstart + Agents/Custom Software pages under `website/`, not in this file. This `CLAUDE.md` is contributor/maintainer-only.
 - The Agent OS website and docs live in `website/` (Astro + Starlight) and deploy to `agentos-sdk.dev` (docs at `agentos-sdk.dev/docs`). The marketing pages and docs were migrated out of `rivet.dev/agent-os` and `rivet.dev/docs/agent-os`, which now 301-redirect to this domain.
 - Docs styling is owned by the shared **`@rivet-dev/docs-theme`** repo (`github.com/rivet-dev/docs-theme`), consumed via `github:rivet-dev/docs-theme#<tag>` and wired in via `...docsTheme(starlight, siteConfig)`. To change any docs styling (palette, header, sidebar, code blocks, fonts), edit that repo and follow its CLAUDE.md release workflow — never restyle docs in `website/src`. This site owns only content + `website/docs.config.mjs` (sidebar icons via each item's `attrs['data-icon']`).
 - Architecture reference docs live in `website/src/content/docs/docs/architecture/` and are surfaced in `website/docs.config.mjs` under Reference → Advanced → Architecture. Treat these pages as the canonical human-facing architecture reference. When architecture behavior changes or new architecture is added, recommend the corresponding docs update to the user; do not proactively edit the docs unless the user asks for docs work or the task explicitly includes it.

Cargo.lock

@@ -18,9 +18,11 @@ agentos-protocol = { workspace = true }
 serde_json = "1.0"
 serde_bare = "0.5"
 secure-exec-sidecar = { workspace = true }
-tokio = { version = "1", features = ["sync", "time"] }
+tokio = { version = "1", features = ["sync", "time", "macros"] }
 tracing = "0.1"
-tracing-subscriber = { version = "0.3", features = ["fmt"] }
+tracing-subscriber = { version = "0.3", features = ["fmt", "env-filter"] }
+tracing-logfmt = { version = "0.3", features = ["ansi_logs"] }
+tracing-appender = "0.2"
 
 [dev-dependencies]
 agentos-bridge = { workspace = true }

crates/agentos-sidecar/Cargo.toml

@@ -109,22 +109,74 @@ impl AcpExtension {
         ctx: ExtensionContext<'_>,
         payload: &[u8],
     ) -> Result<ExtensionResponse, SidecarError> {
+        use tracing::Instrument as _;
         let request = decode_request(payload)?;
-        let response = match request {
-            AcpRequest::AcpCreateSessionRequest(request) => self.create_session(ctx, request).await,
-            AcpRequest::AcpGetSessionStateRequest(request) => {
-                AcpHandlerOutput::response(self.get_session_state(ctx, request).await)
+        let kind = Self::acp_request_kind(&request);
+        let start = std::time::Instant::now();
+        tracing::info!(target: "agentos_sidecar::acp_extension", kind, "ext request received");
+
+        let work = async move {
+            match request {
+                AcpRequest::AcpCreateSessionRequest(request) => {
+                    self.create_session(ctx, request).await
+                }
+                AcpRequest::AcpGetSessionStateRequest(request) => {
+                    AcpHandlerOutput::response(self.get_session_state(ctx, request).await)
+                }
+                AcpRequest::AcpCloseSessionRequest(request) => {
+                    AcpHandlerOutput::response(self.close_session(ctx, request).await)
+                }
+                AcpRequest::AcpSessionRequest(request) => self.session_request(ctx, request).await,
+                AcpRequest::AcpResumeSessionRequest(request) => {
+                    self.resume_session(ctx, request).await
+                }
             }
-            AcpRequest::AcpCloseSessionRequest(request) => {
-                AcpHandlerOutput::response(self.close_session(ctx, request).await)
+        }
+        .instrument(tracing::info_span!(
+            target: "agentos_sidecar::acp_extension",
+            "ext.request",
+            kind
+        ));
+
+        // Stall watchdog: while the request is in flight, warn periodically so a
+        // hang surfaces as a breadcrumb long before the host's 120s frame
+        // timeout. This never interrupts the work itself.
+        tokio::pin!(work);
+        let response = loop {
+            tokio::select! {
+                result = &mut work => break result,
+                _ = tokio::time::sleep(std::time::Duration::from_secs(10)) => {
+                    tracing::warn!(
+                        target: "agentos_sidecar::acp_extension",
+                        kind,
+                        elapsed_ms = start.elapsed().as_millis() as u64,
+                        "ext request still pending — possible stall before response frame",
+                    );
+                }
             }
-            AcpRequest::AcpSessionRequest(request) => self.session_request(ctx, request).await,
-            AcpRequest::AcpResumeSessionRequest(request) => self.resume_session(ctx, request).await,
         };
+
+        tracing::info!(
+            target: "agentos_sidecar::acp_extension",
+            kind,
+            elapsed_ms = start.elapsed().as_millis() as u64,
+            "ext request handled",
+        );
         let payload = encode_response(response.response.unwrap_or_else(error_response))?;
         ExtensionResponse::with_wire_events(payload, response.events)
     }
 
+    /// Stable label for an ACP request kind, used as a tracing field.
+    fn acp_request_kind(request: &AcpRequest) -> &'static str {
+        match request {
+            AcpRequest::AcpCreateSessionRequest(_) => "create_session",
+            AcpRequest::AcpGetSessionStateRequest(_) => "get_session_state",
+            AcpRequest::AcpCloseSessionRequest(_) => "close_session",
+            AcpRequest::AcpSessionRequest(_) => "session_request",
+            AcpRequest::AcpResumeSessionRequest(_) => "resume_session",
+        }
+    }
+
     async fn create_session(
         &self,
         mut ctx: ExtensionContext<'_>,

crates/agentos-sidecar/src/acp_extension.rs

@@ -1,17 +1,90 @@
+use tracing_subscriber::fmt::writer::BoxMakeWriter;
+use tracing_subscriber::{layer::SubscriberExt, util::SubscriberInitExt, EnvFilter};
+
 fn main() {
-    // The sidecar speaks its binary protocol over stdout (see
-    // `secure_exec_sidecar::stdio`), so tracing output MUST go to stderr —
-    // otherwise log lines are interleaved into the frame stream and the client
-    // misreads them as (garbage-length) frames. See crates/CLAUDE.md:
-    // "Control channels must be out-of-band."
-    tracing_subscriber::fmt()
-        .with_writer(std::io::stderr)
-        .with_max_level(tracing::Level::ERROR)
-        .init();
+    init_tracing();
     if let Err(error) =
         secure_exec_sidecar::stdio::run_with_extensions(agentos_sidecar_wrapper::extensions())
     {
         tracing::error!(?error, "agentos-sidecar startup failed");
         std::process::exit(1);
     }
 }
+
+/// `1` => true, anything else => false. Mirrors rivet's `env_flag`.
+fn env_flag(name: &str) -> bool {
+    std::env::var(name).map_or(false, |v| v == "1")
+}
+
+/// Initialize tracing for the sidecar.
+///
+/// Mirrors the rivet logging setup (`rivetkit-napi::init_tracing`): an
+/// `EnvFilter`-gated subscriber with a logfmt formatter, so log level and
+/// verbosity are runtime-configurable instead of hardcoded.
+///
+/// Configuration (all optional):
+/// - level: `AGENTOS_LOG_LEVEL` > `LOG_LEVEL` > `RUST_LOG` > `"info"`.
+/// - format: `RUST_LOG_FORMAT=logfmt` (default) or `text`.
+/// - sink: `AGENTOS_LOG_FILE` (append to that file) else stderr.
+/// - field toggles: `RUST_LOG_{SPAN_NAME,SPAN_PATH,TARGET,LOCATION,MODULE_PATH,ANSI_COLOR}`.
+///
+/// The sink MUST be stderr or a file — NEVER stdout, which carries the
+/// sidecar's binary frame protocol (see crates/CLAUDE.md: "Control channels
+/// must be out-of-band").
+fn init_tracing() {
+    // Level priority: AGENTOS_LOG_LEVEL > LOG_LEVEL > RUST_LOG > "info".
+    let directive = std::env::var("AGENTOS_LOG_LEVEL")
+        .ok()
+        .or_else(|| std::env::var("LOG_LEVEL").ok())
+        .or_else(|| std::env::var("RUST_LOG").ok())
+        .unwrap_or_else(|| "info".to_string());
+    let env_filter = EnvFilter::try_new(&directive).unwrap_or_else(|_| EnvFilter::new("info"));
+
+    // Sink: a file if AGENTOS_LOG_FILE is set, else stderr. Never stdout.
+    let writer: BoxMakeWriter = match std::env::var("AGENTOS_LOG_FILE") {
+        Ok(path) if !path.is_empty() => {
+            let path = std::path::PathBuf::from(path);
+            let dir = path
+                .parent()
+                .filter(|p| !p.as_os_str().is_empty())
+                .map(|p| p.to_path_buf())
+                .unwrap_or_else(|| std::path::PathBuf::from("."));
+            let name = path
+                .file_name()
+                .map(std::ffi::OsString::from)
+                .unwrap_or_else(|| std::ffi::OsString::from("agentos-sidecar.log"));
+            BoxMakeWriter::new(tracing_appender::rolling::never(dir, name))
+        }
+        _ => BoxMakeWriter::new(std::io::stderr),
+    };
+
+    let registry = tracing_subscriber::registry().with(env_filter);
+
+    let text_format = std::env::var("RUST_LOG_FORMAT")
+        .map(|v| v.eq_ignore_ascii_case("text"))
+        .unwrap_or(false);
+
+    if text_format {
+        registry
+            .with(
+                tracing_subscriber::fmt::layer()
+                    .with_ansi(false)
+                    .with_writer(writer),
+            )
+            .init();
+    } else {
+        registry
+            .with(
+                tracing_logfmt::builder()
+                    .with_span_name(env_flag("RUST_LOG_SPAN_NAME"))
+                    .with_span_path(env_flag("RUST_LOG_SPAN_PATH"))
+                    .with_target(env_flag("RUST_LOG_TARGET"))
+                    .with_location(env_flag("RUST_LOG_LOCATION"))
+                    .with_module_path(env_flag("RUST_LOG_MODULE_PATH"))
+                    .with_ansi_color(env_flag("RUST_LOG_ANSI_COLOR"))
+                    .layer()
+                    .with_writer(writer),
+            )
+            .init();
+    }
+}

crates/agentos-sidecar/src/main.rs

@@ -6,34 +6,22 @@ release *args:
 preview-publish REF:
 	gh workflow run .github/workflows/publish.yaml --ref "{{ REF }}"
 
-# Point the workspace at PUBLISHED secure-exec versions (CI/release default).
-secure-exec-pinned:
-	node scripts/secure-exec-dep.mjs pinned
-
-# Point the workspace at the sibling ../secure-exec checkout for local hacking.
+# Point deps at the sibling ../secure-exec checkout for local hacking.
 secure-exec-local:
 	node scripts/secure-exec-dep.mjs local
 
-# Bump the pinned @secure-exec/* npm version (core/s3/google-drive/sandbox).
+# Pin secure-exec to a published version. The @secure-exec/* npm packages and the
+# secure-exec-* crates are always the same version, so this sets both and switches
+# to pinned mode.
 secure-exec-set-version VERSION:
+	node scripts/secure-exec-dep.mjs pinned
 	node scripts/secure-exec-dep.mjs set-secure-exec-version "{{ VERSION }}"
+	node scripts/secure-exec-dep.mjs set-crate-version "{{ VERSION }}"
 
-# Bump the pinned @agentos-software/* software-package npm version.
+# Pin the @agentos-software/* software packages (separate version track).
 agentos-pkgs-set-version VERSION:
 	node scripts/secure-exec-dep.mjs set-agentos-pkgs-version "{{ VERSION }}"
 
-# Bump BOTH scopes at once (only when secure-exec + software publish in lockstep).
-secure-exec-set-all-versions VERSION:
-	node scripts/secure-exec-dep.mjs set-version "{{ VERSION }}"
-
-# Bump the @secure-exec/* crate version requirement (must match the sibling crate version).
-secure-exec-set-crate-version VERSION:
-	node scripts/secure-exec-dep.mjs set-crate-version "{{ VERSION }}"
-
-# Show the current secure-exec dependency mode + pinned versions.
-secure-exec-status:
-	node scripts/secure-exec-dep.mjs status
-
 dev-shell *args:
 	pnpm --filter @rivet-dev/agentos-dev-shell dev-shell -- "$@"
 

justfile

@@ -66,7 +66,8 @@
 		"isolated-vm": "^6.0.0",
 		"long-timeout": "^0.1.1",
 		"minimatch": "^10.2.4",
-		"zod": "^4.1.11"
+		"zod": "^4.1.11",
+		"zod-to-json-schema": "^3.25.2"
 	},
 	"devDependencies": {
 		"@anthropic-ai/claude-agent-sdk": "^0.2.87",
@@ -107,6 +108,7 @@
 		"sandbox-agent": "^0.4.2",
 		"typescript": "^5.7.2",
 		"vitest": "^2.1.8",
-		"ws": "^8.18.0"
+		"ws": "^8.18.0",
+		"zod3": "npm:zod@^3.25.76"
 	}
 }