docs: Limits, Bounds & Observability standard + architecture/limits/debugging docs

CLAUDE.md

@@ -52,6 +52,15 @@ Trust model (decide which side of the boundary something is on before judging wh
 - Isolation is layered (defense in depth), like Cloudflare Workers. Untrusted guest code is isolated *within* the host process by V8/WASM virtualization today; host-level jailing (sandboxing the process itself) is a planned additional layer. Because the in-process layer is load-bearing: keep the embedded V8 patched to current security releases, and never let one isolate take down the shared process — a per-isolate failure (heap OOM, CPU runaway) must terminate that isolate, not abort the host process.
 - Match Cloudflare Workers wherever it makes sense. Use Workers' published behavior as the reference point for isolation semantics, resource limits, and egress defaults — e.g. ~128 MiB memory per isolate, bounded CPU time, default-deny network egress. Resource limits must be bounded by default (never `None`/0 for memory, heap, stack, or CPU time); operators may raise them.
 
+## Limits, Bounds & Observability
+
+Every limit, timeout, bounded queue/buffer, and per-entity collection MUST be bounded by default, warn as it approaches, and fail clearly. This applies to both the secure-exec limits agent-os forwards AND agent-os-owned bounds (ACP/session/frame timeouts, the sidecar event buffer, session/shell-id retention, host-tool registration caps, in-VM adapter log buffers). The canonical statement lives in secure-exec root `CLAUDE.md` → "Limits, Bounds & Observability"; agent-os adds:
+
+- **Forward, don't reimplement.** Resource limits (`limits.resources/jsRuntime/python/wasm/acp/tools/...`) are secure-exec's; agent-os forwards them on the wire and surfaces them in `AgentOsOptions.limits` — never duplicate the enforcement or the defaults TS-side. Agent-os owns only the ACP/session/client-layer bounds.
+- **Warn-on-approach + typed error for agent-os-owned bounds.** ACP method timeouts (`initialize`/`session/new`/`session/prompt`/default — `acp_extension.rs`), the native-sidecar frame timeout, and the event buffer must emit a structured near-threshold signal (default ≥80%) and fail with a typed error that names the limit and how to raise it. ACP timeout errors already carry `data.kind === "acp_timeout"` (keep using `isAcpTimeoutErrorData`); extend the rest to match. The default 120s ACP method timeout is the adapter-stall failure mode — make it observable, not a silent 120s hang.
+- **No unbounded per-entity collections.** Anything that grows per session / process / shell / toolkit (`_sessions`, tracked-process maps, socket-lookup caches, toolkit registries) must be a bounded collection that warns on eviction, or carry an explicit, documented cap. Enforce registration caps (`maxRegisteredToolkits`, etc.) at registration time, not silently at use time. Silent LRU eviction without a log is a footgun.
+- **Host-visible.** A limit warning/error that fires inside the VM or sidecar is useless if it never reaches the host — route it through the agent-process log channel (`onAgentStderr` / the ACP-extension forward) and a structured trace channel, the same path that makes adapter stderr observable.
+
 ## Agent Sessions
 
 - Every public method on `packages/core/src/agent-os.ts` must stay mirrored by RivetKit actor actions after the user confirms the Rivet repo path.

packages/core/CLAUDE.md

@@ -172,4 +172,5 @@ See `.agent/specs/test-structure.md` for the full restructuring plan. Target lay
 
 - **Never guess without concrete logs.** Every assertion about what's happening at runtime must be backed by log output. Add logs at every decision point and trace the full execution path before drawing conclusions. Never assume something is a timeout issue unless there are logs proving the system was actively busy for the entire duration.
 - **Never use CJS transpilation as a workaround** for ESM module loading issues. Fix root causes in the ESM resolver, the `/root/node_modules` mount / kernel VFS, or V8 runtime.
+- **Diagnosing stalls / backpressure / silent hangs:** secure-exec runs a central limit registry (`secure_exec_bridge::queue_tracker`) over the chain of bounded queues (V8→host event channel, per-session frame channel, sidecar stdout/stdin frame queues). A full queue applies backpressure (it blocks the producer), so a "hung" session is often a slow/stuck *consumer* upstream, not a deadlock. The registry emits a structured `WARN` ("bounded limit near capacity…") as any limit crosses ~80%, and resource/heap/CPU breaches surface as typed errors naming the limit. Set `SECURE_EXEC_LOG=warn` (the default) to see near-limit warnings, or `SECURE_EXEC_LOG=debug` for per-limit usage snapshots; secure-exec logs to **stderr** (stdout is the wire protocol). See the **Limits & Observability** architecture doc (`website/src/content/docs/docs/architecture/limits-and-observability.mdx`).
 - **Maintain a friction log** at `.agent/notes/vm-friction.md` for anything that behaves differently from a standard POSIX/Node.js system.

website/docs.config.mjs

@@ -126,6 +126,7 @@ export const siteConfig = {
 								{ slug: "docs/architecture/networking", label: "Networking" },
 								{ slug: "docs/architecture/posix-syscalls", label: "POSIX Syscalls" },
 								{ slug: "docs/architecture/compiler-toolchain", label: "Compiler Toolchain" },
+								{ slug: "docs/architecture/limits-and-observability", label: "Limits & Observability" },
 								{ slug: "docs/system-prompt", label: "System Prompt" },
 								{ slug: "docs/persistence", label: "Persistence & Sleep" },
 							],

website/src/content/docs/docs/architecture/limits-and-observability.mdx

@@ -0,0 +1,89 @@
+---
+title: "Limits & Observability"
+description: "How agentOS bounds resources, applies backpressure, warns before a limit is hit, and surfaces it all to the host."
+skill: true
+---
+
+agentOS runs untrusted, AI-generated code inside disposable VMs. Every resource
+that code can consume is **bounded by default**, and every bound is designed to
+**warn before it is hit**, **fail with a clear error when it is**, and stay
+**inspectable** from one place. This page explains how the limits, backpressure,
+logging, and observability pieces fit together across the stack.
+
+## Where limits live
+
+Limits are owned by **secure-exec** (the VM runtime) and **forwarded** by agentOS
+— the agentOS layer does not reimplement enforcement, it exposes the knobs and
+surfaces the signals.
+
+| Layer | Responsibility |
+| --- | --- |
+| secure-exec kernel | Enforces per-VM resource caps (memory/heap, CPU time, fds, processes, sockets, filesystem bytes, …). |
+| secure-exec sidecar | Owns the bounded queues between the guest, the runtime, and the host; applies backpressure; tracks usage. |
+| agentOS client | Forwards `limits` config to the VM and surfaces limit signals to the caller. |
+
+## Three guarantees for every limit
+
+Every bound — a resource cap, a bounded queue, a timeout, a payload size —
+follows the same contract:
+
+1. **Bounded by default.** Nothing is unbounded out of the box. Memory is capped
+   at ~128 MiB per isolate (Cloudflare Workers parity), CPU is bounded, and every
+   queue has a fixed capacity. Operators may *raise* a cap, but never get an
+   unbounded default.
+2. **Warn on approach.** As usage crosses a threshold (default **≥80%** of
+   capacity), a structured warning is emitted — once per crossing, re-armed only
+   after it drains back below 50% (hysteresis), so a busy limit logs once, not on
+   every operation.
+3. **Clear, typed error on breach.** Exceeding a limit produces an error that
+   names the limit, the observed-vs-cap value, and the config path to raise it —
+   never a bare `EAGAIN`, a silent drop, or a crash.
+
+## Backpressure, not catastrophe
+
+The path from guest code to the host is a **chain of bounded queues**: the V8
+runtime → a per-session frame channel → the V8→host event channel → the sidecar
+stdout frame queue → the host. When a queue fills (a slow host consumer, a chatty
+tool turn), the producer **blocks until the consumer drains a slot** — clean
+backpressure that flows all the way back to the guest. A full queue never
+destroys the session, silently drops data, or crashes the sidecar; a genuinely
+dead consumer surfaces as a typed terminal error instead.
+
+Buffer capacities are sized so that *transient* bursts are absorbed without ever
+engaging backpressure; backpressure is the safety net for a genuinely stuck
+consumer, not a normal-operation event.
+
+## The limit registry
+
+All bounded limits register with a single in-process **limit registry**. Each
+registered limit tracks its live depth, high-water mark, and capacity, and emits
+the near-capacity warning described above. This gives the runtime one place to
+answer two questions:
+
+- *Is a limit about to be hit?* — the registry fires the approach warning.
+- *What is the current usage of everything?* — a registry snapshot lists every
+  limit's depth / high-water / capacity / fill-percent for debugging.
+
+A CI audit fails the build if any limit-shaped constant is not classified and —
+for operator-tunable ones — wired to a config field, so "is everything bounded
+and config-wired?" is verified mechanically rather than by review.
+
+## Logging & host visibility
+
+secure-exec logs to **stderr** (never stdout — stdout is the framed wire
+protocol). The default level is `WARN`, tunable with the `SECURE_EXEC_LOG`
+environment variable (`error` to quiet, `debug` for per-limit usage snapshots).
+Near-limit warnings and backpressure events therefore show up in the sidecar's
+stderr stream, which agentOS forwards to the host.
+
+The limit registry also exposes a structured **warning sink**: a callback that
+fires on the same edge as the log, carrying `{ name, category, observed,
+capacity, fillPercent }`. This is the foundation for host-facing limit
+observability — a structured "a limit is approaching capacity" signal rather than
+a parsed log line.
+
+## See also
+
+- [Resource Limits](/docs/resource-limits) — the full `limits` config surface.
+- [Processes](/docs/architecture/processes) and [Sessions & Persistence](/docs/architecture/sessions-persistence) — the layers the queue chain runs through.
+</content>

website/src/content/docs/docs/resource-limits.mdx

@@ -53,3 +53,22 @@ registry.start();
 - **WASM stack**: deep recursion throws a stack-overflow error in the guest, never a host crash.
 - **Filesystem bytes**: writing past the VFS budget fails with a no-space error to the guest.
 - **Counts (fds / processes / sockets)**: hitting a table cap returns the standard POSIX errno (`EMFILE`, `EAGAIN`, etc.), exactly as a real Linux kernel would under `ulimit`.
+
+## Warnings & observability
+
+Limits are observable, not just enforced. Every bound — resource caps and the
+internal bounded queues alike — is tracked in a central limit registry that:
+
+- **Warns before the limit is hit.** As usage crosses ~80% of a cap, the runtime
+  emits a structured warning (once per crossing, re-armed only after it recovers),
+  so a slow consumer or a runaway guest is visible *before* it fails.
+- **Applies backpressure instead of failing catastrophically.** The internal
+  queues between the guest, the runtime, and the host block their producer when
+  full rather than dropping data or tearing down the session — so a transient
+  burst degrades to "slower", not "broken".
+- **Surfaces through logs.** secure-exec logs to stderr (stdout is the wire
+  protocol); set `SECURE_EXEC_LOG=warn` (the default) to see near-limit warnings
+  or `SECURE_EXEC_LOG=debug` for live per-limit usage snapshots.
+
+See [Limits & Observability](/docs/architecture/limits-and-observability) for the
+full architecture.