OpenClaw full capture (LLP 0157-0162) - #510
Conversation
Technical design covering LLP 0157's two deliverables (the OpenClaw-side steering plugin and the @hypaware/openclaw adapter rework) and its core removal (json_path). Resolves the forks the spec leaves open: where the steering plugin package lives, the before_model_resolve steering precedence that maps onto the three warning causes, whether the adapter still registers a gateway client once attach-probe is gone, single vs two exchange projectors, and cross-shape content-match-key normalization for LLP 0159 settlement. Co-Authored-By: Claude <noreply@anthropic.com>
Refines LLP 0161's design into an eleven-task graph with real code-dependency edges (a shared session reader and match-key module as foundational tasks, the steering plugin and adapter rework split along their actual file/import coupling), rates each task's complexity for the LLP 0022 model-tier seed, and pins every open item LLP 0161 Section 10 named to the task that must resolve it.
New plugin-local module (`hypaware-core/plugins-workspace/openclaw/src/session_file.js`)
implementing the one reader LLP 0158 asks for: a bounded first-line
`type: "session"` header read returning `{ sessionId, cwd, startedAt }`
(non-blank-string fields, absolute-path predicate on cwd shared in
behavior with the Codex `sessionMetaCwd` precedent), plus a full
`type: "message"` transcript iteration. Stays plugin-local per LLP
0158's placement decision until a second plugin needs it.
Tests cover the guard/blank/relative-cwd edge cases LLP 0150's Codex
precedent already names, applied to OpenClaw's flat header shape, plus
the message iteration's skip/normalize behavior.
Task-Id: T3
hypaware.plugin.json drops the write_openclaw_settings permission and
the attach_probe declaration (R7); config_sections[0].summary now
mentions the backfill block. NOTE: write_home is a candidate for
re-audit down to read_home now that the settings-file write is gone
(the LLP 0158 session reader only needs read_home) - flagged here for
a follow-up audit, not resolved in this change.
config.js gains validateBackfillSection (a same-shape copy of
@hypaware/codex's validator: on_join boolean, window_days positive
integer, unknown keys rejected), wired into validateOpenclawConfig.
The stale "no backfill block" comment is gone.
src/settings.js is deleted along with index.js's import of it.
activate() keeps gateway.registerClient({ name: 'openclaw', ... })
but attach() becomes an honest no-op: it writes nothing and reports
that OpenClaw routing is now owned by the
@hypaware/openclaw-steering-plugin npm package, installed on the
OpenClaw side. This keeps `hyp attach openclaw` / `hyp detach
openclaw` / client-descriptor resolution working instead of erroring
`unknown client`.
test/plugins/openclaw-settings-attach.test.js (tested the retired
settings.js directly) is replaced by
test/plugins/openclaw-client-registration.test.js, which proves the
client keeps resolving through activate()'s registered adapter, the
live attach command path, the disk-driven detach path, and the
descriptor map behind client listing/status.
Task-Id: T5
Adds hypaware-core/plugins-workspace/openclaw/src/match_key.js per LLP
0161 design Section 5: canonicalMatchKey (one hash function over a
{kind, identity}[] tuple format), wireMatchKey and sessionMatchKey (the
two shape-specific builders that feed it), and a deliberately separate
ordinal/time fallback matcher (withRoleOrdinals,
buildOrdinalFallbackIndex, matchOrdinalFallback).
The session-side block-kind synonym table (toolCall/toolUse/
function_call/functionCall -> tool_use, redacted_thinking -> thinking)
and the toolResult message-shape reconciliation are built from the real
openclaw package (packages/llm-core/src/types.ts) and its own test
fixtures, not guessed. tool_use/tool_result identity is content-hash
based rather than id-based because openclaw's own
createStandaloneTextToolCallId() proves ids are not stable across
capture routes.
Task-Id: T4
…(T1)
New top-level `@hypaware/openclaw-steering-plugin` package, the OpenClaw-
installed half of LLP 0157/0161's full-capture rework. Registers the two
shadow providers (`hypaware-anthropic` / anthropic-messages,
`hypaware-openai` / openai-completions) via OpenClaw's own
`api.registerProvider({ catalog: { run } })`, with `baseUrl` read from an
env var (HYP_GATEWAY_ENDPOINT, falling back to the fixed default gateway
port) rather than a HypAware kernel import, since this package runs inside
OpenClaw's process, not HypAware's.
The core of this task is `resolveSteering` (src/steering.js): the
four-branch precedence LLP 0161#steering-precedence resolves (shadow-shape
lookup -> canonical-provider check -> DEFERRED_SET membership ->
credential resolution), returning `providerOverride` plus the
`x-hypaware-upstream` request metadata on the terminal steer branch, and
one of `no_credential` / `no_preset` / `deferred` on every pass-through.
Wired into a `before_model_resolve` hook in src/index.js, with a
rate-limited warning ledger (src/warning_ledger.js) for the pass-through
cases per LLP 0149.
OpenClaw's own plugin-manifest/entry-point/hook-registration shape
(`definePluginEntry`, `api.on('before_model_resolve', ...)`,
`hooks.allowConversationAccess` gating, the `openclaw.plugin.json`
manifest fields) was verified against OpenClaw's published plugin docs
before writing openclaw.plugin.json and src/index.js's registration calls;
see the comment block at the top of src/index.js for the two residual
open items (exact before_model_resolve event field names, and the
metadata-return channel) worth reconfirming against a live install.
Credential borrowing (prepareRuntimeAuth) and Anthropic wire parity
(wrapStreamFn) are deliberately out of scope here - LLP 0162 T2, layered
onto this same provider registration.
Unit tests cover resolveSteering's every branch, DEFERRED_SET membership
(including the shape-matching-but-wrong-vendor non-deferred cases:
minimax, synthetic, kimi-coding, openrouter), each of the three warning
causes, and the warning ledger's per-provider+cause rate limiting.
Task-Id: T1
…opic wire (T2)
Layers LLP 0161 Section 2.2's two owner-scoped hooks onto T1's shadow
provider registration.
`prepareRuntimeAuth` (src/runtime_auth.js) resolves the *shadowed* provider's
credential through OpenClaw's public
`openclaw/plugin-sdk/provider-auth-runtime` `resolveApiKeyForProvider` and
returns `{ apiKey, baseUrl, expiresAt? }` for that one request. It caches
nothing, writes nothing, and re-resolves on every call; the shadow-to-real
provider map is derived from the two maps `resolveSteering` already branches
on rather than restated, so a third API shape cannot be steered without also
being borrowable.
`wrapStreamFn` (src/wire_parity.js) mirrors OpenClaw's
`extensions/anthropic/stream-wrappers.ts` for `hypaware-anthropic`: the
default and OAuth beta sets merged as a header-name-keyed `Set` union
(idempotent by construction, R4), `service_tier` / `fastMode` as a payload
patch, and the thinking-prefill strip. `hypaware-openai` gets no wrapper,
per LLP 0148's per-shape scope note.
Open item resolved (LLP 0148 Open questions, LLP 0161 Section 10): does
`@mariozechner/pi-ai` add the default betas itself? It does.
`pi-ai@0.73.1`'s Anthropic provider adds both defaults from its own flags and
prepends `claude-code-20250219` / `oauth-2025-04-20` whenever the key matches
`sk-ant-oat`, the same OAuth predicate OpenClaw uses. So the mirror shrank:
it installs only under OpenClaw's own `needsAnthropicBetaWrapper` condition,
because merging unconditionally would put the interleaved beta back on the
wire for adaptive-thinking models that pi-ai deliberately omits it for, which
is a parity change in the other direction. Both LLPs record the answer.
Three further facts came out of reading the shipped OpenClaw bundle and
pi-ai, and each changed the code (all recorded in LLP 0161 Section 2.2):
- pi-ai merges `options.headers` over its own defaults with `Object.assign`,
so a mirrored `anthropic-beta` replaces rather than extends pi-ai's value
and must therefore carry the full set.
- `service_tier` is a payload field, not a header, and OpenClaw gates it on
`provider === 'anthropic'` plus a public endpoint class. Both halves fail
for a shadow provider on a loopback gateway, which is exactly the loss
LLP 0148 names, so the mirror keeps the conditions it can still evaluate
honestly (the shape, the OAuth and Sonnet 5 carve-outs) and treats the
endpoint as public, which it effectively is: the gateway's `anthropic`
preset forwards to a static `https://api.anthropic.com`.
- The shipped OpenClaw treats `context-1m-2025-08-07` as
`ANTHROPIC_CONTEXT_1M_BETA_LEGACY` and strips it from every emitted set, so
the mirror strips it too. The `context1m` opt-in still decides whether the
mirror runs, exactly as it does for OpenClaw's own wrapper.
Two smaller corrections the borrow could not work without:
- `resolveSyntheticAuth` returns a non-secret placeholder for the two
`hypaware-*` ids. OpenClaw's embedded runner throws
`MissingProviderAuthError` for a provider with no resolvable credential
*before* `prepareRuntimeAuth` runs, so without it the borrow is never
reached. LM Studio's bundled plugin uses the same seam with `custom-local`;
the manifest declares the marker in `nonSecretAuthMarkers` and both ids in
`syntheticAuthRefs`.
- T1's steering hook passed `resolveApiKeyForProvider`'s result straight to
`resolveSteering`'s credential probe. The SDK resolves to a
`ResolvedProviderAuth` record, never a bare key, so the object was always
truthy and the `no_credential` branch could not fire. The probe and the
borrow now share one unwrapping adapter, which is what makes the ledger's
coverage claim true.
`expiresAt` is returned for OAuth-mode borrows only, as a re-resolution
deadline rather than a claim about the token's life: OpenClaw schedules
background re-preparation only when one is present, and
`resolveApiKeyForProvider`'s own OAuth branch refreshes under lock when asked
again. It stays clear of OpenClaw's 5-minute refresh margin so the
re-preparation cannot busy-loop.
The pass-through ledger grew per-record `operation`/`status`/`detail` so the
two new hooks report through the same rate limiter instead of a second copy
of it, keeping LLP 0149's ledger vocabulary for `before_model_resolve` alone.
Tests: 40 new cases across runtime_auth and wire_parity (62 in the package).
Task-Id: T2
…m precedence rung (T6) Adds `openaiUpstreamPreset()` to the openclaw projector, byte-identical in shape to @hypaware/codex's existing openai preset (name, base_url, path_prefix, provider), and registers it in index.js's activate() the same "register iff not already present" way the anthropic preset already is. Both anthropicUpstreamPreset() and openaiUpstreamPreset() gain a match() precedence rung above their existing path/header checks: an unconditional match when the steering plugin's x-hypaware-upstream header names this preset's provider, so steered OpenClaw traffic routes correctly even on a path/header signature shared with another adapter. Claude/Codex traffic, which never sends the header, is unaffected (regression test included, plus an integration-style test against the gateway's real matchUpstream/ compileUpstreams). hypaware.plugin.json's contributes.client.required_upstreams grows from ["anthropic"] to ["anthropic", "openai"], now that the plugin actually registers both presets. Task-Id: T6
json_path was the last-consumer format LLP 0143 accepted removing with no migration (LLP 0157 core removals). Delete the write-side branch in client_detach_disk.js (detachClientFromDisk's json_path dispatch and detachJsonPathMarker) and the read-side branch in status.js (probeClientAttachFromDescriptor's json_path dispatch and parseJsonRecordString), plus the helpers left dead by their removal (parseRecordString, stringEntries, setAtDottedPath, deleteAtDottedPath in client_detach_disk.js) and the now-unused json_util imports in status.js. json and toml formats, and the MALFORMED_MARKER guard, are untouched. Delete test/core/client-detach-json-path.test.js. Task-Id: T10
New `hypaware-core/plugins-workspace/openclaw/src/backfill.js`, `createOpenclawBackfillProvider(opts)`, registered via `ctx.backfills.register(...)` in `activate()` (design Section 7). It scans `~/.openclaw/agents/*/sessions/*.jsonl` through the LLP 0158 shared reader (R9: no private parse), gates each file once on the header's `cwd` through the shared usage-policy resolver (R10), excludes CLI-backend turns with a fail-closed allowlist (R10, LLP 0147), and builds an `AiGatewayProjectedExchange` straight off each record's own fields. The record's native `message_id` goes through verbatim, so a backfilled row never touches `computeMessageId`'s fallback-hash path and lands on the same `message_id`, and therefore the same `part_id`, a settled live row does. That is R11 by construction, and the new test proves it against the real live projector plus the LLP 0159 match keys standing in for T8's enricher. Two implementation-time refinements, landed in LLP 0161 Section 7 with the code: - The allowlist reads a record's effective provider. Only assistant records carry `provider`, so read literally the allowlist would have excluded every user prompt; an unstated record now takes the provider of the turn it belongs to, which excludes a CLI-backend turn whole. - One item per session, not per record, so the materializer's `previous_message_id` chain and tool-call lookup stay whole (the Codex precedent). `native_id` is the session's id accordingly. Open item carried forward, not resolved: no live OpenClaw install was reachable, so the exact `provider`/`api` values a CLI-backend turn records could not be verified. The allowlist fails closed either way, which is why LLP 0161 Section 10 says not to block on it. Task-Id: T9
New hypaware-core/plugins-workspace/openclaw/src/settle.js:
createOpenclawSettlementEnricher(opts) returns { name:
'openclaw-settlement', clientName: 'openclaw', settle(rows, ctx) },
registered in activate() right after registerExchangeProjector (the
claude/src/index.js placement).
Per session (grouped by session_id) it reads one session file through the
T3 LLP 0158 reader, builds a Map<matchKey, message> from the transcript
via T4's sessionMatchKey, and for each row looks up
attributes.openclaw.match_key: content match first, then T4's
ordinal/time fallback as a separate second pass. On a hit the row is
upgraded to native identity (message_id, recomputed part_id, the
header's session container id per LLP 0159) and the spent
openclaw.match_key / gateway.identity_source attributes are stripped.
Independent of match success the session's single header cwd goes
through the shared usage-policy resolver, returning USAGE_POLICY_DROP at
that row's position when policy.class === 'ignore' and logging
plugin.openclaw.usage_policy_drop with the Claude precedent's field
shape (cwd_hash, never a raw cwd).
Two details the design left to implementation time:
- Which file is "the session file". A live row's session_id is the
prompt-head hash (LLP 0144), which the session file never sees, so the
binding is established by CONTENT: the candidate whose transcript
holds the most of the group's match keys wins, ties to the newer file,
and a group no candidate matches binds to nothing (settles nothing,
drops nothing). Weaker signals are not enough to bind because the cwd
verdict that follows the binding drops rows.
- The fallback's ordinal comes from the file, indexed by the row's
message_index (OpenClaw re-sends the whole conversation, so a row's
index in its exchange is its position in the session); a position
whose recorded role disagrees with the row's declines instead of
guessing. A content key two session messages share is owned by
neither, so two rows can never take one native message_id (and with
it one part_id).
session_file.js gains the shared enumeration both this and T9's backfill
need (defaultOpenclawAgentsDir, listOpenclawSessionFiles), keeping the
agents/<id>/sessions/*.jsonl layout in the one reader. LLP 0161 Section 6
records the binding rule and the fallback's ordinal source.
Measured settlement match rate: 1.00 (5/5 rows) against a realistic
session file covering toolCall blocks, a standalone toolResult record
and a multi-block assistant turn - see the PR description for the
open-item note on OpenClaw's JSONL write timing.
Task-Id: T8
`createOpenclawExchangeProjector()` keeps its priority 110 and its `x-hypaware-client` match gate, and gains an internal shape dispatch: `project()` reads `x-hypaware-upstream` to pick `anthropicMessages()` (unchanged) or a new sibling `openaiMessages()`, an OpenAI Chat Completions request/response/SSE-stream parser mirroring `anthropicMessages()`/`reconstructAssistantMessage()` for the other wire. `provider` is no longer hardcoded `'anthropic'`: it is the header's value, falling back to `'anthropic'` only when the header is absent (R6). `openaiMessages()` emits the Anthropic block vocabulary rather than OpenAI's own: `tool_calls` become `tool_use` blocks and a `role: "tool"` envelope becomes a one-block `tool_result` message, because `wireMatchKey`'s per-block reduction only recognizes tool calls and tool results under those kind names - left native, an OpenAI turn could never match the same turn read back out of the session file. For the same reason the leading `system`/`developer` messages are lifted into `system_text` (no system-role record exists in the session file to match a system row against), which also keeps the session hash keyed off the system-prompt head on both shapes. LLP 0161 Section 3.5 records both. Every projected row, being fallback identity, now carries T4's `wireMatchKey` as `attributes.openclaw.match_key` (R8), merged one level deep so it never evicts the `usage` namespace beside it. `usageAttributes` reads both wires' token spellings, subtracting OpenAI's cached reads from its gross `prompt_tokens` so the stored input count is net on both. Tests cover both shapes streamed and non-streamed, the header-absent and unrecognized-header fallbacks, tool-call accumulation across chunks, a truncated stream, match-key stamping on every row of every shape, and that an OpenAI-captured tool call hashes to the same key as its OpenClaw session-file `toolCall` record. Task-Id: T7
T6 grew activate() a second upstream preset registration (openai) on the same import line and in the same registration run T8's settlement enricher attaches to, so the two land as one import and one ordered sequence. The registration-order test asserted the whole call sequence, which pinned a preset COUNT that is Section 3.4's business, not this task's. It now asserts what it is actually about: the enricher sits directly after the exchange projector (the claude/src/index.js placement), with the preset registrations, however many wire shapes there are, ahead of it. Task-Id: T8
…nto HEAD Resolved conflict in openclaw/src/index.js: activate() now registers both the T8 settlement enricher and the T9 backfill provider (additive, non-conflicting features that both touched the same registration block). Updated the T8 adjacency test in openclaw-client-registration.test.js to mock ctx.backfills.register and to expect the backfill registration between the enricher and the client, since that test predates T9's backfill provider and otherwise throws on the now-required ctx.backfills.register call.
Adds `## openclaw_capture` to docs/ACCEPTANCE.md in `## codex_desktop_capture`'s structure (What it proves / does not prove / Requires / Related / Steps / If it fails), covering both routes the OpenClaw adapter offers. Three places where OpenClaw's mechanism differs from Codex's, resolved against the openclaw repo and its plugin/CLI docs rather than guessed: - Step 1 replaces the settings-marker check (there is none, R7) with `openclaw plugins inspect hypaware-openclaw-steering --runtime --json`, the surface that imports the plugin module and reports what `register(api)` registered: both shadow provider ids and the `before_model_resolve` hook. `plugins list` is a cold registry read and cannot answer this. The step also pins the two config entries a human will otherwise miss (`allowConversationAccess`, without which the raw conversation hook never runs for a non-bundled plugin, and `HYP_GATEWAY_ENDPOINT`), and pre-empts the inert `client_attach_missing` warning R7's probe removal leaves behind. - Step 4 names the `openai` row as the on-the-wire proof that `x-hypaware-upstream` arrives: the projector falls back to `anthropic` when the header is absent, so an anthropic row alone cannot distinguish a steered turn from a lost header. - Step 5 exercises the warning ledger against a deferred family, with a config-declared `anthropic-vertex` provider so it needs no cloud credentials (the deferral is decided before any credential resolves), and warns that a Google-family id reports `no_preset` instead because the deferred branch only runs for a shape a shadow already covers. - Step 6 keeps Codex's backfill shape and pass condition (`rows_written: 0` with `rows_skipped >= 1`), preceded by a settled/total query that both guards the pass condition and measures the LLP 0159 open question about real-time session-JSONL appends. Docs only; R12's human run is not this task. Task-Id: T11
…ved (T11) `## openclaw_capture` landed with its step-1 registration check written against guessed `openclaw plugins inspect --runtime --json` field names (`.plugin.status`, `.plugin.providerIds`, `.typedHooks[].name`). OpenClaw documents the report's contents in prose and publishes no key schema, so those selectors would have failed as missing keys rather than as missing registrations: a false negative aimed at HypAware when the plugin was fine. That is exactly the failure this document exists to prevent. Step 1 now keeps the command (it is the right surface: `--runtime` imports the module in a live gateway, plain `inspect` is a cold registry read that passes on a plugin that never loaded) and asserts by token presence in the report instead - two provider ids this repo owns plus OpenClaw's own hook name. It also carries the fallback the design named, for builds without `--runtime`: skip to the live-capture step, where an `openai` row is only reachable through a registered shadow provider and a hook that steered. Two host preconditions the docs made visible are now in Requires: the `plugins.entries.<id>.hooks.allowConversationAccess` gate (it drops a non-bundled plugin's `before_model_resolve` registration silently, leaving only a `pluginDiagnostics` warning) and the OpenClaw version floor that gate implies. Both are failure modes that otherwise read as HypAware bugs. LLP 0161 Section 9's "exact mechanics ... verified at implementation time" is struck through and resolved, matching Section 10's existing convention. Task-Id: T11 Co-Authored-By: Claude <noreply@anthropic.com>
# Conflicts: # src/core/daemon/status.js
REVIEW round 1 —
|
Round 1 gated the steering package's 62 unit tests under `npm test` by importing its test modules from a root shim, which also pulled the package's source into `tsconfig`'s file graph. It reached six of seven source files. `src/index.js` - the entrypoint that registers both shadow providers, wires the credential borrow into `prepareRuntimeAuth`, and installs the `before_model_resolve` steering hook - was still checked by nothing: no test imports it (importing it would execute `definePluginEntry` from a module that does not resolve here), and `tsc` never saw it. The only thing keeping it out was its two `openclaw/plugin-sdk/*` imports. `openclaw` is the host that installs this package, not a dependency of it, so those specifiers do not resolve in this repo and the file failed with two TS2307s and nothing else. `src/types.d.ts` already says it exists to mirror "the slice of OpenClaw's own plugin SDK this package touches", naming exactly those two subpaths, but it only declared the value shapes, never the modules. Declare them, and add the package to `tsconfig`'s `include` so the entrypoint is reached by path rather than only through the test graph. Verified non-vacuous: a type error injected inside `register()` is caught. `npm test` 3228/8 (the known `leave-command` baseline, #512). `npm run typecheck` clean.
REVIEW round 2 (final) -
|
|
Triage rung (LLP 0017) at head
Spot-verified per LLP 0002 rather than trusting prior rounds' prose: |
philcunliffe
added
the
neutral:approved
philcunliffe
added
neutral:stuck
🤖 neutral: stuck - a misrouting hazard found after this PR was marked approvedRetracting my own approval. I flipped this PR to What neutral was doing: the issue-fix rung on #536 (hardening the The defect
export function isOpenaiPath(path) {
return path === '/v1' || path.startsWith('/v1/')
}That matches The chain, each link verified against
So on any install with a configured Demonstrated on Why the review chain missed it
The fix, and why it costs nothingDrop Consider whether What it needs from you
Two smaller items for the same pass, both noted by #537 and neither blocking: How to unstick: reply with a comment on this PR (or push to the branch); neutral monitors this thread and will re-engage with your guidance on its next tick. |
🤖 neutral: raising the severity on this - it is credential exposure, not just misroutingMy earlier stuck report described the With That is a Claude prompt and an Anthropic API key sent to a third-party vendor. Measured blast radius: 128 routing cells with The mechanism is unchanged from my earlier report and independently re-confirmed: The fix still costs nothing. #537's review checked the converse, which no one had: with the steering rung present, Two further notes for the eventual merge-base pass:
For completeness: issue #536's original recommendation to add The ask is unchanged and small: drop |
|
Yeah that seems reasonable, continue |
philcunliffe
removed
the
neutral:stuck
|
Re-engaging with your guidance - taking option (a): dropping |
… literals `openaiUpstreamPreset()`'s `match()` falls through to `isOpenaiPath`, the whole `/v1` tree, so it also answers `/v1/messages`. At `priority: 100` it sorted above a TOML-configured `anthropic` (priority 0) and swallowed Claude traffic on an ordinary install, forwarding the prompts and their `x-api-key` / `Bearer sk-ant-` credentials to api.openai.com. The priority bought nothing it was needed for: at 0 the prefix-length tiebreak puts the narrower `/v1/messages` ahead of `/v1`, and the `x-hypaware-upstream` steering rung still reaches the preset on the bare-origin `/chat/completions` path the steering plugin actually produces. The test that guarded this preset asserted its literal field values, which is why two review rounds and a triage missed the defect: `priority === 100` was exactly what the test wanted to see. Replaced with route assertions over the table an install really compiles (`mergeUpstreams` + `compileUpstreams` + `matchUpstream`), covering a configured `anthropic` with no configured `openai`, steered `/chat/completions`, and unsteered Claude/Codex traffic. Verified by mutation: re-adding `priority: 100` reddens three of the new tests. `mergeUpstreams` is exported from the ai-gateway source so the test can compile the real merged table instead of hand-building one. Also records the config-wins seam the fix does not close (an operator-configured `openai` replaces the preset and its steering rung, at either priority) and refreshes the stale comment claiming Codex's registration declares no priority. Co-Authored-By: Claude <noreply@anthropic.com>
|
Triage rung (LLP 0017) re-run on Verified, ground-truth:
All four residuals classified non-blocking (full evidence and reasoning posted on #536, which already tracks this PR's deferred findings — reused rather than duplicated):
New findings and one factual staleness note (issue #536's item #1 still quotes the now-removed Follow-up: #536 |
philcunliffe
added
the
neutral:approved
|
Backfill imports 0 sessions against real OpenClaw session filesRan this branch's backfill provider against a live
Cause: the message envelope is read one level too high
const provider = nonBlankString(row.provider)Real OpenClaw v3 session files nest them under The only top-level It fails silently because this lands exactly on the fail-closed path the doc comment describes ("A file where no record ever states a provider projects nothing, which is the fail-closed direction
Why CI is green
JSON.stringify({
type: 'message',
id: 'msg-2',
role: 'assistant',
provider: 'anthropic',
usage: { input_tokens: 10, output_tokens: 20 },
content: [{ type: 'text', text: 'hello' }],
})The suite asserts against an invented envelope, so it cannot catch this. Fixing the reader without re-grounding the fixtures will just move the failure into the tests. Suggest deriving fixtures from a real captured session file (redacted), which would also pin the Expected result after the fixSession Note on the two probe sessions
|
…p, json_path revival (#570) * Design: OpenClaw two-lane capture (LLP 0172) * Plan: OpenClaw two-lane capture executable tasks (LLP 0173) * openclaw config: validate sweep_cron and quiesce_ms in backfill section validateBackfillSection now accepts sweep_cron (a 5-field cron expression, validated with core's shared isCronExpression grammar so a malformed schedule is rejected the same way a sink's config.schedule is) and quiesce_ms (non-negative integer) alongside the existing on_join/window_days keys, added together so the unknown-key rejection loop recognizes both from the same merge. Task-Id: T6 * Restore json_path attach-probe format, add sweep field to BackfillContribution hypaware-plugin-kernel-types.d.ts: PluginAttachProbeManifest.format regains 'json_path' (removed by LLP 0143 after #212's orphaning danger), plus the new container_path, provider_keys, and cache_glob fields the format needs, reusing the existing marker_header. The comment at the removal site is revised, not deleted: it now explains that the runtime support LLP 0173 T2/T3 add closes the gap #212 warned about, so a manifest can only declare this format once both sides exist. BackfillContribution gains an optional sweep?: { cron: string } field for the daemon's periodic sweep (LLP 0173 T9), absent-by-default for every existing contribution. Both additions are purely additive: npm run typecheck (tsc --noEmit over the whole tree, including hypaware-core/plugins-workspace/openclaw) and npm test pass unchanged, proving no existing consumer's typecheck shifted. Task-Id: T1 * OpenClaw attach writes the two provider overrides, refusing to merge LLP 0169 reverses LLP 0152's premise: there is a real, reversible settings write for OpenClaw again, so the adapter's honest no-op attach() has nothing left to be honest about. New hypaware-core/plugins-workspace/openclaw/src/attach.js exports createOpenclawAttach({homeDir, env, fs}), mirroring the Claude adapter's attach() shape (same AiGatewayClientAttachContext, same withSpan('client.attach', ...), same dry-run branch). It reads openclaw.json through the one core settings-path seam (so $OPENCLAW_HOME relocation resolves the same file the manifest's probe will), refuses with {status:'failed', reason} when models.providers.anthropic or .openai is already there, and otherwise writes both entries whole from attachCtx.endpoint: bare origin for anthropic, endpoint + '/v1' for openai, each with the x-hypaware-upstream marker header and the mandatory empty models array. The refusal check runs entirely before the single atomicWriteFile, so there is no partial write to roll back, and it returns rather than throws, which is the whole mechanism by which a refuse during attach-on-join warns instead of failing the join. Every other key in openclaw.json is carried through by reference. Both output modes end with the 'openclaw gateway restart' instruction, since a --json caller is as blocked on the restart as a human is. index.js drops the no-op body, STEERING_PLUGIN_NAME, ROUTING_OWNED_BY_STEERING_PLUGIN_MESSAGE and the @ref LLP 0143#decision block, and wires activate() to the new effect. The registered attach() keeps the kernel's Promise<void> contract, so the outcome object is dropped there on purpose: both callers already derive success from a throw plus the one-line JSON the effect writes. Tests cover the refusal (including that the file is byte-identical after it, and that it never throws), the exact two-entry shape with the bare-origin/+v1 asymmetry, key preservation, the restart instruction on both output modes, dry-run, $OPENCLAW_HOME, and the missing/malformed config hard failures. The two attach tests in openclaw-client-registration.test.js are retargeted at the new behavior so the suite stays green; T10 owns that file's fuller rewrite. Task-Id: T4 * daemon/status.js: restore json_path attach-probe read branch Restores the probe.format === 'json_path' read branch removed by LLP 0143 / PR #510, parallel to the existing json/toml branches in probeClientAttachFromDescriptor: navigate container_path + each provider_keys entry, read headers[marker_header], and report attached when it equals the expected provider key for at least one configured key. Pure read, no ownership/backup concerns. Task-Id: T3 * openclaw backfill: quiesce window skips recently-modified session files listSessionFiles(agentsDir) gains an optional quiesceBeforeMs cutoff, and runOpenclawBackfill() computes it once per run as Date.now() - quiesceMs. quiesceMs resolves from the plugin's own config.backfill.quiesce_ms, defaulting to 180000ms (QUERY_FLUSH_DEBOUNCE_MS plus a one-minute margin), so a run never imports a session file OpenClaw is still mid-write on, or a settlement pass is still mid-flush against. Composes with the existing effectiveProviders/partitionByBackend CLI-backend logic (R10) rather than replacing it. Task-Id: T8 * Lane B sweep metadata: openclaw backfill provider + narrowed runner ctx createOpenclawBackfillProvider now populates the contribution's opt-in sweep field from config.backfill?.sweep_cron, defaulting to every 5 minutes when absent (LLP 0172#lane-b-sweep, R7). src/core/commands/backfill.js's runBackfillProvider, runProvider, and resolveOwnersForRun now declare a new BackfillRunnerContext interface (env, config, storage, backfills, backfillMaterializers) instead of the full CommandRunContext, a pure structural narrowing so the daemon-side sweep driver (LLP 0173 T9) can build one without assembling registries it never uses. Existing hyp backfill CLI-path and onboarding-finale call sites keep typechecking and passing unchanged. Task-Id: T7 * json_path detach returns: ownership, backup-not-discard, best-effort cache purge LLP 0143 pulled the `json_path` branch out of the disk-driven undo because LLP 0152 left nothing on disk for it to reverse. LLP 0169 reverses that premise, so the branch comes back - reshaped for the two provider entries attach now writes, not the single shadow provider of the old design. `detachJsonPathProviders` judges each `provider_keys` entry on what it points at, because this format has no HypAware-owned marker to replay: its undo record IS the entry. Ours (the gateway's own `baseUrl`, in either the bare origin or `+ /v1` spelling, with `marker_header` naming its own key, in the shape attach produces) is deleted. Anything else present at our key is backed up to a `_hypaware_detach_backup.<key>` sibling inside the same container before the live key goes, following the `prev_malformed` precedent of LLP 0163: never discard a value HypAware did not write. That closes the json/toml-vs-json_path asymmetry LLP 0163 flagged as worth its own look, converging on the outcome without the top-level marker key LLP 0163 correctly ruled out for this client. The derived caches (`cache_glob`, relative to the client's config home) are then purged of the same keys, best-effort: they do not self-heal, so a partial purge beats none, and one unreadable cache file is logged and skipped rather than failing a detach whose settings half already landed. An unknown gateway base URL has no safe default here - guessing either way silently deletes a foreign value or reports a finished detach over a client still routed at a dead port - so it refuses (`EXPECTED_BASE_URL_UNKNOWN`). Both callers degrade correctly: `reverse()` keeps the marker, `hyp detach` prints the reason. Both real callers thread the base URL: `detachClientViaCore` resolves it through the same three rungs manual attach already walks (live `localEndpoint()`, configured `listen`, the daemon's persisted bound port), every one optional so detach keeps working with the gateway capability unloaded; `reverse()` passes the `ctx.endpoint` `perform()` attached with. Task-Id: T2 * OpenClaw manifest: restore json_path attach_probe, retire steering-plugin copy hypaware.plugin.json gains contributes.client.attach_probe (design 1.4: json_path format, .openclaw/openclaw.json settings file, models.providers container, anthropic/openai provider keys, x-hypaware-upstream marker header, agents/*/agent/models.json cache glob). description and picker[0].summary drop every @hypaware/openclaw-steering-plugin reference and state the two capture tiers directly: live gateway capture once attached, plus a periodic transcript sweep. Claude's manifest gains the LLP 0167#onboarding line naming the claude-cli/<model> case OpenClaw's CLI-backend exclusion produces, so a user knows which picker entry an OpenClaw-routed Claude Code session belongs to. projector.js's UPSTREAM_HEADER comment no longer credits the deleted steering plugin; it now describes the config-override write attach() itself makes. Adds a manifest-shape test asserting attach_probe parses to the exact design-1.4 fields and that description/summary no longer match /openclaw-steering-plugin/. Updates the one existing assertion this manifest change makes false (the R7 "no attach_probe" descriptor check) to the restored json_path shape; the remaining behavioral rewrites of that test file are T10's scope. Task-Id: T5 * Delete openclaw-steering-plugin/ (LLP 0172 Section 5, R9) Removes openclaw-steering-plugin/ in full (src/, test/, package.json, openclaw.plugin.json, .d.ts files) and test/plugins/openclaw-steering-plugin.test.js: Lane A's config-override entries make OpenClaw route to the gateway on its own, so the credential-borrowing runtime auth shim, the live wire-parity mirror, the steering decision logic, the live warning ledger, and the gateway endpoint resolver that fed them no longer have a purpose. Also drops tsconfig.json's stray "openclaw-steering-plugin" include entry (line 19), not named in the design's own deletion inventory but found verifying the deletion against the real tree; leaving it would have left a dead include path. docs/ACCEPTANCE.md and test/plugins/openclaw-manifest.test.js still mention the package name (an onboarding rewrite reserved for T13, and a T5 regression test asserting the manifest no longer references it, respectively); neither is in this task's file list. Task-Id: T11 * openclaw-client-registration.test.js: finish T4/T5's deferred rewrite T4 and T5 already retargeted the two attach() no-op tests and the descriptor's attach_probe assertion to keep the suite green while they landed; this closes the two pieces both left for T10: - The "honest no-op" detach test's comment still credited the retired R7 no-attach_probe guard. Since T5 restored the manifest's json_path attach_probe, the no-op this test actually observes on a fresh temp HOME is detachClientFromDisk's absent-settings-file guard instead. Corrected the comment to say so. - Added a companion case that stages a real openclaw.json via the actual createOpenclawAttach() effect, then drives the same hyp detach CLI entry point (buildClientDescriptorMap's real manifest descriptor -> detachClientFromDisk's json_path branch) and asserts the ownership-based detachJsonPathProviders (T2) actually fires: changed:true, the removed baseUrl, and both provider entries gone from the file while everything else in it is untouched. Task-Id: T10 * Daemon sweep driver: run sweep-bearing backfill providers on the tick loop New `src/core/daemon/backfill_sweep.js`. `createBackfillSweepDriver({backfills, backfillMaterializers, env, config, storage})`'s `tick({now})` walks `backfills.list()`, skips any contribution with no `sweep` field or a cron that is not due (`cronMatches`, the sink driver's own due-check), and fires `runBackfillProvider` per due contribution with a `sweep-<name>-<now>` dev run id. Runs are fired unblocked: `tick()` resolves once each run has been started, never once one finishes, so a provider's transcript scan cannot stall the sink snapshots, the source-detail refresh, or `persist()` later in the same tick. Both settlements are handled, so a failing run is a logged `backfill.sweep_failed` record (component `openclaw`, operation `backfill.sweep`, `error_kind`) rather than an unhandled rejection that would take the daemon process down. A malformed `sweep.cron` is logged and treated as not due rather than thrown, so one provider's bad metadata cannot skip the rest of the list. `runtime.js`'s `runTick()` calls `await sweepDriver.tick({now})` directly after the existing sink-driver tick, riding the same `DEFAULT_TICK_INTERVAL_MS` 60-second loop: a `*/5 * * * *` schedule only needs a due-check once a minute, so this opens no second timer to start, drain, and account for at shutdown. Also repairs the typecheck this task's branch point already failed: T7's `sweep: { cron: opts.config?.backfill?.sweep_cron ?? ... }` does not compile, because the plugin's config slice is a `JsonObject` and every step below its root is a `JsonValue`. Read through a `resolveSweepCron` mirroring the `resolveQuiesceMs` helper already sitting beside it; behavior is unchanged. Externally blocked for real capture: until PR #552 (issue #543) merges, the LLP 0158 reader still reads OpenClaw v3 fields flat, so a sweep projects nothing from a real transcript. These tests passing is not evidence that it does. Tests: the due-check fires only sweep-bearing, cron-due contributions and builds the narrowed `BackfillRunnerContext` from the daemon's own runtime fields; a rejected run neither throws out of `tick()` nor lands as an unhandled rejection, and a never-settling run does not block the tick. A separate wiring test boots a real daemon with a fixture plugin whose contribution opts into a sweep and proves the tick actually runs it, which no unit test of the driver can show. Task-Id: T9 * docs/ACCEPTANCE.md: rewrite openclaw_capture for two-lane capture Drops the steering-plugin link/enable setup and the before_model_resolve/hooks.allowConversationAccess version-gate language (the plugin is deleted; Lane A depends on no OpenClaw hook API). Adds a setup step running `hyp attach --client openclaw` followed by the restart instruction it prints, a sweep step (detach to strip the live route, confirm the row is absent, confirm it lands within one sweep interval past the quiesce window), and a zero-duplicate assertion (a turn both lanes observe resolves to exactly one row for its part_id, proven against the daemon's own scheduler rather than a manual `hyp backfill`). Re-confirms LLP 0167#verify-results items 1, 3, and 4 against the current tree's attach/detach behavior instead of assuming them. Drops the retired deferred-provider-family warning-ledger step (LLP 0171 retires R13; no ledger) and the shadow-provider-id failure mode (Lane A overrides the existing anthropic/openai entries, it does not register new ids). States in the section's own Requires that the sweep/dedupe steps need PR #552 merged (the LLP 0158 reader still parses OpenClaw v3 flat), and the client_attach status-row re-confirmation needs PR #553 merged (a now-probed openclaw otherwise falls back to pre-#553 status behavior). This is a doc; the test is a human's successful run against a real OpenClaw install, which this change cannot perform. It is specified against what T2 (detach), T4 (attach), and T5 (manifest) actually implement in this tree, read directly from hypaware-core/plugins-workspace/openclaw/src/attach.js, src/core/config/client_detach_disk.js, and hypaware-core/plugins-workspace/openclaw/hypaware.plugin.json. Task-Id: T13 * Add backfill_openclaw_fixture hermetic smoke for Lane B sweep New backfill_openclaw_fixture.js under hypaware-core/smoke/flows, mirroring backfill_claude_fixture.js / backfill_codex_fixture.js: writes a minimal OpenClaw v3 session JSONL in the nested-message-envelope shape (PR #552's reader) under a temp agents/<id>/sessions/ tree with a controllable mtime, drives the real createBackfillSweepDriver (T9) through a cron-due tick, and asserts (a) a file inside the default 180000ms quiesce window is skipped, (b) a file backdated past it is captured with native message identity, and (c) rerunning the sweep on a later cron-due tick (forcing a fresh devRunId, so the ai-gateway materializer's dedupe genuinely re-scans committed partitions) nets zero new rows for the already-written part_ids. Driving a real, non-dry-run sweep write for the first time (T9's own tests only ever exercised a mocked runBackfill seam) surfaced a latent bug: writeRows/flushDataset read ctx.query, which BackfillRunnerContext never carried and the daemon's createBackfillSweepDriver(...) call never supplied, so any real sweep write actually crashed on "Cannot read properties of undefined (reading 'getDataset')" in both the smoke and the real daemon path. Threaded query through BackfillRunnerContext, BackfillSweepDriverOptions, createBackfillSweepDriver, and the daemon's sweepDriver construction, and updated LLP 0172's field enumeration and ctx samples (Sections 4.3/4.4) to match. Extended the existing T9 unit tests (test/core/daemon-backfill-sweep.test.js) to cover the new required field and its passthrough. Task-Id: T12 * openclaw-backfill.test.js: backdate fixture mtimes to close a quiesce-window race (#570) listSessionFiles compares stat.mtimeMs <= Date.now() even when a test sets config.backfill.quiesce_ms: 0 to opt out of the quiesce gate for something unrelated: 0ms only removes the margin, not the comparison. A fixture written moments earlier could race the provider's own later Date.now() call across two different clocks and occasionally lose, projecting 0 items instead of 1. Confirmed non-deterministic: PR #570's commit 0c62a21 produced both a green and a red `test (24)` CI run from the identical commit. writeSession now backdates every fixture's mtime by a small, fixed margin (FIXTURE_MTIME_MARGIN_MS), comfortably clearing the race while staying far below the real 180000ms default quiesce window, so the tests that rely on genuine freshness against that default are unaffected. The one test that writes its session file outside writeSession (the OPENCLAW_HOME relocation test) gets the same backdate applied directly. * Review round 1: make an attach refusal observable, and three smaller fixes Finding 1 (major). The registered `attach()` wrapper discarded the effect's `OpenclawAttachOutcome`, and both callers infer success from "did it throw", so a refusal recorded a `done` marker whose endpoint and assets_key matched forever: the join never retried even after the user cleared the conflicting `models.providers` entry, while the json_path attach probe kept reporting `not attached`. `hyp attach --client openclaw` printed the refusal and exited 0. LLP 0172 1.3 is authoritative (it promises the `{status:'failed', reason}` outcome is recorded and retried), so the wrapper now rethrows a failed outcome: `perform()`'s catch turns it back into that shape (recorded, warned, retried, the join's other actions untouched) and `runClientLifecycle`'s catch makes it exit 1. Rethrowing at the wrapper rather than teaching `perform()` to parse the adapter payload is what fixes both callers, since the CLI hands the adapter `ctx.stdout` directly and captures nothing to inspect. LLP 0172 1.3 gains the translation step it left implicit; the test that locked in the swallow now asserts the retryable failure, plus the reconciler and exit-code halves. Finding 2. `clientConfigHome` took the first segment of the settings path's home-relative form, which is not the config home when `$OPENCLAW_HOME` is nested inside `$HOME`: the cache glob then matched nothing and the purge silently no-opped while the settings half reported success. Derive it by stripping the manifest's own `settings_file` tail instead, the exact inverse of what `resolveClientSettingsPath` joined on. Regression test uses a two-segment `OPENCLAW_HOME`. Finding 3. `listSessionFiles`'s JSDoc claimed the CLI path runs unfiltered. It does not: `runOpenclawBackfill` computes the quiesce cutoff on every run, and `run()` is the single entrypoint for the CLI, the onboarding finale, and the sweep. Name `plan()` as the only unfiltered caller. No behavior change. Finding 4. The sweep fired a due provider with no record of what was still running, so a pass outliving its cron interval got a second concurrent run against the same datasets and mid-flush spool. Add the `maintenanceInFlight` guard shape, widened to a Set because the driver fires one run per provider, with a `backfill.sweep_skipped` / `already_running` record and clearing on both settlements. LLP 0172 4.4 states the re-entrancy rule it had left out. Co-Authored-By: Claude <noreply@anthropic.com> * Review round 2: thread the plugin config, make attach idempotent, name the sweep's component Three review findings, each with a doc edit in the same commit. 1. `sweep_cron` and `quiesce_ms` were validated then discarded. `activate()` built the backfill contribution without passing `ctx.config`, so both keys this PR adds to `validateBackfillSection` resolved to the hardcoded `*/5 * * * *` / 180000ms defaults at runtime, with no diagnostic. The existing unit tests handed `config` straight to the factory, so the missing wiring was invisible to them; the new test starts from an activation. 2. `attach()` refused on bare key presence, so it was not idempotent over its own output. This PR's manifest `attach_probe` is what makes openclaw eligible for attach-on-join, and `isCurrent()` re-performs attach on an ephemeral-port rebind (LLP 0086) or an asset-set change (LLP 0107). Every re-perform then refused: the marker churned to `failed`, `hyp attach openclaw` exited 1, and `openclaw.json` stayed pinned to the dead port while the marker-header probe still reported `attached: true`. The refusal is now ownership-aware, on the self-identifying triple detach already tests before deleting. `isOwnedProviderEntry`/`ownedBaseUrls` move out of `client_detach_disk.js` into a shared `src/core/config/provider_entry_ownership.js` so the two halves cannot disagree about the same file. Attach passes no base-URL set (on a drift re-attach its own entry carries the old origin); detach still passes one, because there the wrong answer deletes a value HypAware never wrote. Everything that fails the test still refuses, including `null`, a foreign entry, and a hand-edited one that merely kept the header. 3. The generic sweep driver stamped `component: 'openclaw'` on all five of its records while logging as `backfill-sweep`. It fires any contribution carrying a `sweep` field, so a second opt-in would have been misattributed. `component` now names the emitting module; plugin identity already rides `hyp_plugin` and `provider`. Docs updated to match: LLP 0167#attach-detach, LLP 0169's decision bullet and summary, LLP 0171 R2, LLP 0172 sections 1.2 and 2.2, LLP 0173's implementer note on the sweep's telemetry pair. Co-Authored-By: Claude <noreply@anthropic.com> --------- Co-authored-by: test <test@test.com> Co-authored-by: Claude <noreply@anthropic.com>
2 participants
OpenClaw full capture: shadow-provider steering, cross-shape projection, settlement, and backfill.
Implements the
openclaw-full-capturechange set: spec LLP 0157, decisions LLP 0158 / 0159, design LLP 0161, executable plan LLP 0162.See
llp/0161-openclaw-full-capture.design.mdfor the technical design andllp/0162-openclaw-full-capture.plan.mdfor the eleven-task breakdown.Draft while the remaining tasks land. Opened by neutral so the change set has a PR to reconcile against; held for a human merge.
Change-Set: openclaw-full-capture