feat: add visual Agent Eval system

.agents/skills/lingxiloop-eval-change/SKILL.md

@@ -0,0 +1,46 @@
+---
+name: lingxiloop-eval-change
+description: Implement, review, or verify LingxiLoop Agent Eval suites, baselines, deterministic runtime gates, evaluator contracts, trace ingestion and sanitization, Eval persistence, or the Admin Eval Dashboard. Use for changes under eval/, server/src/eval/, Eval scripts/tests, Eval API/DB surfaces, or Eval Dashboard paths; use when an Agent OS, prompt, RAG, tool, approval, Canvas, or multi-Agent change needs regression coverage.
+---
+
+# LingxiLoop Eval Change
+
+Build Eval evidence that can detect a regression in the current Agent behavior, keep observations safe to persist, and run only the owning CI scope.
+
+## Workflow
+
+1. Read [references/eval-contracts.md](references/eval-contracts.md) before changing a suite, baseline, runtime observation, persistence, or comparison behavior.
+2. Invoke `$lingxiloop-verify-change` and run its classifier against the intended diff. Confirm Eval paths produce `ci.eval=true`. Accept `ci.evalFocused=true` only when every changed path is Eval-owned. Shared Agent OS, DB, API, Admin shell, or integration-runner files must fail closed to their owning checks; package manifests, workflows, and classifier changes require `ci.fullMatrix=true`.
+3. Choose the lightest truthful execution mode:
+   - Use frozen inline observations only to test evaluator, parser, sanitizer, gate, and report semantics.
+   - Use the deterministic Agent OS runtime harness for merge-blocking behavior coverage. Exercise the real runtime with `MemoryHostAdapter`, `ScriptedModelDriver`, and a deterministic Kernel/Host seam; do not call external models or networks.
+   - Use real-model Eval only for prompt/model quality that deterministic assertions cannot represent. Keep it manual or scheduled unless an explicitly provisioned stable CI contract exists.
+4. Add or update a versioned Case when behavior, a failure mode, or a production bug is newly in scope. Keep inputs, expectations, scenario identity, and thresholds reviewable in `eval/suites/`.
+5. Update a baseline only after the new behavior is intentionally accepted. Never raise/lower a baseline merely to silence a regression. Compare run, dimension, and Case deltas before accepting it.
+6. Preserve the real trace chain: input, decision, model hop, IPython cell, Host Bridge action, Approval/Canvas activity, and final answer. Use runtime durations when available; do not substitute evaluator compute time.
+7. Sanitize before persistence or report creation. RAG results may retain sourceId, chunkId, marker, title, position, and bounded status/count metadata, but never excerpts or retrieved content. Allowlist ordinary tool results and redact secrets, authorization, message bodies, stdout/stderr, and oversized payloads.
+8. Run focused evidence from the matrix below and report which scopes were intentionally skipped. Expand to owning or full-matrix evidence whenever `$lingxiloop-verify-change` classifies a shared/high-risk path or the user asks for it.
+
+## Focused verification
+
+Run these for every Eval change:
+
+```bash
+npm run guard:brand
+npm run guard:agent-os
+npm run guard:llm-tracked
+npm run lint
+npm run server:typecheck
+npm run test:eval
+npm run eval:check
+```
+
+Add `npm run typecheck && npm run build` for the Eval Dashboard. Add `npm run test:integration:eval` when Eval-owned service or persistence behavior changes. The fail-closed classifier is the source of truth for whether full unit, full integration, Compose, vendored Open Notebook, desktop packaging, or the complete matrix is also required.
+
+## Completion bar
+
+- `eval:check` includes both the frozen harness self-test and a deterministic real Agent OS runtime gate.
+- Runtime fixtures fail when required prompt/context input, routing, RAG, tool selection, or Approval behavior no longer reaches the model/runtime seam.
+- Artifacts identify commit, prompt, and model targets and expose per-stage and per-Case regressions.
+- Stored and generated observations pass excerpt/secret checks.
+- CI uploads both Eval reports and runs only the classified scope on pull requests. Package manifest, selector, or workflow changes and `main`, manual, or release callers run the full matrix.

.agents/skills/lingxiloop-eval-change/agents/openai.yaml

@@ -0,0 +1,6 @@
+interface:
+  display_name: "LingxiLoop Eval Change"
+  short_description: "Build and verify deterministic Agent Eval changes"
+  default_prompt: "Use $lingxiloop-eval-change to implement or review this LingxiLoop Eval change with focused evidence."
+policy:
+  allow_implicit_invocation: true

.agents/skills/lingxiloop-eval-change/references/eval-contracts.md

@@ -0,0 +1,64 @@
+# LingxiLoop Eval Contracts
+
+## Authorities and ownership
+
+- `server/src/eval/contracts.ts` owns the versioned input, observation, trace, dimension, failure-category, and report contract.
+- `server/src/eval/evaluator.ts` owns deterministic scoring. A configured required stage must fail when evidence is missing; an unconfigured optional stage remains skipped and must not inflate the score.
+- `server/src/eval/trace.ts` owns Host Action allowlisting, RAG metadata extraction, deduplication, truncation, and redaction.
+- `server/src/eval/harness.ts` owns baseline validation and run/stage/Case regression checks.
+- `server/src/eval/service.ts` and `server/src/db/migrate.ts` own durable Eval ingestion and schema behavior.
+- `eval/suites/` and `eval/baselines/` are reviewable, versioned test data. `scripts/run-agent-eval.ts` replays frozen observations; `scripts/run-agent-runtime-eval.ts` runs the current Agent OS before evaluation.
+
+## Suite and baseline decisions
+
+Add a Case when introducing a supported behavior, covering a fixed bug, or protecting a failure boundary. Prefer small orthogonal Cases over one fixture that asserts many unrelated behaviors.
+
+Use frozen observations for evaluator mechanics only. They prove that scoring, failure classification, comparison, and report generation work; they do not prove the current Agent runtime still behaves correctly.
+
+Use deterministic runtime Cases for merge gates. Each Case should:
+
+- run `AgentOSRuntime` with the in-memory Host and scripted model seam;
+- assert required system-instruction and model-item fragments so prompt/context wiring affects the result;
+- cross the actual IPython/Host Bridge/Approval boundary when that behavior is under test;
+- avoid wall-clock-sensitive scoring, external model calls, and network access;
+- convert the captured runtime outcome/actions/events into the same `EvalObservation` contract as persisted runs.
+
+Use model Eval for semantic qualities that deterministic checks cannot judge reliably. Record model and prompt versions, pin inputs and evaluator configuration, budget cost, and keep the run manual or scheduled by default.
+
+Baseline changes require an intentional reviewed result. Preserve `referenceVersion`, per-dimension reference/minimums, per-Case reference/minimums, and `maximumScoreDrop`. Inspect regressions by Case and dimension before updating. Do not overwrite history or hide a missing/failed Case by loosening its floor.
+
+## Trace and persistence safety
+
+The desired trace is:
+
+`test input -> Agent decision -> model call -> IPython cell -> Host Bridge action -> Approval or Canvas worker -> final answer`
+
+Capture actual runtime status and duration where the runtime exposes them. Evaluator execution duration is not Agent stage latency.
+
+For `knowledge.search` and automatic knowledge context, retain only identity and traceability metadata such as sourceId, chunkId, marker, title, position, count, and bounded status fields. Drop excerpts and retrieved source content before the observation reaches a report or database write.
+
+For other tools, use explicit bounded sanitization. Redact keys matching secrets/tokens/authorization/cookies, content/body/messages, stdout/stderr, HTML/Markdown, and payloads. Limit depth, item count, string length, and object key count. A sanitizer test should use unmistakable sentinel text and assert the serialized observation and artifact do not contain it.
+
+Dynamic `knowledge.search` results must merge with automatic citations and deduplicate by sourceId, chunkId, and marker. Answer citation scoring must resolve only markers present in the sanitized citation metadata.
+
+## Comparison and CI
+
+Every report should identify target commit, prompt version, and model. Compare two targets at run, dimension, and Case levels, and retain categorized failures such as missing RAG source, bad citation, wrong tool, Approval violation, routing/Canvas failure, timeout, and cost regression.
+
+Run the classifier before choosing checks:
+
+```bash
+node .agents/skills/lingxiloop-verify-change/scripts/classify-change.mjs \
+  --base origin/main --include-worktree --format json
+```
+
+Expected focused commands:
+
+- Evaluator/trace/harness/runtime suite: `npm run test:eval` and `npm run eval:check`.
+- Eval persistence/API/migration: `npm run test:integration:eval` with dedicated PostgreSQL and Redis.
+- Eval Dashboard: frontend typecheck and production build.
+- All Eval TypeScript: lint, server typecheck, Agent OS architecture guard, and LLM ledger guard.
+
+Focused means every changed file is Eval-owned: `eval/`, `server/src/eval/`, Eval-specific tests and runners, `src/admin/EvalPage.tsx`, the Eval Skill, or the Eval guide. Do not infer hunk ownership from a shared filename. Changes to Agent OS runtime, DB migration, API/Admin shell, integration infrastructure, root docs, or shared config must fail closed to their owning checks. Package manifests, workflows, and classifier changes run the full matrix once before the dependency/selector change is trusted.
+
+Open Notebook scope, Compose smoke, full serial integration, and Windows/macOS packaging remain path-owned checks for ordinary pull requests. The reusable quality workflow also runs the full matrix for package-manifest or selector changes, `main`, manual, and release callers.

.agents/skills/lingxiloop-verify-change/SKILL.md

@@ -5,7 +5,7 @@ description: "Classify a LingxiLoop diff and run the smallest credible verificat
 
 # Verify a LingxiLoop Change
 
-Select evidence from the actual outgoing scope. CI owns the exhaustive platform matrix, while local verification must exercise the narrowest check that would fail for the changed behavior.
+Select evidence from the actual outgoing scope. Pull-request CI consumes the classifier's `ci` plan, while `main`, manual, and release callers own the exhaustive platform matrix. Local verification must exercise the narrowest check that would fail for the changed behavior.
 
 ## Classify the scope
 
@@ -20,6 +20,7 @@ node .agents/skills/lingxiloop-verify-change/scripts/classify-change.mjs --base
 - With `--base`, compare the verified merge base to `--head` or `HEAD`. The script never guesses or fetches a base.
 - Add `--include-worktree` only when local changes belong to that committed range.
 - Use `--format json` when another tool needs the versioned report.
+- Read the JSON `ci` object when planning automation. `evalFocused` is fail-closed and valid only when every path is Eval-owned. Shared runtime/DB/API/integration files restore their owning checks; package manifests, workflows, or classifier changes set `fullMatrix`.
 
 Read [references/check-matrix.md](references/check-matrix.md) before changing the classifier mapping or when a category needs manual interpretation.
 

.agents/skills/lingxiloop-verify-change/references/check-matrix.md

@@ -7,6 +7,7 @@ The classifier uses paths as a deterministic first pass. Inspect diff content be
 | Category | Typical paths | Minimum evidence |
 | --- | --- | --- |
 | `docs` | root Markdown, `docs/`, project Skills | brand guard and link/content inspection |
+| `eval` | `eval/`, `server/src/eval/`, Eval tests/scripts/persistence, Admin Eval Dashboard | focused Eval unit tests, frozen harness plus deterministic Agent OS runtime gate, server typecheck; focused Eval integration for persistence and frontend build for Dashboard |
 | `frontend` | `src/`, `public/`, `website/`, Vite/Tailwind entry files | lint, frontend typecheck, owning tests; build when bundling or runtime entry behavior changes |
 | `server` | general `server/` runtime | lint, server typecheck, owning unit tests |
 | `agent-os-im-canvas` | Agent OS, agents, IM, Canvas, message-stream seams | Agent OS and LLM ledger guards, server typecheck, focused unit tests, reliability integration |
@@ -17,12 +18,15 @@ The classifier uses paths as a deterministic first pass. Inspect diff content be
 
 Categories overlap deliberately. `agent-os-im-canvas` and `database-tenant` are specialized views of server risk, not evidence that a change is automatically cross-domain.
 
+The Eval fast path is fail-closed. It applies only when every path is Eval-owned: versioned suites/baselines, `server/src/eval/`, Eval-specific tests/runners, `src/admin/EvalPage.tsx`, the Eval Skill, or the Eval guide. Shared Agent OS, DB migration, API/Admin shell, integration-runner, root config/docs, workflow, and classifier paths cannot prove hunk ownership and therefore restore their owning checks. Package manifests, workflows, and classifier changes set `ci.fullMatrix=true` so dependency and selector changes are exercised by the complete matrix before they are trusted.
+
 ## Escalation rules
 
 Recommend a full CI approximation when any of these applies:
 
 - a runtime migration or latest-schema sentinel changes;
 - workflow, dependency, package, Docker, Compose, version, or release machinery changes;
+- the CI workflow or its change classifier changes;
 - two or more primary runtime domains change in one diff;
 - vendored source or provenance changes;
 - the user explicitly asks for a full rehearsal or a CI failure is being reproduced.
@@ -32,6 +36,9 @@ The local approximation is the applicable subset of brand, architecture and ledg
 ## Selection details
 
 - Use direct owning test files when the relationship is clear. Otherwise run `npm test`; do not guess a narrow test from a similar filename.
+- Run `npm run test:eval` and `npm run eval:check` for Eval changes. The first is focused unit evidence; the second combines frozen evaluator/harness replay with a deterministic current Agent OS runtime regression.
+- Run `npm run test:integration:eval` for Eval schema, service, API, and persistence changes. Do not serialize the full integration directory for an Eval-focused pull request.
+- Run frontend typecheck and `npm run build` when the Eval Dashboard changes.
 - Run `npm run test:integration` for schema, tenant authorization, durable work, IM routing, Host Bridge recovery, or multi-service persistence behavior when PostgreSQL and Redis are available.
 - Run `npm run guard:agent-os` for the active Agent OS composition and tool boundary.
 - Run `npm run guard:llm-tracked` whenever a server-side cloud LLM call or its wrapper can change.