Skip to content
Open
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
4 changes: 4 additions & 0 deletions Makefile
Original file line number Diff line number Diff line change
Expand Up @@ -361,6 +361,10 @@ e2e-dashboard: ## Click the real dashboard page in a real browser (needs Playwri
@node --experimental-strip-types test/e2e-dashboard.mjs allow && \
node --experimental-strip-types test/e2e-dashboard.mjs reject

.PHONY: e2e-standalone-panes
e2e-standalone-panes: ## Prove the opt-in loopback page draws MEASUREMENTS and PROPOSALS (needs Playwright + Chromium; opt-in, not part of verify)
@node --experimental-strip-types test/e2e-standalone-panes.mjs

.PHONY: e2e-in-ui
e2e-in-ui: ## Click Allow/Reject on the in-UI page against a RUNNING profile (DSH_URL=… PROOF_DIR=…)
@test -n "$(DSH_URL)" || { echo "DSH_URL is required: paste the 'dsh web:' line from the server log"; exit 2; }
Expand Down
4 changes: 3 additions & 1 deletion README.md
Original file line number Diff line number Diff line change
Expand Up @@ -211,7 +211,9 @@ decision.

For headless and CI there is still an opt-in loopback page — set
`dashboard.standalone: true` in the patch row — which serves the same components
on `127.0.0.1` with its own token. Details:
on `127.0.0.1` with its own token. It is built from its own entry
(`web/standalone.tsx`) against the same `web/app.tsx`, so both surfaces carry
every panel; `make e2e-standalone-panes` proves it in a real browser. Details:
[`docs/VERIFY-DASHBOARD.md`](docs/VERIFY-DASHBOARD.md).

---
Expand Down
22 changes: 12 additions & 10 deletions assets/assistant-ui/dashboard.js

Large diffs are not rendered by default.

57 changes: 57 additions & 0 deletions docs/KNOWN-ISSUES.md
Original file line number Diff line number Diff line change
Expand Up @@ -88,6 +88,7 @@ rather than a list of letters:
| 1cd | The first three proposals, read against the history they came from |
| 1cc | `recommendations` was forwarded by nobody, set by nobody, and configured by nobody |
| 1ce | The run history was written next to the server, not next to the work |
| 1cf | The opt-in loopback page served a component set frozen ten days ago |


### 1a. Nothing in this repo ran on the model it was supposed to run on
Expand Down Expand Up @@ -1706,6 +1707,62 @@ panel shows the LAST workspace that recorded. Keying those by workspace is the
upgrade path; today every deployment runs one workspace per process, and this is
the same loopback posture the default already documented.

### 1cf. The opt-in loopback page served a component set frozen ten days ago

**Observed:** nothing. The standalone page (`dashboard.standalone: true`)
mounted, rendered its approval thread, its workspace tree and its run list, and
answered a decision — every assertion anyone had written against it passed. It
had no **MEASUREMENTS** pane and no **PROPOSALS** pane, which is the whole point
of watching a run on a loopback page with no harness UI beside it.

**Why:** `assets/assistant-ui/dashboard.js` had not been written by the builder
since `3973e3a` (2026-09-23). Commit `633c1e2` folded the dashboard into the DSH
UI, repointed `web/build.mjs` at `web/entry.tsx` and dropped its `outfile: …,
'dashboard.js'` line — while keeping `dashboard.js` in `assets/`, because the
standalone opt-in was meant to keep working. The fold also moved the
self-mount (`createRoot(document.getElementById('root'))`) and the SSE transport
out of `web/app.tsx` into the host-only `web/entry.tsx`, so **no source in the
tree could build a working standalone bundle any more**. Every component added
afterwards (`08f1cb5` metrics, `237ea70` proposals) lands in `web/app.tsx`,
which is `client.js`'s input and not `dashboard.js`'s.

**Why it survived `make dashboard-bundle` and a staleness check.** The check in
`test/assistant-ui.test.ts` hashes `client.js`'s inputs, and `web/standalone.tsx`
was not an input of anything — there was no such file. So: run the builder, and
it does not touch the artefact; change the page, and the hash does not move.
Both halves are individually defensible and jointly invisible.

**Measured, not reasoned.** Serving the committed artefact in a real browser
with a snapshot carrying `metrics` and `recommendations` rendered
WORKSPACES/ACTIVITY/APPROVAL THREAD and nothing else: `pane-metrics: 0`,
`pane-proposals: 0`. Same server, same snapshot, same page, with a bundle built
from today's `web/standalone.tsx`: both panes drawn. That is the whole proof.

**Fix:** `web/standalone.tsx` — 135 lines, mounting today's `DashboardApp` over
the three endpoints the loopback server already serves (`/api/events`,
`/api/state`, `POST /api/approvals/:id`), with the same `DashboardSource` seam
the host entry uses. React is bundled here rather than external, which is the
opposite of the host entry and for the opposite reason (no other React on this
page). `web/build.mjs` now writes both artefacts from one shared esbuild
literal, differing only in `external`, and `web/standalone.tsx` joins
`SOURCE_INPUTS` so the staleness hash covers the loopback page too.

**Two checks, because either alone is a comment.** `test/assistant-ui.test.ts`
asserts the builder writes `dashboard.js` again, from `web/standalone.tsx`, and
that the hash covers that entry — that runs on a bare clone in CI.
`test/e2e-standalone-panes.mjs` drives the real `startDashboard` in real
Chromium and asserts both panes appear once the server has something to show;
that one needs a browser, so it is opt-in like its siblings.

**The finding inside the finding, recorded because it cost the fixture twice.**
A hand-written `MetricsSummary` threw
`Cannot read properties of undefined (reading 'length')` inside `MetricsPanel`:
`alerts`, and each axis' `latest` and `samples`, are required fields, and a
fixture that remembers only the interesting ones produces a shape the server
never emits. The test now asks `summarize()` for its own metrics from five
invented records — shorter, and the only version that cannot drift from the real
shape without failing here.

### 1c. `make profile` hung for nine minutes with the registry unreachable

**Observed:** `bash scripts/make-profile.sh webz --port 4596` printed
Expand Down
18 changes: 18 additions & 0 deletions docs/VERIFY-DASHBOARD.md
Original file line number Diff line number Diff line change
Expand Up @@ -143,6 +143,24 @@ e2e-dashboard (rejected): the Reject click resolved the ask rejected
A captured frame of the three card states (brief ready / still writing /
unavailable) is committed at [`assistant-ui-dashboard.png`](assistant-ui-dashboard.png).

**And the two rail panes, on the loopback page.** `make e2e-standalone-panes`
drives a real `startDashboard` in real Chromium and asserts MEASUREMENTS and
PROPOSALS both appear once the server has a roll-up and a recommendation to
show. It exists because for ten days the loopback page answered approvals
perfectly with **no such panes at all**: its bundle had no builder left after the
fold into the DSH UI, so it served the component set from 2026-09-23 (KNOWN-ISSUES
§1cf). It fails today if the artefact goes stale again — verified by reverting to
the frozen bundle, which mounts the page and reports both panes absent.

```
$ make e2e-standalone-panes
mounted: 297 chars of the page body
panes: runs=1
panes: metrics=1 proposals=1
console/network errors: none
standalone panes: mounted, with MEASUREMENTS and PROPOSALS drawn
```

What changed in the mechanism, and why it is still the same guarantee: the
approval card is now rendered by assistant-ui's `MessagePrimitive.Parts` with a
`tools.Override` component, and the click reaches the server through
Expand Down
Binary file added docs/evidence/standalone-panes.png
Loading
Sorry, something went wrong. Reload?
Sorry, we cannot display this file.
Sorry, this file is invalid so it cannot be displayed.
2 changes: 1 addition & 1 deletion src/plugin.ts
Original file line number Diff line number Diff line change
Expand Up @@ -25,9 +25,9 @@
*/

import type { Context } from '@deepseek-ai/cordis'
import type { Agent, AgentRegistry, PreStepDecision } from '@deepseek-ai/dsh-agent'

Check failure on line 28 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Cannot find module '@deepseek-ai/dsh-agent' or its corresponding type declarations.
import { boundContextSummary, createUserMessage } from '@deepseek-ai/dsh-llm'

Check failure on line 29 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Cannot find module '@deepseek-ai/dsh-llm' or its corresponding type declarations.
import type { ContextFormed, LlmCallConfig, UserMessage } from '@deepseek-ai/dsh-llm'

Check failure on line 30 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Cannot find module '@deepseek-ai/dsh-llm' or its corresponding type declarations.

/**
* Declare this plugin's message producer identity.
Expand All @@ -45,12 +45,12 @@
* non-first-party plugin of this name, so rows written before this fix and rows
* written after it agree, instead of splitting the transcript across two kinds.
*/
declare module '@deepseek-ai/dsh-llm' {

Check failure on line 48 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Invalid module name in augmentation, module '@deepseek-ai/dsh-llm' cannot be found.
interface MessageSourceMap {
'plugin:feature-loop': { kind: 'plugin:feature-loop' } & ContextFormed
}
}
import type { PostToolDecision, PreToolDecision, ToolExecution } from '@deepseek-ai/dsh-tools'

Check failure on line 53 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Cannot find module '@deepseek-ai/dsh-tools' or its corresponding type declarations.
import { LoopBudget } from './budget.ts'
import type { BudgetSnapshot, UsageReading } from './budget.ts'
import { PhaseAllocator } from './phase-budget.ts'
Expand Down Expand Up @@ -1021,7 +1021,7 @@
* @param stopArmed - whether the operator's stop sentinel is set.
* @returns proceed or deny. Never `ask` — that is what YOLO means.
*/
export function gateEnforce(
function gateEnforce(
policy: FeatureLoopPolicy,
toolName: string,
args: unknown,
Expand Down Expand Up @@ -2711,7 +2711,7 @@
// receives the carrier's events with this set, and it is the documented
// switch for "receive regardless of context filter checks" — so the
// registration is not relying on the root context happening to be untagged.
return ctx.on('session/event', (session: unknown, event: unknown) => {

Check failure on line 2714 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Argument of type '"session/event"' is not assignable to parameter of type 'keyof Events'.
const end = asTurnEnd(event)
if (end === undefined) return
const key = `${sessionIdOf(session)}#${String(end.turn)}`
Expand Down Expand Up @@ -2743,7 +2743,7 @@
}, { global: true })
})()

const disposeStep = ctx.on('agent/pre-step', async ({ agent, turn, step }, next) => {

Check failure on line 2746 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Binding element 'step' implicitly has an 'any' type.

Check failure on line 2746 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Binding element 'turn' implicitly has an 'any' type.

Check failure on line 2746 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Binding element 'agent' implicitly has an 'any' type.

Check failure on line 2746 in src/plugin.ts

View workflow job for this annotation

GitHub Actions / typecheck (harness-free src/ modules)

Argument of type '"agent/pre-step"' is not assignable to parameter of type 'keyof Events'.
const policy = policyFor(agent)
// Price the settled attempts of the step(s) before this boundary FIRST:
// a verdict computed against a budget that has not yet been told what the
Expand Down
46 changes: 43 additions & 3 deletions test/assistant-ui.test.ts
Original file line number Diff line number Diff line change
Expand Up @@ -16,7 +16,7 @@
import { strict as assert } from 'node:assert'
import { test } from 'node:test'
import { createHash } from 'node:crypto'
import { readFileSync, statSync } from 'node:fs'
import { existsSync, readFileSync, statSync } from 'node:fs'
import { dirname, join } from 'node:path'
import { fileURLToPath } from 'node:url'

Expand Down Expand Up @@ -156,9 +156,12 @@ test('the checked-in bundle was built from the sources in the tree', () => {
assert.ok(recorded !== undefined,
'MANIFEST.txt must record sources-sha256 — run `make dashboard-bundle`')

// Recompute the builder's hash over the same inputs, in the same order.
// Recompute the builder's hash over the same inputs, in the same order. Both
// entries are named: `web/standalone.tsx` builds `dashboard.js`, and a hash
// that omitted it let that artefact drift from source without a word (§1cf).
const inputs = [
'web/entry.tsx', 'web/app.tsx', 'web/plugin.css', 'web/shell.css', 'web/start-target.ts',
'web/entry.tsx', 'web/standalone.tsx', 'web/app.tsx', 'web/plugin.css',
'web/shell.css', 'web/start-target.ts',
]
const now = createHash('sha256')
.update(inputs.map((f) => readFileSync(join(repo, f))).join('\u0000'))
Expand All @@ -171,6 +174,43 @@ test('the checked-in bundle was built from the sources in the tree', () => {
+ 'serves the old code while every other check passes.')
})

test('the standalone bundle has a builder, and it covers its own entry', () => {
// §1cf. The loopback page's bundle had NO builder between `633c1e2` (which
// repointed `web/build.mjs` at `web/entry.tsx` and dropped the `outfile` line,
// keeping `dashboard.js` for the standalone opt-in) and now — so it served the
// component set frozen at `3973e3a`, with no metrics pane and no proposals
// pane, and nothing failed. Two checks, because either one alone is a comment:
//
// 1. the builder writes `dashboard.js` again, and
// 2. the staleness hash covers `web/standalone.tsx`, so a change to the page
// the standalone entry mounts fails the check above rather than waiting
// for somebody to notice a missing pane.
//
// `test/e2e-standalone-panes.mjs` is the half that needs a browser; this is
// the half that runs in CI on a bare clone, and it fails in the same commit
// either way.
const here = dirname(fileURLToPath(import.meta.url))
const repo = join(here, '..')
const build = readFileSync(join(repo, 'web/build.mjs'), 'utf8')

assert.match(build, /outfile:\s*join\(outDir, 'dashboard\.js'\)/,
'the builder must write the standalone bundle again — the loopback page serves it')
assert.match(build, /entryPoints:\s*\[join\(root, 'web\/standalone\.tsx'\)\]/,
'and it must be built from the standalone entry, which mounts DashboardApp '
+ 'over the loopback endpoints')
assert.ok(existsSync(join(repo, 'web/standalone.tsx')),
'web/standalone.tsx is the entry the builder above names')

// The hash below is over `client.js` inputs. Without the standalone entry in
// that list, a page change that only affects the loopback surface is invisible
// to it — which is precisely how ten days of components went missing from one
// surface without a failure.
const staleness = build.match(/const SOURCE_INPUTS = \[([^\]]*)\]/s)?.[1] ?? ''
assert.ok(staleness.includes("'web/standalone.tsx'"),
'the staleness hash must cover web/standalone.tsx, or a change that only '
+ 'affects the loopback page is invisible to it')
})

test('the sizes the manifest recorded are still the sizes on disk', () => {
// The hash above catches every source change. This catches the other
// direction: a file edited or truncated WITHOUT a rebuild. Both matter, and
Expand Down
177 changes: 177 additions & 0 deletions test/e2e-standalone-panes.mjs
Original file line number Diff line number Diff line change
@@ -0,0 +1,177 @@
/**
* §1cf: the standalone (loopback) page renders TODAY's components.
*
* The artefact this replaces was frozen at `3973e3a` (2026-09-23) and had no
* metrics pane and no proposals pane on a page whose whole purpose is watching
* a run without the harness UI. This drives the real `startDashboard` in a real
* browser and asserts the two panes are MOUNTED, which a source-level assertion
* cannot do — the whole failure was a component that exists in `web/app.tsx` and
* was absent from the served bundle.
*
* The panes are absent-by-design until there is something to show, so the
* assertions are split: with an empty snapshot the page must still MOUNT (the
* bundle loaded and ran), and with metrics + recommendations set it must draw
* both. `startDashboard`'s second parameter is the very `DashboardState` the
* plugin feeds in production, so this drives the production path rather than a
* fake server.
*
* Run: node --experimental-strip-types test/e2e-standalone-panes.mjs
*/
import { createRequire } from 'node:module'
import { existsSync } from 'node:fs'
import { homedir } from 'node:os'
import { join } from 'node:path'

import { DashboardState, startDashboard } from '../src/dashboard.ts'

function resolvePlaywrightCore() {
const candidates = [
join(homedir(), 'node_modules/playwright-core'),
join(homedir(), 'work/harvey/freepeak/deepseek-harness/node_modules/playwright-core'),
]
return candidates.find(p => existsSync(join(p, 'package.json'))) ?? 'playwright-core'
}

function resolveChrome() {
const installed = '/Applications/Google Chrome.app/Contents/MacOS/Google Chrome'
return existsSync(installed) ? installed : undefined
}

const { chromium } = createRequire(join(homedir(), 'x.js'))(resolvePlaywrightCore())

/**
* Every child that opens `#root` and nothing else: no harness, no client bundle
* (that one is loaded by the harness through `__ModuleLoader__`, and this page
* is not it), no host React. What runs is `assets/assistant-ui/dashboard.js`
* alone — the file the loopback server serves.
*/
// The plugin's own state object, passed the way `apply()` passes it.
const state = new DashboardState()
const dash = startDashboard({ enabled: true, port: 0, answers: true }, state)
await dash.ready
const executablePath = resolveChrome()
const browser = await chromium.launch(executablePath === undefined ? {} : { executablePath })
let failures = 0
try {
const page = await (await browser.newContext({ viewport: { width: 1440, height: 1100 } })).newPage()
const errors = []
page.on('pageerror', e => { errors.push(`pageerror: ${e.message}`) })
page.on('response', r => { if (r.status() >= 400) errors.push(`HTTP ${String(r.status())} ${r.url()}`) })

await page.goto(`${dash.url}?token=${encodeURIComponent(dash.token)}`, { waitUntil: 'domcontentloaded' })
await page.waitForTimeout(4000)

const text = await page.evaluate(() => document.getElementById('root')?.innerText ?? '')
console.log(`mounted: ${String(text.length)} chars of the page body`)
console.log(`panes: runs=${String(await page.locator('.pane-runs').count())}`)

// The bundle ran. Before this change the same page served a frozen artefact
// that also mounted, so "mounted" is necessary and not sufficient — the panes
// below are the assertion that actually distinguishes the two bundles.
if (text.length === 0) {
console.error('FAIL: #root is empty — the standalone bundle did not mount at all')
failures += 1
}
if (await page.locator('.pane-runs').count() === 0) {
console.error('FAIL: the run rail is absent — the page rendered without its own components')
failures += 1
}

// The two panes, with data. Both are absent-by-design on an empty snapshot
// (§1bq's rule: absent is not empty), so the assertion cannot be "the pane
// exists" — it is "the pane appears once the server has something to show",
// which is the only thing that distinguishes a current bundle from a stale
// one at runtime.
{
state.recordStep('sess-panes', {
label: 'Write the release note',
cwd: '/private/tmp/workspace-demo',
workspaceLabel: 'workspace-demo',
steps: 4,
maxSteps: 8,
spentUSD: 0.0042,
budgetUSD: 1,
state: 'running',
})
// Computed by `summarize`, not hand-written: this fixture threw
// `Cannot read properties of undefined (reading 'length')` in MetricsPanel
// because a hand-built summary left out `alerts` and each axis' `latest` /
// `samples`. The server's own producer cannot omit a required field, so
// asking it for the values is both shorter and the only version of this
// fixture that can go stale silently.
const { summarize } = await import('../src/metrics.ts')
const record = (steps, costUSD, wallMs, outcome) => ({
runId: `sess-${String(steps)}-${String(Math.round(costUSD * 1e6))}`,
startedAt: 1_700_000_000_000,
endedAt: 1_700_000_000_000 + wallMs,
pass: 1,
passes: 1,
taskKey: 'pane-fixture',
outcome,
steps,
maxSteps: 8,
costUSD,
budgetUSD: 1,
unpricedSteps: 0,
byRoute: { 'onegw/execution': { steps, usd: costUSD } },
stepLatencyMs: [],
wallMs,
signals: [],
judgeScores: [],
reviewFraction: 0.125,
specFingerprint: 'pane-fixture',
})
state.setMetrics(summarize([
record(7, 0.0029, 203_000, 'goal-met'),
record(8, 0.0040, 260_000, 'goal-met'),
record(8, 0.0081, 260_000, 'goal-met'),
record(12, 0.0060, 420_000, 'budget-stop'),
record(6, 0.0031, 180_000, 'model-stop'),
]))
state.setRecommendations([{
lever: 'prompt-cache',
current: 'off',
proposed: 'on',
evidence: 'no cache reads recorded',
confidence: 0.6562,
}])
await page.waitForTimeout(2500)

const metrics = await page.locator('#metrics').count()
const proposals = await page.locator('#proposals').count()
console.log(`panes: metrics=${String(metrics)} proposals=${String(proposals)}`)
if (metrics === 0) {
console.error('FAIL: #metrics is absent with a roll-up set — the served bundle predates the panel')
failures += 1
} else {
console.log(`MEASUREMENTS:\n${await page.locator('#metrics').innerText()}`)
}
if (proposals === 0) {
console.error('FAIL: #proposals is absent with a recommendation set — the served bundle predates the panel')
failures += 1
} else {
console.log(`PROPOSALS:\n${await page.locator('#proposals').innerText()}`)
}
}

// A 401 here means the shell's own bootstrap read the page without a token,
// and an SSE 404 means the loopback source never opened its stream — either
// is a page that renders once and then goes stale forever.
const real = errors.filter(e => !/favicon/.test(e))
console.log(`console/network errors: ${real.length === 0 ? 'none' : JSON.stringify(real.slice(0, 6))}`)
if (real.some(e => /pageerror/.test(e))) {
console.error('FAIL: the page threw — a bundle that cannot run is worse than a stale one')
failures += 1
}
await page.screenshot({ path: 'docs/evidence/standalone-panes.png', fullPage: true })
} finally {
await browser.close()
await dash.stop()
}

if (failures > 0) {
console.error(`standalone panes: ${String(failures)} failure(s) — §1cf says the loopback page is frozen`)
process.exitCode = 1
} else {
console.log('standalone panes: mounted, with MEASUREMENTS and PROPOSALS drawn')
}
Loading
Loading