Skip to content

feat(harness): declarative node config — set up ESP32 CSI nodes from one YAML file - #2545

Open
matthewxfz3 wants to merge 8 commits into
ruvnet:mainfrom
matthewxfz3:feat/ruview-node-config
Open

matthewxfz3 wants to merge 8 commits into
ruvnet:mainfrom
matthewxfz3:feat/ruview-node-config

Conversation

@matthewxfz3

@matthewxfz3 matthewxfz3 commented Nov 13, 2026 •

Copy link
Copy Markdown

Current CI repair — 2026-11-19

Current head 73640844e0b8acff47c9a75bb5d9a7b6cf9794a9 incorporates main13b4788 and runner PR #2583 at6bff4df9 through a normal history-preserving merge. Land/reconcile #2583 first; no server main merge or release occurred here.

  • Fix the Windows tilde-path test to use native path.join semantics; runtime node configuration and device-ID conflict/retirement guards are unchanged.
  • Explicit package-budget proposal for review: 480KiB (491520B), up from the prior448KiB. Measured published artifact479863B includes a new44024B node-config runtime module and leaves11657B headroom. No package content was silently removed. The hard bound and source-map prohibition remain enforced; the original over-budget failure is retained.
  • Local Node24.19/macOS: full harness322PASS/5existingSKIP; security152PASS/3existingSKIP; all65 manifest files verify; actual pack gate PASS and oversized/source-map negative controls FAIL as expected; actionlint and secret/diff checks PASS. The first run reproduced pre-existing /var vs /private/var fixture failures; canonical TMPDIR=/private/tmp passes without source changes or skipped tests.
  • Fresh exact-head hosted checks, including Windows and npm pack, are pending. Existing formal approvals onb02ef79 and earlier tests are historical and do not qualify this head. No hardware check was rerun by this repair lane.

The original feature description and contributor's earlier hardware evidence below are preserved as historical context.

Summary

Bringing up an ESP32 node today means a manual chain per board: pick the image, run esptool, run provision.py with the right node ID, TDM slot and target, then check the stream. This PR makes setup config-driven. One YAML file (~/.ruview/nodes.yaml) declares the network, aggregator, firmware and every node by MAC. ruview node apply brings an attached board to that declared state, reusing the existing pieces: flashFirmware, provision.py and the UDP capture.

  • ruview node plan: validate the config and show what each node will get. Read-only.
  • ruview node apply: read the board's MAC, resolve its node (config, then registry, then next free ID), and run the configured steps: flash → provision → validate. --firmware-only flashes and validates while keeping the settings. Works with --port, --all or --watch.
  • ruview node status: read-only per-node CSI rate, loss and RSSI from UDP, joined with :8032/ota/status.
  • Every write prints its plan and needs a typed yes, or --confirm. Without a terminal (pipes, --json, MCP), only the plan is printed.
  • Secrets are file paths passed straight to provision.py, so the harness never reads them. Inline passwords and env-var sources are refused.
  • Strict YAML subset, zero runtime deps. Anchors, tags, block scalars, tabs, duplicate and unknown keys fail closed.
  • Fixes found on hardware:
    • the capture summary includes each node's sender address
    • summarizeBootLog parses the current CSI Node … — vX.Y.Z banner
    • the CLI finds its checkout when cwd is outside the repo

Template: firmware/esp32-csi-node/nodes.example.yaml.

Test plan

Hardware: two ESP32-S3 boards (QFN56 rev v0.2, 16 MB flash, 8 MB PSRAM, WCH CH343), macOS 15, firmware 0.8.15 from a local build/ with the DevKitC overlay. MACs, IPs, SSID and paths are redacted.

Unit tests

$ node --test test/node-config.test.mjs test/firmware.test.mjs
ℹ tests 28
ℹ pass 28
ℹ fail 0

Package tests. Both failures also fail on main on macOS, from /var vs /private/var tmpdir comparisons in the hosts tests:

$ npm test
ℹ tests 308
ℹ pass 301
ℹ fail 2

Plan from a real config (read-only):

$ ruview node plan
Node plan — ~/.ruview/nodes.yaml
  WiFi: HomeNet-2G · target 192.168.1.10:5005 · steps device,driver,wifi
  Secrets: wifi file ~/.ruview/wifi-pass; ota not configured — remote OTA and config_push stay disabled
  Firmware: local-build …/firmware/esp32-csi-node/build (ESP32-S3 8MB; bootloader:unverified, partition-table:unverified, otadata:unverified, app:unverified)
  Validate: wait 60s, measure 15s, min 20 Hz

  NODE  MAC                ZONE    TDM  TIER
  1     aa:bb:cc:00:00:01  zone-a  0/2  2
  2     aa:bb:cc:00:00:02  -       1/2  2
  ! no ota.psk_file: provisioning disables OTA updates and remote config_push on these nodes

Apply, prompt declined: nothing is written (registry unchanged):

$ ruview node apply
  … reading MAC on /dev/cu.usbmodemB1
✔ apply /dev/cu.usbmodemB1 · MAC aa:bb:cc:00:00:01 · node 1 (zone-a) [config]
  { "target": "192.168.1.10:5005",
    "provision": ["python","provision.py","--port","/dev/cu.usbmodemB1","--chip","esp32s3",
      "--ssid","HomeNet-2G","--password-file","~/.ruview/wifi-pass","--target-ip","192.168.1.10",
      "--target-port","5005","--node-id","1","--tdm-slot","0","--tdm-total","2","--edge-tier","2","--zone","zone-a"] }
  ? write WiFi/node settings to aa:bb:cc:00:00:01 (node 1) on /dev/cu.usbmodemB1? Type yes to continue [y/N] n
  skipped — nothing was written

New, unlisted board, full apply. MEASURED; reproduce with ruview node apply, validate stage plus boot log. Captured just before the command rename (fleet → node); the code path is unchanged:

$ ruview node apply --watch --confirm --json
{ "ok": true, "flow": "apply", "port": "/dev/cu.usbmodemB2", "mac": "aa:bb:cc:00:00:02",
  "node_id": 2, "node_source": "auto-assigned",
  "stages": {
    "flash":    { "ok": true, "panic": false },
    "wifi":     { "ok": true, "target": "192.168.1.10:5005", "ota": false },
    "validate": { "ok": true, "csiRateHz": 30, "csiLossFraction": 0.3411, "rssiMean": -49.5, "freqMhz": 2462,
                  "evidence": "MEASURED: live UDP CSI from this node decoded on this host" } } }

Registry trail for that board (no secrets):

$ grep aa:bb:cc:00:00:02 ~/.ruview/esp32-setup/devices.jsonl | jq -c '{event, node_id, health}'
{"event":"node_assigned","node_id":2,"health":null}
{"event":"chip_probed","node_id":2,"health":null}
{"event":"flashed","node_id":2,"health":null}
{"event":"provisioned","node_id":2,"health":null}
{"event":"validated","node_id":2,"health":"healthy"}

Read-only status (MEASURED; reproduce with ruview node status; node 1 was unplugged):

$ ruview node status --json | jq -c '.summary, (.nodes[] | {node_id, firmware, http, csiRateHz, health})'
"1/2 known nodes healthy (min 20 Hz)"
{"node_id":1,"firmware":null,"http":"unreachable","csiRateHz":0,"health":"offline"}
{"node_id":2,"firmware":"0.8.15","http":"ok","csiRateHz":31.19,"health":"healthy"}

Firmware-only on both boards. The flash succeeds and the version parses. Validation reports the UDP port held by another receiver (see follow-ups):

$ ruview node apply --firmware-only --confirm --json | jq -c '{node_id, fw: .stages.flash.firmwareVersion, reason}'
{"node_id":2,"fw":"0.8.15","reason":"port_in_use"}
{"node_id":1,"fw":"0.8.15","reason":"port_in_use"}

Follow-ups

  • A busy UDP port should be reported as a skipped validation rather than a failure, and --firmware-only should honour steps.
  • MEASURED (reproduce with ruview esp32 --seconds 15): both nodes show 34–37% CSI sequence loss on channel 11. validate.max_loss exists but is unset by default.
  • CLI-only, with no MCP tool yet. Firmware builds stay with Docker.

🤖 Generated with Claude Code

Maintainer follow-up — 2026-11-13

At Dragan's request, repair 3e1af4f9 fixes the three findings in review 5888074936. Final head b02ef79ad1be51042da985cfe686e36d4bbc88a7 also reconciles current main 094e1400 via a normal merge commit; pre-existing conflicts are resolved, upstream C5 safeguards/package0.9.3/test concurrency are preserved, and generated manifests are rebuilt.

  • Interactive confirmation is bound to the displayed MAC and node ID. A changed board/assignment requires a new plan before registry, flash or NVS writes.
  • Restored and explicit assignments are checked against effective config/registry ownership. Duplicate IDs fail closed; malformed assignments are ignored; ambiguous status is id-conflict.
  • Registry-only nodes remain visible/count unhealthy when offline; explicit reassignment does not duplicate historical IDs.
  • Docker publishing and external CI notifications are main-push-only, matching the reviewed server2513 guard. Feature/PR verification does not publish or notify.

Eleven regressions were added: eight fail on original59cb and pass on repair; three positive/compatibility controls remain passing. Final reconciled validation on Node24.19.0/macOS: 41 focused tests pass; full package318pass/5existing skips/0fail with TMPDIR=/private/tmp. The two default macOS temp-path alias hosts-test failures were reproduced on original and repaired source; canonical TMPDIR runs the full suite without exclusions. Harness65-file manifest verification, Gitleaks and whitespace checks pass. GitHub reports mergeable:true; no hosted checks/workflow runs are reported for this head yet, so hosted CI is not claimed green. No real USB/UDP, firmware/app/device operation or main merge was performed.

Replaces the per-board esptool/provision.py chain with a declarative config applied by MAC.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>

@proffesor-for-testing proffesor-for-testing left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

REMAND at 59cb729. Existing targeted tests pass (28/28), but three independent synthetic production-function probes expose gaps. No real serial ports, UDP listeners or devices were opened; author branch unchanged.

  1. P1 — bind interactive confirmation to the displayed board/plan. EXECUTED. harness/ruview/src/cli.js:289-299 obtains an unconfirmed plan, asks permission for its MAC/node, then calls applyBoard again without the expected MAC/plan. node-config.js:522 re-probes and adopts whatever board now occupies the path. Probe: plan MAC A/node1; replace mocked MAC with B/node2 on the same port before the confirmed call; the provisioning mock receives --node-id 2 and the result is ok:true. Re-identification is necessary, but a changed identity must invalidate the old confirmation. Pass an immutable expected MAC/node/plan into confirmed execution and reject mismatch before flash, registry assignment or NVS writes; test replacement between plan and confirmation.

  2. P1 — registry restoration can duplicate an explicitly configured node ID. EXECUTED. harness/ruview/src/node-config.js:332-333 restores the last registry ID without checking current assignments. Probe: registry MAC B has node4; YAML now assigns node4 to different MAC A; resolveNode(config, B, registry) returns B/node4, source registry. Applying both boards produces the same stream identity. Reject/reconcile conflicting restored IDs against the full current config and active registry map before planning or writes; add stale-registry versus explicit-config collision controls.

  3. P2 — offline auto-assigned nodes disappear from status and can yield false success. EXECUTED. harness/ruview/src/node-config.js:610-614 builds registry assignments but omits their IDs from the iteration set. Probe: empty YAML node list, registry node_assigned MAC B/node4, empty mocked capture. nodeStatus returns ok:true, nodes:[], 0/0 known nodes healthy. Include active known registry IDs in status and retain an offline/no-CSI row; test disappearance/reappearance independently of explicit YAML entries.

Applicability to the managed Mac/radar lane: this is a generic CSI CLI, not a dedicated MR60 installer. Its example aggregator UDP5005 must not be copied into the current raw55063/radar5005 split without explicit configuration. Status also competes for the live receiver port; no live status/apply qualification was attempted. No C5, signed-app, USB replacement or physical radar acceptance claim is made.

Verification: frozen Git archive of this head, including its exact YAML example; node --test --test-reporter=dot test/node-config.test.mjs test/firmware.test.mjs PASS28. Separate Node imports of production normalizeNodeConfig, resolveNode, nodeStatus, applyBoard used temporary synthetic registry/config and mocked exec/capture/fetch to reproduce the findings above. Full package suite and hardware checks were not rerun.

@proffesor-for-testing proffesor-for-testing left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO BLOCK for the three repaired review findings at 3e1af4f. This is the repair author's bounded recheck, not an independent or formal human approval.

EXECUTED: eleven new regressions; eight fail on frozen original59cb and all pass on the repair. Focused39/39; full package314pass/5existing skips/0fail with canonical TMPDIR; original default-temp hosts failures independently reproduced. Manifest65-file integrity, Gitleaks and diff checks pass. Checked the actual CLI caller passes displayed MAC/node into confirmed execution before any registry/device writes; covered same-board assignment drift, registry/config and registry/registry collisions, explicit reassignment, malformed records, offline→online status and ambiguous identity reporting.

The original REMAND remains an accurate historical verdict for59cb; its three findings are closed at this new head. CI status and original hardware/receiver qualification limits remain separate; no physical operation or merge.

@proffesor-for-testing proffesor-for-testing left a comment

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

NO BLOCK within the repaired findings/current-main reconciliation scope at b02ef79. Repair-author recheck; not an independent or formal human approval.

The three original findings are closed: displayed MAC/node binds confirmation before side effects; effective retained/config ownership rejects collisions; offline registry nodes remain visible/unhealthy and ambiguous IDs are explicit. Eleven new regressions include eight original-source failures and three compatibility controls. Final reconciled head:41focused PASS; full package318PASS/5existing skips/0FAIL with canonical TMPDIR; manifest65-file integrity, Gitleaks and diff checks PASS.

Normal merge of main094 resolves the original PR's existing conflicts, preserves upstream strict CSI callback evidence/C5 watchdog/security-eFuse/flash-capacity protections and package0.9.3 bounded test concurrency. GitHub mergeable=true. Hosted checks/runs are not reported yet; no hosted-CI pass or physical qualification is inferred. Main and installed app/helper/devices were not changed. Earlier reviews remain historical for their exact heads.

matthewxfz

This comment was marked as resolved.

matthewxfz

This comment was marked as outdated.

…d board's ID

plan resolved against an empty registry, so it passed configs apply then refused, and a gone board's ID could only be freed by hand-editing devices.jsonl.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
@matthewxfz3

Copy link
Copy Markdown
Author

Thanks for the review and the repair commits. Follow-up at 578ca7b for two gaps in the new node-ID conflict logic:

  1. plan and apply disagreed. planNodes resolved against an empty registry, so plan passed a config that apply then refused with node_id_conflict. plan now reads the same registry and reports conflicts (ok: false, conflicts[], STATE column).
  2. A gone board's retained ID could not be released except by hand-editing devices.jsonl. This adds a top-level retired: [<mac>]. Retired MACs are dropped from the retained inventory, so a replacement or auto-assign can take the ID. A retired board plugged back in is refused (node_retired), and a MAC can't be both retired and under nodes:. The conflict error now points at retired:.

Also:

  • reverted the redundant summarizeBootLog regex and duplicate test, so firmware.js and its test match main
  • isolated the tests from the operator's real ~/.ruview registry
$ node --test test/node-config.test.mjs test/firmware.test.mjs
ℹ tests 45
ℹ pass 45
ℹ fail 0

The 5 new regressions fail on b02ef79's node-config.js and pass here. npm test gives 320/327; the 2 failures are the pre-existing macOS /var vs /private/var hosts tests. Manifest verified.

One question on the ci.yml change in 3e1af4f (Docker build and Notify gated to pushes on main): is that meant to be part of this PR? It changes CI for every PR, so it might be cleaner as its own change.

@proffesor-for-testing

Copy link
Copy Markdown
Collaborator

NO BLOCK in the reviewed node-ID conflict/retirement delta at578ca7b2232a6e20441717b9bfd25d2c738dd3a3.

EXECUTED:45 focused node-config/firmware tests passed; the complete harness run passed322 tests with5 skips after installing the pinned development dependencies and using canonical /private/tmp for macOS fixtures. The first full run lacked @metaharness/flywheel and hit existing /var versus /private/var fixture assumptions; those were environment/portability failures, not changes to this PR. No source changes were needed for this review.

STATIC: plan now resolves against the same registry as apply; retired MACs release their retained IDs and are refused if selected again; normalized config prevents active/retired overlap; offline inventory excludes retired entries. The updated boot-banner expression still recognizes the current S3 banner. Explicit confirmation, MAC re-probe and firmware identity checks remain.

No hardware, USB, flashing, Wi-Fi or live CSI acceptance performed. Hosted checks are not reported for this head. This is a scoped software review, not merge authorization or hardware qualification.

@proffesor-for-testing

Copy link
Copy Markdown
Collaborator

Current-head CI/review reconciliation at 578ca7b2232a6e20441717b9bfd25d2c738dd3a3:

  • The absent checks are explained by nine workflows awaiting execution approval (action_required), not nine executed test failures. Examples: CI43516143966, Security43516144078, QEMU43516143822. API check-runs count is0 at this head.
  • Both recorded formal approvals apply to older b02ef79ad1be51042da985cfe686e36d4bbc88a7. They predate the current plan/apply conflict and retirement changes; do not treat them as a fresh exact-head approval.
  • My November14 scoped local review at578ca7b remains historical executed evidence (45 focused tests; full harness322 pass/5 skip with pinned dependencies/canonical macOS fixtures), not hosted qualification or hardware acceptance. No source change since that reviewed head was found.
  • The reviewer's CI-scope question also remains relevant: this PR restricts the advisory Docker job and write-capable Notify job to pushes on main. That is a repository-wide workflow change and needs an explicit scope decision during current-head review; the node-config review alone does not settle it.

Next steps: approve the intended workflow executions through the repository's normal process, obtain current-head results, and refresh the maintainer review on578ca7b (or its replacement). No workflow approval, source edit, merge or device action performed in this reconciliation.

@proffesor-for-testing

Copy link
Copy Markdown
Collaborator

CI/review reconciliation at unchanged head 578ca7b2232a6e20441717b9bfd25d2c738dd3a3 (2026-11-18): execution approval for all nine waiting workflows succeeded. This was an execution gate, not proof that the acting user lacked repository write access.

The old PR-event CI checks out 6e0a481423bf3eb44444337a4952d6841999c9af, merging this head into old main 094e1400719ded1283c67d23b9c737f6738a3244; the checkout log confirms those exact parents. To qualify today's combination without editing the contributor branch, GitHub's new merge candidate e44e97bf844db617c1d10c13b7388c86ecf9dcfa (parents current main 13b47883a73d5070f35a3f04aec2ffa778b47e29 and PR head 578ca7b…, tree 3bdd82b5367329215af32550832fc18b96137e96) is pushed unchanged to feat/2545-current-main-ci-20261118. No competing PR or main merge. Full current-main CI/security/bench qualification is running. Five still-running old-base workflows were cancelled to avoid duplicate full qualification; their completed results are preserved.

Two newly executed failures remain actionable:

  1. Package footprint: npm packages43516143964 fails on both Node20/22: packed contents are 479,863 bytes, exceeding the reviewed 458,752-byte ceiling by21,111 bytes (.github/workflows/npm-packages.yml:49). Trim/package the new capability more narrowly or explicitly review a justified new footprint budget; do not simply bypass the gate. The passing unit/security tests did not exercise this pack-size gate.
  2. Windows portability: flywheel43516143894 fails harness/ruview/test/node-config.test.mjs:85: expected /home/u/p, actual \\home\\u\\p. Compare against join('/home/u', 'p') (the test already imports join) so it tests tilde expansion using native path semantics; retain the real normalized-path assertion.

The dedicated device-e2e workflow, CSI policy and fix-marker runs passed. These are source/test checks, not physical hardware qualification. The prior code-only NO BLOCK does not mean current CI is green; formal approvals on b02ef79… also remain historical for this newer head. The two CI guards in this PR restrict Docker and Notify to push/main; they were not widened or changed by this follow-up. No contributor source, approval, release or installed helper was changed.

@proffesor-for-testing

Copy link
Copy Markdown
Collaborator

Current-main candidate qualification completed on e44e97bf844db617c1d10c13b7388c86ecf9dcfa:

The separate npm package-size and Windows assertion failures above remain open. The qualification branch's push filters do not execute those same PR-only package/flywheel jobs, so its green full CI cannot clear them. Contributor branch and reviewed size ceiling remain unchanged; no main merge/release was performed.

Justinwold88 and others added 2 commits November 19, 2026 02:38
Run the MQTT test broker from the eclipse-mosquitto:2.0.18 image, build
Espressif QEMU on ubuntu-24.04, and enable git long paths before cargo
on Windows. Plan item 0.6.

Co-Authored-By: Claude Opus 5.5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01T8hca9bTB4puaeBstXR9CE
@proffesor-for-testing

Copy link
Copy Markdown
Collaborator

Taking the remaining CI repairs under Dragan's selected server lane. Current contributor head578ca7b2232a6e20441717b9bfd25d2c738dd3a3 is preserved; please flag conflicting source/workflow edits.

Planned normal update in an isolated worktree: incorporate current-main and #2583's shared runner repairs without a main merge; correct the existing Windows tilde-expansion test to native path semantics; propose an explicit480KiB package ceiling for the new node-config capability. Local npm pack reproduces479863bytes (468.6KiB), including44024bytes for the new runtime module; the old448KiB ceiling predates that capability. The proposed480KiB ceiling retains a hard bound with11657bytes headroom. This is an explicit feature-footprint change for review, not a claim that the old ceiling passed or an already-approved policy.

Runtime node-config behavior, device-ID conflict/retirement guards and security manifests remain unchanged. Preserve full package, harness, Windows, broker/QEMU/fuzz qualification; no device flashing, contributor-history rewrite, main merge or release. Will provide new exact head, pack inventory, tests and fresh CI before treating ready.

Owner Dragan; agent Codex; host DraganMac; session ruview-server-repairs-20261119; lease until14:25UTC.

This branch has not been deployed

No deployments
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

4 participants