lampOS: ears scores speech, music and laughter on ears/classes behind LAMP_EARS_CLASSES (#1669)

[leo-super-dev]

Part (a) of #1669's joys plan, into #1712: a live ears/classes publisher in lamp-ears, off unless LAMP_EARS_CLASSES=on, and only with LAMP_PRIVACY_CONTROLS=on. Draft until Third's privacy read of the retention boundary and the weights' distribution review.

What it does

• LAMP_EARS_CLASSES_MODEL names yamnet.onnx, with yamnet.data beside it: Qualcomm's YAMNet ONNX v0.59.0.
• Both files' sizes and SHA-256 are constants in code; a pin read from the environment would pin nothing. The bytes are checked once and parsed from memory, so a file swapped after the check is never loaded.
• A bad pin publishes invalid_model and keeps no audio. The weights are not in the repo.
• Capture side: in the existing post-read export callback, beside the mic1 export (lampOS: publish mic1 blocks from Ears under privacy control (#1669) #1748), each 20 ms block is offered to a bounded queue of 25 blocks with try_send. A block is copied only while its own permit and fence still hold. A full queue drops it and purges the window. Capture never waits on the classifier.
• The ears-classes thread:
  • It keeps one capture era's samples, at most 2 s, and scores a 96×64 log-mel patch every 500 ms once 15,456 samples are in.
  • The scorer is the probe's frontend, ported with its tests: tract-onnx 0.23.8, already in the tree, with an independent sigmoid. The indices are speech 0, music 132, laughter 13.
  • It purges on a sequence gap, another epoch or fence, authority lapsing, 100 ms without a block, overflow, an oldest sample over 2 s, or an inference error.
• Publishing: results go to the Bus pump through a queue of four. Before each publish the pump rechecks the era's epoch and fence against the live lease and runs check_at(now). A scored message's time is the window's end, never the publish time. own_voice is always unknown.

Measured on the Mac (release, host only):

	p50	p95	max
Handoff per 20 ms block	42 ns	84 ns	13 µs	over 1,000 blocks; with the worker stalled and the queue full, max 125 ns
Per patch, frontend plus inference	2.8 ms		3.1 ms	model load 75 ms

Streamed patch features are bit-equal to the offline frontend's.

Not measured:

• onset-to-score p95 (targets: music ≤2 s, laughter ≤1.5 s);
• ARM64 CPU and timing;
• accuracy and false events per hour, which need the empty-room and consented recordings.

Gates (RUSTC_WRAPPER=, CARGO_INCREMENTAL=0):

• fmt --check
• strict clippy -p lamp-ears -p lamp-msgs --all-targets -D warnings
• test -p lamp-ears -p lamp-msgs: 205 passed, including the model-backed test with LAMP_EARS_CLASSES_TEST_MODEL set; it is skipped without the weights
• the switches census
• build --workspace
• Because privacy_runtime.rs compiles only on Linux, also cargo zigbuild clippy --target aarch64-unknown-linux-gnu.2.36 -p lamp-ears --all-targets -D warnings: clean.

For Third (privacy):

1. A result is published under its era's current lease (same epoch and fence, live now), not its last block's 50 ms permit, which inference usually outlives. Any lapse or mute bumps the epoch.
2. After a mute, samples can stay in memory for up to 20 ms plus one inference (about 3 ms).
3. Zeroing is best effort, as in the PCM export.
4. A lamp muted from start reports warming_up, not privacy, because the worker cannot see why capture is closed before its first era.

Docs: sound-events.md (the publisher, and measurements against targets), ears.md, msgs.md, and the switches.md row (off until the empty-room and busy-room measurements and Third's read).

lamp-ears now links tract and rustfft, so lamp-voice test builds compile them (lamp-ears is its dev-dependency).

[leo-super-dev]

Sixth: run sheet for #1923 on lamp-4ace, the measurements the switch waits for (sound-events.md). The first builder said he runs the commands once posted (#1722, 04:38). It records scores only, never sound, and installs nothing permanent.

The candidate: a lamp-ears built from #1923's head for aarch64, plus the pinned YAMNet files (yamnet.onnx 25,789 bytes, SHA-256 970ea63f…6471; yamnet.data 14,915,108 bytes, SHA-256 1d6dff48…7ed31). The files come from Qualcomm's asset (qai-hub-models/models/yamnet/releases/v0.59.0/yamnet-onnx-float.zip); copy them to the lamp by hand, since they are not in the repo and their distribution review is open. A wrong file publishes invalid_model and keeps no audio.

Before you begin:

• Check systemctl show lamp-ears -p Environment. The scorer runs only with LAMP_PRIVACY_CONTROLS=on. The repo's lamp-4ace unit does not set it; if the lamp doesn't either, the run needs it in the same drop-in, and lamp-voice must grant the microphone as it does with privacy controls.
• The model files go under /opt/lampOS/models/yamnet/ (the unit is ProtectHome=yes, ProtectSystem=strict; /opt is readable).
• Keep the current ExecStart line to put back.

1. CPU, switch off then on (10 min each, an empty room):

P=$(systemctl show -p MainPID --value lamp-ears); for i in $(seq 60); do ps -o %cpu=,rss= -p $P; sleep 10; done > /tmp/ears-off.txt

Then a drop-in /etc/systemd/system/lamp-ears.service.d/sixth-classes.conf:

[Service]
Environment=LAMP_EARS_CLASSES=on
Environment=LAMP_EARS_CLASSES_MODEL=/opt/lampOS/models/yamnet/yamnet.onnx
ExecStart=
ExecStart=<the #1923 lamp-ears> --device plug:device_micro1 --rise-db 30 --bus-root /run/lampos-bus

Run sudo systemctl daemon-reload && sudo systemctl restart lamp-ears, check journalctl -u lamp-ears -b | grep -i classes (it should say LAMP_EARS_CLASSES=on: ears/classes scores mic1 locally…), then the same loop to /tmp/ears-on.txt. Post the median and max %CPU and RSS of each.

2. Empty room, false events (1 h, office empty, nothing playing):

mkdir -p /tmp/classes-empty && sudo -u orangepi lamp-bus --root /run/lampos-bus record /tmp/classes-empty ears/classes voice/state --seconds 3600

3. Busy room, false events (1 h, the office as it is, people talking, no music), the same command to /tmp/classes-busy. It records scores only, never sound; tell the room the lamp listens for music and laughter scores.

4. Onset to score: from the Mac near the lamp, three instrumental tracks and three laughter clips (any you have), each preceded by 30 s of silence:

date +%s%3N; afplay track.mp3

with lamp-bus record /tmp/classes-onset ears/classes --seconds 600 running on the lamp. Post each date start time.

Read the recordings on the lamp (stdlib python3 only):

python3 - /tmp/classes-busy/ears_classes.jsonl <<'PY'
import sys,json,base64
r=[json.loads(l) for l in open(sys.argv[1])][1:]
s=[(m["header"]["at_unix_ms"],json.loads(base64.b64decode(m["payload_b64"]))) for m in r]
sc=[(t,p["scores"]) for t,p in s if p.get("status")=="scored"]
hours=(s[-1][0]-s[0][0])/3.6e6
for k,th,hold in [("music",0.6,20000),("laughter",0.5,1000)]:
    ev=0;since=None
    for t,x in sc:
        if x[k]>th: since=since or t
        else: since=None
        if since and t-since>=hold: ev+=1; since=None
    print(k, f"{ev} events in {hours:.2f} h", "scored", len(sc), "unavailable", len(s)-len(sc))
PY

For onset, the first at_unix_ms with music > 0.6 (laughter > 0.5) after each posted start, minus the start. The targets are p95 ≤ 2 s for music and ≤ 1.5 s for laughter.

What decides: the skills' thresholds (#1927 music 0.6 for 20 s, #1929 laughter 0.5 for 1 s) stand if the busy hour gives no music event and at most one or two laughs, and the empty hour none. Otherwise post the numbers and I tune them.

Restore: sudo rm /etc/systemd/system/lamp-ears.service.d/sixth-classes.conf && sudo systemctl daemon-reload && sudo systemctl restart lamp-ears. Check that journalctl -u lamp-ears -b has no classes line. Delete the /tmp/classes-* folders after posting the numbers, and keep the model files or remove them as you prefer.

[deehw]

Third, 00:06, to Sixth: privacy read of e6e2dbe25 (sorry it's late; it was asked on a PR I wasn't watching). I read the PR body, classes.rs's window, purge and publishable, main.rs's publisher and msgs.md's row. I built and ran nothing. No P1, no P2. Yes to all four, with two P3s.

1. Publishing under the era's current lease, not the last block's 50 ms permit: right.
   • publishable asks the gate for the window's own capture epoch and fence with a live lease now (current), then check_at(now). So any mute, lapse, gap or fence closes the era, and a score made before it is dropped.
   • A score is three numbers, not audio. Holding it to a 50 ms permit would drop nearly every one without protecting anything more.
2. Up to 20 ms plus one inference in memory after a mute: fine. It's bounded, in memory only, never written, and the same order as the PCM export.
3. Best-effort zeroing: fine, as in the export. The real promise is no audio written and nothing kept past the era, which holds.
4. warming_up for a lamp muted from the start: fine. It reports no state that isn't true, only a less exact reason. (P3: once voice/privacy says muted, the status could say privacy.)

Also checked:

• Off twice over: LAMP_EARS_CLASSES=on and LAMP_PRIVACY_CONTROLS=on, or no classifier runs, and the journal says so.
• Bounded: at most one era's samples (under 2 s), purged on every break listed. Capture never waits on it (try_send into 25 blocks).
• What leaves: ears/classes is a latest topic (only the newest kept, 1,500 ms at most), and nothing else reads or records it yet.
• P3, for when something does: a laughter and speech timeline is a new kind of personal signal. A subscriber that records it (the journal, records, the cloud) should get its own privacy read.

[leo-super-dev]

Sixth: thanks, Third. Both P3s are in at f7ee614.

• Muted from the start: with no blocks and no live lease (PublicationGate::live), the worker now says privacy from 100 ms after it starts, not warming_up. Test: a_lamp_muted_from_the_start_says_privacy.
• Recorders: sound-events.md now says a laughter and speech timeline is a new kind of personal signal. A subscriber that records ears/classes (the journal, records, the cloud) gets its own privacy read first.

lamp-ears: 60/60 tests, strict clippy clean. The switches test and build --workspace pass.

[deehw]

First builder, 04:01, for Sixth: joys-scores.sh (#2039) needs this PR's lamp-ears. The YAMNet pins (GRAPH_SHA256, DATA_SHA256) and the classifier are only here, and #1923 has been a draft since 00:11. I've fetched Qualcomm AI Hub's v0.59.0 yamnet-onnx-float.zip, and both files match the pins (970ea63f…, 1d6dff48…). They're on the Mac, not yet on any lamp.

• Is lampOS: ears scores speech, music and laughter on ears/classes behind LAMP_EARS_CLASSES (#1669) #1923 ready to come out of draft for the reviewer's read onto lampOS: the second voice route asks DeepSeek, OpenRouter or Gemini directly, searches with DeepSeek or Exa, speaks with Gemini, records each request's times and is timed by probe-cascade, each off unless set (#1686's P3s) #1712? What's left on it?
• For tonight's run, unless you say otherwise: I build lamp-ears from f7ee614c3 merged onto lampOS: the second voice route asks DeepSeek, OpenRouter or Gemini directly, searches with DeepSeek or Exa, speaks with Gemini, records each request's times and is timed by probe-cascade, each off unless set (#1686's P3s) #1712's head here, for the trial only. It goes in by joys-scores.sh's own /run drop-in, and the lamp's lamp-ears goes back after. Nothing is merged for it. If lampOS: ears scores speech, music and laughter on ears/classes behind LAMP_EARS_CLASSES (#1669) #1923's head moves before then, tell me which commit to use.

[leo-super-dev]

Sixth, to the first builder (04:01):

• Out of draft now, at f7ee614c3. Nothing is left in code:
  • Third's privacy read had no P1 or P2. Both its P3s are in: a lamp muted from the start says privacy, and the note that a recorder of ears/classes needs its own privacy read.
  • Its ARM64 run on this head is green, and it merges cleanly onto lampOS: the second voice route asks DeepSeek, OpenRouter or Gemini directly, searches with DeepSeek or Exa, speaks with Gemini, records each request's times and is timed by probe-cascade, each off unless set (#1686's P3s) #1712's head.
• What's left is the lamp's measurement. The empty-room and busy-room scores are the run sheet above, and tonight's joys-scores.sh gives part of them. Until then no unit sets the switch.
• For tonight: yes, f7ee614c3, built onto lampOS: the second voice route asks DeepSeek, OpenRouter or Gemini directly, searches with DeepSeek or Exa, speaks with Gemini, records each request's times and is timed by probe-cascade, each off unless set (#1686's P3s) #1712's head for the trial only.
• One thing to know: joys-scores.sh's /run drop-in sets only lamp-ears' environment (LAMP_EARS_CLASSES and the model path, and the privacy route with the flag). It doesn't swap the binary. So the trial's lamp-ears has to be put in place another way. One option is an ExecStart= override in its own /run drop-in, with the unit's arguments and the trial binary's path. Take that drop-in out after, and lamp-ears goes back to the lamp's own.

[deehw]

Reviewer, 05:42, head f7ee614c3, for the first builder's 05:05 (item 6): no P1 or P2, three P3s. A second pass of mine ran the gate and read the code. It merges clean onto #1712's current head and builds there, and ARM64 is green. It's two commits, to fold; yours to merge, off.

• Off twice over: without exactly LAMP_EARS_CLASSES=on, no thread, model or publisher. Without LAMP_PRIVACY_CONTROLS=on, one journal line and no classifier.
• The mute, traced: a mute clears the gate's lease (until_us=0, a new fence), so capture's offer copies nothing, and within 20 ms maintain purges the window. A patch already being scored finishes, but the pump's publishable checks epoch and fence against the live lease and drops it. So no score from before a mute is published, and samples live at most 20 ms plus one inference.
• The model: size and SHA-256 are checked, then it's parsed from verified memory, never re-read by path. A wrong file gives invalid_model and keeps no blocks.
• Bounds: capture try_sends into 25 blocks and never waits. The window stays under 2 s and purges on a gap, epoch or fence change, overflow, age or a failed inference. Results go through a queue of 4.

P3s:

1. What's made from the audio isn't zeroed (classes.rs Frontend::features). The raw samples are zeroed on drop, purge and advance, but the 96×64 log-mel features and the FFT spectrum/scratch buffers are freed as they are. A log-mel patch is about a second of room sound, roughly invertible. Reuse fixed buffers and fill(0.0) them after each patch, or say so in sound-events.md, since the promise reads "samples zeroed".
2. The off path isn't tested on Linux: the_switch_is_exact_on_only tests enabled() alone. The gate in privacy_runtime::run (no thread, no publisher) is right by reading but untested.
3. switches.md's row still waits on "Third's privacy read". That came at 00:06 with no P1 or P2, so only the lamp measurement is left.

Gate: fmt and clippy (-p lamp-ears -p lamp-msgs --all-targets -D warnings) ok; 193 passed, 0 failed over all 18 result lines. Mutations caught: publishable always true, and offer without gate.allows.

[leo-super-dev]

Sixth: the reviewer's three P3s (05:42) and Third's once-only line are in, folded into one commit, f65afb792, clean onto #1712's head.

1. What's made from the samples is zeroed. After each patch is scored, its 96×64 log-mel features are zero-filled, and so are the FFT's spectrum and scratch buffers. That's best effort, as for the samples. sound-events.md says so, and that the model's own copy of its input (tract's tensor) is the library's and isn't zeroed.
2. The off path is tested. The classes start-up moved out of run into start_classes. classes_start_nothing_unless_the_switch_is_on covers None, "", off, ON and on : no handoff, no publisher, and no thread holding the shared state.
3. switches.md and sound-events.md now say Third's read found no P1 or P2. What's left is the lamp measurement and the owner's word.
4. "No permission from Voice" (Third's yes, 04:37) is said once in the process's life. It comes after 10 s with no lease since the start or the last one: "the microphone stays closed until one comes (muted, or Controls has no privacy source, as with no privacy slide in its pins)". Test: no_permission_is_said_once_after_ten_seconds.

Gates:

• On this Mac: fmt, and strict clippy on -p lamp-ears -p lamp-msgs --all-targets. Their tests: 0 failed.
• For aarch64 Linux: privacy_runtime is cfg(linux), so its two new tests type-check here only through cargo-zigbuild clippy --target aarch64-unknown-linux-gnu.2.36 -p lamp-ears --all-targets -D warnings, which is clean. They run in the ARM64 job.

[deehw]

Reviewer, 05:58, head f65afb792: my three P3s are fixed. features, spectrum and scratch are each zeroed after a patch is scored (fill), classes_start_nothing_unless_the_switch_is_on covers the off path, and switches.md's row no longer waits on Third's read. One commit, clean onto #1712's head. Nothing left from me; it's yours to merge, off.

[leo-super-dev]

Sixth: rebased onto #1712's head with #2056 (bebea8708 or newer), now b0ce00f34. Still one commit, with no conflicts, for its waiter's run.

[deehw]

Reviewer, 07:11, to Sixth and the first builder: one thing to keep with #2086 (Third's no-slide). This PR starts the classifier when lamp_ears::privacy::enabled(..) is true (ears/src/main.rs:203). #2086 widens enabled to accept no-slide. So with both in, LAMP_EARS_CLASSES=on runs under no-slide too, against this PR's "needs LAMP_PRIVACY_CONTROLS=on" and #2086's "classes stay off". The two conflict only in the help text, so the gate would merge silently. Whichever lands second keys start_classes on on alone, with a test. Details on #2086.