Revised RADE verification procedure - #76
Conversation
…section verification_procedure.md: formal procedure for integration verification before Phase D OTA testing. Covers software loopback (mandatory), OTAC and OTC (at least one mandatory for hardware integrations). Requires clean signal path (transfer function of 1) in both directions, loss.py from this repo, and wav/all.wav as the test file. template.md: checklist form for testers to complete and submit as evidence of a verified integration. README.md: replace verbose prose with a short intro linking to the procedure, plus the baseline command lines. Baseline framing updated to require re-running with current version rather than using a cached loss value. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
|
@peterbmarks @tmiw - please take a look and tell me what you think. From our work to date, I think Peter you may have already the results to pass and Mooneer you just need an OTAC or OTC, repeated across the target OSes. |
|
Looks good @drowe67. I went straight from Level 1 Software loopback (virtual audio cable) to Level 3 RF over the coax. I agree that Level 2 is useful. It would be good to mention the value of --plot to see if there are outliers before (are they ever after?) that can be clipped off with --clip_start. For me it varies but I need -clip_start 8. |
|
Actually, what about systems that can only support RX? (Example: freedv-ka9q.) Would an OTC/OTAC test still be needed, just with another approved app as the TX? |
|
I think we should have some sample modem.wav files and their features.f32. Then people testing receivers, such as WebSDRs could check their performance. |
|
The good news is that freedv-gui is giving me a loss of 0.083 now for software loopback since I adjusted the scaling (tmiw/freedv-backend@3cf3cc8). (Only tested on macOS so far.) No +6 dB gain needed. Hopefully OTC/OTAC will give me similar results. |
To update, I was able to do a successful OTAC test on macOS as well this morning (0.088). I still had problems doing an OTC test so I'll need to investigate that further. One note: it still seemed to be pretty sensitive to RX signal levels. But it looks like I was still using RADE main and not dr-v2-rx-gain, so that could be it. EDIT: I tried dr-v2-rx-gain and had to adjust levels again to get a pass. I suspect I was already borderline with the test out of main, so I'll take a look and see if there's anything more I can clean up on my end. |
…cal OR Explains the rationale for the L2/L3 choice: OTAC covers all application integration paths; OTC is the appropriate alternative for SDRs and hardware radios where OTAC is impractical (e.g. Flex). At least one is mandatory. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…ixes - Add 'Worked Example of Loss Tests' section to verification procedure, showing clip value selection with before/after loss plots - Add automated pass/fail subsection using --compare/--delta for CI use - loss.py: print FAIL and exit(1) for --compare failures; replace quit() with sys.exit(1) throughout for correct CI return codes Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
…an PNGs Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Good idea. Take a look at the new "Worked Example of Loss Tests" example in cae3fb3 |
|
Yes, that's very helpful. |
Tested pipeline: lpcnet_demo -> tx2.py -> f32toint16.py --real -> sox -> WAV -> rade_rx_wav gives loss 0.083, consistent with baseline. Co-Authored-By: Claude Sonnet 4.6 <noreply@anthropic.com>
Excellent question. Pls take a look at 1e47cbd and see if that works. |
A Tx wave file would go out of date silently if the software changes in any way. Instead, how about the programmatic approach in 1e47cbd. We're already generating the feature file as part of the software reference, so just one more step to a wav. |
Yep, that works. |
|
@peterbmarks @tmiw are you happy for me to merge this PR, and start following this procedure? I'm keen to keep this moving along so we can get to the Phase D PTT Beta tests. I feel we're converging nicely. Heads up that I have a lot of travel in August, including two weeks completely off grid. |
|
I'm fine to merge. |
|
Yes please merge. |
I used Claude for that (for the worked example). |
|
@peterbmarks @tmiw - are you intending to submit tests results using the agreed template/following the verification procedure for the various software projects you maintain that integrate RADE V2? I'm keen to keep this moving along so we can get to the Phase D PTT Beta tests. |
Yes, but I have some real life stuff going on so it'll be a few days minimum. I'm also working through an issue with freedv-gui where I'm getting intermittent loss figures slightly above the cutoff so that may delay things too. |
Of course - take your time, life and family comes first. I'm also off walking for the next 7 days so will have limited bandwidth. The intermittent issues are just what this test is designed to flush out. Just wanted to make sure verification is in the pipeline. |
|
Yes I will. I'll get it done this week. Driving to Melbourne after the morning net today so won't be immediately. I think it's a good thing to model for other porting users. |
3 participants
Claude and I have worked up a revised verification procedure, based on lessons learned from peterbmarks/radae_nopy#14.