fix(bench): retry shell wake writes and require pymavlink >= 2.4.42#27852
fix(bench): retry shell wake writes and require pymavlink >= 2.4.42#27852mrpollo wants to merge 3 commits into
Conversation
…eout Running the suite from main against a release/1.17 FMUv6X-RT board (#27840) failed boot_health and flight_mission at 'shell open' while reboot_loop passed the same check 10/10 in the same run. The failing opens were the sessions immediately following a closed one, so the single wake write was being consumed before the firmware nsh pipes were reading and no amount of waiting could recover it. Re-send the wake write every second until the deadline, strip stale BENCHOPEN wake lines from subsequent command output, and replace the 5s hardcoded at every call site with one px4bench.SHELL_OPEN_TIMEOUT (10s) so the budget is tunable in one place. Signed-off-by: Ramon Roche <mrpollo@gmail.com>
log_transfer crashed with a raw ImportError on setups whose pymavlink predates the mavftp module (#27840). mavftp first shipped in pymavlink 2.4.42, so raise the floor in pyproject.toml, Tools/setup/requirements.txt and the README, and guard the import in px4bench.ftp so an old install reports a clear upgrade hint instead of a traceback. Both mavftp consumers now import it through the guard. Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The 1.17 run in #27840 read 'SDLOG_UTC_OFFSET not found in downloaded set' as the parameter not existing on that release. It does exist there; the download itself was the anomaly (19 params total, where a booted board reports hundreds, since the autopilot lists used params only). Name the downloaded count and that semantics in the failure so the next tiny param set is diagnosed as a degraded session, not a missing param. Signed-off-by: Ramon Roche <mrpollo@gmail.com>
|
I re-ran the suite from this branch against the same 1.17 FMUv6X-RT bench (same firmware,
I first suspected USB CDC packet loss ( This board heartbeats from two components on the same sysid: Possible fix: |
Follow-up to the release/1.17 run in #27840. Digging into that log, the shell failures are not the 5s-vs-8.8s boot timing suggested there: preflight opened the same shell seconds after the flash boot, reboot_loop passed the identical 5s open 10/10, and flight_mission failed its probe moments after reboot_loop had just verified the shell on the same boot. The pattern is that the session immediately following a closed one fails, meaning the single wake write gets consumed before the firmware nsh pipes are reading, and no amount of waiting recovers a dropped write.
MavlinkShell.open()now re-sends its newline plus sentinel wake write every second until the deadline, and staleBENCHOPENechoes are stripped from subsequent command output. The timeout hardcoded at every call site is now a singlepx4bench.SHELL_OPEN_TIMEOUT(10s).mavftpmodule, so the floor is raised inpyproject.toml,Tools/setup/requirements.txtand the README, and the import is guarded inpx4bench.ftpso an old install prints an upgrade hint instead of the raw traceback seen in the issue.param_existsfailure now reports the downloaded count and that the autopilot lists used params only.SDLOG_UTC_OFFSETdoes exist on release/1.17; the real anomaly in that run was a 19-param download (plus heartbeats from system 0), which points at a degraded MAVLink session, not a missing parameter.Verified with a mock SERIAL_CONTROL endpoint that drops writes for 2.5s: the old single-write open fails, the retrying open succeeds, and the timeout path stays bounded. The missing-mavftp path was tested by blocking the module import.
@farhangnaderi could you re-run the suite from this branch against your 1.17 FMUv6X-RT bench? boot_health, flight_mission and log_transfer should now pass or at least fail with a clearer story. If param_stress still reports a tiny download, please grab
param statusandmavlink statusfrom the nsh console on that boot; the 19-param set and the sysid 0 heartbeats are worth their own investigation.