Skip to content

fix(bench): retry shell wake writes and require pymavlink >= 2.4.42#27852

Open
mrpollo wants to merge 3 commits into
mainfrom
mrpollo/px4bench-27840-fixes
Open

fix(bench): retry shell wake writes and require pymavlink >= 2.4.42#27852
mrpollo wants to merge 3 commits into
mainfrom
mrpollo/px4bench-27840-fixes

Conversation

@mrpollo

@mrpollo mrpollo commented Jul 9, 2026

Copy link
Copy Markdown
Contributor

Follow-up to the release/1.17 run in #27840. Digging into that log, the shell failures are not the 5s-vs-8.8s boot timing suggested there: preflight opened the same shell seconds after the flash boot, reboot_loop passed the identical 5s open 10/10, and flight_mission failed its probe moments after reboot_loop had just verified the shell on the same boot. The pattern is that the session immediately following a closed one fails, meaning the single wake write gets consumed before the firmware nsh pipes are reading, and no amount of waiting recovers a dropped write.

  • MavlinkShell.open() now re-sends its newline plus sentinel wake write every second until the deadline, and stale BENCHOPEN echoes are stripped from subsequent command output. The timeout hardcoded at every call site is now a single px4bench.SHELL_OPEN_TIMEOUT (10s).
  • pymavlink 2.4.42 is the first release shipping the mavftp module, so the floor is raised in pyproject.toml, Tools/setup/requirements.txt and the README, and the import is guarded in px4bench.ftp so an old install prints an upgrade hint instead of the raw traceback seen in the issue.
  • The param_exists failure now reports the downloaded count and that the autopilot lists used params only. SDLOG_UTC_OFFSET does exist on release/1.17; the real anomaly in that run was a 19-param download (plus heartbeats from system 0), which points at a degraded MAVLink session, not a missing parameter.

Verified with a mock SERIAL_CONTROL endpoint that drops writes for 2.5s: the old single-write open fails, the retrying open succeeds, and the timeout path stays bounded. The missing-mavftp path was tested by blocking the module import.

@farhangnaderi could you re-run the suite from this branch against your 1.17 FMUv6X-RT bench? boot_health, flight_mission and log_transfer should now pass or at least fail with a clearer story. If param_stress still reports a tiny download, please grab param status and mavlink status from the nsh console on that boot; the 19-param set and the sysid 0 heartbeats are worth their own investigation.

mrpollo added 3 commits July 9, 2026 08:09
…eout

Running the suite from main against a release/1.17 FMUv6X-RT board
(#27840) failed boot_health and flight_mission at 'shell open' while
reboot_loop passed the same check 10/10 in the same run. The failing
opens were the sessions immediately following a closed one, so the
single wake write was being consumed before the firmware nsh pipes were
reading and no amount of waiting could recover it. Re-send the wake
write every second until the deadline, strip stale BENCHOPEN wake lines
from subsequent command output, and replace the 5s hardcoded at every
call site with one px4bench.SHELL_OPEN_TIMEOUT (10s) so the budget is
tunable in one place.

Signed-off-by: Ramon Roche <mrpollo@gmail.com>
log_transfer crashed with a raw ImportError on setups whose pymavlink
predates the mavftp module (#27840). mavftp first shipped in pymavlink
2.4.42, so raise the floor in pyproject.toml, Tools/setup/requirements.txt
and the README, and guard the import in px4bench.ftp so an old install
reports a clear upgrade hint instead of a traceback. Both mavftp
consumers now import it through the guard.

Signed-off-by: Ramon Roche <mrpollo@gmail.com>
The 1.17 run in #27840 read 'SDLOG_UTC_OFFSET not found in downloaded
set' as the parameter not existing on that release. It does exist there;
the download itself was the anomaly (19 params total, where a booted
board reports hundreds, since the autopilot lists used params only).
Name the downloaded count and that semantics in the failure so the next
tiny param set is diagnosed as a degraded session, not a missing param.

Signed-off-by: Ramon Roche <mrpollo@gmail.com>
@github-actions github-actions Bot added kind:bug Something is broken or behaving incorrectly. scope:tools Scripts, developer tools, packaging, or setup helpers. labels Jul 9, 2026
@farhangnaderi farhangnaderi self-assigned this Jul 9, 2026
@farhangnaderi

Copy link
Copy Markdown
Contributor

I re-ran the suite from this branch against the same 1.17 FMUv6X-RT bench (same firmware, v1.17.0-6-g6737fe1c37), with pymavlink upgraded to 2.4.49. Results are as below:

TEST            STATUS   TIME(s)  DETAIL
----------------------------------------------------------------------
boot_health     PASS         5.7
param_stress    FAIL         2.2  exit code 1
mission_stress  PASS        10.1
storage_stress  PASS        11.8
log_transfer    PASS        29.1
reboot_loop     PASS        91.2
flight_mission  SKIP         2.1  skipped by the test (see its output above)
----------------------------------------------------------------------
5 passed, 1 failed, 1 skipped

param status on the board shows: 1135/2034 (used/total) params. So it's not a missing-parameter problem I suppose.

I first suspected USB CDC packet loss (mavlink status did show a rough rx-loss number on that instance), but that theory doesn't hold up: I reproduced px4bench.connect()'s exact code path standalone and the real mechanism came out to be component addressing.

This board heartbeats from two components on the same sysid:

srcSystem=1 srcComponent=1   type=2  autopilot=12 (PX4)              <- the real autopilot
srcSystem=1 srcComponent=236 type=34 autopilot=8  (MAV_AUTOPILOT_INVALID)  <- some other onboard component

Possible fix: px4bench.connect() should explicitly resolve and pin target_component to the heartbeat where autopilot != MAV_AUTOPILOT_INVALID (normally compid=1), instead of leaving it at pymavlink's default broadcast 0, and downstream param/mission calls should use that pinned target.

@farhangnaderi farhangnaderi removed their assignment Jul 10, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

kind:bug Something is broken or behaving incorrectly. scope:tools Scripts, developer tools, packaging, or setup helpers.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants