Skip to content

[Celestica] Leh800bcls: Fix CPU-originated packet targeting failure in QueuePerHost tests on multi-NPU platforms - #1500

Open
gang-tao wants to merge 1 commit into
facebook:mainfrom
gang-tao:leh800bcls_agent_fix21
Open

[Celestica] Leh800bcls: Fix CPU-originated packet targeting failure in QueuePerHost tests on multi-NPU platforms#1500
gang-tao wants to merge 1 commit into
facebook:mainfrom
gang-tao:leh800bcls_agent_fix21

Conversation

@gang-tao

Copy link
Copy Markdown
Contributor

Pre-submission checklist

  • [ ✓ ] I've ran the linters locally and fixed lint errors related to the files I modified in this PR. You can install the linters by running pip install -r requirements-dev.txt && pre-commit install
  • [ ✓ ] pre-commit run
clang-format.............................................................Passed
shellcheck...........................................(no files to check)Skipped
shfmt................................................(no files to check)Skipped
trim trailing whitespace.................................................Passed
fix end of files.........................................................Passed
check yaml...........................................(no files to check)Skipped
check json...........................................(no files to check)Skipped
check for merge conflicts................................................Passed
ruff check...........................................(no files to check)Skipped
ruff format..........................................(no files to check)Skipped
Prevent sai_impl in fboss manifest.......................................Passed

Summary

In Multi-Switch (Split Agent) configurations, QueuePerHost hardware tests (specifically AgentQueuePerHostRouteTest) failed when run on secondary NPUs (such as NPU1). The test expects 2 packets to be queued on the target port's queue, but instead gets 0.

/var/FBOSS/fboss/fboss/agent/test/utils/QueuePerHostTestUtils.cpp:180: Failure
Expected equality of these values:
  pktsOnQueue
    Which is: 0
  2
Google Test trace:
/var/FBOSS/fboss/fboss/agent/test/agent_hw_tests/AgentQueuePerHostRouteTests.cpp:238: v6

Root Cause

When injecting CPU-originated (switched) packets during the verification phase, QueuePerHostTestUtils invoked sendPacketSwitchedAsync(std::move(pkt)) without specifying a target SwitchID. In multi-NPU environments, this single-switch fallback routes all CPU-switched packets to Switch 0. Consequently, when the test runner ran on Switch 1, packets were routed to Switch 0 instead of Switch 1, leaving Switch 1's queues empty and causing the test to fail.

Solution

In QueuePerHostTestUtils.cpp, retrieve the correct SwitchID of the active port under test using the SwSwitch ScopeResolver. Explicitly pass this SwitchID to sendPacketSwitchedAsync(std::move(pkt), {switchId}), ensuring CPU-originated switched packets are targeted to the correct active hardware agent.

Test Plan

Tests running on NPU0 and NPU1 on the Leh800bcls platform both passed:

[root@localhost log]# tail -n 17 AgentHwTest_Npu[01]Cold_agent-t0-npu0-failed-queueperhost.cases/multi_switch_agen*.log
==> AgentHwTest_Npu0Cold_agent-t0-npu0-failed-queueperhost.cases/multi_switch_agent_hw_test_agent-t0-npu0-failed-queueperhost.cases_20260809_122708.log <==
Running all tests took 0:05:56.840817 between 2026-08-09 12:27:10.006518 and 2026-08-09 12:33:06.847335
[ PASSED ] cold_boot.AgentQueuePerHostL2Test.VerifyHostToQueueMappingClassID (25322 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyHostToQueueMappingClassIDsAfterResolveBlock (21342 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyHostToQueueMappingClassIDsWithResolve (26214 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyHostToQueueMappingClassIDsWithResolveBlock (22977 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyTtldCounter (29213 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.RemovePendingNeighborDoesNotCrash (21435 ms)
[ PASSED ] cold_boot.AgentQueuePerHostRouteTest.VerifyHostToQueueMappingClassID (43077 ms)
[ PASSED ] cold_boot.AgentQueuePerHostRouteTest.VerifyHostToQueueMappingClassIDBlock (34728 ms)
Summary:
   PASSED : 8
   FAILED : 0
   SKIPPED : 0
   TIMEOUT : 0

Test output stored at: hwtest_results_2026_Aug_09-12_33_06_PM.csv
++ set +x

==> AgentHwTest_Npu1Cold_agent-t0-npu0-failed-queueperhost.cases/multi_switch_agent_hw_test_agent-t0-npu0-failed-queueperhost.cases_20260809_121812.log <==
Running all tests took 0:06:01.242345 between 2026-08-09 12:18:13.878375 and 2026-08-09 12:24:15.120720
[ PASSED ] cold_boot.AgentQueuePerHostL2Test.VerifyHostToQueueMappingClassID (24117 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyHostToQueueMappingClassIDsAfterResolveBlock (21603 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyHostToQueueMappingClassIDsWithResolve (25446 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyHostToQueueMappingClassIDsWithResolveBlock (22178 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.VerifyTtldCounter (31942 ms)
[ PASSED ] cold_boot.AgentQueuePerHostTest.RemovePendingNeighborDoesNotCrash (22789 ms)
[ PASSED ] cold_boot.AgentQueuePerHostRouteTest.VerifyHostToQueueMappingClassID (43139 ms)
[ PASSED ] cold_boot.AgentQueuePerHostRouteTest.VerifyHostToQueueMappingClassIDBlock (34766 ms)
Summary:
   PASSED : 8
   FAILED : 0
   SKIPPED : 0
   TIMEOUT : 0

Test output stored at: hwtest_results_2026_Aug_09-12_24_15_PM.csv
++ set +x

@gang-tao
gang-tao requested a review from a team as a code owner August 11, 2026 01:34
@meta-cla meta-cla Bot added the CLA Signed label Aug 11, 2026
@meta-codesync

meta-codesync Bot commented Aug 11, 2026

Copy link
Copy Markdown
Contributor

@jchallag has imported this pull request. If you are a Meta employee, you can view this in D115625598.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant