prov/efa: Introduce PPS enhancement interface#12225
Open
shijin-aws wants to merge 6 commits intoofiwg:mainfrom
Open
prov/efa: Introduce PPS enhancement interface#12225shijin-aws wants to merge 6 commits intoofiwg:mainfrom
shijin-aws wants to merge 6 commits intoofiwg:mainfrom
Conversation
Add enable_high_pps field to efa_env struct, gated by the undocumented FI_EFA_ENABLE_HIGH_PPS environment variable. This allows controlled rollout of PPS optimization before firmware deployment is complete. Signed-off-by: Shi Jin <sjina@amazon.com>
Define FI_EFA_WR_HIGH_PPS as a provider-specific operation flag (bit 60) in fi_ext_efa.h. Applications pass this flag in fi_writemsg() to hint the device to optimize for higher message rate on RDMA write operations. Signed-off-by: Shi Jin <sjina@amazon.com>
In efa_data_path_direct_post_write(), set the processing hint bit in the TX WQE metadata ctrl3 field when both efa_env.enable_high_pps and FI_EFA_WR_HIGH_PPS are set. Add efa_send_wr_set_processing_hint_mask() helper and the EFA_IO_TX_META_DESC_PROCESSING_HINT_MASK definition for the new ctrl3 field in efa_io_defs.h. Signed-off-by: Shi Jin <sjina@amazon.com>
Implement the rdma-core data path integration for the high PPS processing hint on RDMA write operations. During QP creation, set EFADV_WR_EX_WITH_PROCESSING_HINT in wr_flags to enable the WQE-level hint setter. In efa_ibv_post_write(), call efadv_wr_set_processing_hint() when both the feature flag and FI_EFA_WR_HIGH_PPS are set. Add configure checks for efadv_qp_from_ibv_qp_ex, wr_flags, and EFADV_WR_PROCESSING_HINT_BURST_PPS_SENSITIVE. All changes are guarded by HAVE_EFADV_WR_PROCESSING_HINT for backward compatibility with older rdma-core versions. Signed-off-by: Shi Jin <sjina@amazon.com>
Add fi_efa_rma_bw, an EFA-specific RMA bandwidth test that supports write and writedata operations with EFA-specific features such as the FI_EFA_WR_HIGH_PPS flag. Unlike fi_rma_bw, this test uses a nonblocking benchmark loop that interleaves posting and completion polling to keep the pipeline full, similar to the approach used by rdma-core/perftest. This avoids blocking at window boundaries and maximizes throughput. Signed-off-by: Shi Jin <sjina@amazon.com>
Use FI_EFA_ENABLE_HIGH_PPS=1 explicitly to enable the feature Disable shm because it is a efa-only test. Signed-off-by: Shi Jin <sjina@amazon.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
A series of commit that introduce an operation level interface to allow user enhance the packet per second (PPS) .