fabtests: Add new fi_mr_abort fabtest by a-szegel · Pull Request #12113 · ofiwg/libfabric

a-szegel · 2026-04-03T23:50:14Z

The man pages currently say:

Outstanding operations posted to the endpoint when fi_close is called will be discarded. Discarded operations will silently be dropped, with no completions reported. Memory buffers that the user associated with the endpoint by calling libfabric's transmission interfaces must not be released, deregistered, or reused until either a completion is generated or fi_close on the respective endpoint returns. Additionally, a provider may discard previously completed operations from the associated completion queue(s). The behavior to discard completed operations is provider specific.

but the EFA provider would like to support closing MR's when data transfer operations are in-flight.

This fabtest has some efa-specific provider error codes in it... but it should still be usable by every provider. I can move it to prov/efa if it is not generic enough to be shared across providers.

a-szegel · 2026-04-15T06:53:06Z

bot:aws:retest ... testing to see if AWS PR CI is working.

a-szegel · 2026-04-16T20:12:23Z

bot:aws:retest

talavr-amazon · 2026-04-16T20:36:48Z

+ * Test aborting in-flight RMA operations by closing MRs.
+ *
+ * Test modes:
+ *   - Initiator close: client posts RMA ops, client closes its local MRs


Would have been easier to review if the different tests are in different commits

talavr-amazon · 2026-04-16T20:38:49Z

+
+#include "shared.h"
+
+enum cancel_mode {


nit: close order?

talavr-amazon · 2026-04-16T20:39:30Z

+ */
+struct op_ctx {
+	struct fi_context2 context;
+	int mr_idx;	/* which mr_slot this op belongs to */


nit: unsigned here and for completed?

talavr-amazon · 2026-04-16T20:39:53Z

+	struct fid_mr *mr;
+	void *desc;
+	uint64_t key;
+	int posted;	/* number of ops posted using this MR */


nit: unsigned?

talavr-amazon · 2026-04-16T20:40:42Z

+static struct mr_slot *slots;
+static struct op_ctx *op_arr;
+static int *cancel_order;
+static int num_mrs = 8192;


nit: same and below

talavr-amazon · 2026-04-16T20:41:19Z

+/* Remote side MR info */
+static struct fi_rma_iov *remote_arr;
+
+#define MR_ABORT_KEY_BASE 0x1000


nit: static variables as well?

talavr-amazon · 2026-04-16T20:41:36Z

 	pytest/efa/conftest.py \
 	pytest/efa/efa_common.py \
 	pytest/efa/test_flood_peer.py \
+	pytest/efa/test_mr_abort.py \


should this be part of the next commit?

talavr-amazon · 2026-04-16T20:42:47Z

+
+	srand(time(NULL));
+
+	while ((op = getopt(argc, argv,


nit: a function for parsing options? or at least for the help section?

talavr-amazon · 2026-04-16T20:43:38Z

+	case TEST_ABORT:
+		for (i = 0; i < opts.iterations; i++) {
+			ret = register_mrs(remote_access_for_op());
+			if (ret)


log failures?

sunkuamzn · 2026-04-23T21:05:53Z

+import pytest
+from common import ClientServerTest
+
+failing_test_skip = pytest.mark.skip(reason="efa/efa-direct fabrics do not currently pass fi_mr_abort testing")


nit: failing_test_skip is generic but the message is specific to fi_mr_abort test

sunkuamzn · 2026-04-23T21:09:20Z

+#define MR_ABORT_KEY_BASE 0x1000
+#define CQ_TIMEOUT_MS 30000
+
+static uint64_t local_access_for_op(void)


Can you reuse ft_info_to_mr_access?

sunkuamzn · 2026-04-23T21:11:30Z

+/*
+ * Test 1: Fill-and-abort (RMA)
+ *
+ * Initiator mode: fill TX queue with RMA ops, then close local MRs.


There are two sets of terms: initiator, target and client, server. Test will be easier to understand if there's only one set.

sunkuamzn · 2026-04-23T22:29:02Z

+ * In target-close mode, the server actively closes MRs when
+ * signaled by the client.
+ */
+static int run_server(void)


A lot of code is duplicated between run_client and run_server. Switching on the test_mode first and then calling the relevant function for the server/client is better.

sunkuamzn · 2026-04-23T23:11:47Z

+			       remote_fi_addr, &o->context);
+}
+
+static void shuffle(int *arr, int n)


consider making this a utility function in common code

Add a new functional test that verifies behavior when memory regions are closed while operations are in-flight. This tests the abort-via- MR-close approach for canceling RDMA operations. Test modes (-T): abort - Fill TX queue, close MRs, verify all ops get completions partial - 2 MRs on same buffer, close 1, verify the other survives send - Same as abort but with fi_send/fi_recv tagged - Same as abort but with fi_tsend/fi_trecv Supports both initiator-side close (-R initiator) where the sender closes its local MRs, and target-side close (-R target) where the receiver closes its remote MRs via write-with-imm signaling. Configurable cancel order (-C reverse|random), operations per MR (-N), MR slot count (-W), and transfer size (-S). Server teardown order (-D ep_first|mr_first) controls whether the endpoint is closed before or after MR deregistration. EFA curently requires ep_first for send/tagged tests since posted RX buffer MRs cannot be deregistered while the endpoint is still open. The reuse check (write + read round-trip after abort) only runs for RMA test modes since send/tagged do not request FI_RMA capability. Signed-off-by: Seth Zegelstein <szegel@amazon.com>

Add pytest/efa/test_mr_abort.py with functional tests covering MR close during in-flight operations across multiple dimensions: test_mr_abort: RMA write/read/writedata × reverse/random cancel order × initiator/target close × 1 or 4 ops per MR test_mr_abort_partial: 2 MRs on same buffer, close 1 test_mr_abort_send: fi_send at 4KB/64KB/256KB × reverse/random × initiator/target × 1 or 4 ops per MR test_mr_abort_tagged: fi_tsend at 4KB/64KB/256KB × reverse/random × initiator/target × 1 or 4 ops per MR Skips efa-direct+tagged (unsupported), efa-direct sends >8KB (max send size), and target-close for send/tagged (EFA does not currently support canceling posted RX buffers). All tests except test_mr_abort_partial are marked skip pending EFA firmware/provider fixes for MR abort support. Parametrized over fabric (efa, efa-direct) via existing conftest. Signed-off-by: Seth Zegelstein <szegel@amazon.com>

Add printf calls between posting, MR close, and CQ drain in the writedata abort path. These introduce a small delay that gives the EFA device time to begin processing writedata WQEs before MR deregistration, exposing a firmware bug where the flush of partially-processed writedata WQEs triggers a QP internal error (prov_errno=2). Without these prints the client races through post/close so fast that all ops are canceled before reaching the device, masking the bug. The prints will be removed once the firmware issue is resolved. Signed-off-by: Seth Zegelstein <szegel@amazon.com>

a-szegel requested review from a team and j-xiong April 3, 2026 23:50

a-szegel force-pushed the add-mr-abort-fabtest branch 2 times, most recently from 29260c4 to 7b6aadf Compare April 7, 2026 16:57

a-szegel force-pushed the add-mr-abort-fabtest branch from 7b6aadf to d89fc4b Compare April 15, 2026 13:03

talavr-amazon reviewed Apr 16, 2026

View reviewed changes

sunkuamzn reviewed Apr 23, 2026

View reviewed changes

a-szegel added 3 commits May 8, 2026 23:02

a-szegel force-pushed the add-mr-abort-fabtest branch from d89fc4b to 79b59f9 Compare May 8, 2026 23:04


		#include "shared.h"

		enum cancel_mode {

Conversation

a-szegel commented Apr 3, 2026

Uh oh!

a-szegel commented Apr 15, 2026

Uh oh!

a-szegel commented Apr 16, 2026

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

3 participants