prov/verbs: Automatic GPU-NIC affinity by gad-arbel · Pull Request #12203 · ofiwg/libfabric

gad-arbel · 2026-04-30T15:01:20Z

No description provided.

j-xiong

In general, looks good, see the nitpicks in line.

Please sign the commits (use git commit -s) otherwise the DCO check won't pass.

It would be nice to have a few sentences in the commit message to describe what the commit does. For the first commit, consider changing the title to prov/verbs: Add infrastructure for ..., and the commit message can be a little bit more detailed than the rest.

The appveyor CI failure is due to setenv and unsetenv not being available on Windows. Need to implement wrappers (using _putenv_s) in include/windows/osd.h.

I would prefer the new test being standalone instead of being part of the existing getinfo test. This way the new test can be configured to only run on machines that makes the test meaningful.

j-xiong · 2026-05-01T01:21:14Z

+	.nic_affinity_policy	= "none",
+	.affinity_device		= NULL,


Try to keep the = aligned like the code segment above.

I think we have different tab width. Hope that I fixed what you meant.

j-xiong · 2026-05-01T01:29:00Z

+	if (!ancestor) {
+		return VRB_PROXIMITY_UNKNOWN;
+	}
+
+	if (ancestor->type == HWLOC_OBJ_BRIDGE) {
+		return VRB_PROXIMITY_BRIDGE;
+	}
+
+	if (ancestor->type == HWLOC_OBJ_PACKAGE) {
+		return VRB_PROXIMITY_PACKAGE;
+	}


Convention here is not to use { and } around single line body for conditional statements.

j-xiong · 2026-05-01T01:29:13Z

+	if (nic_a->proximity < nic_b->proximity) {
+		return -1;
+	}
+	if (nic_a->proximity > nic_b->proximity) {
+		return 1;
+	}


j-xiong · 2026-05-01T01:30:09Z

+	for (cur = *info; cur; cur = cur->next) {
+		entries_count++;
+	}
+	if (entries_count == 0) {
+		return FI_SUCCESS;
+	}


Introduce a NIC affinity framework to enable NIC ordering in fi_getinfo() results based on GPU-NIC proximity. The 'none' policy preserves current behavior (no reordering) and serves as the default for backward compatibility. It establishes the infrastructure for policy-based handlers that will be added in subsequent commits. Signed-off-by: Gad Arbel <gad.arbel@intel.com>

j-xiong · 2026-05-04T21:46:33Z

The setenv/unsetenv wrappers for windows need to be duplicated in fabtests as well (fabtests/include/windows/osd.h).

Can we make a new test for the affinity settings and leave the original getinfo test unchanged?

gad-arbel · 2026-05-05T13:40:32Z

The setenv/unsetenv wrappers for windows need to be duplicated in fabtests as well (fabtests/include/windows/osd.h).

Can we make a new test for the affinity settings and leave the original getinfo test unchanged?

Thanks for the Windows tip - that was helpful.
Regarding the test separation, I've created the new fi_nic_affinity_test.c file and it compiles and runs successfully. I'm currently working through some cleanup issues in the new test file, but the functionality appears to be working correctly. I'll have it polished and ready for review shortly.

Implement the 'manual' policy that allows users to specify explicit GPU-to-NIC mappings via a configuration file. This policy reads a text-based config file containing PCI address to NIC name mappings and reorders fi_getinfo() results to prioritize the mapped NIC. Signed-off-by: Gad Arbel <gad.arbel@intel.com>

Implement the 'auto' policy that automatically detects and prioritizes NICs based on PCIe topology proximity to the target GPU device using hwloc. This policy calculates PCIe proximity levels between the GPU and each NIC, then reorders the fi_getinfo() list to place closer NICs first. Signed-off-by: Gad Arbel <gad.arbel@intel.com>

Add hwloc library detection and linking to the verbs provider build system. Hwloc is optional and used by the 'auto' policy to query PCIe topology for automatic GPU-NIC proximity detection. When hwloc is not available, the 'auto' policy falls back to 'none' behavior. Signed-off-by: Gad Arbel <gad.arbel@intel.com>

Add test suite for GPU-NIC affinity policies. Tests validate that the affinity feature does not break existing functionality by checking failure states and verifying that all original fi_info entries are preserved in the result list. Tests do not verify reordering correctness - the goal is to ensure that in the worst case, users receive a different permutation of results rather than missing or corrupted data. Signed-off-by: Gad Arbel <gad.arbel@intel.com>

j-xiong · 2026-05-09T00:36:47Z

- name:   fi_nic_affinity_test -p "tcp"
  timestamp: 20260507-201418+0000
  result: Fail
  time:   1
  server_cmd:  fi_nic_affinity_test -p "tcp"
  server_stdout: |
    fi_nic_affinity_test: invalid option -- 'p'
    Usage: fi_nic_affinity_test [options]
    Unit tests for GPU-NIC affinity
    Options:
      -h	Display this help message

gad-arbel closed this Apr 30, 2026

gad-arbel deleted the automatic_gpu_nic_affinity branch April 30, 2026 16:01

gad-arbel restored the automatic_gpu_nic_affinity branch April 30, 2026 16:02

gad-arbel reopened this Apr 30, 2026

j-xiong reviewed May 1, 2026

View reviewed changes

gad-arbel force-pushed the automatic_gpu_nic_affinity branch from d48ba04 to c66e17f Compare May 4, 2026 08:11

gad-arbel force-pushed the automatic_gpu_nic_affinity branch 3 times, most recently from c3fb7b3 to f936ae3 Compare May 4, 2026 15:01

gad-arbel force-pushed the automatic_gpu_nic_affinity branch from f936ae3 to 887cb0d Compare May 5, 2026 13:36

gad-arbel force-pushed the automatic_gpu_nic_affinity branch 2 times, most recently from 581dcc6 to 84fbcdb Compare May 6, 2026 10:03

gad-arbel added 4 commits May 7, 2026 02:01

gad-arbel force-pushed the automatic_gpu_nic_affinity branch from 84fbcdb to 9e2efa6 Compare May 7, 2026 09:11

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

prov/verbs: Automatic GPU-NIC affinity#12203

prov/verbs: Automatic GPU-NIC affinity#12203
gad-arbel wants to merge 5 commits intoofiwg:mainfrom
gad-arbel:automatic_gpu_nic_affinity

gad-arbel commented Apr 30, 2026

Uh oh!

j-xiong left a comment •

edited

Loading

Uh oh!

j-xiong May 1, 2026

Uh oh!

gad-arbel May 4, 2026

Uh oh!

j-xiong May 1, 2026

Uh oh!

gad-arbel May 4, 2026

Uh oh!

j-xiong May 1, 2026

Uh oh!

gad-arbel May 4, 2026

Uh oh!

j-xiong May 1, 2026

Uh oh!

gad-arbel May 4, 2026

Uh oh!

j-xiong commented May 4, 2026

Uh oh!

gad-arbel commented May 5, 2026 •

edited

Loading

Uh oh!

j-xiong commented May 9, 2026 •

edited

Loading

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

Conversation

gad-arbel commented Apr 30, 2026

Uh oh!

j-xiong left a comment • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

Choose a reason for hiding this comment

Uh oh!

j-xiong commented May 4, 2026

Uh oh!

gad-arbel commented May 5, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

j-xiong commented May 9, 2026 • edited Loading Uh oh! There was an error while loading. Please reload this page.

Uh oh!

Uh oh!

Reviewers

Assignees

Labels

Projects

Milestone

Development

Uh oh!

2 participants

j-xiong left a comment •

edited

Loading

gad-arbel commented May 5, 2026 •

edited

Loading

j-xiong commented May 9, 2026 •

edited

Loading