[Bug]: Crashlooping nvidia-peermem-ctr: could not insert 'nvidia_peermem': Invalid argument

[nicdumz]

Describe the bug

Filing a bug after the conversation at #2844 thanks

On a GB300 on-prem cluster we're seeing constant crashloops of nvidia-peermem-ctr:

DRIVER_ARCH is aarch64
modprobe: ERROR: could not insert 'nvidia_peermem': Invalid argument
failed to load nvidia-peermem module

Recent dmesg -T | grep -i -E 'peermem|ib_core|peer_mem':

[Tue Sep  8 17:16:02 2026] nvidia_peermem: disagrees about version of symbol ib_register_peer_memory_client
[Tue Sep  8 17:16:02 2026] nvidia_peermem: Unknown symbol ib_register_peer_memory_client (err -22)
[Tue Sep  8 17:16:02 2026] nvidia_peermem: disagrees about version of symbol ib_unregister_peer_memory_client
[Tue Sep  8 17:16:02 2026] nvidia_peermem: Unknown symbol ib_unregister_peer_memory_client (err -22)
[Tue Sep  8 17:21:18 2026] nvidia_peermem: disagrees about version of symbol ib_register_peer_memory_client
[Tue Sep  8 17:21:18 2026] nvidia_peermem: Unknown symbol ib_register_peer_memory_client (err -22)
[Tue Sep  8 17:21:18 2026] nvidia_peermem: disagrees about version of symbol ib_unregister_peer_memory_client
[Tue Sep  8 17:21:18 2026] nvidia_peermem: Unknown symbol ib_unregister_peer_memory_client (err -22)

lsmod | grep -E 'nvidia|mlx5|ib_':

mlx5_dpll             196608  0
mlx5_vdpa             196608  0
ib_ipoib              262144  0
ib_cm                 327680  2 rdma_cm,ib_ipoib
ib_umad               262144  0
mlx5_fwctl            262144  0
fwctl                 262144  1 mlx5_fwctl
mlx5_ib               655360  0
ib_uverbs             327680  2 rdma_ucm,mlx5_ib
mlx5_core            3145728  3 mlx5_dpll,mlx5_fwctl,mlx5_ib
mlxfw                 262144  1 mlx5_core
mlxdevm               655360  1 mlx5_core
ib_core               720896  8 rdma_cm,ib_ipoib,iw_cm,ib_umad,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm
mlx_compat            196608  15 mlx5_dpll,rdma_cm,ib_ipoib,mlxdevm,mlx5_fwctl,iw_cm,ib_umad,mlx5_vdpa,fwctl,ib_core,rdma_ucm,ib_uverbs,mlx5_ib,ib_cm,mlx5_core
nvidia_modeset       2097152  0
nvidia_uvm           1966080  8
nvidia              15138816  55 nvidia_uvm,gdrdrv,nvidia_modeset
video                 262144  1 nvidia_modeset
ecc                   196608  1 nvidia
nvidia_cspmu          196608  0
arm_cspmu_module      262144  1 nvidia_cspmu
macsec                262144  1 mlx5_ib
psample               262144  1 mlx5_core
tls                   327680  1 mlx5_core
pci_hyperv_intf       196608  1 mlx5_core

To Reproduce

No clear way to get to those errors. Half of the nodes seem to load the module, half do not. No reasonable split of configs / kernels / ... so far explained this split.

Expected behavior

Not crash :-)

Environment (please provide the following information):

• GPU Operator Version: v26.7.0
• OS: Ubuntu24.04
• Kernel Version: 6.8.0-124-generic-64k or 6.8.0-136-generic-64k
• Container Runtime Version: containerd 2.2.3
• Kubernetes Distro and Version: k8s v1.35.3

DOCA: doca3.5.0-26.07-0.7.7.0-0-ubuntu24.04-arm64

@rajathagasthya LMK if a full debug bundle would be helpful

@lalitadithya FWIW: in this case nvsentinel is not actually picking up any errors, interestingly. The only "signal" from an administrator perspective is that pods are crashlooping continuously (which perhaps should be enough, but ... do consider if detection here would make sense thanks).

[nicdumz]

Tue Sep  8 17:46:04 2026
+-----------------------------------------------------------------------------------------+
| NVIDIA-SMI 595.71.05              Driver Version: 595.71.05      CUDA Version: 13.2     |
+-----------------------------------------+------------------------+----------------------+
| GPU  Name                 Persistence-M | Bus-Id          Disp.A | Volatile Uncorr. ECC |
| Fan  Temp   Perf          Pwr:Usage/Cap |           Memory-Usage | GPU-Util  Compute M. |
|                                         |                        |               MIG M. |
|=========================================+========================+======================|
|   0  NVIDIA GB300                   On  |   00000008:06:00.0 Off |                    0 |
| N/A   41C    P0            238W / 1400W |  267060MiB / 284208MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   1  NVIDIA GB300                   On  |   00000009:06:00.0 Off |                    0 |
| N/A   41C    P0            238W / 1400W |  267060MiB / 284208MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   2  NVIDIA GB300                   On  |   00000018:06:00.0 Off |                    0 |
| N/A   41C    P0            242W / 1400W |  267036MiB / 284208MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+
|   3  NVIDIA GB300                   On  |   00000019:06:00.0 Off |                    0 |
| N/A   41C    P0            242W / 1400W |  267060MiB / 284208MiB |      0%      Default |
|                                         |                        |             Disabled |
+-----------------------------------------+------------------------+----------------------+

[lalitadithya]

in this case nvsentinel is not actually picking up any errors, interestingly. The only "signal" from an administrator perspective is that pods are crashlooping continuously (which perhaps should be enough, but ... do consider if detection here would make sense thanks).

ah, yeah. I think this is one of those cases where we don't see an XID. But we do have an option to monitor and cordon nodes based on status of GPU operator pods -- https://docs.nvidia.com/nvsentinel/v1.22.0/components/monitoring-critical-operators/ but this is not something we enabled by default since some operators have their own automation to handle node bring up.

[nicdumz]

I would eventually want to write a longer writeup, eventually, but I have an approximative root cause:

• Some network-operator upgrade about a week ago, did change network drivers (this update did not go well and caused some network driver crashes due to undersized RAM limits, etc)
• I believe this left on the machine older gpu/peermem modules compiled against an older ABI
• Which is brittle and then fails as soon as modules are unloaded

Recovering is somewhat involved, we had a slow and gradual script going something like:

1. Opt out node from nvsentinel https://docs.nvidia.com/nvsentinel/runbooks/driver-upgrades/#1-disable-nvsentinel-management-on-target-nodes
2. cordon node, drain gpu workloads
3. delete gpu driver pod, which then recreates it, forces a driver recompilation against the right ABI
4. wait for all gpu-operator pods to come back healthy (dgcm in particular)
5. Then uncordon, opt into nvsentinel

Note that a naive delete of the gpu driver pod on an (undrained) affected node turns nvidia-peermem-ctr pod crashes into initContainer crashes where the init complains that it cannot unload modules

[tariq1890]

@nicdumz Have you looked into DMA-BUF yet? Given that you're on kernel 6.8 and using driver 595.91.07, DMA-BUF should easily be supported. It replaces nvidia_peermem and simplifies the GPUDirectRDMA significantly.

From the gpu-operator docs page:

To support GPUDirect RDMA, userspace CUDA APIs are required. The kernel mode support is provided by one of two approaches: DMA-BUF from the Linux kernel or the legacy nvidia-peermem kernel module. NVIDIA recommends using the DMA-BUF rather than using the nvidia-peermem kernel module from the GPU Driver.

[tariq1890]

If you'd like to give DMA-BUF a try, please ensure to set the .Values.driver.rdma.enabled to false.

(I understand this seems counter-intuitive, but at the time of implementing this feature, the we had chosen to externalise the enablement of the nvidia_peermem kernel module in the driver container as the Value.driver.rdma in our ClusterPolicy API)

[nicdumz]

@tariq1890 we saw that and were curious for sure (cc @jgehring), as the promise of simplified ops load is definitely interesting.

We're slightly confused about https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.2/gpu-operator-rdma.html#common-prerequisites though, in particular "GPU Driver: An Open Kernel module driver is required."

Anything you could share on what is needed here on the gpu-operator side? Must we also set driver.useOpenKernelModules=true?

[tariq1890]

We're slightly confused about https://docs.nvidia.com/datacenter/cloud-native/gpu-operator/24.9.2/gpu-operator-rdma.html#common-prerequisites though, in particular "GPU Driver: An Open Kernel module driver is required."

All the supported driver branches (R580 being the minimum supported branch at the time of this comment) use the open kernel module by default. We have been defaulting to the open modules for a while now. We can look into updating the documentation to better reflect this.

Anything you could share on what is needed here on the gpu-operator side? Must we also set driver.useOpenKernelModules=true?

No, you should just have open modules installed by default.