[Bug]: Deadlock with DRA on node reboot

[just1not2]

Describe the bug
There is a circular dependency when using DRA on node reboots when running uninstall in the GPU driver init containers:

• the DRA plugin requires GPU drivers (and/or the toolkit, which depends on it):

[dra plugin logs]
2026-07-20T10:42:10Z  /driver-root (/run/nvidia/driver on host): nvidia-smi: not found, libnvidia-ml.so.1: not found, current contents: [lib].

• during a reboot, the GPU driver container restarts and its k8s-driver-manager init container tries the uninstall that deletes all GPU consumers

[gpu-driver-container logs]
time=2026-07-20T10:41:53Z level=info msg=Identifying GPU pods to delete

• to delete a GPU consumer that uses DRA, it needs the DRA plugin to be registered (CDI/MIG cleanup); if the latter is not running, the pod stays in Terminating forever

[pod events]
  Warning  FailedPrepareDynamicResources  26s (x18 over 21m)  kubelet             Failed to prepare dynamic resources: prepare dynamic resources: DRA driver gpu.nvidia.com is not registered

Note that this is not a DRA issue, on kubelet side this behaviour is intentional: kubernetes/kubernetes#129402 (comment)

To Reproduce

• run any workload that request a GPU on a DRA-enabled GPU infrastructure
• reboot the associated node
• check that both gpu-driver pod is pending in its k8s-driver-manager phase

Expected behavior
The k8s-driver-manager should support this use case and not cause a deadlock

Environment (please provide the following information):

• k8s-driver-manager source (Commit SHA or image digest): main
• Host OS: all
• Kernel Version: all
• NVIDIA Driver Version: all
• Container Runtime Version: all
• CPU Architecture: all
• GPU Model(s): all

[tariq1890]

@just1not2 I see that you're deploying a k8s-driver-manager image built from the main branch. We don't really recommend this. Ideally, we wait till the next version of k8s-driver-manager is released and we use it through the gpu-operator.

In the upcoming version of gpu-operator, we plan on supporting the automatic eviction of the DRA driver kubelet plugin pod(if deployed via the standalone helm chart). See this PR for more details. With the third-party GPU client label feature and this change, I believe your use case will be addressed. You will, however, have to wait till the next version of GPU Operator is released where it all comes together.

[tariq1890]

Reopening after discussing the issue with @just1not2 on the Kubernetes slack. The k8s-driver-manager first takes down all of the gpu clients/operands and then attempts to evict GPU workloads. This does not seem to work for DRA-allocated pods as they get stuck in Terminating as the dra driver kubelet plugin was terminated prior to the pod evict.

[rajathagasthya]

Fixed in #217 (and #219).