[Bug]: Helm: gpu-feature-discovery DaemonSet missing volumeMounts for nvidiaDriverRoot + custom env needed

Description:
In a Pure CDI architecture (deviceListStrategy: cdi-cri) without the nvidia-container-runtime wrapper, infrastructure pods like GFD are "blind" to NVIDIA libraries because they do not trigger CDI injection.

The Problem:
While the Helm chart provides the nvidiaDriverRoot variable to define the host path and creates the Volumes entry, it only adds the volumeMounts to the nvidia-device-plugin container. The gpu-feature-discovery container is missing these mounts.

Without the mount and a corresponding LD_PRELOAD to the mount path, GFD fails to load libnvidia-ml.so.1 on any system not using the legacy runtime wrapper.

Proposed Fix:
Sync the GFD template with the Device Plugin template to include:

volumeMounts: Mount nvidiaDriverRoot to a neutral path (e.g., /driver-root).

env: Add an optional custom env list so user can specify LD_PRELOAD pointing to the library within that mount to preload needed library(ies).

@plevart

Thank you @plevart for raising an issue. In regards to your statement

In a Pure CDI architecture (deviceListStrategy: cdi-cri) without the nvidia-container-runtime wrapper, infrastructure pods like GFD are "blind" to NVIDIA libraries because they do not trigger CDI injection.

The is not entirely true. The GFD pod leverages a CDI device named management.nvidia.com/gpu=all CDI device. This CDI device is injected into the pod since it has the nvidia runtime class and has NVIDIA_VISIBLE_DEVICES (this holds true for GPU Operator versions v25.10.0 and above). This CDI device spec is generated by the nvidia-container-toolkit pod and you can see its contents in /var/run/cdi/management.nvidia.com-gpu.yaml on your host.

I would recommend looking at the following areas

Container runtime config file. Ensure that the config file has been overridden correctly and you can see the nvidia runtime handler
You see the management CDI device spec file generated in the /var/run/cdi/ host directory.

Both of the above require the toolkit daemonset pod to run successfully.

As I said, I'm not running the GPU operator. I'm not configuring special runtime (I'm using plain CRI-O). I just deployed node-feature-discovery Helm chart (standalone) with almost default values and nvidia-device-plugin Helm chart with the following values:

gfd:
  enabled: true
  env:
    # when deviceListStrategy=cdi-cri, there is no runc wrapper to inject the library
    - name: LD_PRELOAD
      value: /driver-root/usr/lib64/libnvidia-ml.so.1
nfd:
  enabled: false # we deploy nfd separately
affinity:
  nodeAffinity:
    requiredDuringSchedulingIgnoredDuringExecution:
      nodeSelectorTerms:
      - matchExpressions:
        # Use the EXACT label NFD generated on your node
        - key: feature.node.kubernetes.io/pci-0302_10de.present
          operator: In
          values: ["true"]
config:
  map:
    default: |-
      version: v1
      sharing:
        timeSlicing:
          resources:
          - name: nvidia.com/gpu
            replicas: 2
deviceDiscoveryStrategy: "nvml"
deviceListStrategy: cdi-cri
migStrategy: none
nvidiaDriverRoot: "/"
securityContext:
  privileged: true

This configuration and the proposed additions to Helm chart enables a working configuration.

Navigation Menu

Repository navigation

[Bug]: Helm: gpu-feature-discovery DaemonSet missing volumeMounts for nvidiaDriverRoot + custom env needed #1677

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Bug]: Helm: gpu-feature-discovery DaemonSet missing volumeMounts for nvidiaDriverRoot + custom env needed #1677

Description

Activity

tariq1890 commented on Jun 1, 2026

plevart commented on Jun 2, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions