Repository navigation
GPU ressource is required when nvml plugin is activated #37
Description
Describe the bug
In the last version of Helm chart (0.4.5), it is not possible to deploy alumet with the nvml plugin.
If you want to add the nvml plugin you have to modify daemonset for alumet client by requested gpu resources:
'imagePullPolicy: IfNotPresent
resources:
{{- toYaml .Values.resources | nindent 12 }}
limits:
nvidia.com/gpu: 1
volumeMounts:
- name: sysfs
mountPath: /sys'
But with this update, you request gpu resources on every nodes, to avoid deployement of alumet-client on nodes that do not have gpu resources, you can use tolerations. Nevertheless, you can not deployed alumet-client on all nodes of your k8s cluster if some nodes have gpu resources and other none.
How to solve this issue ? can we test the existing node resources at deployment step and activate/deactivate nvml plugin ? create 2 daemonset (one with gpu and one without) ?
The other problem is the name of library libnvidia-ml.so:
When GPU resources are requested in daemonset, the library libnvidia-ml.so is not present with this name in the pod. The libray name has a suffix with the nvidia driver version. For example on 2 different kubernetes I got the names:
libnvidia-ml.so.560.35.03
libnvidia-ml.so.570.148.08
It is possible in the daemonset to create a link or make a copy of this library:
'containers:
- name: alumet-relay-client
image: {{ $global.image.registry }}/alumet-agent:{{ .Values.image.version | default (printf "%s-%.f_%s" .Chart.AppVersion .Values.image.revision .Values.image.osVersion) }}
lifecycle:
postStart:
exec:
# command: ["/usr/bin/find", "/", "-name", "libnvidia*.so*"]
command: ["cp", "/usr/lib/x86_64-linux-gnu/libnvidia-ml.so.560.35.03", "/usr/lib/x86_64-linux-gnu/libnvidia-ml.so"]
args: ["--config", "/etc/alumet/alumet-agent.toml"]'
Environment information
OS: kubernetes
Alumet version²: helm chart 0.4.5 but all older version are concerned
Enabled plugins: nvml
To Reproduce
Steps to reproduce the behavior:
- deploy alumet using the helm install comand with nvm plugin (--set alumet-relay-client.plugins.nvml.enable="true")
- alumet-client crash at startup
Expected behavior
no crash in alumet client.
The crash is due to the fact that the library libnvidia-ml.so is not present in the pod becaus no GPU resources was asked in the daemonset.
Logs / Output
Defaulted container "alumet-relay-client" out of: alumet-relay-client, wait-for-alumet-server (init)
[2026-02-25T15:22:35Z INFO alumet_agent] Starting Alumet agent 'alumet-agent' v0.9.3 (2026-02-11)
[2026-02-25T15:22:35Z INFO alumet::agent::builder] Initializing the plugins...
[2026-02-25T15:22:35Z INFO alumet::agent::builder] 4 plugins initialized.
[2026-02-25T15:22:35Z INFO alumet::agent::builder] Starting the plugins...
[2026-02-25T15:22:35Z INFO plugin_k8s] List of pods refreshed.
Error: startup failure
Caused by:
0: plugin failed to start: nvml v0.5.0
1: NVML initialization failed, please check your driver (do you have a dekstop/server NVidia GPU?)
2: a libloading error occurred: libnvidia-ml.so: cannot open shared object file: No such file or directory
3: libnvidia-ml.so: cannot open shared object file: No such file or directory
Activity
TheElectronWill commentedon Mar 5, 2026
In my opinion, something is wrong with the setup if
libnvidia-ml.sois not found. Usually, it's a symlink to the versioned .so (not a copy). But even without the symlink, I believe that it should work (lookup by name).Could you provide more information about your environment? In particular:
LD_LIBRARY_PATHenvironment variableezychlagre commentedon Mar 9, 2026
for information on the host (OS unbuntu 22.04.5) where k8s is installed the linux package installed for nvidia are:
Note that the variable
LD_LIBRARY_PATHis not set.TheElectronWill commentedon Apr 29, 2026
Option 1
Add a gpu limit (see limits/requests for gpu) so that the gpu-operator for k8s injects the library in the pod's container
Problem: this allocates 1 gpu to the Alumet client, which we do not want.
Option 2
Mount the library in the container.
Problem: the libc of the container is not necessarily ABI-compatible with the libc of the host, which will cause issues with the loading of the library
Option 3 (preferred)
Install nvml in the container, when we need to use it. Since it's only loaded on demand, it should even work without a gpu (just don't enable the plugin).
Problem: increase the size of the container (but not that much IMO)
titouanj commentedon May 11, 2026
Only partially fixed in #47
We need to consider option 3 or another one