Invalidate status files before rescheduling operands - #263
Open
rajathagasthya wants to merge 1 commit into
Open
rajathagasthya wants to merge 1 commit into
rajathagasthya wants to merge 1 commit into
Conversation
The operator-validator and driver containers remove the readiness files under /run/nvidia/validations in their preStop hooks. Those hooks do not run when the containers are removed outside the kubelet, for example when a CRI-O minor version upgrade wipes every pod sandbox on the node. The files survive on the host tmpfs. When k8s-driver-manager then re-enables the operand labels, the operand pods are recreated together with the driver pod, before the driver is installed. A stale toolkit-ready lets an operand init container pass while the kernel module is unloaded, so the nvidia runtime creates the main container without the driver libraries. The MIG manager then fails NVML initialization with ERROR_LIBRARY_NOT_FOUND and leaves the node at mig.config.state=failed until the pod is deleted. Remove the files right after the GPU operator components have been evicted. The validators are confirmed deleted at that point, so nothing recreates the files until they are rescheduled against the replacement driver. The paths that do not evict, because no driver is loaded or because eviction itself failed, leave the files alone: a validator that is still running would not rewrite them. The set removed is what the two preStop hooks remove: the *-ready files written by the validators, plus the driver's .driver-ctr-ready and .driver-daemons-status. Other hidden markers, such as cc-manager's .cc-manager-ctr-ready, are left in place because k8s-driver-manager does not restart their owners. The sandbox validator waits on that marker before writing cc-manager-ready, and cc-manager can share a node with the vfio-manager DaemonSet, which also runs this init container. workload-type is not a readiness marker and is also left in place. In a normal driver rollout both preStop hooks have already run and the removal is a no-op. On the fast path that skips the module unload, removing .driver-ctr-ready means the validator waits for the new driver container's startup probe to rewrite it, which is the ordering a normal rollout already has. Removal failures are logged and ignored in order to not abort driver rollout. Signed-off-by: Rajath Agasthya <ragasthya@nvidia.com>
Coverage Report for CI Build 34541594029Coverage increased (+0.7%) to 11.806%Details
Uncovered Changes
Coverage RegressionsNo coverage regressions found. Coverage Stats
💛 - Coveralls |
rajathagasthya
marked this pull request as ready for review
September 10, 2026 23:36
cdesiniotis
reviewed
Sep 11, 2026
cdesiniotis
reviewed
Sep 11, 2026
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
3 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
The operator-validator and driver containers remove the readiness files under
/run/nvidia/validationsin their preStop hooks. Those hooks do not run when the containers are removed outside the kubelet, for example when a CRI-O minor version upgrade wipes every pod sandbox on the node. The files survive on the host tmpfs. When k8s-driver-manager then re-enables the operand labels, the operand pods are recreated together with the driver pod, before the driver is installed. A staletoolkit-readylets an operand init container pass while the kernel module is unloaded, so the nvidia runtime creates the main container without the driver libraries. The MIG manager then fails NVML initialization withERROR_LIBRARY_NOT_FOUNDand leaves the node atmig.config.state=faileduntil the pod is deleted.Remove the files right after the GPU operator components have been evicted. The validators are confirmed deleted at that point, so nothing recreates the files until they are rescheduled against the replacement driver. The paths that do not evict, because no driver is loaded or because eviction itself failed, leave the files alone: a validator that is still running would not rewrite them.
The set removed is what the two preStop hooks remove: the
*-readyfiles written by the validators, plus the driver's.driver-ctr-readyand.driver-daemons-status. Other hidden markers, such as cc-manager's.cc-manager-ctr-ready, are left in place because k8s-driver-manager does not restart their owners.workload-typeis not a readiness marker and is also left in place.In a normal driver rollout, both preStop hooks have already run and the removal is a no-op. On the fast path that skips the module unload, removing
.driver-ctr-readymeans the validator waits for the new driver container's startup probe to rewrite it, which is the ordering a normal rollout already has.Removal failures are logged and ignored in order to not abort driver rollout.
Related: NVIDIA/gpu-operator#2874 (PR NVIDIA/gpu-operator#2881)