Ignore faulty GPUs when listing GPUs - #66
Conversation
On servers with some of the GPUs broken, we want to continue using the other GPUs. However, when computing the topology, it fails on GPU Lost state. Signed-off-by: g.beausire <g.beausire@criteo.com>
|
cc @rajatchopra, sorry for the ping but you are the last one that merged something on the repo, do you mind taking a look? |
I reread the goal of the PR, operator already knows about the broken GPUs. OS my comments are not valid in this case
| for i := 0; i < nvml.NVLINK_MAX_LINKS; i++ { | ||
| state, ret := dev.GetNvLinkState(i) | ||
| if ret == nvml.ERROR_NOT_SUPPORTED || ret == nvml.ERROR_INVALID_ARGUMENT { | ||
| if ret == nvml.ERROR_NOT_SUPPORTED || ret == nvml.ERROR_INVALID_ARGUMENT || ret == nvml.ERROR_GPU_IS_LOST { |
There was a problem hiding this comment.
Could we add a higher-level test demonstrating the expected operator flow: topology discovery tolerates ERROR_GPU_IS_LOST, the known-broken GPU is excluded from the allocation set, and BestEffort successfully evaluates the remaining healthy GPUs? This would document the caller-side filtering assumption and why incomplete links involving the broken GPU are safe.
There was a problem hiding this comment.
Please see the nvml.h documentation for the C symbol nvmlDeviceGetNvLinkState
/**
* Retrieves the state of the device's NvLink for the link specified
*
* For Pascal &tm; or newer fully supported devices.
*
* @param device The identifier of the target device
* @param link Specifies the NvLink link to be queried
* @param isActive \a nvmlEnableState_t where NVML_FEATURE_ENABLED indicates that
* the link is active and NVML_FEATURE_DISABLED indicates it
* is inactive
*
* @return
* - \ref NVML_SUCCESS if \a isActive has been set
* - \ref NVML_ERROR_UNINITIALIZED if the library has not been successfully initialized
* - \ref NVML_ERROR_INVALID_ARGUMENT if \a device or \a link is invalid or \a isActive is NULL
* - \ref NVML_ERROR_NOT_SUPPORTED if the device doesn't support this feature
* - \ref NVML_ERROR_UNKNOWN on any unexpected error
*/
nvmlReturn_t DECLDIR nvmlDeviceGetNvLinkState(nvmlDevice_t device, unsigned int link, nvmlEnableState_t *isActive);
NVML_ERROR_GPU_IS_LOST is not in the list of possible errors returned.
There was a problem hiding this comment.
Thanks I completely missed this.
4 participants
We have the case where a server has some of the GPUs in a broken state, however we want to continue using the other GPUs. While the nvidia operator is aware that these GPUs are out, it fails when computing GPU topology.
The goal of this PR is to skip broken GPUs when computing the topology