[Feature]: Align the default ignored XID policy with NVIDIA XID Catalog `RESTART_APP` and `IGNORE` actions

[ymygogogo]

Align the default ignored XID policy with NVIDIA XID Catalog RESTART_APP and IGNORE actions

Summary

The default XID health policy in k8s-device-plugin appears inconsistent with the recovery guidance in the NVIDIA XID Catalog.

In particular, XIDs whose Catalog immediate action is RESTART_APP or IGNORE are not consistently included in the device plugin's default ignored XID list:

https://github.com/NVIDIA/k8s-device-plugin/blob/main/internal/rm/health.go

XID 94 and XID 109 are concrete examples of the inconsistency:

XID	NVIDIA XID Catalog immediate action	Current device-plugin behavior
94	RESTART_APP	Not ignored; the GPU is marked unhealthy
109	RESET_GPU	Ignored; the GPU remains healthy and allocatable

XID 94 has a Catalog action of RESTART_APP, but it is not ignored by the device plugin. XID 109 has a Catalog action of RESET_GPU, but it is currently ignored.

I would like to propose that XIDs classified by the Catalog with an immediate action of RESTART_APP or IGNORE be ignored by default by the device plugin. Other XIDs would continue through the existing device health path unless explicitly overridden by the operator.

Relevant XID definitions

XID 94

XID 94 is described as:

• ROBUST_CHANNEL_CONTAINED_ERROR
• the error is contained to a single application;
• other applications are not affected;
• immediate action: RESTART_APP;
• the GPU may be reset later when convenient.

Because the failure is contained to one application, marking the entire GPU unhealthy immediately removes GPU capacity from Kubernetes even though the device may still be usable by other workloads.

This behavior has also been discussed in:

#1014

XID 109

XID 109 is described as:

• ROBUST_CHANNEL_CTXSW_TIMEOUT_ERROR
• context-switch timeout;
• immediate action: RESET_GPU;
• investigatory action: CONTACT_SUPPORT.

However, XID 109 is currently part of the device plugin's default ignored XID list, so it does not mark the GPU unhealthy.

XID 109 was added to the ignored list by:

#1171

The PR explains that XID 109 may also be caused by an application error. However, this appears different from the current NVIDIA XID Catalog recommendation, which classifies it as requiring a GPU reset.

NVIDIA XID Catalog reference:

https://docs.nvidia.com/deploy/xid-errors/analyzing-xid-catalog.html

Observed impact

With the current default policy:

1. An XID whose recommended action is only RESTART_APP can cause the device plugin to mark the entire GPU unhealthy.
2. An XID whose recommended action is IGNORE may also remove GPU capacity if it is absent from the hardcoded ignored list.
3. The GPU does not automatically become healthy again when the affected application exits or restarts.
4. Restoring the GPU may require restarting the device-plugin pod or another explicit recovery operation.
5. XID 109, although associated with RESET_GPU guidance, is ignored and leaves the GPU allocatable.

This creates an operational inconsistency:

• application-contained or explicitly ignored errors can remove the GPU from scheduling;
• a GPU-reset-level XID is ignored by default.

Expected behavior

I would expect the default policy to distinguish between application-contained errors and GPU/device-level errors.

For example:

Failure scope	Expected device-plugin behavior
Catalog immediate action is RESTART_APP	Do not mark the whole GPU unhealthy by default
Catalog immediate action is IGNORE	Do not mark the whole GPU unhealthy by default
GPU/device-level error such as XID 109	Enter the device health path unless explicitly disabled by the operator

This issue is not requesting that k8s-device-plugin perform application restarts or GPU resets.

The question is only whether the device should be marked unhealthy and removed from Kubernetes allocatable capacity.

Questions

1. Is the current difference between XID 94 and XID 109 intentional?
2. Is there additional guidance, outside the NVIDIA XID Catalog, that explains why XID 109 is ignored while XID 94 is treated as a critical device error?
3. Would it be acceptable to add the XIDs whose Catalog immediate action is RESTART_APP or IGNORE to the default ignored XID list?

Proposed initial change

As an initial change, I propose updating the device plugin's static default ignored XID list to include the XIDs whose immediate action in the current NVIDIA XID Catalog is either:

• RESTART_APP; or
• IGNORE.

This would be a static, reviewed list in the source code. It would not introduce a runtime dependency on the Catalog spreadsheet.

The intended behavior would be:

• application-contained errors do not immediately mark the entire GPU unhealthy;
• errors explicitly classified as IGNORE do not affect allocatable GPU capacity;
• the affected application can restart or exit when the Catalog action is RESTART_APP;
• operators can still explicitly enable health handling for these XIDs if their environments require stricter isolation.

The existing XID 109 policy can be discussed separately. This avoids combining two policy changes into one PR.

Willingness to contribute

If the maintainers agree with this direction, I would be happy to submit a PR updating the device plugin's default ignored XID list for both Catalog action categories: RESTART_APP and IGNORE.

I can keep the PR limited to:

• adding the current Catalog XIDs categorized as RESTART_APP or IGNORE to the static default ignored list;
• adding or updating the relevant unit tests;
• documenting that the list is derived from these two Catalog action categories;
• preserving the existing operator override mechanisms through DP_DISABLE_HEALTHCHECKS and DP_ENABLE_HEALTHCHECKS.

Any change to the existing XID 109 policy can be handled in a separate discussion or PR.

Related issues and proposals

• XID 94 causes GPU resources to remain unhealthy:
  GPU resources are not recovered even XID error is resolved #1014
• XID 109 added to the default ignored list:
  Ignore XID error 109 #1171
• Proposal for more configurable XID health-check behavior:
  Add advanced health check configuration to config file #1340