Repository navigation
Question about MIG config persistent #343
Description
Hi team, I am setting MIG in a100 GPU but blocked at the very beginning which is the Enable MIG in the machine. could you please help look at this configuration problem when you have time? Appreciated!
here is the init nvidia-smi out put:
every time i config the gpus to the below and then reboot the config about mig is back to the init config.
nvidia-smi -a
Timestamp : Thu Nov 3 05:46:34 2022
Driver Version : 450.203.03
CUDA Version : 11.0
Attached GPUs : 8
GPU 00000000:07:00.0
Product Name : A100-SXM4-40GB
Product Brand : Tesla
Display Mode : Enabled
Display Active : Disabled
Persistence Mode : Enabled
MIG Mode
Current : Disabled
Pending : Disabled
Accounting Mode : Disabled
Accounting Mode Buffer Size : 4000
Driver Model
Current : N/A
Pending : N/A
Serial Number : 1562021001122
GPU UUID : GPU-36c9561a-02ef-5c4d-3733-f0a56c42dc45
Minor Number : 0
VBIOS Version : 92.00.45.00.03
MultiGPU Board : No
Board ID : 0x700
GPU Part Number : 692-2G506-0200-002
Inforom Version
Image Version : G506.0200.00.04
OEM Object : 2.0
ECC Object : 6.16
Power Management Object : N/A
GPU Operation Mode
Current : N/A
Pending : N/A
GPU Virtualization Mode
Virtualization Mode : None
Host VGPU Mode : N/A
IBMNPU
Relaxed Ordering Mode : N/A
PCI
Bus : 0x07
Device : 0x00
Domain : 0x0000
Device Id : 0x20B010DE
Bus Id : 00000000:07:00.0
Sub System Id : 0x134F10DE
GPU Link Info
PCIe Generation
Max : 4
Current : 4
Link Width
Max : 16x
Current : 16x
Bridge Chip
Type : N/A
Firmware : N/A
Replays Since Reset : 0
Replay Number Rollovers : 0
Tx Throughput : 0 KB/s
Rx Throughput : 0 KB/s
Fan Speed : N/A
Performance State : P0
...
Ecc Mode
Current : Enabled
Pending : Enabled
ECC Errors
Volatile
SRAM Correctable : 0
SRAM Uncorrectable : 0
DRAM Correctable : 0
DRAM Uncorrectable : 0
Aggregate
SRAM Correctable : 0
SRAM Uncorrectable : 0
DRAM Correctable : 0
DRAM Uncorrectable : 0
Retired Pages
Single Bit ECC : N/A
Double Bit ECC : N/A
Pending Page Blacklist : N/A
Remapped Rows
Correctable Error : 0
Uncorrectable Error : 0
Pending : No
Remapping Failure Occurred : No
Bank Remap Availability Histogram
Max : 640 bank(s)
High : 0 bank(s)
Partial : 0 bank(s)
Low : 0 bank(s)
None : 0 bank(s)
Temperature
GPU Current Temp : 31 C
GPU Shutdown Temp : 92 C
GPU Slowdown Temp : 89 C
GPU Max Operating Temp : 85 C
Memory Current Temp : 30 C
Memory Max Operating Temp : 95 C
Power Readings
Power Management : Supported
Power Draw : 57.13 W
Power Limit : 400.00 W
Default Power Limit : 400.00 W
Enforced Power Limit : 400.00 W
Min Power Limit : 100.00 W
Max Power Limit : 400.00 W
Clocks
Graphics : 210 MHz
SM : 210 MHz
Memory : 1215 MHz
Video : 585 MHz
Applications Clocks
Graphics : 1095 MHz
Memory : 1215 MHz
Default Applications Clocks
Graphics : 1095 MHz
Memory : 1215 MHz
Max Clocks
Graphics : 1410 MHz
SM : 1410 MHz
Memory : 1215 MHz
Video : 1290 MHz
Max Customer Boost Clocks
Graphics : 1410 MHz
Clock Policy
Auto Boost : N/A
Auto Boost Default : N/A
Processes : None
Activity
klueska commentedon Nov 3, 2022
Individual MIG configurations do not survive a node reboot (only the overall MIG mode setting of a GPU).
It is therefore recommended to configure MIG using the
nvidia-mig-partedtool and it's complementarysystemdservice to ensure MIG configurations are reapplied across a reboot.https://github.com/NVIDIA/mig-parted/tree/main/deployments/systemd
slow-zhang commentedon Nov 3, 2022
thank you @klueska
slow-zhang commentedon Nov 3, 2022
is the
only the overall MIG mode setting of a GPUmeans config bynvidia-smi mig 1?klueska commentedon Nov 3, 2022
Yes, that survives a node reboot, the individual configurations made after MIG mode is enabled do not.
slow-zhang commentedon Nov 3, 2022
this is cool! thank you Kevin, let me have a try
slow-zhang commentedon Nov 3, 2022
this is strange, i have run the following commands but the MIG for all GPU is still disabled after reboot
after reboot the status is
klueska commentedon Nov 3, 2022
What did nvidia-smi show after running
nvidia-smi -mig 1but before the rebootslow-zhang commentedon Nov 3, 2022
klueska commentedon Nov 4, 2022
Sorry, your response is garbled, so I don't quite understand. Can you paste the exact output of:
followed by the exact output of
before rebooting
klueska commentedon Nov 4, 2022
In any case, as I mentioend before, I would recommend against using
nvidia-smito do your MIG partitioning, and instead usemig-parted:https://github.com/NVIDIA/mig-parted
It handles all of the edge cases where trying to use
nvidia-smidirectly for MIG management breaks down.slow-zhang commentedon Nov 28, 2022
thank you @klueska , i no will continue to work with Enbaling MIG for A100, and here is the outputs
# nvidia-smi -mig 1 # for all gpu request to connect with running Fabric Manager instance failed with error 111 failed to connect to Fabric Manager instance. Warning: MIG mode is in pending enable state for GPU 00000000:07:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:07:00.0 Warning: MIG mode is in pending enable state for GPU 00000000:0F:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:0F:00.0 Warning: MIG mode is in pending enable state for GPU 00000000:47:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:47:00.0 Warning: MIG mode is in pending enable state for GPU 00000000:4E:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:4E:00.0 Warning: MIG mode is in pending enable state for GPU 00000000:87:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:87:00.0 Warning: MIG mode is in pending enable state for GPU 00000000:90:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:90:00.0 Warning: MIG mode is in pending enable state for GPU 00000000:B7:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:B7:00.0 Warning: MIG mode is in pending enable state for GPU 00000000:BD:00.0:Timeout Reboot the system or try nvidia-smi --gpu-reset to make MIG mode effective on GPU 00000000:BD:00.0the nvidia-smi output after i run
nvidia-smi -mig 1andnvidia-smi --gpu-resetafter reboot, the output of nvidia-smi
github-actions commentedon Feb 28, 2024
This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed.
riverind commentedon Jul 5, 2024
i want to make smi disable,
nvidia-smi -mig 0
however, Warning: MIG mode is in pending disable state for GPU
and nvidia-smi -L also have mig
riverind commentedon Jul 5, 2024
thanks for your help to how to enable smi
github-actions commentedon May 6, 2026
This issue is stale because it has been open 90 days with no activity. This issue will be closed in 30 days unless new comments are made or the stale label is removed. To skip these checks, apply the "lifecycle/frozen" label.
shivamerla commentedon Jun 1, 2026
@slow-zhang As @klueska commented, MIG mode is persistent across reboot and should stay enabled. please re-open this if you are still seeing the issue.