Enabling MIG mode on a VM with a GPU in Compute Cloud

MIG (Multi-Instance GPU) is an NVIDIA technology that allows splitting a single graphics accelerator into several independent GPU instances. Each GPU instance gets its dedicated computing blocks, L2 cache, and HBM2 memory. Instances are isolated from each other at the hardware level, so a load on one instance does not affect the performance of the others.

For a detailed description of this technology and its splitting schema, see the NVIDIA MIG User Guide page.

With MIG, you can:

• Run multiple independent ML or inference jobs on a single VM without them affecting each other in terms of performance and memory.
• Use your GPU in a more efficient way because no individual job can load the entire GPU.
• Guarantee the allocation of GPU resources among several users or processes.

MIG is supported for the following platforms:

• AMD EPYC™ with NVIDIA® Ampere® A100 (gpu-standard-v3).
• Gen2 (gpu-standard-v3i).
• GPU PLATFORM V4 (gpu-standard-v4).

To enable MIG mode on a VM with a GPU:

1. Create a VM:

   1. Create a VM with a GPU based on one of the following platforms:

      • AMD EPYC™ with NVIDIA® Ampere® A100 (gpu-standard-v3).
      • Gen2 (gpu-standard-v3i).
      • GPU PLATFORM V4 (gpu-standard-v4).

      Tip

      Make sure the cloud has enough quotas for the total number of GPUs of the correct platform, RAM, number of vCPUs, and SSD size to create a VM. Use Yandex Cloud Quota Manager to run this check.

   2. Connect to the VM over SSH.

   3. Install NVIDIA drivers version 535.

2. Disable NVLink on the GPUs.

   1. Create a file named /etc/modprobe.d/nonvlink.conf with the following contents:

      options nvidia NVreg_NvLinkDisable=1
      

   2. Restart the NVIDIA driver:

      Warning

      From now on, run all commands as the root user or use sudo.

      sudo systemctl stop nvidia-persistenced
      sudo modprobe -r nvidia_uvm nvidia_drm nvidia_modeset nvidia
      sudo modprobe nvidia
      sudo systemctl start nvidia-persistenced
      nvidia-smi
      

   3. Make sure NVLink is off:

      grep NvLink /proc/driver/nvidia/params
      

      Expected result:

      NvLinkDisable: 1
      

3. Reset the status of the GPUs:

   sudo nvidia-smi -r
   

   Result:

   GPU 00000000:8B:00.0 was successfully reset.
   GPU 00000000:8C:00.0 was successfully reset.
   All done.
   

   For more on possible errors, see Troubleshooting.

4. Enable MIG mode for all the VM GPUs:

   sudo nvidia-smi -mig 1
   

   Result:

   Enabled MIG Mode for GPU 00000000:8B:00.0
   Enabled MIG Mode for GPU 00000000:8C:00.0
   All done.
   

   Note

   If the output alerts you with Warning: persistence mode is disabled on device ..., start nvidia-persistenced:

   sudo systemctl start nvidia-persistenced
   

5. Split the GPUs into GPU instances. The example below creates two GPU instances with the 1g.10gb profile on each GPU:

   sudo nvidia-smi mig -cgi 19,1g.10gb -C
   

   Result:

   Successfully created GPU instance ID 13 on GPU  0 using profile MIG 1g.10gb (ID 19)
   Successfully created compute instance ID  0 on GPU  0 GPU instance ID 13 using profile MIG 1g.10gb (ID  0)
   Successfully created GPU instance ID 11 on GPU  0 using profile MIG 1g.10gb (ID 19)
   Successfully created compute instance ID  0 on GPU  0 GPU instance ID 11 using profile MIG 1g.10gb (ID  0)
   ...
   

   For a list of available splitting profiles for NVIDIA® A100, see NVIDIA MIG User Guide. Alternatively, you can use the nvidia-smi mig -lgip command.

6. Check your new GPU instances:

   nvidia-smi -L
   

   Result:

   GPU 0: NVIDIA A100-SXM4-80GB (UUID: GPU-9c73eb7b-8d6d-6c0e-dc13-97873e2ede40)
     MIG 1g.10gb     Device  0: (UUID: MIG-a686709d-6dc2-542a-8851-0d2508448c10)
     MIG 1g.10gb     Device  1: (UUID: MIG-53864714-0a66-5df2-9007-9268e30d9a72)
   GPU 1: NVIDIA A100-SXM4-80GB (UUID: GPU-c92e2d5d-4a69-f1cd-4df0-305f8606705a)
     MIG 1g.10gb     Device  0: (UUID: MIG-d5a46a2e-1d77-575d-99e3-1f96c8555a2e)
     MIG 1g.10gb     Device  1: (UUID: MIG-b8479d6e-d08f-5a92-9a24-0946db72b2a0)
   

Your GPU instances are ready to use. To run computations on a specific instance, provide its UUID into the CUDA_VISIBLE_DEVICES environment variable.

TroubleshootingTroubleshooting

"Not Supported" GPU reset error"Not Supported" GPU reset error

You may get the following error when running sudo nvidia-smi -r:

The following GPUs could not be reset:
  GPU 00000000:8B:00.0: Not Supported
  GPU 00000000:8C:00.0: Not Supported

This error indicates an active NVLink on your GPUs. Disable it and try resetting them again.

"In use by another client" GPU reset error"In use by another client" GPU reset error

You may get the following error when running sudo nvidia-smi -r:

The following GPUs could not be reset:
  GPU 00000000:8B:00.0: In use by another client
  GPU 00000000:8C:00.0: In use by another client

2 devices are currently being used by one or more other processes (e.g., Fabric Manager, CUDA application, graphics application such as an X server, or a monitoring application such as another instance of nvidia-smi). Please first kill all processes using these devices and all compute applications running in the system.

This means your GPUs are currently in use by another process, e.g., the nvidia_drm module with enabled modeset=1, or a user process holding a lock on the device.

To fix the issue:

1. Check if nvidia_drm is running with modeset=1:

   cat /sys/module/nvidia_drm/parameters/modeset
   

   If the output returns Y, disable modeset. Do to this, create a file named /etc/modprobe.d/nomodeset.conf with the following contents:

   options nvidia-drm modeset=0
   

2. Update initramfs (Initial RAM File System):

   sudo update-initramfs -u
   

3. Identify the processes holding a lock on the device:

   sudo ls -l /proc/*/fd/* | grep /dev/nvidia
   
   Example of the command output

   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/3 -> /dev/nvidiactl
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/4 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/5 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/6 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/7 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/13 -> /dev/nvidiactl
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/17 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/18 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/19 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/20 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/24 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/25 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/27 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/28 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/29 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/3 -> /dev/nvidiactl
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/39 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/43 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/8 -> /dev/nvidia-modeset
   

4. Terminate these processes by running sudo kill -9 <process_PID>.

5. Try resetting your GPUs again:

   sudo nvidia-smi -r
   

sudo grep -R "modeset=1" /etc

"Unknown Error" when running nvidia-smi commands"Unknown Error" when running nvidia-smi commands

If nvidia-smi commands, such as nvidia-smi --gpu-reset, return an Unknown Error, restart nvidia-persistenced:

sudo systemctl restart nvidia-persistenced

"Warning: persistence mode is disabled on device" error when enabling MIG mode"Warning: persistence mode is disabled on device" error when enabling MIG mode

If the nvidia-smi -mig 1 command output alerts you with Warning: persistence mode is disabled on device ..., start nvidia-persistenced:

sudo systemctl start nvidia-persistenced

Useful linksUseful links

• Graphics processing units (GPUs)
• Creating a VM with a GPU
• Installing NVIDIA drivers
• NVIDIA driver update guide