FAQ on graphics processing units (GPUs) in Compute Cloud

• What should I do if there is a GPU failure on the VM?

• "Not Supported" GPU reset error

• "In use by another client" GPU reset error

• "Unknown Error" when running nvidia-smi commands

• "Warning: persistence mode is disabled on device" error when enabling MIG mode

• How do I test a GPU cluster physical state?

• How do I run parallel tasks in a GPU cluster?

• How do I test InfiniBand throughput?

What should I do if there is a GPU failure on the VM?What should I do if there is a GPU failure on the VM?

Try stopping and restarting the VM. This is usually more effective than a reboot, since when you reboot a VM, it remains on the same host where the GPU issue occurred.

"Not Supported" GPU reset error"Not Supported" GPU reset error

You may get the following error when running sudo nvidia-smi -r:

The following GPUs could not be reset:
  GPU 00000000:8B:00.0: Not Supported
  GPU 00000000:8C:00.0: Not Supported

This error indicates an active NVLink on your GPUs. Disable it and try resetting them again.

"In use by another client" GPU reset error"In use by another client" GPU reset error

You may get the following error when running sudo nvidia-smi -r:

The following GPUs could not be reset:
  GPU 00000000:8B:00.0: In use by another client
  GPU 00000000:8C:00.0: In use by another client

2 devices are currently being used by one or more other processes (e.g., Fabric Manager, CUDA application, graphics application such as an X server, or a monitoring application such as another instance of nvidia-smi). Please first kill all processes using these devices and all compute applications running in the system.

This means your GPUs are currently in use by another process, e.g., the nvidia_drm module with enabled modeset=1, or a user process holding a lock on the device.

To fix the issue:

1. Check if nvidia_drm is running with modeset=1:

   cat /sys/module/nvidia_drm/parameters/modeset
   

   If the output returns Y, disable modeset. Do to this, create a file named /etc/modprobe.d/nomodeset.conf with the following contents:

   options nvidia-drm modeset=0
   

2. Update initramfs (Initial RAM File System):

   sudo update-initramfs -u
   

3. Identify the processes holding a lock on the device:

   sudo ls -l /proc/*/fd/* | grep /dev/nvidia
   
   Example of the command output

   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/3 -> /dev/nvidiactl
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/4 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/5 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/6 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 12:01 /proc/1574/fd/7 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/13 -> /dev/nvidiactl
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/17 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/18 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/19 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/20 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/24 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/25 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/27 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/28 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/29 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/3 -> /dev/nvidiactl
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/39 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/43 -> /dev/nvidia0
   lrwx------ 1 root             root             64 Jun 17 11:59 /proc/904/fd/8 -> /dev/nvidia-modeset
   

4. Terminate these processes by running sudo kill -9 <process_PID>.

5. Try resetting your GPUs again:

   sudo nvidia-smi -r
   

sudo grep -R "modeset=1" /etc

"Unknown Error" when running nvidia-smi commands"Unknown Error" when running nvidia-smi commands

If nvidia-smi commands, such as nvidia-smi --gpu-reset, return an Unknown Error, restart nvidia-persistenced:

sudo systemctl restart nvidia-persistenced

"Warning: persistence mode is disabled on device" error when enabling MIG mode"Warning: persistence mode is disabled on device" error when enabling MIG mode

If the nvidia-smi -mig 1 command output alerts you with Warning: persistence mode is disabled on device ..., start nvidia-persistenced:

sudo systemctl start nvidia-persistenced

How do I test a GPU cluster physical state?How do I test a GPU cluster physical state?

• Test the InfiniBand ports.
• Test the network.

For more information, see Testing a GPU cluster physical state.

How do I run parallel tasks in a GPU cluster?How do I run parallel tasks in a GPU cluster?

To run parallel tasks in a GPU cluster:

1. Connect to each VM over SSH and install Open MPI and NCCL.
2. On the main VM, build NVIDIA tests and set up passwordless SSH keys.
3. Add a public key to authorized_keys on each VM.
4. On the main VM, run the mpirun command specifying the VM IP addresses and number of GPUs.

For more information, see Running parallel tasks in a GPU cluster.

How do I test InfiniBand throughput?How do I test InfiniBand throughput?

To check InfiniBand throughput, create and run a script for starting perftest tests with numactl.

For more information, see Testing InfiniBand throughput.