Graphics processing units (GPUs)

• How do I test a GPU cluster physical state?

• How do I run parallel tasks in a GPU cluster?

• How do I test InfiniBand throughput?

• What should I do if there is a GPU failure on the VM?

How do I test a GPU cluster physical state?How do I test a GPU cluster physical state?

• Test the InfiniBand ports.
• Test the network.

For more information, see Testing a GPU cluster physical state.

How do I run parallel tasks in a GPU cluster?How do I run parallel tasks in a GPU cluster?

To run parallel tasks in a GPU cluster:

1. Connect to each VM over SSH and install Open MPI and NCCL.
2. On the main VM, build NVIDIA tests and set up passwordless SSH keys.
3. Add a public key to authorized_keys on each VM.
4. On the main VM, run the mpirun command specifying the VM IP addresses and number of GPUs.

For more information, see Running parallel tasks in a GPU cluster.

How do I test InfiniBand throughput?How do I test InfiniBand throughput?

To check InfiniBand throughput, create and run a script for starting perftest tests with numactl.

For more information, see Testing InfiniBand throughput.

What should I do if there is a GPU failure on the VM?What should I do if there is a GPU failure on the VM?

Try stopping and restarting the VM. This is usually more effective than a reboot, since when you reboot a VM, it remains on the same host where the GPU issue occurred.