Testing a GPU cluster physical state

Test the state of InfiniBand portsTest the state of InfiniBand ports

1. Connect to the VM over SSH.

2. Install the infiniband-diags package:

   sudo apt update
   sudo apt install infiniband-diags
   

3. Run the ibstatus command:

   Result:

   state:           4: ACTIVE
   phys state:      5: LinkUp
   

4. Make sure the phys state parameter is set to LinkUp for all ports.

Test network performanceTest network performance

To test the data transfer rate between GPUs on different VMs:

1. Install the perftest package on each test VM:

   sudo apt install perftest
   

2. Connect to the first VM over SSH.

3. Run this command:

   ib_send_bw --report_gbits
   

4. Connect to the second VM over SSH.

5. Run this command:

   ib_send_bw <first_VM_internal_IP> --report_gbits
   

   Result:

   #bytes     #iterations    BW peak[Gb/sec]    BW average[Gb/sec]   MsgRate[Mpps]
   65536      1000             245.54             244.08             0.465536
   

6. Make sure the output shows non-zero values for these parameters:

   • BW average[Gb/sec]: Average transfer rate
   • MsgRate[Mpps]: Message frequency