High-performance computing (HPC) on preemptible VMs

HPC clusters are used for computational purposes, for scientific research and computation-intensive scientific calculations in particular. A computer cluster is a set of servers (computing nodes) connected over a network. Each computing node has a number of multicore processors, local memory, and runs its own operating system instance. The most common are homogeneous clusters where all nodes are identical in their architecture and performance.

Follow this guide to create a cluster of preemptible VMs to perform a shared computational task. For example, to solve a system of linear equations using the Jacobi method.

To create a cluster and run a computational task:

1. Prepare your cloud.
2. Create a master VM in the cloud.
3. Prepare the VM's cluster.
4. Create a cluster.
5. Create a task for computations in the cluster.
6. Run and analyze the computations.
7. Delete the resources you created.

Prepare your cloudPrepare your cloud

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost for hosting servers includes:

• Fee for multiple continuously running VMs (see Yandex Compute Cloud pricing).
• Fee for using a dynamic or static public IP address (see Yandex Virtual Private Cloud pricing).

Create a master VM in the cloudCreate a master VM in the cloud

Create a VMCreate a VM

To create a VM:

1. In the management console, select the folder to create your VM in.

2. In the list of services, select Compute Cloud.

3. In the left-hand panel, select Virtual machines.

4. Click Create virtual machine.

5. Under Boot disk image, select the Ubuntu image.

6. Under Location, select an availability zone to place your VM in.

7. Under Disks and file storages, select SSD as the boot disk type.

8. Under Computing resources, navigate to the Custom tab and specify parameters for your current computational tasks:

   • Platform: Intel Ice Lake
   • vCPU: 4
   • Guaranteed vCPU performance: 100%
   • RAM: 4 GB
   • Additional: Preemptible

9. Under Network settings:

   • In the Subnet field, specify the ID of a subnet in the new VM’s availability zone or select a cloud network from the list.

     • Each network must have at least one subnet. If there is no subnet, create one by selecting Create subnet.

     • If you do not have a network, click Create network to create one:

       • In the window that opens, specify the network name and select the folder to host the network.
       • (Optional) Select the Create subnets option to automatically create subnets in all availability zones.
       • Click Create network.

   • In the Public IP field, select Auto to assign the VM a random external IP address from the Yandex Cloud pool or select a static address from the list if you reserved one in advance.

10. Under Access, specify the information required to access the VM:

    • In the Login field, enter the name of the user to create on the VM, e.g., ubuntu.

      Alert

      Do not use root or other usernames reserved by the operating system. To perform operations requiring superuser permissions, use the sudo command.

    • In the SSH key field, paste the contents of the public key file.

      You need to create a key pair for the SSH connection yourself. To learn how, see Connecting to a VM via SSH.

11. Under General information, specify the VM name. For clarity, enter master-node.

12. Click Create VM.

Set up the VMSet up the VM

1. Use SSH to connect to the VM and change to administrator mode in the console:

   sudo -i
   

2. Update the repository and install the required utilities:

   apt update
   apt install -y net-tools htop libopenmpi-dev nfs-common
   

3. Exit admin mode and generate SSH keys for access between the VMs:

   exit
   ssh-keygen -t ed25519
   

4. Add the generated key to the list of allowed ones:

   cd ~/.ssh
   cat id_ed25519.pub >> authorized_keys
   

Prepare the VM's clusterPrepare the VM's cluster

Create a clusterCreate a cluster

1. In the management console, go to Disks.
2. To the right of the master-node VM disk, click and select Create snapshot. Enter the name: master-node-snapshot. After the snapshot is created, it appears in the list under Snapshots.
3. Go to Instance groups and click Create group of virtual machines.
4. Create a VM group:
   • In the Name field, enter a name for the future VM group, e.g., compute-group.
   • In the Service account field, add a service account to the instance group. If you do not have a service account, click Create new account, enter a name, and click Create.
   • In the Availability zone field, select the availability zone the master-node VM is in. Make sure the VMs are in the same availability zone to reduce latency between them.
   • Under Instance template, click Define. This opens a screen for creating a template.
     • Under Disks, select Add disk. In the window that opens, specify:
       • Type: SSD
       • Contents: From the created snapshot named master-node-snapshot
     • Under Computing resources, specify the same configuration as that of the master VM:
       • Platform: Intel Ice Lake
       • vCPU: 4
       • Guaranteed vCPU performance: 100%
       • RAM: 4 GB
       • Additional: Preemptible
     • Under Network settings, specify the same network and subnet as those of the master VM. Leave Auto for the IP address type.
     • Under Access, specify the information required to access the VM:
       • In the Login field, enter your preferred login for the user to create on the VM.
       • In the SSH key field, paste your public SSH key. You need to create a key pair for the SSH connection yourself. To learn how, see Connecting to a VM via SSH.
     • Click Save. This returns you to the instance group creation screen.
5. Under Scaling, select the number of instances to be created. Specify 3 instances.
6. Click Create.

Test the clusterTest the cluster

Log in via SSH to each VM in compute-group and make sure you can access themaster-node VM from them via SSH:

ping master-node
ssh master-node

Configure the NFSConfigure the NFS

To allow the VMs to use the same source files, create a shared network directory using NFS:

1. Log in to the master-node VM via SSH and install the NFS server:

   ssh <master-node VM public IP address>
   sudo apt install nfs-kernel-server
   

2. Create a shared directory for the VMs:

   mkdir ~/shared
   

3. Open the /etc/exports file with a text editor, e.g., nano:

   sudo nano /etc/exports
   

4. Add an entry to the file to enable access to the shared directory:

   /home/<username>/shared *(rw,sync,no_root_squash,no_subtree_check)
   

   Save the file.

5. Apply the settings and restart the service:

   sudo exportfs -a
   sudo service nfs-kernel-server restart
   

Mount directories on group VMsMount directories on group VMs

On each VM in compute-group, mount the directory you created:

1. Create a shared directory and mount the directory with the master-node VM on it:

   mkdir ~/shared
   sudo mount -t nfs master-node:/home/<username>/shared ~/shared
   

2. Make sure that the directory is successfully mounted:

   df -h
   

   Result:

   Filesystem                                   Size  Used  Avail  Use%  Mounted on
   ...
   master-node:/home/<username>/shared  13G   1.8G  11G    15%   /home/<username>/shared
   

Create a task for computations in the clusterCreate a task for computations in the cluster

1. Log in to the master-node VM via SSH, go to the shared directory, and download the task.c source file with a computational task:

   cd ~/shared
   wget https://raw.githubusercontent.com/cloud-docs-writer/examples/master/hpc-on-preemptible/task.c
   

   This code solves a system of linear equations using the Jacobi method. The task has one distributed implementation using MPI.

2. Compile the source file into an executable file:

   mpicc task.c -o task
   

   As a result, the task executable file should appear in the shared directory.

Run and analyze the computationsRun and analyze the computations

1. Run the task on two cores using only the master-node VM resources:

   mpirun -np 2 task
   

   When the task is completed, the program displays the time spent performing it:

   JAC1 STARTED
   1: Time of task=45.104153
   0: Time of task=45.103931
   

2. Run the task on four cores using only the master-node VM resources:

   mpirun -np 4 task
   

   Result:

   JAC1 STARTED
   1: Time of task=36.562328
   2: Time of task=36.562291
   3: Time of task=36.561989
   0: Time of task=36.561695
   

3. Run the task on four cores using the resources of two VMs with two cores per VM. To do this, run the task with the -host key that accepts parameters expressed as <VM IP address>:<number of cores>[,<ip>:<cores>[,...]]:

   mpirun -np 4 -host localhost:2,<VM IP address>:2 task
   

   Result:

   JAC1 STARTED
   0: Time of task=24.539981
   1: Time of task=24.540288
   3: Time of task=24.540619
   2: Time of task=24.540781
   

4. Similarly, you can continue to increase the number of VMs and cores in use and see how distributed computing can significantly speed up task execution.

Delete the resources you createdDelete the resources you created

To stop paying for the deployed server and VM group you created, simply delete the master-node VM and compute-group

If you reserved a static public IP address specifically for this VM:

1. Select Virtual Private Cloud in your folder.
2. Go to the IP addresses tab.
3. Find the required IP address, click , and select Delete.