Cluster node maintenance with zero resource loss

During maintenance and update, you may need to restart or recreate Managed Service for Kubernetes cluster nodes. In which case node resources become publicly available again and can be taken by other users.

Reservation is especially relevant for limited availability resources, such as GPUs. If the GPU configuration turns out occupied during the GPU node restart, the start will terminate with an error.

There are two ways for you to preserve your resources during maintenance:

• Using a reserved instance pool: Suitable for any node restart or recreation scenarios.
• Using a maintenance task: Suitable if there is a node maintenance task in Compute Cloud.

Using a reserved instance poolUsing a reserved instance pool

The reserved instance pool feature is at the Preview stage.

Node resources can be reserved using a reserved instance pool. In this case, during node restart or recreation, its resources will remain in the pool and not go public: the node will start on exactly the same resources.

1. Create a reserved instance pool of size 0 with a configuration of existing nodes, e.g., with required GPU parameters.

2. Expand the pool to as many nodes as you need by activating the --allow-pending-slots resource wait option. This parameter allows you to add slots, even if there are not enough free resources in the availability zone temporarily.

   Slots that did not get enough resources during the expansion will enter the pending status.

3. Attach the nodes of the existing Managed Service for Kubernetes cluster to the new pool.

The Managed Service for Kubernetes cluster nodes will occupy free slots in the pool. Next, the nodes will be started or recreated on the pool’s reserved resources.

Using a maintenance taskUsing a maintenance task

Compute Cloud performs periodical maintenance of the hardware your VMs need to operate. Sometimes maintenance implies restarting the GPU VMs.

To complete the maintenance task, GPU VM resources are reserved automatically and not made public again on restart: for maintenance it is enough to remove load from the node and perform a restart via Compute Cloud.

1. To figure out which node requires maintenance, look up maintenance tasks.

2. Make sure kubectl is connected to the right cluster and get a list of nodes:

   kubectl config current-context
   kubectl get nodes
   

   The node name in the command output matches the VM name in Compute Cloud.

3. Evict the pods from the node using kubectl drain:

   kubectl drain <node_name> \
     --ignore-daemonsets \
     --delete-emptydir-data
   

   Where <node_name> is the Kubernetes node name.

   After the command is executed, the node will enter the SchedulingDisabled status: new pods will no longer be created on it, and current active pods will be evicted to other cluster nodes.

   For more on pod eviction, see Evicting pods from nodes.

4. Run the maintenance task immediately. The VM will restart but Compute Cloud will keep the GPU resources assigned to it.

5. Use the management console or this command to allow pods to be put on the node:

   kubectl uncordon <node_name>
   

   The node will once again become available for new pod scheduling.

Useful linksUseful links

• Evicting pods from nodes
• Maintenance of GPU VMs
• Managing maintenance of GPU VMs
• Creating a group with nodes from a Yandex Compute Cloud reserved instance pool