Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Migrating Managed Service for Kubernetes resources to a different availability zone

Written by
Updated at July 8, 2024

Note

We are gradually deprecating the ru-central1-c availability zone. For more information about development plans for availability zones and migration options, see this Yandex Cloud blog post.

To migrate Managed Service for Kubernetes resources from one availability zone to another:

  1. Migrate a master host.
  2. Migrate the node group and the pod workloads.

Getting started

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

If you have already installed the CLI, update to its latest version.

yc components update

Migrate your master host to a different availability zone

Migration of a master host depends on its type: zonal or regional.

Migrating a zonal master host

A zonal master is located in one availability zone. You can only migrate the master host from the ru-central1-c zone to the ru-central1-d zone. During migration, the master host is restarted, and the Kubernetes server API becomes briefly unavailable. Once migrated, the master host's public and private IP addresses remain the same; the cluster and node group are not recreated.

The zonal master is moved to the new subnet with the same internal IP address it had in the old subnet. This IP address remains reserved in the old subnet as well, so you cannot delete it. Later on, you will be able to move such a subnet to a new availability zone, or it will be moved to the ru-central1-d zone automatically as soon as the ru-central1-c zone is closed.

To migrate a zonal master to a different availability zone:

  1. Create a subnet in the ru-central1-d availability zone:

    yc vpc subnet create \
   --folder-id <folder_ID> \
   --name <subnet_name> \
   --zone ru-central1-d \
   --network-id <network_ID> \
   --range <subnet_CIDR>

    In the command, specify the following subnet parameters:

    • --folder-id: Folder ID.
    • --name: Subnet name.
    • --zone: Availability zone.
    • --network-id: ID of the network the new subnet belongs to.
    • --range: List of IPv4 addresses for outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.

  2. Migrate your master host to a different availability zone:

    yc managed-kubernetes cluster update \
   --folder-id <folder_ID> \
   --id <cluster_ID> \
   --master-location subnet-id=<new_subnet_ID>,zone=ru-central1-d

    Where:

    • --folder-id: Folder ID.
    • --id: Cluster ID.
    • --master-location: Master hosting parameters:
      • subnet-id: New subnet ID.
      • zone: Availability zone.

Alert

To make sure your cluster and any of its node groups are not recreated (unless it is your intention to do so), analyze the output of the terraform plan and terraform apply commands before applying the configuration.

You can migrate a master host to a different availability zone without recreating the cluster only if the configuration file contains the master_location section.

  1. In the cluster configuration file, update the format of the cluster location section (zonal), but do not change the parameter values:

    Old format:

    resource "yandex_kubernetes_cluster" "<cluster_name>" {
   ...
   master {
      ...
      zonal {
         subnet_id = yandex_vpc_subnet.my-old-subnet.id
         zone      = "ru-central1-c"
      }
   }
   ...
}

    New format:

    resource "yandex_kubernetes_cluster" "<cluster_name>" {
   ...
   master {
      ...
      master_location {
         subnet_id = yandex_vpc_subnet.my-old-subnet.id
         zone      = "ru-central1-c"
      }
   }
   ...
}

  2. Make sure no Terraform resource parameter changes have been detected:

    terraform plan

    If Terraform detects the changes, validate all cluster parameter values: they should match the current state.

  3. Add the new network's manifest to the cluster configuration file and change the cluster location:

    resource "yandex_vpc_subnet" "my-new-subnet" {
   name           = "<network_name>"
   zone           = "ru-central1-d"
   network_id     = yandex_vpc_network.k8s-network.id
   v4_cidr_blocks = ["<subnet_CIDR>"]
}

...

resource "yandex_kubernetes_cluster" "<cluster_name>" {
   ...
   master {
      ...
      master_location {
         subnet_id = yandex_vpc_subnet.my-new-subnet.id
         zone      = "ru-central1-d"
      }
   }
   ...
}

    The old subnet for the cluster, my-old-subnet, is replaced by my-new-subnet with these parameters:

    • name: Subnet name.
    • zone: ru-central1-d availability zone.
    • network_id: ID of the network the new subnet belongs to.
    • v4_cidr_blocks: List of IPv4 addresses to deal with outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.

  4. Check that the configuration file is correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  5. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

Migrating a regional master host

A regional master host is created and distributed across three subnets in different availability zones. You can change only one availability zone for a regional master host: from ru-central1-c to ru-central1-d. Three subnets and availability zones are specified for host migration. During migration, the master host keeps running, the cluster and node group not being recreated.

The internal IP address for the regional master host is assigned automatically in one of the three hosting subnets. After migration, the master retains its internal IP address. If this IP address was allocated in the ru-central1-c zone subnet, the master is moved to the ru-central1-d zone with the previous IP address reserved in the old subnet. You cannot delete such a subnet, but later on you will be able to move it to a new availability zone, or it will be moved to the ru-central1-d zone automatically as soon as the ru-central1-c zone is closed.

To migrate a regional master host to a different set of availability zones:

  1. Create a subnet in the ru-central1-d availability zone:

    yc vpc subnet create \
   --folder-id <folder_ID> \
   --name <subnet_name> \
   --zone ru-central1-d \
   --network-id <network_ID> \
   --range <subnet_CIDR>

    In the command, specify the following subnet parameters:

    • --folder-id: Folder ID.
    • --name: Subnet name.
    • --zone: Availability zone.
    • --network-id: ID of the network the new subnet belongs to.
    • --range: List of IPv4 addresses for outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.

  2. Migrate your master host to a different set of availability zones:

    yc managed-kubernetes cluster update \
   --folder-id <folder_ID> \
   --id <cluster_ID> \
   --master-location subnet-id=<subnet_ID>,zone=ru-central1-a \
   --master-location subnet-id=<subnet_ID>,zone=ru-central1-b \
   --master-location subnet-id=<new_subnet_ID>,zone=ru-central1-d

    Where:

    • --folder-id: Folder ID.

    • --id: Cluster ID.

    • --master-location: Master hosting parameters:

      • subnet-id: Subnet ID.
      • zone: Availability zone.

      Specify the appropriate subnet for each availability zone.

Alert

To make sure your cluster and any of its node groups are not recreated (unless it is your intention to do so), analyze the output of the terraform plan and terraform apply commands before applying the configuration.

You can migrate a master host to a different availability zone without recreating the cluster only if the configuration file contains the master_location section.

  1. In the cluster configuration file, update the format of the cluster location section (regional), but do not change the parameter values:

    Old format:

    resource "yandex_kubernetes_cluster" "<cluster_name>" {
   ...
   master {
      ...
      regional {
         region = "ru-central1"
         location {
            subnet_id = yandex_vpc_subnet.my-subnet-a.id
            zone      = yandex_vpc_subnet.my-subnet-a.zone
         }
         location {
            subnet_id = yandex_vpc_subnet.my-subnet-b.id
            zone      = yandex_vpc_subnet.my-subnet-b.zone
         }
         location {
            subnet_id = yandex_vpc_subnet.my-subnet-c.id
            zone      = yandex_vpc_subnet.my-subnet-c.zone
         }
      }
   }
   ...
}

    New format:

    resource "yandex_kubernetes_cluster" "<cluster_name>" {
   ...
   master {
      ...
      master_location {
         subnet_id = yandex_vpc_subnet.my-subnet-a.id
         zone      = yandex_vpc_subnet.my-subnet-a.zone
      }
      master_location {
         subnet_id = yandex_vpc_subnet.my-subnet-b.id
         zone      = yandex_vpc_subnet.my-subnet-b.zone
      }
      master_location {
         subnet_id = yandex_vpc_subnet.my-subnet-c.id
         zone      = yandex_vpc_subnet.my-subnet-c.zone
      }
   }
   ...
}

  2. Make sure no Terraform resource parameter changes have been detected:

    terraform plan

    If Terraform detects changes, validate all cluster parameter values: they should match the current state.

  3. Add the new subnet's manifest to the cluster configuration file and change the cluster location:

    resource "yandex_vpc_subnet" "my-subnet-d" {
   name           = "<network_name>"
   zone           = "ru-central1-d"
   network_id     = yandex_vpc_network.k8s-network.id
   v4_cidr_blocks = ["<subnet_CIDR>"]
}

...

resource "yandex_kubernetes_cluster" "k8s-cluster" {
   ...
   master {
      ...
      master_location {
         subnet_id = yandex_vpc_subnet.my-subnet-a.id
         zone      = yandex_vpc_subnet.my-subnet-a.zone
      }
      master_location {
         subnet_id = yandex_vpc_subnet.my-subnet-b.id
         zone      = yandex_vpc_subnet.my-subnet-b.zone
      }
      master_location {
         subnet_id = yandex_vpc_subnet.my-subnet-d.id
         zone      = yandex_vpc_subnet.my-subnet-d.zone
      }
      ...
   }
...
}

    The cluster's subnet named my-subnet-c is replaced with my-subnet-d that has the following parameters:

    • name: Subnet name.
    • zone: Availability zone.
    • network_id: ID of the network the new subnet belongs to.
    • v4_cidr_blocks: List of IPv4 addresses to deal with outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.

  4. Check that the configuration file is correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  5. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

Migrate the node group and the pod workloads to a different availability zone

Prepare a node group and proceed to migration using one of the following methods:

  • Migrating a node group directly to the new availability zone. It depends on the type of workload in the pods:

    • Stateless workload: The functioning of applications in the pods during migration depends on how the workload is distributed among the cluster nodes. If the pods reside both in the node group you are migrating and the groups for which the availability zone remains the same, the applications will continue to run. If the pods only reside in the group you are migrating, both the pods and the applications in them will have to be stopped for a short while.

      Examples of stateless workloads include the web server, Yandex Application Load Balancer Ingress controller, and REST API applications.

    • Stateful workloads: The pods and applications will have to be stopped for a short while, regardless of how the workload is distributed among the cluster nodes.

      Examples of stateful workloads include databases and storages.

  • Gradually migrating a stateless and stateful workload to the new node group. It involves creating a new node group in the new availability zone and gradually discontinuing the old nodes. This way, you can monitor the workload transfer.

Getting started

  1. Check if the nodeSelector, affinity, or topology spread constraints strategies are being used to assign the pods to the group's nodes. For more information on strategies, see High availability and fault tolerance and the Kubernetes documentation. To check how a pod is assigned to its associated node and unlink them:

    1. In the management console, select the folder with your Managed Service for Kubernetes cluster.

    2. In the list of services, select Managed Service for Kubernetes.

    3. Go to the cluster page and find the Workload section.

    4. On the Pods tab, open the pod's page.

    5. Go to the YAML tab.

    6. Check if the pod manifest contains the following parameters and Kubernetes labels in them:

      • Parameters:

        • spec.nodeSelector
        • spec.affinity
        • spec.topologySpreadConstraints

      • Kubernetes labels set in the parameters:

        • failure-domain.beta.kubernetes.io/zone: <availability_zone>
        • topology.kubernetes.io/zone: <availability_zone>

      If a configuration has at least one of these parameters and that parameter contains at least one of the listed Kubernetes labels, that configuration prevents node group and workload migration.

    7. Check if the pod manifest has any dependencies from:

      • Availability zone you are migrating resources from.
      • Specific nodes within the group.

    8. If you find any of these settings, assignments, or dependencies, remove them from the pod configuration:

      1. Copy the YAML configuration from the management console.

      2. Create a local YAML file and paste the configuration into it.

      3. Remove any availability zone assignments from the configuration. For example, if the spec.affinity parameter sets the failure-domain.beta.kubernetes.io/zone Kubernetes label, remove it.

      4. Apply the new configuration:

        kubectl apply -f <path_to_YAML_file>

      5. Make sure the pod status changed to Running:

        kubectl get pods

    9. Check and update the configuration of each pod by repeating these steps.

  2. Transfer persistent data, such as databases, message queues, monitoring and log servers, to the new availability zone.

Migrating a stateless workload

  1. Create a subnet in the new availability zone and migrate the node group:

    1. Create a subnet:

      yc vpc subnet create \
   --name <subnet_name> \
   --zone <availability_zone> \
   --network-id <network_ID> \
   --range <subnet_CIDR>

      Where:

      • --name: Subnet name.
      • --zone: Availability zone (ru-central1-a, ru-central1-b, or ru-central1-d).
      • --network-id: ID of the network the new subnet belongs to.
      • --range: List of IPv4 addresses for outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.

    2. Move the node group to the new availability zone. The example below shows a command for moving a group residing in a single zone:

      yc managed-kubernetes node-group update \
   --id <node_group_ID> \
   --location zone=<availability_zone>,subnet-id=<subnet_ID> \
   --network-interface subnets=<subnet_ID>,`
        `ipv4-address=nat,`
        `security-group-ids=[<security_group_IDs>]

      Where:

      • id: ID of the node group to move to a different availability zone.
      • zone: Availability zone you want to move your node group to (ru-central1-a, ru-central1-b, or ru-central1-d).
      • subnet-id and subnets: ID of the new subnet you created earlier.
      • ipv4-address: Method of assigning an IPv4 address. The nat value allows assigning public and internal IP addresses to nodes.
      • security-group-ids: List of security group IDs.

      Warning

      If you want to keep the values of other network parameters for the node group, specify them in the network-interface parameter as well. Otherwise, the group can be recreated with default values. For more information, see Updating a Managed Service for Kubernetes node group.

      It is important to provide the ipv4-address and security-group-ids parameters in the command: this will assign public IP addresses to the node group and keep security groups within it.

      This command recreates the nodes within their group in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size versus the original node count.

      How to move a node group residing in different availability zones

      In this case, use the following command:

      yc managed-kubernetes node-group update \
   --id <node_group_ID> \
   --location zone=<availability_zone>,subnet-id=<subnet_ID> \
   ...
   --location zone=<availability_zone>,subnet-id=<subnet_ID> \
   --network-interface subnets=[<subnet_IDs>],`
        `ipv4-address=nat,`
        `security-group-ids=[<security_group_IDs>]

      Where:

      • id: ID of the node group to move to a different availability zone.
      • zone: Availability zone (ru-central1-a, ru-central1-b, or ru-central1-d). Specify the location parameters for each availability zone that will host the node group.
      • subnet-id and subnets: IDs of the subnets for the specified availability zones.
      • ipv4-address: Method of assigning an IPv4 address. The nat value allows assigning public and internal IP addresses to nodes.
      • security-group-ids: List of security group IDs.

    Alert

    To make sure your node group is not recreated (unless it is your intention to do so), analyze the output of the terraform plan and terraform apply commands before applying the configuration.

    You can migrate a node group to a different availability zone without recreating it only if the configuration file contains the allocation_policy section.

    1. Make the following changes to the configuration file:

      • Add a new subnet manifest (yandex_vpc_subnet resource) in the availability zone to which you want to move the node group.
      • Change the node group location parameters (yandex_kubernetes_node_group resource):
        • allocation_policy.location.subnet_id: Remove this parameter if it is in the manifest.
        • allocation_policy.location.zone: Specify the availability zone you want to move the node group to.
        • instance_template.network_interface.subnet_ids: Specify the new subnet ID. Add this parameter if it is not in the manifest.
      resource "yandex_vpc_subnet" "my-new-subnet" {
   name           = "<subnet_name>"
   zone           = "<availability_zone>"
   network_id     = "<network_ID>"
   v4_cidr_blocks = ["<subnet_CIDR>"]
}
...
resource "yandex_kubernetes_node_group" "k8s-node-group" {
   allocation_policy {
      location {
         zone = "<availability_zone>"
      }
   }
   ...
   instance_template {
      network_interface {
         subnet_ids = [yandex_vpc_subnet.my-new-subnet.id]
         ...
      }
      ...
   }
   ...
}

      Where:

      • name: Subnet name.
      • zone: Availability zone you want to move your node group to (ru-central1-a, ru-central1-b, or ru-central1-d).
      • network_id: ID of the network the new subnet belongs to.
      • v4_cidr_blocks: List of IPv4 addresses to deal with outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.
      • subnet_ids: ID of the new subnet.

    2. Check that the configuration file is correct.

      1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

      2. Run the command:

        terraform validate

        If there are errors in the configuration files, Terraform will point to them.

    3. Confirm updating the resources.

      1. Run the command to view planned changes:

        terraform plan

        If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

        1. Run the command:

          terraform apply

        2. Confirm the update of resources.

        3. Wait for the operation to complete.

    The group nodes will be recreated in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size versus the original node count.

  2. Make sure the pods are running in the migrated node group:

    kubectl get po --output wide

    The output of this command displays the nodes on which your pods are currently running.

Migrating a stateful workload

The migration process is based on scaling the StatefulSet controller. To migrate a stateful workload:

  1. Get a list of StatefulSet controllers to find the name of the one you need:

    kubectl get statefulsets

  2. Get the number of pods managed by the controller:

    kubectl get statefulsets <controller_name> \
   -n default -o=jsonpath='{.status.replicas}'

    Save the obtained value. You will need it to scale the StatefulSet controller once the migration of your stateful workload is complete.

  3. Reduce the number of pods to zero:

    kubectl scale statefulset <controller_name> --replicas=0

    This way, you will disable the pods that use disks. In addition, running this command saves a PersistentVolumeClaim (PVC) Kubernetes API object.

  4. For the PersistentVolume object (PV) associated with PersistentVolumeClaim, change the value of the persistentVolumeReclaimPolicy parameter from Delete to Retain to prevent accidental data loss.

    1. Get the name of the PersistentVolume object:

      kubectl get pv

    2. Edit the PersistentVolume object:

      kubectl edit pv <PV_name>

  5. Check if the PersistentVolume object manifest contains the spec.nodeAffinity parameter:

    kubectl get pv <PV_name> --output='yaml'

    If the manifest contains the spec.nodeAffinity parameter with an availability zone specified in it, save this parameter. You will need to specify it in a new PersistentVolume object.

  6. Create a snapshot representing a point-in-time copy of the PersistentVolume disk. For more information about snapshots, see the Kubernetes documentation.

    1. Get the name of the PersistentVolumeClaim:

      kubectl get pvc

    2. Create a snapshot.yaml file with the snapshot manifest and specify the PersistentVolumeClaim name in it:

      apiVersion: snapshot.storage.k8s.io/v1
kind: VolumeSnapshot
metadata:
   name: new-snapshot-test-<number>
spec:
   volumeSnapshotClassName: yc-csi-snapclass
   source:
      persistentVolumeClaimName: <PVC_name>

      If you are creating several snapshots for different PersistentVolumeClaim objects, specify the <number> (consecutive number) to make a unique metadata.name value for each snapshot.

    3. Create a snapshot:

      kubectl apply -f snapshot.yaml

    4. Check that the snapshot has been created:

      kubectl get volumesnapshots.snapshot.storage.k8s.io

    5. Make sure the VolumeSnapshotContent Kubernetes API object has been created:

      kubectl get volumesnapshotcontents.snapshot.storage.k8s.io

  7. Get the snapshot ID:

    yc compute snapshot list

  8. Create a VM disk from the snapshot:

    yc compute disk create \
   --source-snapshot-id <snapshot_ID> \
   --zone <availability_zone>

    In the command, specify the availability zone to which the Managed Service for Kubernetes node group is being migrated.

    Save the following parameters from the command's output:

    • Disk ID from the id field.
    • Disk type from the type_id field.
    • Disk size from the size field.

  9. Create a PersistentVolume Kubernetes API object from the new disk:

    1. Create a persistent-volume.yaml file with the PersistentVolume manifest:

      apiVersion: v1
kind: PersistentVolume
metadata:
   name: new-pv-test-<number>
spec:
   capacity:
      storage: <PersistentVolume_size>
   accessModes:
      - ReadWriteOnce
   csi:
      driver: disk-csi-driver.mks.ycloud.io
      fsType: ext4
      volumeHandle: <disk_ID>
   storageClassName: <disk_type>

      In the file, specify the parameters of the disk created from the snapshot:

      • spec.capacity.storage: Disk size.

      • spec.csi.volumeHandle: Disk ID.

      • spec.storageClassName: Disk type. Specify the type in accordance with the following table:

        Type of the disk created from the snapshot Disk type for the YAML file
        network-ssd yc-network-ssd
        network-ssd-nonreplicated yc-network-ssd-nonreplicated
        network-nvme yc-network-nvme
        network-hdd yc-network-hdd

      If you are creating several PersistentVolume objects, specify the <number> (consecutive number) to make a unique metadata.name value for each snapshot.

      If you saved the spec.nodeAffinity parameter earlier, add it to the manifest and specify the availability zone to which the Managed Service for Kubernetes node group is being migrated. If you omit this parameter, the workload can run in a different availability zone where PersistentVolume is not available. This will result in a run error.

      Example of the spec.nodeAffinity parameter:

      spec:
   ...
   nodeAffinity:
      required:
         nodeSelectorTerms:
         - matchExpressions:
            - key: failure-domain.beta.kubernetes.io/zone
              operator: In
              values:
                 - ru-central1-d

    2. Create a PersistentVolume object:

      kubectl apply -f persistent-volume.yaml

    3. Make sure the PersistentVolume has been created:

      kubectl get pv

      The new-pv-test-<number> object will appear in the command's output.

    4. If you specified the spec.nodeAffinity parameter in the manifest, make sure to apply it for the PersistentVolume object:

      1. In the management console, select the folder with your Managed Service for Kubernetes cluster.
      2. In the list of services, select Managed Service for Kubernetes.
      3. Go to the cluster page and find the Storage section.
      4. On the Persistent Volumes tab, find the new-pv-test-<number> object and look up the Availability zone field value. It must specify an availability zone. A dash means there is no assignment to an availability zone.

    5. If you omitted the spec.nodeAffinity parameter in the manifest, you can add it by editing the PersistentVolume object:

      kubectl edit pv new-pv-test-<number>

  10. Create a PersistentVolumeClaim object from the new PersistentVolume:

    1. Create a persistent-volume-claim.yaml file with the PersistentVolumeClaim manifest:

      apiVersion: v1
kind: PersistentVolumeClaim
metadata:
   name: <PVC_name>
spec:
   accessModes:
      - ReadWriteOnce
   resources:
      requests:
         storage: <PV_size>
   storageClassName: <disk_type>
   volumeName: new-pv-test-<number>

      In the file, set the following parameters:

      • metadata.name: Name of the PersistentVolumeClaim that you used to create the snapshot. You can get this name by running the kubectl get pvc command.
      • spec.resources.requests.storage: Size of the PersistentVolume; matches the size of the created disk.
      • spec.storageClassName: Disk type of the PersistentVolume; matches the disk type of the new PersistentVolume.
      • spec.volumeName: Name of the PersistentVolume object based on which the PersistentVolumeClaim object is created. You can get this name by running the kubectl get pv command.

    2. Delete the original PersistentVolumeClaim so you can replace it:

      kubectl delete pvc <PVC_name>

    3. Create a PersistentVolumeClaim:

      kubectl apply -f persistent-volume-claim.yaml

    4. Make sure the PersistentVolumeClaim has been created:

      kubectl get pvc

      The output of the PersistentVolumeClaim command will contain the size you specified in the YAML file.

  11. Create a subnet in the new availability zone and migrate the node group:

    1. Create a subnet:

      yc vpc subnet create \
   --name <subnet_name> \
   --zone <availability_zone> \
   --network-id <network_ID> \
   --range <subnet_CIDR>

      Where:

      • --name: Subnet name.
      • --zone: Availability zone (ru-central1-a, ru-central1-b, or ru-central1-d).
      • --network-id: ID of the network the new subnet belongs to.
      • --range: List of IPv4 addresses for outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.

    2. Move the node group to the new availability zone. The example below shows a command for moving a group residing in a single zone:

      yc managed-kubernetes node-group update \
   --id <node_group_ID> \
   --location zone=<availability_zone>,subnet-id=<subnet_ID> \
   --network-interface subnets=<subnet_ID>,`
        `ipv4-address=nat,`
        `security-group-ids=[<security_group_IDs>]

      Where:

      • id: ID of the node group to move to a different availability zone.
      • zone: Availability zone you want to move your node group to (ru-central1-a, ru-central1-b, or ru-central1-d).
      • subnet-id and subnets: ID of the new subnet you created earlier.
      • ipv4-address: Method of assigning an IPv4 address. The nat value allows assigning public and internal IP addresses to nodes.
      • security-group-ids: List of security group IDs.

      Warning

      If you want to keep the values of other network parameters for the node group, specify them in the network-interface parameter as well. Otherwise, the group can be recreated with default values. For more information, see Updating a Managed Service for Kubernetes node group.

      It is important to provide the ipv4-address and security-group-ids parameters in the command: this will assign public IP addresses to the node group and keep security groups within it.

      This command recreates the nodes within their group in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size versus the original node count.

      How to move a node group residing in different availability zones

      In this case, use the following command:

      yc managed-kubernetes node-group update \
   --id <node_group_ID> \
   --location zone=<availability_zone>,subnet-id=<subnet_ID> \
   ...
   --location zone=<availability_zone>,subnet-id=<subnet_ID> \
   --network-interface subnets=[<subnet_IDs>],`
        `ipv4-address=nat,`
        `security-group-ids=[<security_group_IDs>]

      Where:

      • id: ID of the node group to move to a different availability zone.
      • zone: Availability zone (ru-central1-a, ru-central1-b, or ru-central1-d). Specify the location parameters for each availability zone that will host the node group.
      • subnet-id and subnets: IDs of the subnets for the specified availability zones.
      • ipv4-address: Method of assigning an IPv4 address. The nat value allows assigning public and internal IP addresses to nodes.
      • security-group-ids: List of security group IDs.

    Alert

    To make sure your node group is not recreated (unless it is your intention to do so), analyze the output of the terraform plan and terraform apply commands before applying the configuration.

    You can migrate a node group to a different availability zone without recreating it only if the configuration file contains the allocation_policy section.

    1. Make the following changes to the configuration file:

      • Add a new subnet manifest (yandex_vpc_subnet resource) in the availability zone to which you want to move the node group.
      • Change the node group location parameters (yandex_kubernetes_node_group resource):
        • allocation_policy.location.subnet_id: Remove this parameter if it is in the manifest.
        • allocation_policy.location.zone: Specify the availability zone you want to move the node group to.
        • instance_template.network_interface.subnet_ids: Specify the new subnet ID. Add this parameter if it is not in the manifest.
      resource "yandex_vpc_subnet" "my-new-subnet" {
   name           = "<subnet_name>"
   zone           = "<availability_zone>"
   network_id     = "<network_ID>"
   v4_cidr_blocks = ["<subnet_CIDR>"]
}
...
resource "yandex_kubernetes_node_group" "k8s-node-group" {
   allocation_policy {
      location {
         zone = "<availability_zone>"
      }
   }
   ...
   instance_template {
      network_interface {
         subnet_ids = [yandex_vpc_subnet.my-new-subnet.id]
         ...
      }
      ...
   }
   ...
}

      Where:

      • name: Subnet name.
      • zone: Availability zone you want to move your node group to (ru-central1-a, ru-central1-b, or ru-central1-d).
      • network_id: ID of the network the new subnet belongs to.
      • v4_cidr_blocks: List of IPv4 addresses to deal with outgoing and incoming traffic, e.g., 10.0.0.0/22 or 192.168.0.0/16. Make sure the addresses are unique within the network. The minimum subnet size is /28, the maximum subnet size is /16. Only IPv4 is supported.
      • subnet_ids: ID of the new subnet.

    2. Check that the configuration file is correct.

      1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

      2. Run the command:

        terraform validate

        If there are errors in the configuration files, Terraform will point to them.

    3. Confirm updating the resources.

      1. Run the command to view planned changes:

        terraform plan

        If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

        1. Run the command:

          terraform apply

        2. Confirm the update of resources.

        3. Wait for the operation to complete.

    The group nodes will be recreated in the specified availability zone and subnet. When recreating, the deployment settings are considered: the maximum number of nodes by which you can increase or decrease the group size versus the original node count.

  12. Restore the original number of pods managed by the StatefulSet controller:

    kubectl scale statefulset <controller_name> --replicas=<pod_count>

    The pods will be launched in the migrated node group.

    In the command, specify the following parameters:

    • Name of the StatefulSet controller. You can get it by running the kubectl get statefulsets command.
    • Number of pods prior to scaling down the controller.

  13. Make sure the pods are running in the migrated node group:

    kubectl get po --output wide

    The output of this command displays the nodes on which your pods are currently running.

  14. Delete the unused PersistentVolume object (having the Released status).

    1. Get the name of the PersistentVolume object:

      kubectl get pv

    2. Delete the PersistentVolume object:

      kubectl delete pv <PV_name>

Gradually migrating a stateless and stateful workload

See below how to gradually migrate a workload from the old node group to the new one. For instructions on migrating the PersistentVolume and PersistentVolumeClaim objects, see the Migrating a stateful workload subsection.

  1. Create a new Managed Service for Kubernetes node group in the new availability zone.

  2. Disable running new pods in the old node group:

    kubectl cordon -l yandex.cloud/node-group-id=<old_node_group_ID>

  3. For each node in the old node group, run the following command:

    kubectl drain <node_name> --ignore-daemonsets --delete-emptydir-data

    The pods will gradually move to the new node group.

  4. Make sure the pods are running in the new node group:

    kubectl get po --output wide

  5. Delete the old node group.

Previous
Setting up Time-Slicing GPUs
Next
Using Yandex Cloud modules in Terraform