Contact UsGet started

Managed Service for Kubernetes cluster backups in Object Storage

Written by
Updated at July 8, 2024

Data in Managed Service for Kubernetes clusters is securely stored and replicated within the Yandex Cloud infrastructure. However, you can back up data from Managed Service for Kubernetes cluster node groups at any time and store them in Yandex Object Storage or other types of storage.

You can create backups of Managed Service for Kubernetes cluster node group data using the Velero tool. It supports working with Yandex Cloud disks using the Kubernetes CSI driver and helps create snapshots of disks and volumes.

When working with Velero that is installed manually, you can use nfs, emptyDir, local, or any other type of volumes without built-in support for snapshots. To use such a volume type, install Velero with the restic plugin. Velero installed from Cloud Marketplace does not include the restic plugin.

In this article, you will learn how to create a backup of a Managed Service for Kubernetes cluster node group using Velero, save it in Object Storage, and restore it in a node group in a different cluster:

  1. Create a backup of your Managed Service for Kubernetes node group.
  2. Recover a node group of another Managed Service for Kubernetes cluster from a backup.

If you no longer need the resources you created, delete them.

Getting started

Prepare the infrastructure

  1. Create security groups for the Managed Service for Kubernetes cluster and its node groups.

    You must configure these security groups for each cluster and node group.

    Warning

    The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.

  2. Create two Managed Service for Kubernetes clusters. When creating them, specify the security groups prepared in advance.

    One Managed Service for Kubernetes cluster will be used for creating a node group backup and the other one for restoring data from the backup.

    If you intend to use your clusters within the Yandex Cloud network, there is no need to allocate a public IP address to them. To allow connections from outside the network, assign a public IP to the clusters.

  3. Create a node group in each cluster. When creating, allocate each group a public IP address and specify the security groups prepared in advance.

  4. Create a bucket in Object Storage.

  5. Create a service account with the compute.admin role for the folder to work with Velero.

  6. Grant the service account READ and WRITE permissions to a bucket in Object Storage. To do this, configure the bucket ACL.

  7. Create a static access key for the service account and save its ID and value. You will not be able to get the key value again.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the velero-backup.tf configuration file to the same working directory.

    This file describes:

    • Network.

    • Subnet.

    • Two Managed Service for Kubernetes clusters and their node groups.

    • Service account required to use the Managed Service for Kubernetes clusters and node groups.

    • Security groups which contain rules required for the Managed Service for Kubernetes cluster and its node groups.

      These security groups will be used by both clusters.

      Warning

      The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.

    • Service account with the compute.admin role to work with Velero.

    • Static access key for the service account used to work with Velero.

    • Bucket in Object Storage.

  6. In velero-backup.tf, specify:

    • folder_id: ID of the folder to create resources in.
    • k8s_version: Kubernetes version 1.22 or higher.
    • zone_a_v4_cidr_blocks: CIDR of the subnet hosting the Managed Service for Kubernetes clusters.
    • sa_name_k8s: Name of the Managed Service for Kubernetes clusters' service account.
    • sa_name_velero: Name of the service account to work with Velero.
    • storage_sa_id: ID of the service account with the storage.admin role. It will be used to create a bucket in Object Storage with READ and WRITE permissions in the ACL for the sa_name_velero service account.
    • bucket_name: Object Storage bucket name.

  7. Run the terraform init command in the directory with the configuration file. This command initializes the provider specified in the configuration files and enables you to use the provider resources and data sources.

  8. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  9. Create the required infrastructure:

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

  10. Get and save the ID and value of the static access key for the service account used to work with Velero.

    • Key ID:

      terraform output -raw access_key

    • Key value:

      terraform output -raw secret_key

Configure additional settings

  1. Select a Velero client for your platform based on the compatibility table.

  2. Download the Velero client, extract the contents of the archive, and install it. For more information about installation, see the Velero documentation.

  3. Make sure the Velero client has been installed. To do this, run the following command:

    velero version

    Result:

    Client:
        Version: v1.10.3
        Git commit: 18ee078dffd9345df610e0ca9f61b31124e93f50

Backups

To back up the Managed Service for Kubernetes node group data:

  1. Install kubectl and configure it to work with the first Managed Service for Kubernetes cluster.

    If a cluster has no public IP address assigned and kubectl is configured via the cluster's private IP address, run kubectl commands on a Yandex Cloud VM that is in the same network as the cluster.

  2. Install the Velero application as follows:

    Install the Velero application by following this guide. In the Object Storage bucket name field, specify the bucket you created earlier.

    Warning

    If the name of the namespace where Velero is installed is not velero, additionally use the --namespace <Velero namespace> parameter for all subsequent commands.

    1. If you do not have the Yandex Cloud command line interface yet, install and initialize it.

      The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

    2. Create a service account for Velero to run.

    3. Assign it the storage.editor role to access Yandex Object Storage.

    4. Create a static access key for the service account in JSON format and save it to the sa-key.json file:

      yc iam access-key create \
   --service-account-name=<service_account_name> \
   --format=json > sa-key.json

    5. Create a file named credentials with the previously received static key data:

      [default]
  aws_access_key_id=<key_ID>
  aws_secret_access_key=<secret_part_of_key>

    6. Install the Velero server in the Managed Service for Kubernetes cluster:

      kubectl label volumesnapshotclasses.snapshot.storage.k8s.io yc-csi-snapclass \
velero.io/csi-volumesnapshot-class="true" && \
velero install \
  --backup-location-config s3Url=https://storage.yandexcloud.net,region=ru-central1 \
  --bucket <bucket_name> \
  --plugins velero/velero-plugin-for-aws:v1.3.0,velero/velero-plugin-for-csi:v0.2.0 \
  --provider aws \
  --secret-file ./credentials \
  --features=EnableCSI \
  --use-volume-snapshots=true \
  --snapshot-location-config region=ru-central1 \
  --use-restic

      Where:

      • --backup-location-config: Backup storage parameters, i.e., the URL of Object Storage storage and region.
      • --bucket: Name of the Object Storage backup storage bucket you previously created.
      • --plugins: Plugin images for AWS API compatibility.
      • --provider: Name of the object storage provider.
      • --secret-file: Full path to static access key data.
      • --features: List of enabled features.
      • --snapshot-location-config: Availability zone where disk snapshots will be located.
      • (Optional) --use-restic: Enables the restic plugin.

      Result:

      CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: already exists, proceeding
CustomResourceDefinition/backups.velero.io: created
...
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

    7. Make sure the Velero pod status has changed to Running:

      kubectl get pods --namespace velero

  3. Back up data from the Managed Service for Kubernetes cluster node group:

    velero backup create my-backup

    Result:

    Backup request "my-backup" submitted successfully.
Run `velero backup describe my-backup` or `velero backup logs my-backup` for more details.

  4. Wait for the backup to complete. The value displayed in the STATUS field will be Completed.

    velero backup get

    Result:

    NAME       STATUS     ERRORS  WARNINGS  CREATED                        EXPIRES  STORAGE LOCATION  SELECTOR
my-backup  Completed  0       0         2020-10-19 17:13:25 +0300 MSK  29d      default           <none>

Restoring data from backups

To restore data from the Managed Service for Kubernetes cluster node group:

  1. Configure kubectl to work with the second Managed Service for Kubernetes cluster.

    If a cluster has no public IP address assigned and kubectl is configured via the cluster's private IP address, run kubectl commands on a Yandex Cloud VM that is in the same network as the cluster.

  2. Install the Velero application as follows:

    Install the Velero application by following this guide. In the Object Storage bucket name field, specify the bucket you created earlier.

    Warning

    If the name of the namespace where Velero is installed is not velero, additionally use the --namespace <Velero namespace> parameter for all subsequent commands.

    1. If you do not have the Yandex Cloud command line interface yet, install and initialize it.

      The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

    2. Create a service account for Velero to run.

    3. Assign it the storage.editor role to access Yandex Object Storage.

    4. Create a static access key for the service account in JSON format and save it to the sa-key.json file:

      yc iam access-key create \
   --service-account-name=<service_account_name> \
   --format=json > sa-key.json

    5. Create a file named credentials with the previously received static key data:

      [default]
  aws_access_key_id=<key_ID>
  aws_secret_access_key=<secret_part_of_key>

    6. Install the Velero server in the Managed Service for Kubernetes cluster:

      kubectl label volumesnapshotclasses.snapshot.storage.k8s.io yc-csi-snapclass \
velero.io/csi-volumesnapshot-class="true" && \
velero install \
  --backup-location-config s3Url=https://storage.yandexcloud.net,region=ru-central1 \
  --bucket <bucket_name> \
  --plugins velero/velero-plugin-for-aws:v1.3.0,velero/velero-plugin-for-csi:v0.2.0 \
  --provider aws \
  --secret-file ./credentials \
  --features=EnableCSI \
  --use-volume-snapshots=true \
  --snapshot-location-config region=ru-central1 \
  --use-restic

      Where:

      • --backup-location-config: Backup storage parameters, i.e., the URL of Object Storage storage and region.
      • --bucket: Name of the Object Storage backup storage bucket you previously created.
      • --plugins: Plugin images for AWS API compatibility.
      • --provider: Name of the object storage provider.
      • --secret-file: Full path to static access key data.
      • --features: List of enabled features.
      • --snapshot-location-config: Availability zone where disk snapshots will be located.
      • (Optional) --use-restic: Enables the restic plugin.

      Result:

      CustomResourceDefinition/backups.velero.io: attempting to create resource
CustomResourceDefinition/backups.velero.io: already exists, proceeding
CustomResourceDefinition/backups.velero.io: created
...
Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.

    7. Make sure the Velero pod status has changed to Running:

      kubectl get pods --namespace velero

  3. Make sure the data backup is displayed in the new Managed Service for Kubernetes cluster:

    velero backup get

    Result:

    NAME       STATUS     ERRORS  WARNINGS  CREATED                        EXPIRES  STORAGE LOCATION  SELECTOR
my-backup  Completed  0       0         2020-10-19 17:13:25 +0300 MSK  29d      default           <none>

  4. Restore data from the backup:

    velero restore create my-restore \
  --exclude-namespaces velero \
  --from-backup my-backup

    Where:

    • --exclude-namespaces: List of namespaces to exclude from the recovery process.
    • --from-backup: Name of the backup for recovery.

    Result:

    Restore request "my-restore" submitted successfully.
Run `velero restore describe my-restore` or `velero restore logs my-restore` for more details.

  5. Wait for the backup restoration to complete. The value displayed in the STATUS field will be Completed.

    velero get restore

    Result:

    NAME        BACKUP     STATUS     STARTED                        COMPLETED                      ERRORS  WARNINGS  CREATED                        SELECTOR
my-restore  my-backup  Completed  2020-10-20 14:04:55 +0300 MSK  2020-10-20 14:05:22 +0300 MSK  0       23        2020-10-20 14:04:55 +0300 MSK  <none>

Delete the resources you created

If you no longer need the resources you created, delete them:

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the velero-backup.tf configuration file.

  3. Make sure the Terraform configuration files are correct using this command:

    terraform validate

    If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

    This will delete all the resources described in the velero-backup.tf configuration file.

Previous
Backup to Object Storage via Veritas Backup Exec
Next
Deploying the GlusterFS parallel file system in high availability mode