Managed Service for Kubernetes cluster backups in Object Storage
Data in Managed Service for Kubernetes clusters is securely stored and replicated within the Yandex Cloud infrastructure. However, you can back up data from Managed Service for Kubernetes cluster node groups at any time and store them in Yandex Object Storage or other types of storage.
You can create backups of Managed Service for Kubernetes cluster node group data using the Velero
When working with Velero that is installed manually, you can use nfs
In this article, you will learn how to create a backup of a Managed Service for Kubernetes cluster node group using Velero, save it in Object Storage, and restore it in a node group in a different cluster:
- Create a backup of your Managed Service for Kubernetes node group.
- Recover a node group of another Managed Service for Kubernetes cluster from a backup.
If you no longer need the resources you created, delete them.
Getting started
Prepare the infrastructure
-
Create security groups for the Managed Service for Kubernetes cluster and its node groups.
You must configure these security groups for each cluster and node group.
Warning
The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.
-
Create two Managed Service for Kubernetes clusters. When creating them, specify the security groups prepared in advance.
One Managed Service for Kubernetes cluster will be used for creating a node group backup and the other one for restoring data from the backup.
If you intend to use your clusters within the Yandex Cloud network, there is no need to allocate a public IP address to them. To allow connections from outside the network, assign a public IP to the clusters.
-
Create a node group in each cluster. When creating, allocate each group a public IP address and specify the security groups prepared in advance.
-
Create a service account with the
compute.admin
role for the folder to work with Velero. -
Grant the service account READ and WRITE permissions to a bucket in Object Storage. To do this, configure the bucket ACL.
-
Create a static access key for the service account and save its ID and value. You will not be able to get the key value again.
-
If you do not have Terraform yet, install it.
-
Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.
-
Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it
. -
Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.
-
Download the velero-backup.tf
configuration file to the same working directory.This file describes:
-
Two Managed Service for Kubernetes clusters and their node groups.
-
Service account required to use the Managed Service for Kubernetes clusters and node groups.
-
Security groups which contain rules required for the Managed Service for Kubernetes cluster and its node groups.
These security groups will be used by both clusters.
Warning
The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.
-
Service account with the
compute.admin
role to work with Velero. -
Static access key for the service account used to work with Velero.
-
Bucket in Object Storage.
-
In
velero-backup.tf
, specify:folder_id
: ID of the folder to create resources in.k8s_version
: Kubernetes version 1.22 or higher.zone_a_v4_cidr_blocks
: CIDR of the subnet hosting the Managed Service for Kubernetes clusters.sa_name_k8s
: Name of the Managed Service for Kubernetes clusters' service account.sa_name_velero
: Name of the service account to work with Velero.storage_sa_id
: ID of the service account with thestorage.admin
role. It will be used to create a bucket in Object Storage withREAD and WRITE
permissions in the ACL for thesa_name_velero
service account.bucket_name
: Object Storage bucket name.
-
Run the
terraform init
command in the directory with the configuration file. This command initializes the provider specified in the configuration files and enables you to use the provider resources and data sources. -
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Create the required infrastructure:
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console
. -
-
Get and save the ID and value of the static access key for the service account used to work with Velero.
-
Key ID:
terraform output -raw access_key
-
Key value:
terraform output -raw secret_key
-
Configure additional settings
-
Select a Velero client
for your platform based on the compatibility table . -
Download the Velero client, extract the contents of the archive, and install it. For more information about installation, see the Velero documentation
. -
Make sure the Velero client has been installed. To do this, run the following command:
velero version
Result:
Client: Version: v1.10.3 Git commit: 18ee078dffd9345df610e0ca9f61b31124e93f50
Backups
To back up the Managed Service for Kubernetes node group data:
-
Install kubectl
and configure it to work with the first Managed Service for Kubernetes cluster.If a cluster has no public IP address assigned and
kubectl
is configured via the cluster's private IP address, runkubectl
commands on a Yandex Cloud VM that is in the same network as the cluster. -
Install the Velero application as follows:
Yandex Cloud MarketplaceManuallyInstall the Velero application by following this guide. In the Object Storage bucket name field, specify the bucket you created earlier.
Warning
If the name of the namespace where Velero is installed is not
velero
, additionally use the--namespace <Velero namespace>
parameter for all subsequent commands.-
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the
--folder-name
or--folder-id
parameter. -
Create a service account for Velero to run.
-
Assign it the
storage.editor
role to access Yandex Object Storage. -
Create a static access key for the service account in JSON format and save it to the
sa-key.json
file:yc iam access-key create \ --service-account-name=<service_account_name> \ --format=json > sa-key.json
-
Create a file named
credentials
with the previously received static key data:[default] aws_access_key_id=<key_ID> aws_secret_access_key=<secret_part_of_key>
-
Install the Velero server in the Managed Service for Kubernetes cluster:
kubectl label volumesnapshotclasses.snapshot.storage.k8s.io yc-csi-snapclass \ velero.io/csi-volumesnapshot-class="true" && \ velero install \ --backup-location-config s3Url=https://storage.yandexcloud.net,region=ru-central1 \ --bucket <bucket_name> \ --plugins velero/velero-plugin-for-aws:v1.3.0,velero/velero-plugin-for-csi:v0.2.0 \ --provider aws \ --secret-file ./credentials \ --features=EnableCSI \ --use-volume-snapshots=true \ --snapshot-location-config region=ru-central1 \ --use-restic
Where:
--backup-location-config
: Backup storage parameters, i.e., the URL of Object Storage storage and region.--bucket
: Name of the Object Storage backup storage bucket you previously created.--plugins
: Plugin images for AWS API compatibility.--provider
: Name of the object storage provider.--secret-file
: Full path to static access key data.--features
: List of enabled features.--snapshot-location-config
: Availability zone where disk snapshots will be located.- (Optional)
--use-restic
: Enables the restic plugin.
Result:
CustomResourceDefinition/backups.velero.io: attempting to create resource CustomResourceDefinition/backups.velero.io: already exists, proceeding CustomResourceDefinition/backups.velero.io: created ... Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
-
Make sure the Velero pod status has changed to
Running
:kubectl get pods --namespace velero
-
-
Back up data from the Managed Service for Kubernetes cluster node group:
velero backup create my-backup
Result:
Backup request "my-backup" submitted successfully. Run `velero backup describe my-backup` or `velero backup logs my-backup` for more details.
-
Wait for the backup to complete. The value displayed in the
STATUS
field will beCompleted
.velero backup get
Result:
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR my-backup Completed 0 0 2020-10-19 17:13:25 +0300 MSK 29d default <none>
Restoring data from backups
To restore data from the Managed Service for Kubernetes cluster node group:
-
Configure kubectl to work with the second Managed Service for Kubernetes cluster.
If a cluster has no public IP address assigned and
kubectl
is configured via the cluster's private IP address, runkubectl
commands on a Yandex Cloud VM that is in the same network as the cluster. -
Install the Velero application as follows:
Yandex Cloud MarketplaceManuallyInstall the Velero application by following this guide. In the Object Storage bucket name field, specify the bucket you created earlier.
Warning
If the name of the namespace where Velero is installed is not
velero
, additionally use the--namespace <Velero namespace>
parameter for all subsequent commands.-
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the
--folder-name
or--folder-id
parameter. -
Create a service account for Velero to run.
-
Assign it the
storage.editor
role to access Yandex Object Storage. -
Create a static access key for the service account in JSON format and save it to the
sa-key.json
file:yc iam access-key create \ --service-account-name=<service_account_name> \ --format=json > sa-key.json
-
Create a file named
credentials
with the previously received static key data:[default] aws_access_key_id=<key_ID> aws_secret_access_key=<secret_part_of_key>
-
Install the Velero server in the Managed Service for Kubernetes cluster:
kubectl label volumesnapshotclasses.snapshot.storage.k8s.io yc-csi-snapclass \ velero.io/csi-volumesnapshot-class="true" && \ velero install \ --backup-location-config s3Url=https://storage.yandexcloud.net,region=ru-central1 \ --bucket <bucket_name> \ --plugins velero/velero-plugin-for-aws:v1.3.0,velero/velero-plugin-for-csi:v0.2.0 \ --provider aws \ --secret-file ./credentials \ --features=EnableCSI \ --use-volume-snapshots=true \ --snapshot-location-config region=ru-central1 \ --use-restic
Where:
--backup-location-config
: Backup storage parameters, i.e., the URL of Object Storage storage and region.--bucket
: Name of the Object Storage backup storage bucket you previously created.--plugins
: Plugin images for AWS API compatibility.--provider
: Name of the object storage provider.--secret-file
: Full path to static access key data.--features
: List of enabled features.--snapshot-location-config
: Availability zone where disk snapshots will be located.- (Optional)
--use-restic
: Enables the restic plugin.
Result:
CustomResourceDefinition/backups.velero.io: attempting to create resource CustomResourceDefinition/backups.velero.io: already exists, proceeding CustomResourceDefinition/backups.velero.io: created ... Velero is installed! ⛵ Use 'kubectl logs deployment/velero -n velero' to view the status.
-
Make sure the Velero pod status has changed to
Running
:kubectl get pods --namespace velero
-
-
Make sure the data backup is displayed in the new Managed Service for Kubernetes cluster:
velero backup get
Result:
NAME STATUS ERRORS WARNINGS CREATED EXPIRES STORAGE LOCATION SELECTOR my-backup Completed 0 0 2020-10-19 17:13:25 +0300 MSK 29d default <none>
-
Restore data from the backup:
velero restore create my-restore \ --exclude-namespaces velero \ --from-backup my-backup
Where:
--exclude-namespaces
: List of namespaces to exclude from the recovery process.--from-backup
: Name of the backup for recovery.
Result:
Restore request "my-restore" submitted successfully. Run `velero restore describe my-restore` or `velero restore logs my-restore` for more details.
-
Wait for the backup restoration to complete. The value displayed in the
STATUS
field will beCompleted
.velero get restore
Result:
NAME BACKUP STATUS STARTED COMPLETED ERRORS WARNINGS CREATED SELECTOR my-restore my-backup Completed 2020-10-20 14:04:55 +0300 MSK 2020-10-20 14:05:22 +0300 MSK 0 23 2020-10-20 14:04:55 +0300 MSK <none>
Delete the resources you created
If you no longer need the resources you created, delete them:
- Delete the clusters Managed Service for Kubernetes.
- If you reserved public static IP addresses for the Managed Service for Kubernetes clusters, delete them.
- Delete the Object Storage bucket.
- Delete the service account used to work with Velero.
-
In the terminal window, go to the directory containing the infrastructure plan.
-
Delete the
velero-backup.tf
configuration file. -
Make sure the Terraform configuration files are correct using this command:
terraform validate
If there are any errors in the configuration files, Terraform will point them out.
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
This will delete all the resources described in the
velero-backup.tf
configuration file. -