Managing backups in Managed Service for OpenSearch
Managed Service for OpenSearch enables you to create index backups using both the Yandex Cloud tools and the OpenSearch snapshot mechanism. For more information about snapshots, see this OpenSearch guide
Creating backups with the Yandex Cloud tools
You can create backups and use existing backups to restore clusters.
Managed Service for OpenSearch also creates automatic hourly backups.
Getting a list of backups
You can get a list of backups created for the past 14 days.
To get a list of cluster backups:
- In the management console
, navigate to the relevant folder. - Navigate to the Managed Service for OpenSearch service.
- Click the name of your cluster and select the
Backups tab.
To get a list of all backups in a folder:
- In the management console
, navigate to the relevant folder. - Navigate to the Managed Service for OpenSearch service.
- Select
Backups.
If you do not have the Yandex Cloud CLI installed yet, install and initialize it.
By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.
To get a list of all backups in a folder, run this command:
yc managed-opensearch backup list
Result:
+----------------------+---------------------+-------------------+---------------------+
| ID | CREATED AT | SOURCE CLUSTER ID | STARTED AT |
+----------------------+---------------------+-------------------+---------------------+
| c9qlk4v13uq7******** | 2024-01-09 14:38:34 | c9qpm4i******** | 2024-01-09 14:38:28 |
| c9qpm90p3pcg******** | 2024-01-09 13:38:31 | c9qpm4i******** | 2024-01-09 13:38:28 |
+----------------------+---------------------+-------------------+---------------------+
If you want to limit the backup list displayed after running the command, provide --limit <number_of_records> in the command. For example, if the output of the yc managed-opensearch backup list command takes up several screens, run the yc managed-opensearch backup list --limit 5 command. In this case, the output will contain the list of five most recent backups.
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
To get a list of cluster backups:
-
Call the Cluster.ListBackups method, e.g., via the following cURL
request:curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters/<cluster_ID>/backups'You can get the cluster ID with the list of clusters in the folder.
-
View the server response to make sure your request was successful.
-
-
To get a list of backups for all clusters in a folder:
-
Call the Backup.List method, e.g., via the following cURL
request:curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/backups' \ --url-query folderId=<folder_ID>You can get the folder ID with the list of folders in the cloud.
-
View the server response to make sure your request was successful.
-
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
To get a list of cluster backups:
-
Call the ClusterService.ListBackups method, e.g., via the following gRPCurl
request:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.opensearch.v1.ClusterService.ListBackupsYou can get the cluster ID with the list of clusters in the folder.
-
View the server response to make sure your request was successful.
-
-
To get a list of backups for all clusters in a folder:
-
Call the BackupService.List method, e.g., via the following gRPCurl
request:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/backup_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "folder_id": "<folder_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.opensearch.v1.BackupService.ListYou can get the folder ID with the list of folders in the cloud.
-
View the server response to make sure your request was successful.
-
Getting backup information
To get information about a backup of an existing cluster:
- In the management console
, navigate to the relevant folder. - Navigate to the Managed Service for OpenSearch service.
- Click the name of your cluster and select the
Backups tab.
To get information about a backup of a previously deleted cluster:
- In the management console
, navigate to the relevant folder. - Navigate to the Managed Service for OpenSearch service.
- Select
Backups.
If you do not have the Yandex Cloud CLI installed yet, install and initialize it.
By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.
To get information about a cluster backup:
-
Get the backup ID with the list of all backups in the folder:
yc managed-opensearch backup listYou will see the ID in the
IDcolumn of the command output. -
Get information about the backup you need:
yc managed-opensearch backup get <backup_ID>Command output example:
id: c9qlk4v13uq7******** folder_id: b1g86q4m5vej******** source_cluster_id: c9qpm4i******** started_at: "2024-01-09T10:38:28.683Z" created_at: "2024-01-09T10:38:31.685Z" indices: - .mdb-sli - .opendistro_security - .kibana_1 - .opendistro-job-scheduler-lock - .opensearch-observability - .opendistro-ism-config opensearch_version: 2.8.0 indices_total: "6"
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
Call the Backup.Get method, e.g., via the following cURL
request:curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/backups/<backup_ID>'You can get the backup ID with the list of backups.
-
View the server response to make sure your request was successful.
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
Call the BackupService.Get method, e.g., via the following gRPCurl
request:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/backup_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "backup_id": "<backup_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.opensearch.v1.BackupService.GetYou can get the backup ID with the list of backups.
-
View the server response to make sure your request was successful.
Creating a backup
- In the management console
, navigate to the relevant folder. - Navigate to the Managed Service for OpenSearch service.
- Click the name of your cluster and select the
Backups tab. - Click
Create backup.
This will initiate creating a backup without any additional confirmation.
If you do not have the Yandex Cloud CLI installed yet, install and initialize it.
By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.
To create a backup of cluster data, run this command:
yc managed-opensearch cluster backup <cluster_name_or_ID>
You can get the cluster name and ID with the list of clusters in the folder.
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
Call the Cluster.Backup method, e.g., via the following cURL
request:curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters/<cluster_ID>:backup'You can get the cluster ID with the list of clusters in the folder.
-
View the server response to make sure your request was successful.
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
Call the ClusterService.Backup method, e.g., via the following gRPCurl
request:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.opensearch.v1.ClusterService.BackupYou can get the cluster ID with the list of clusters in the folder.
-
View the server response to make sure your request was successful.
Warning
While you are creating a backup, the cluster performance may degrade.
Restoring a cluster from a backup
Warning
For clusters running an unsupported DBMS version, restoring from backups is not available.
When you restore a cluster from a backup, you create a new cluster with the backup data. If your folder lacks resources to create such a cluster, you will not be able to restore your data from the backup.
When creating a cluster, specify all the required settings.
Before you begin, assign the following roles to your Yandex Cloud account:
- managed-opensearch.restorer or higher for the backup folder and the new cluster folder.
- iam.serviceAccounts.user or higher if restoring a cluster attached to a service account.
To restore an existing cluster from a backup:
- In the management console
, navigate to the relevant folder. - Navigate to the Managed Service for OpenSearch service.
- Click the name of your cluster and select the
Backups tab. - Click
for the backup you need and then click Restore cluster. - Configure the new cluster.
- Click Restore cluster.
To restore a previously deleted cluster from a backup:
- In the management console
, navigate to the relevant folder. - Navigate to the Managed Service for OpenSearch service.
- Select
Backups. - Find the backup you need using the backup creation time and cluster ID. The ID column contains IDs in
<cluster_ID>:<backup_ID>format. - Click
for the backup you need and then click Restore cluster. - Configure the new cluster.
- Click Restore cluster.
Managed Service for OpenSearch will start creating the cluster from the backup.
If you do not have the Yandex Cloud CLI installed yet, install and initialize it.
By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.
To restore a cluster from a backup:
-
Get the backup ID with the list of all backups in the folder:
yc managed-opensearch backup listResult:
+----------------------+---------------------+-------------------+---------------------+ | ID | CREATED AT | SOURCE CLUSTER ID | STARTED AT | +----------------------+---------------------+-------------------+---------------------+ | c9qlk4v13uq7******** | 2024-01-09 14:38:34 | c9qpm4i******** | 2024-01-09 14:38:28 | | ... | +----------------------+---------------------+-------------------+---------------------+You will see the backup ID in the
IDcolumn. TheCREATED ATcolumn shows the backup completion time inyyyy-mm-dd hh:mm:ssformat. -
Make the following request to create a cluster from a backup:
yc managed-opensearch cluster restore --backup-id <backup_ID>You can also run this command, providing the parameters you set when creating a cluster. For the description of these parameters, see Creating a cluster.
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
Create a file named
body.jsonand paste the following code into it:{ "backupId": "<backup_ID>", "folderId": "<folder_ID>", "name": "<cluster_name>", "environment": "<environment>", "networkId": "<network_ID>", "configSpec": { "version": "<OpenSearch_version>", "adminPassword": "<admin_password>", "opensearchSpec": { "nodeGroups": [ { "name": "<OpenSearch_host_group_name>", "resources": { "resourcePresetId": "<host_class>", "diskSize": "<storage_size_in_bytes>", "diskTypeId": "<disk_type>" }, "roles": ["<role_1>","<role_2>"], "hostsCount": "<number_of_hosts>", "zoneIds": [ "<availability_zone_1>", "<availability_zone_2>", "<availability_zone_3>" ], "subnetIds": [ "<subnet_1_ID>", "<subnet_2_ID>", "<subnet_3_ID>" ] } ] }, "dashboardsSpec": { "nodeGroups": [ { "name": "<Dashboards_host_group_name>", "resources": { "resourcePresetId": "<host_class>", "diskSize": "<storage_size_in_bytes>", "diskTypeId": "<disk_type>" }, "hostsCount": "<number_of_hosts>", "zoneIds": [ "<availability_zone_1>", "<availability_zone_2>", "<availability_zone_3>" ], "subnetIds": [ "<subnet_1_ID>", "<subnet_2_ID>", "<subnet_3_ID>" ] } ] } } }Where:
-
backupId: ID of the backup you are restoring the cluster from. You can get it with the list of backups. -
folderId: Folder ID. You can get it with the list of folders in the cloud. -
name: Cluster name. -
environment: Cluster environment,PRODUCTIONorPRESTABLE. -
networkId: ID of the network where the cluster will be deployed. -
configSpec: Cluster settings:-
version: OpenSearch version. -
adminPassword:adminpassword.The password must include three groups of characters out of these four:
- Lowercase Latin letters
- Uppercase Latin letters
- Numbers
- Special characters
The password must be from 10 to 72 characters long.
-
opensearchSpec:OpenSearchhost group settings that contain an array ofnodeGroupselements, one per host group. Each element has the following structure:-
name: Host group name. -
resources: Cluster resources:resourcePresetId: Host class.diskSize: Disk size, in bytes.diskTypeId: Disk type.
-
roles: List of host roles. A cluster must include at least one group ofDATAhosts and one group ofMANAGERhosts. This can be a single group with two roles or multiple groups with different roles. -
hostsCount: Number of hosts in the group. The minimum number ofDATAhosts is one, while the minimum number ofMANAGERhosts is three. -
zoneIds: List of availability zones the cluster hosts are located in. -
subnetIds: List of subnet IDs.
-
-
dashboardsSpec:Dashboardshost group settings that contain an array ofnodeGroupselements that has the same structure asopensearchSpec.nodeGroups, except for therolesparameter. TheDashboardshosts have only one role,DASHBOARDS, so there is no need to specify it.
-
-
-
Call the Cluster.Restore method, e.g., via the following cURL
request:curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters:restore' \ --data "@body.json" -
View the server response to make sure your request was successful.
-
Get an IAM token for API authentication and put it in an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
Create a file named
body.jsonand paste the following code into it:{ "backup_id": "<backup_ID>", "folder_id": "<folder_ID>", "name": "<cluster_name>", "environment": "<environment>", "network_id": "<network_ID>", "config_spec": { "version": "<OpenSearch_version>", "admin_password": "<admin_password>", "opensearch_spec": { "node_groups": [ { "name": "<OpenSearch_host_group_name>", "resources": { "resource_preset_id": "<host_class>", "disk_size": "<storage_size_in_bytes>", "disk_type_id": "<disk_type>" }, "roles": ["<role_1>","<role_2>"], "hosts_count": "<number_of_hosts>", "zone_ids": [ "<availability_zone_1>", "<availability_zone_2>", "<availability_zone_3>" ], "subnet_ids": [ "<subnet_1_ID>", "<subnet_2_ID>", "<subnet_3_ID>" ] } ] }, "dashboards_spec": { "node_groups": [ { "name": "<Dashboards_host_group_name>", "resources": { "resource_preset_id": "<host_class>", "disk_size": "<storage_size_in_bytes>", "disk_type_id": "<disk_type>" }, "hosts_count": "<number_of_hosts>", "zone_ids": [ "<availability_zone_1>", "<availability_zone_2>", "<availability_zone_3>" ], "subnet_ids": [ "<subnet_1_ID>", "<subnet_2_ID>", "<subnet_3_ID>" ] } ] } } }Where:
-
backup_id: ID of the backup you are restoring the cluster from. You can get it with the list of backups. -
folder_id: Folder ID. You can get it with the list of folders in the cloud. -
name: Cluster name. -
environment: Cluster environment,PRODUCTIONorPRESTABLE. -
network_id: ID of the network where the cluster will be deployed. -
config_spec: Cluster settings:-
version: OpenSearch version. -
admin_password:adminpassword.The password must include three groups of characters out of these four:
- Lowercase Latin letters
- Uppercase Latin letters
- Numbers
- Special characters
The password must be from 10 to 72 characters long.
-
opensearch_spec:OpenSearchhost group settings that contain an array ofnodeGroupselements, one per host group. Each element has the following structure:-
name: Host group name. -
resources: Cluster resources:resource_preset_id: Host class.disk_size: Disk size, in bytes.disk_type_id: Disk type.
-
roles: List of host roles. A cluster must include at least one group ofDATAhosts and one group ofMANAGERhosts. This can be a single group with two roles or multiple groups with different roles. -
hosts_count: Number of hosts in the group. The minimum number ofDATAhosts is one, while the minimum number ofMANAGERhosts is three. -
zone_ids: List of availability zones the cluster hosts are located in. -
subnet_ids: List of subnet IDs.
-
-
dashboards_spec:Dashboardshost group settings that contain an array ofnode_groupselements that has the same structure asopensearch_spec.node_groups, except for therolesparameter. TheDashboardshosts have only one role,DASHBOARDS, so there is no need to specify it.
-
-
-
Call the ClusterService.Restore method, e.g., via the following gRPCurl
request:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d @ \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.opensearch.v1.ClusterService.Restore \ < body.json -
View the server response to make sure your request was successful.
Backing up using snapshots
Work with snapshots via the OpenSearch public API
Getting a snapshot list
-
Find the repository containing snapshot backups in the OpenSearch repository list:
GET https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/_allIf the repository you need is not on the list, add it.
-
Get a list of snapshots in the repository:
GET https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/_allEach snapshot is a single backup.
Creating a snapshot
-
In the OpenSearch repository list, find the repository where you want to create a snapshot backup:
GET https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/_allIf the repository you need is not on the list, add it.
-
Create a snapshot
of the specific data or cluster in this repository:PUT https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/<snapshot_name>
Restoring a cluster from a snapshot
Warning
When restoring a cluster from a snapshot, the OpenSearch version in the cluster must be the same or higher than the OpenSearch version where the snapshot was taken.
-
Create an OpenSearch cluster with the required configuration but do not populate it with data.
When creating a cluster, select:
-
Number and class of hosts as well as the storage size and disk type based on the snapshot size and performance requirements.
-
OpenSearch version used to make the snapshot or a higher one.
-
-
Close any open indexes using the OpenSearch API
:POST: https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/<index_name>/_closeTo restore an entire cluster, close all open indexes. To restore individual indexes, close only those indexes.
-
Get a list of backups and find the required snapshot.
-
Start restoring
an entire cluster or individual data indexes and streams from this snapshot.