Contact UsGet started
© 2024 Direct Cursus Technology L.L.C.

Managing backups in Managed Service for OpenSearch

Written by
Updated at December 11, 2024

Managed Service for OpenSearch enables you to create index backups using both Yandex Cloud tools and the OpenSearch snapshot mechanism. For more information about snapshots, see the OpenSearch documentation.

Creating backups with Yandex Cloud tools

You can create backups and restore clusters from existing backups.

Managed Service for OpenSearch also creates automatic hourly backups.

Getting a list of backups

You can get a list of backups created for the past 14 days.

To get a list of cluster backups:

  1. Go to the folder page and select Managed Service for OpenSearch.
  2. Click the name of the cluster you need and select the  Backups tab.

To get a list of all backups in a folder:

  1. Go to the folder page and select Managed Service for OpenSearch.
  2. Select  Backups.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of all backups in the folder, run this command:

yc managed-opensearch backup list

Result:

+----------------------+---------------------+-------------------+---------------------+
|          ID          |      CREATED AT     | SOURCE CLUSTER ID |      STARTED AT     |
+----------------------+---------------------+-------------------+---------------------+
| c9qlk4v13uq7******** | 2024-01-09 14:38:34 | c9qpm4i********   | 2024-01-09 14:38:28 |
| c9qpm90p3pcg******** | 2024-01-09 13:38:31 | c9qpm4i********   | 2024-01-09 13:38:28 |
+----------------------+---------------------+-------------------+---------------------+

If you want to limit the backup list displayed after running the command, include the --limit <number_of_records> flag in the command. For example, if the output of the yc managed-opensearch backup list command takes up several screens, run the yc managed-opensearch backup list --limit 5 command. In this case, the output will contain the list of five most recent backups.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. To get a list of cluster backups:

    1. Use the Cluster.ListBackups method and make a request, e.g., via cURL:

      curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters/<cluster_ID>/backups'

      You can get the cluster ID with a list of clusters in the folder.

    2. View the server response to make sure the request was successful.

  3. To get a list of backups for all the clusters in a folder:

    1. Use the Backup.List method and make a request, e.g., via cURL:

      curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/backups' \
    --url-query folderId=<folder_ID>

      You can request the folder ID with the list of folders in the cloud.

    2. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. To get a list of cluster backups:

    1. Use the ClusterService.ListBackups call and make a request, e.g., via gRPCurl:

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.opensearch.v1.ClusterService.ListBackups

      You can get the cluster ID with a list of clusters in the folder.

    2. View the server response to make sure the request was successful.

  4. To get a list of backups for all the clusters in a folder:

    1. Use the BackupService.List call and make a request, e.g., via gRPCurl:

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/backup_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "folder_id": "<folder_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.opensearch.v1.BackupService.List

      You can request the folder ID with the list of folders in the cloud.

    2. View the server response to make sure the request was successful.

Getting information about backups

To get information about the backup of an existing cluster:

  1. Go to the folder page and select Managed Service for OpenSearch.
  2. Click the name of the cluster you need and select the  Backups tab.

To get information about the backup of a previously deleted cluster:

  1. Go to the folder page and select Managed Service for OpenSearch.
  2. Select  Backups.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get information about the backup of a cluster:

  1. Retrieve the backup ID with a list of all backups in the folder:

    yc managed-opensearch backup list

    You will see the ID in the ID column of the command output.

  2. Get information about the backup you need:

    yc managed-opensearch backup get <backup_ID>

    Command output example:

    id: c9qlk4v13uq7********
folder_id: b1g86q4m5vej********
source_cluster_id: c9qpm4i********
started_at: "2024-01-09T10:38:28.683Z"
created_at: "2024-01-09T10:38:31.685Z"
indices:
  - .mdb-sli
  - .opendistro_security
  - .kibana_1
  - .opendistro-job-scheduler-lock
  - .opensearch-observability
  - .opendistro-ism-config
opensearch_version: 2.8.0
indices_total: "6"

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Use the Backup.Get method and make a request, e.g., via cURL:

    curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/backups/<backup_ID>'

    You can get the backup ID together with a list of backups.

  3. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Use the BackupService.Get call and make a request, e.g., via gRPCurl:

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/backup_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "backup_id": "<backup_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.opensearch.v1.BackupService.Get

    You can get the backup ID together with a list of backups.

  4. View the server response to make sure the request was successful.

Creating a backup

  1. Go to the folder page and select Managed Service for OpenSearch.
  2. Click the name of the cluster you need and select the  Backups tab.
  3. Click  Create backup.

The service will start creating a backup without an additional confirmation.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a backup of cluster data, run this command:

yc managed-opensearch cluster backup <cluster_name_or_ID>

You can request the cluster name and ID with the list of clusters in the folder.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Use the Cluster.Backup method and make a request, e.g., via cURL:

    curl \
    --request POST \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters/<cluster_ID>:backup'

    You can get the cluster ID with a list of clusters in the folder.

  3. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Use the ClusterService.Backup call and make a request, e.g., via gRPCurl:

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.opensearch.v1.ClusterService.Backup

    You can get the cluster ID with a list of clusters in the folder.

  4. View the server response to make sure the request was successful.

Warning

While you are creating your backup, the cluster performance might degrade.

Restoring clusters from backups

When you restore a cluster from a backup, you create a new cluster with the backup data. If the folder has insufficient resources to create such a cluster, you will not be able to restore from the backup.

When creating a new cluster, set all required parameters.

Before you begin, make sure your Yandex Cloud account has the iam.serviceAccounts.user role or higher. You will need this role if the cluster you want to restore is linked to a service account.

To restore an existing cluster from a backup:

  1. Go to the folder page and select Managed Service for OpenSearch.
  2. Click the name of the cluster you need and select the  Backups tab.
  3. Click for the backup you need and click Restore cluster.
  4. Set up the new cluster.
  5. Click Restore cluster.

To restore a previously deleted cluster from a backup:

  1. Go to the folder page and select Managed Service for OpenSearch.
  2. Select  Backups.
  3. Find the backup you need using the backup creation time and cluster ID. The ID column contains IDs formatted as <cluster_ID>:<backup_IP>.
  4. Click for the backup you need and click Restore cluster.
  5. Set up the new cluster.
  6. Click Restore cluster.

Managed Service for OpenSearch will launch the operation to create a cluster from the backup.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To restore a cluster from a backup:

  1. Retrieve the backup ID with a list of all backups in the folder:

    yc managed-opensearch backup list

    Result:

    +----------------------+---------------------+-------------------+---------------------+
|          ID          |      CREATED AT     | SOURCE CLUSTER ID |      STARTED AT     |
+----------------------+---------------------+-------------------+---------------------+
| c9qlk4v13uq7******** | 2024-01-09 14:38:34 | c9qpm4i********   | 2024-01-09 14:38:28 |
| ...                                                                                  |
+----------------------+---------------------+-------------------+---------------------+

    You will see the backup ID in the ID column. The CREATED AT column shows the backup completion time in yyyy-mm-dd hh:mm:ss format.

  2. Request creating a cluster from a backup:

    yc managed-opensearch cluster restore --backup-id <backup_ID>

    You can also run the command with the parameters you use when creating a cluster. For the description of such parameters, see Creating a cluster.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Create a file named body.json and add the following contents to it:

    {
    "backupId": "<backup_ID>",
    "folderId": "<folder_ID>",
    "name": "<cluster_name>",
    "environment": "<environment>",
    "networkId": "<network_ID>",
    "configSpec": {
        "version": "<OpenSearch_version>",
        "adminPassword": "<admin_user_password>",
        "opensearchSpec": {
            "nodeGroups": [
                {
                    "name": "<OpenSearch_host_group_name>",
                    "resources": {
                        "resourcePresetId": "<host_class>",
                        "diskSize": "<storage_size_in_bytes>",
                        "diskTypeId": "<disk_type>"
                    },
                    "roles": ["<role_1>","<role_2>"],
                    "hostsCount": "<number_of_hosts>",
                    "zoneIds": [
                        "<availability_zone_1>",
                        "<availability_zone_2>",
                        "<availability_zone_3>"
                    ],
                    "subnetIds": [
                        "<subnet_1_ID>",
                        "<subnet_2_ID>",
                        "<subnet_3_ID>"
                    ]
                }
            ]
        },
        "dashboardsSpec": {
            "nodeGroups": [
                {
                    "name": "<Dashboards_host_group_name>",
                    "resources": {
                        "resourcePresetId": "<host_class>",
                        "diskSize": "<storage_size_in_bytes>",
                        "diskTypeId": "<disk_type>"
                    },
                    "hostsCount": "<number_of_hosts>",
                    "zoneIds": [
                        "<availability_zone_1>",
                        "<availability_zone_2>",
                        "<availability_zone_3>"
                    ],
                    "subnetIds": [
                        "<subnet_1_ID>",
                        "<subnet_2_ID>",
                        "<subnet_3_ID>"
                    ]
                }
            ]
        }
    }
}

    Where:

    • backupId: ID of the backup copy you are restoring the cluster from. You can get the ID together with a list of backups.

    • folderId: Folder ID. You can request it with a list of folders in the cloud.

    • name: Cluster name.

    • environment: Cluster environment, PRODUCTION or PRESTABLE.

    • networkId: ID of the network the cluster will be in.

    • configSpec: Cluster settings:

      • version: OpenSearch version.

      • adminPassword: admin user password.

      • opensearchSpec: Settings for OpenSearch host groups. Contain an array of nodeGroups elements. one for each host group. Each element has the following structure:

        • name: Host group name.

        • resources: Cluster resources:

        • roles: List of host roles. A cluster must include at least one group of DATA hosts and one group of MANAGER hosts. This can be a single group with the two roles, or different groups with the roles.

        • hostsCount: Number of hosts in the group. It must have at least one DATA host and at least three MANAGER hosts.

        • zoneIds: List of availability zones the cluster hosts are located in.

        • subnetIds: List of subnet IDs.

      • dashboardsSpec: Settings for Dashboards host groups. Contains an array of nodeGroups elements, the array structure matches opensearchSpec.nodeGroups structure. The roles parameter is the only exception: Dashboards hosts can only have one role, DASHBOARDS, so you do not need to specify it directly.

  3. Use the Cluster.Restore method and make a request, e.g., via cURL:

    curl \
    --request POST \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters:restore' \
    --data "@body.json"

  4. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Create a file named body.json and add the following contents to it:

    {
    "backup_id": "<backup_ID>",
    "folder_id": "<folder_ID>",
    "name": "<cluster_name>",
    "environment": "<environment>",
    "network_id": "<network_ID>",
    "config_spec": {
        "version": "<OpenSearch_version>",
        "admin_password": "<admin_user_password>",
        "opensearch_spec": {
            "node_groups": [
                {
                    "name": "<OpenSearch_host_group_name>",
                    "resources": {
                        "resource_preset_id": "<host_class>",
                        "disk_size": "<storage_size_in_bytes>",
                        "disk_type_id": "<disk_type>"
                    },
                    "roles": ["<role_1>","<role_2>"],
                    "hosts_count": "<number_of_hosts>",
                    "zone_ids": [
                        "<availability_zone_1>",
                        "<availability_zone_2>",
                        "<availability_zone_3>"
                    ],
                    "subnet_ids": [
                        "<subnet_1_ID>",
                        "<subnet_2_ID>",
                        "<subnet_3_ID>"
                    ]
                }
            ]
        },
        "dashboards_spec": {
            "node_groups": [
                {
                    "name": "<Dashboards_host_group_name>",
                    "resources": {
                        "resource_preset_id": "<host_class>",
                        "disk_size": "<storage_size_in_bytes>",
                        "disk_type_id": "<disk_type>"
                    },
                    "hosts_count": "<number_of_hosts>",
                    "zone_ids": [
                        "<availability_zone_1>",
                        "<availability_zone_2>",
                        "<availability_zone_3>"
                    ],
                    "subnet_ids": [
                        "<subnet_1_ID>",
                        "<subnet_2_ID>",
                        "<subnet_3_ID>"
                    ]
                }
            ]
        }
    }
}

    Where:

    • backup_id: ID of the backup copy you are restoring the cluster from. You can get the ID together with a list of backups.

    • folder_id: Folder ID. You can request it with a list of folders in the cloud.

    • name: Cluster name.

    • environment: Cluster environment, PRODUCTION or PRESTABLE.

    • network_id: ID of the network the cluster will be in.

    • config_spec: Cluster settings:

      • version: OpenSearch version.

      • admin_password: admin user password.

      • opensearch_spec: Settings for OpenSearch host groups. Contain an array of nodeGroups elements. one for each host group. Each element has the following structure:

        • name: Host group name.

        • resources: Cluster resources:

        • roles: List of host roles. A cluster must include at least one group of DATA hosts and one group of MANAGER hosts. This can be a single group with the two roles, or different groups with the roles.

        • hosts_count: Number of hosts in the group. It must have at least one DATA host and at least three MANAGER hosts.

        • zone_ids: List of availability zones the cluster hosts are located in.

        • subnet_ids: List of subnet IDs.

      • dashboards_spec: Settings for Dashboards host groups. Contains an array of node_groups elements, the array structure matches opensearch_spec.node_groups structure. The roles parameter is the only exception: Dashboards hosts can only have one role, DASHBOARDS, so you do not need to specify it directly.

  4. Use the ClusterService.Restore call and make a request, e.g., via gRPCurl:

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d @ \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.opensearch.v1.ClusterService.Restore \
    < body.json

  5. View the server response to make sure the request was successful.

Backups using snapshots

To work with snapshots, use the public API OpenSearch and a bucket in Object Storage to store them.

Retrieving a snapshot list

  1. Find the repository containing snapshot backups in the OpenSearch repository list:

    GET https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/_all

    If the required repository is not on the list, connect it.

  2. Get a list of snapshots in the repository:

    GET https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/_all

    Each snapshot is a single backup.

Creating a snapshot

  1. In the OpenSearch repository list, find the repository where you want to create the snapshot:

    GET https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/_all

    If the required repository is not on the list, connect it.

  2. Create a snapshot of the required data or cluster in the selected repository:

    PUT https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/<snapshot_name>

Restoring a cluster from a snapshot

Warning

When restoring a cluster from a snapshot, the OpenSearch version in the cluster must be equal to or higher than the OpenSearch version where the snapshot was taken.

  1. Create a new OpenSearch cluster in the required configuration, but do not populate it with data.

    When creating a cluster, select:

    • The number and class of hosts as well as the size and type of storage based on snapshot size and performance requirements.

    • The OpenSearch version used to make the snapshot or higher.

  2. Close the open indexes using the OpenSearch API:

    POST: https://admin:<password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/<index_name>/_close

    To restore an entire cluster, close all open indexes. To restore individual indexes, close only those indexes.

  3. Retrieve a list of backups and find the required snapshot.

  4. Start restoring an entire cluster or individual data indexes and streams from the required snapshot.

Previous
Stopping and starting a cluster
Next
Configuring access to Object Storage