Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Managing backups in Managed Service for Elasticsearch

Written by
Updated at November 18, 2024

Warning

Yandex Managed Service for Elasticsearch is unavailable as of April 11, 2024.

You can create an OpenSearch cluster in Yandex Cloud as an alternative to Elasticsearch.

Managed Service for Elasticsearch enables you to create index backups using both Yandex Cloud tools and the Elasticsearch snapshot mechanism. For more information about snapshots, see the Elasticsearch documentation.

Creating backups with Yandex Cloud tools

You can create backups and restore clusters from existing backups.

Managed Service for Elasticsearch also creates automatic hourly backups.

Getting a list of backups

To get a list of cluster backups:

  1. Go to the folder page and select Managed Service for Elasticsearch.
  2. Click the cluster name and open the Backups tab.

To get a list of all backups in a folder:

  1. Go to the folder page and select Managed Service for Elasticsearch.
  2. Select the Backups tab.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of Elasticsearch cluster backups available in the default folder, run the command:

yc managed-elasticsearch backup list

Result:

+----------------------+----------------------+----------------------+----------------------+
|          ID          |      CREATED AT      |  SOURCE CLUSTER ID   |      STARTED AT      |
+----------------------+----------------------+----------------------+----------------------+
| c9qlk4v13uq7******** | 2020-08-10T12:00:00Z | c9qlk4v13uq7******** | 2020-08-10T11:55:17Z |
| c9qpm90p3pcg******** | 2020-08-09T22:01:04Z | c9qpm90p3pcg******** | 2020-08-09T21:30:00Z |
+----------------------+----------------------+----------------------+----------------------+

To get a list of cluster backups, use the listBackups REST API method for the Cluster resource or the ClusterService/ListBackups gRPC API call and provide the cluster ID in the clusterId parameter of your request.

To get a list of backups for all the Managed Service for Elasticsearch clusters in the folder, use the list REST API method for the Backup resource or the BackupService/List gRPC API call and provide the folder ID in the folderId parameter of your request.

You can get the cluster ID with a list of clusters in the folder.

Getting information about backups

To get information about the backup of an existing cluster:

  1. Go to the folder page and select Managed Service for Elasticsearch.
  2. Click the cluster name and open the Backups tab.

To get information about the backup of a previously deleted cluster:

  1. Go to the folder page and select Managed Service for Elasticsearch.
  2. Select the Backups tab.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get information about a Elasticsearch cluster backup, run the command:

yc managed-elasticsearch backup get <backup_ID>

You can retrieve the backup ID with a list of backups.

To get information about a backup, use the get REST API method for the Backup resource or the BackupService/Get gRPC API call and provide the backup ID in the backupId parameter of your request.

To find out the ID, retrieve a list of backups.

Creating a backup

  1. Go to the folder page and select Managed Service for Elasticsearch.
  2. Click the cluster name and open the Backups tab.
  3. Click Create backup.

The service will start creating a backup without an additional confirmation.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a cluster backup:

  1. View a description of the CLI create Elasticsearch backup command:

    yc managed-elasticsearch cluster backup --help

  2. Request a backup to be created by specifying the cluster name or ID:

    yc managed-elasticsearch cluster backup <cluster_name_or_ID>

    You can get the cluster ID and name with a list of clusters.

To create a backup, use the backup REST API method for the Cluster resource or the ClusterService/Backup gRPC API call and provide the cluster ID in the clusterId parameter of your request.

You can get the cluster ID with a list of clusters in the folder.

Warning

While you are creating your backup, the cluster performance might degrade.

Restoring clusters from backups

When you restore a cluster from a backup, you create a new cluster with the backup data. If the folder has insufficient resources to create such a cluster, you will not be able to restore from the backup. You can manage the restore speed using Elasticsearch tools.

When creating a new cluster, set all required parameters.

To restore an existing cluster from a backup:

  1. Go to the folder page and select Managed Service for Elasticsearch.
  2. Click the cluster name and open the Backups tab.
  3. Click for the backup you need, then click Restore cluster.
  4. Set up the new cluster. You can select a folder for the new cluster from the Folder list.
  5. Click Restore cluster.

To restore a previously deleted cluster from a backup:

  1. Go to the folder page and select Managed Service for Elasticsearch.
  2. Select the Backups tab.
  3. Find the backup you need using the backup creation time and cluster ID. The Name column contains IDs formatted as <cluster_ID>:<backup_ID>.
  4. Click for the backup you need, then click Restore cluster.
  5. Set up the new cluster. You can select a folder for the new cluster from the Folder list.
  6. Click Restore cluster.

Managed Service for Elasticsearch will launch the operation to create a cluster from the backup.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To restore a cluster from a backup:

  1. View a description of the CLI restore Elasticsearch cluster command:

    yc managed-elasticsearch cluster restore --help

  2. Getting a list of available Elasticsearch cluster backups:

    yc managed-elasticsearch backup list

    Result:

    +--------------------------+----------------------+----------------------+----------------------+
|            ID            |      CREATED AT      |  SOURCE CLUSTER ID   |      STARTED AT      |
+--------------------------+----------------------+----------------------+----------------------+
| c9qlk4v13uq79r9********  | 2020-08-10T12:00:00Z | c9qlk4v13uq7******** | 2020-08-10T11:55:17Z |
| ...                                                                                           |
+--------------------------+----------------------+----------------------+----------------------+

    The backup completion time is shown in the CREATED AT column of the list of available backups, in yyyy-mm-ddThh:mm:ssZ format (2020-08-10T12:00:00Z in the example above).

  3. Request the creation of a cluster from a backup:

    yc managed-elasticsearch cluster restore \
   --backup-id=<backup_ID> \
   --name=<cluster_name> \
   --environment=<environment> \
   --network-name=<network_name> \
   --host zone-id=<availability_zone>,`
         `subnet-id=<subnet_ID>,`
         `assign-public-ip=<public_access>,`
         `type=<host_role> \
   --datanode-resource-preset=<Data_node_host_class> \
   --datanode-disk-size=<Data_node_storage_size_in_GB> \
   --datanode-disk-type=<Data_node_disk_type> \
   --masternode-resource-preset=<Master_node_host_class> \
   --masternode-disk-size=<Master_node_storage_size_in_GB> \
   --masternode-disk-type=<Master_node_disk_type> \
   --admin-password=<admin_user_password>

    Where:

    • --backup-id: Backup ID.

    • --name: Cluster name.

    • --environment: Environment:

      • PRESTABLE: For testing purposes. The prestable environment is similar to the production environment and likewise covered by the SLA, but it is the first to get new functionalities, improvements, and bug fixes. In the prestable environment, you can test compatibility of new versions with your application.
      • PRODUCTION: For stable versions of your apps.

    • --network-name: Network name.

    • --host: Host parameters:

      • zone-id: Availability zone.

      • subnet-name: Subnet name. Specify if two or more subnets are created in the selected availability zone.

      • assign-public-ip: Flag to set if public access to the host is required. It can be either true or false.

      • type: Host role: datanode or masternode.

    • --datanode-resource-preset: Class of hosts with the Data node role.

    • --datanode-disk-size: Storage size in GB for hosts with the Data node role.

    • --datanode-disk-type: Disk type for hosts with the Data node role:

      • network-hdd
      • network-ssd
      • local-ssd
      • network-ssd-nonreplicated

    • --masternode-resource-preset: Class of hosts with the Master node role.

    • --masternode-disk-size: Storage size in GB for hosts with the Master node role.

    • --masternode-disk-type: Disk type for hosts with the Master node role. Only network-ssd is available.

    • --admin-password: admin user password.

      This is a special user that is required for managing clusters and cannot be deleted. It is assigned the superuser role and can perform any operations on clusters.

      Tip

      To perform routine tasks, we recommend creating individual users. For more information, see Managing users.

To restore a cluster from a backup, use the restore REST API method for the Cluster resource or the ClusterService/Restore gRPC API call and provide the following in the request:

  • ID of the backup you need, in the backupId parameter. To find out the ID, get a list of cluster backups.
  • Name of the new cluster that will contain the data recovered from the backup, in the name parameter. It must be unique within the folder.
  • Cluster environment in the environment parameter.
  • Cluster configuration in the configSpec parameter.
  • Configuration of the cluster hosts in one or more hostSpecs parameters.
  • Network ID in the networkId parameter.

Backups using snapshots

To work with snapshots, use the Elasticsearch public API and a bucket in Object Storage to store them.

Retrieving a snapshot list

  1. Find the repository containing snapshot backups in the Elasticsearch repository list:

    GET https://admin:<password>@<host_FQDN>:9200/_snapshot/_all

    If the repository you need is not on the list, connect it.

  2. Get a list of snapshots in the repository:

    GET https://admin:<password>@<host_FQDN>:9200/_snapshot/<repository>/_all

    Each snapshot is a single backup.

Creating a snapshot

  1. In the Elasticsearch repository list, find the repository where you want to create the snapshot:

    GET https://admin:<password>@<host_FQDN>:9200/_snapshot/_all

    If the repository you need is not on the list, connect it.

  2. Create a snapshot of the required data or cluster in the selected repository:

    PUT https://admin:<password>@<host_FQDN>:9200/_snapshot/<repository>/<snapshot>

Restoring a cluster from a snapshot

Warning

When restoring from snapshots, the following restrictions apply:

  • The Elasticsearch version in the cluster must be equal to or higher than the Elasticsearch version where the snapshot was taken.
  • To restore the indexes, be sure that the major Elasticsearch version in the cluster is no more than one unit higher than the major Elasticsearch version where the snapshot was taken. For example, you can use version 6.0 (but not 7.0) to recover indexes created in version 5.0.

  1. Create a new Elasticsearch cluster in the required configuration, but do not populate it with data.

    When creating a cluster, select:

    • The number and class of hosts as well as the storage size and disk type based on snapshot size and performance requirements. If necessary, upgrade your host class or increase cluster storage size.

    • The Elasticsearch version used to make the snapshot or higher.

  2. Close the open indexes using the Elasticsearch API:

    POST: https://admin:<password>@<host_FQDN>:9200/<index>/_close

    To restore an entire cluster, close all open indexes. To restore individual indexes, close only those indexes.

  3. Retrieve a list of backups and find the required snapshot.

  4. Start restoring an entire cluster or individual data indexes and streams from the required snapshot.

For more information about restoring from snapshots, see the Elasticsearch documentation.

Previous
Migrating hosts to a different availability zone
Next
Configuring access to Object Storage