Migrating to Managed Service for Elasticsearch using snapshots

Managed Service for Elasticsearch clusters support snapshots. This allows you to migrate data from another Elasticsearch cluster to them. For more information about snapshots, see the Elasticsearch documentation.

To migrate data from the Elasticsearch source cluster to the Managed Service for Elasticsearch target cluster:

1. Set up a work environment.
2. Create a snapshot in the source cluster.
3. Restore the snapshot in the target cluster.
4. Complete your migration.

If you no longer need the resources you are using, delete them.

Set up a working environmentSet up a working environment

Create the required resourcesCreate the required resources

Complete the configuration and check access to resourcesComplete the configuration and check access to resources

1. Set up the bucket ACL:

   1. In the Select a user drop-down list, specify the created service account.
   2. Select the READ and WRITE permissions for the selected service account.
   3. Click Add.
   4. Click Save.

2. Install the repository-s3 plugin on all source cluster hosts.

3. For the repository-s3 plugin to work, restart Elasticsearch and Kibana on all source cluster hosts.

4. Make sure you can connect to the Managed Service for Elasticsearch target cluster using the Elasticsearch API and Kibana.

5. Make sure the Elasticsearch source cluster can access the internet.

Create a snapshot on the source clusterCreate a snapshot on the source cluster

1. Connect the bucket as a snapshot repository on the source cluster:

   1. Add data about the static access key to the Elasticsearch keystore.

      Run the procedure on all hosts of the source cluster.

      Add the following:

      • Key ID:

        $ES_PATH/bin/elasticsearch-keystore add s3.client.default.access_key
        

      • Secret key:

        $ES_PATH/bin/elasticsearch-keystore add s3.client.default.secret_key
        

        Note

        The path to Elasticsearch ($ES_PATH) depends on the selected installation method. To find out the path to the installed Elasticsearch, see the installation documentation (for example, for DEB and RPM).

   2. Upload the data from the keystore:

      curl -X POST "https://<source_cluster_FQDN>:9200/_nodes/reload_secure_settings"
      

   3. Register the repository:

      curl "https://<source_cluster_FQDN>:9200/_snapshot/<repository_name>" \
           -X PUT \
           -H 'Content-Type: application/json' -d '
             {
               "type": "s3",
               "settings": {
                 "bucket": "<bucket_name>",
                 "endpoint": "storage.yandexcloud.net"
               }
             }'
      

   For more information about connecting the repository, see the plugin documentation.

   Alert

   If a bucket is registered in an Elasticsearch cluster as a snapshot repository, do not edit the bucket contents manually as this will disrupt the Elasticsearch snapshot mechanism.

2. Start creating a snapshot on the source cluster. You can create a snapshot of the entire cluster or some of the data. Place the snapshot in the repository created in the previous step. For more information, see the Elasticsearch documentation.

   Example of creating a snapshot named snapshot_1 for the entire cluster:

   curl -X PUT \
        "https://<source_cluster_FQDN>:9200/_snapshot/<repository_name>/snapshot_1?wait_for_completion=true&pretty"
   

   Creating a snapshot may take a long time. Track the progress of the operation using Elasticsearch tools, such as:

   curl -X GET \
        "https://<source_cluster_FQDN>:9200/_snapshot/<repository_name>/snapshot_1/_status?pretty"
   

Restore a snapshot on the target clusterRestore a snapshot on the target cluster

1. Configure access to the bucket with snapshots for the target cluster. Use the service account you previously created.

2. Connect the Object Storage bucket to the target cluster as a snapshot storage:

   curl "https://admin:<admin_user_password>@<host_FQDN>:9200/_snapshot/<repository_name>" \
        -X PUT \
        -H 'Content-Type: application/json' -d '
          {
            "type": "s3",
            "settings": {
              "bucket": "<bucket_name>",
              "endpoint": "storage.yandexcloud.net"
            }
          }'
   

3. Select how to restore an index on the target cluster.

   With the default settings, an attempt to restore an index will fail in a cluster where the same-name index is already open. Even in Managed Service for Elasticsearch clusters without user data, there are open system indexes (such as .apm-custom-link or .kibana_*, etc.), which may interfere with the restore operation. To avoid this, use one of the restore policies:

   • Existing system indices are not migrated. Only indices created on the source cluster by the user are involved in the import process.

   • Delete and restore: Existing indexes are closed and deleted with new empty indexes created with the same names and then populated with data from a snapshot.

   • Rename on restore: Existing indexes are not affected, new indexes are created with other names. Snapshot data is restored to the new indexes.

   Warning

   Closing all indexes will make Kibana temporarily unavailable. Once the system indexes are opened, Kibana will be available again.

   For example, the following command closes all indexes in the target cluster:

   curl -X POST \
        "https://admin:<admin_user_password>@<target_cluster_FQDN>:9200/_all/_close?pretty"
   

   Example of restoring the entire snapshot:

   curl -X POST \
        "https://admin:<admin_user_password>@<target_cluster_FQDN>:9200/_snapshot/<repository_name>/snapshot_1/_restore"
   

4. Start restoring data from the snapshot on the target cluster. You can restore the entire snapshot or individual indexes. For more information, see the Elasticsearch documentation.

   Example of restoring a snapshot with indication of the custom indexes to be restored on the target cluster:

   curl -X POST \
        -H 'Content-Type: application/json' 'https://admin:<admin_user_password>@<target_cluster_FQDN>:9200/_snapshot/<repository_name>/snapshot_1/_restore' -d '
        {
          "indices": "<list_of_indexes>"
        }'
   

   Where list of indexes is a list of comma-separated indexes to restore, e.g., my_index*, my_index_2.*. Transferring only the user indexes will enable you to avoid errors when restoring the snapshot. System indexes are not affected.

   Restoring a snapshot may take a long time. Track the progress of the operation using Elasticsearch tools, such as:

   curl -X GET \
        "https://admin:<admin_user_password>@<target_cluster_FQDN>:9200/_snapshot/<repository_name>/snapshot_1/_status?pretty"
   

5. If necessary, after the restore operation is completed, open all closed indexes.

   For example, the following command opens all indexes in the target cluster:

   curl -X POST \
        "https://admin:<admin_user_password>@<target_cluster_FQDN>:9200/_all/_open?pretty"
   

Complete your migrationComplete your migration

1. Make sure that all required data has been transferred to the Managed Service for Elasticsearch target cluster.

   You can check this, for example, using Kibana.

2. If necessary, disable the snapshot repository on the side of the source and target clusters.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need: