Migrating data from a third-party Elasticsearch cluster using the Reindex API

Managed Service for Elasticsearch clusters support reindexing via the Reindex API. You can specify another Elasticsearch cluster as your data reindexing source. This type of reindexing is referred to as remote. It can be used to move existing indexes, aliases, or data streams from a third party source to a Managed Service for Elasticsearch target cluster.

To migrate data from the source cluster in Elasticsearch to the target cluster in Managed Service for Elasticsearch:

1. Configure the target cluster.
2. Start reindexing.

If you no longer need the resources you created, delete them.

Getting startedGetting started

1. Create a Managed Service for Elasticsearch target cluster with any suitable configuration.

   Enable cluster host access via public IPs.

   Warning

   The Elasticsearch version on the target cluster must be the same as or higher than on the source cluster.

2. Make sure you can connect to the Managed Service for Elasticsearch target cluster using the Elasticsearch API and Kibana.

3. Make sure the Elasticsearch source cluster can access the internet.

4. If a source cluster is using a self-signed certificate to encrypt the connection, add it as an extension to the target cluster.

5. Create a user with the monitoring_user and the viewer roles in the target cluster.

Configure the target clusterConfigure the target cluster

1. Create a role with the create_index and write privileges for all indexes (*).

2. Create a user and assign this role to them.

   Tip

   In Managed Service for Elasticsearch clusters, you can use Reindex API as the admin user with the superuser role; however, it is more secure to create separate users with limited privileges for each job. For more information, see Managing Elasticsearch users.

3. Update DBMS settings:

   • Reindex remote whitelist: Specify the source cluster IP or FQDN, such as:

     192.168.0.1:9200, example.com:9200
     

   • (Optional) Reindex SSL CA path: Specify the local path to the imported certificate as /etc/elasticsearch/extensions/<extension_name>/<certificate_name>.

Start reindexingStart reindexing

1. Retrieve the list of hosts in the target cluster.

2. To start reindexing, run a request against the target cluster's host with the Master node role:

   curl --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
        --request POST "https://<IP_or_FQDN_of_target_cluster_Master_Node_host>:9200/_reindex?pretty" \
        --header 'Content-Type: application/json' \
        --data '{
          "source": {
            "remote": {
              "host": "https://<IP_or_FQDN_of_source_cluster_Master_Node_host>:9200",
              "username": "<username_in_the_source_cluster>",
              "password": "<user_password_in_the_source_cluster>"
            },
            "index": "<source_cluster_index_alias_or_data_stream_name>"
          },
          "dest": {
            "index": "<target_cluster_index_alias_or_data_stream_name>"
          }
        }'
   

   To transfer several indexes, use a for loop:

   for index in <space-separated_list_of_index_alias_or_data_stream_names>; do
     curl --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
          --request POST "https://<target_cluster_Mater_Node_IP_or_FQDN>:9200/_reindex?pretty" \
          --header 'Content-Type: application/json' \
          --data '{
            "source": {
              "remote": {
                "host": "https://<source_cluster_Mater_Node_IP_or_FQDN>:9200",
                "username": "<username_in_the_source_cluster>",
                "password": "<user_password_in_the_source_cluster>"
              },
              "index": "'$index'"
            },
            "dest": {
              "index": "'$index'"
            }
          }'
   done
   

Delete the resources you no longer needDelete the resources you no longer need

Delete the resources you no longer need to avoid paying for them:

• If you reserved public static IPs for cluster access, release and delete them.
• If you used a Yandex Object Storage bucket to import a self-signed certificate, clear and delete it.