Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch

There are three mechanisms to migrate data from a source Elasticsearch cluster to a target Yandex Managed Service for OpenSearch cluster:

• Yandex Data Transfer service

  This method is good for any Elasticsearch cluster.

  For an example of this kind of migration, see Migrating data to OpenSearch using Yandex Data Transfer.

• Snapshots

  This method is good for Elasticsearch cluster versions 7.11 or lower.

  To learn more about snapshots, see the OpenSearch documentation.

• Remote reindexing (reindex data)

  You can use this mechanism to move your existing indexes, aliases, or data streams. This method is good for all Elasticsearch clusters of version 7.

Migration using snapshotsMigration using snapshots

To migrate data from a source cluster in Elasticsearch to a target cluster in Managed Service for OpenSearch using snapshots:

1. Create a snapshot in the source cluster.
2. Restore the snapshot in the target cluster.
3. Complete your migration.

If you no longer need the resources you are using, delete them.

Getting startedGetting started

Prepare the infrastructurePrepare the infrastructure

Complete the configuration and check access to resourcesComplete the configuration and check access to resources

1. Set up the bucket ACL:

   1. In the Select a user drop-down list, specify the created service account.
   2. Select the READ and WRITE permissions for the selected service account.
   3. Click Add.
   4. Click Save.

2. Set up the Elasticsearch source cluster:

   1. Install the plugin repository-s3 on all cluster hosts.

   2. For the repository-s3 plugin to work, restart the Elasticsearch and Kibana services on all cluster hosts.

   3. Make sure the Elasticsearch source cluster can access the internet.

3. Install an SSL certificate.

4. Make sure you can connect to the target Managed Service for OpenSearch cluster using the OpenSearch API and Dashboards.

Create a snapshot on the source clusterCreate a snapshot on the source cluster

1. Connect the bucket as a snapshot repository on the source cluster:

   1. Add the static access key information to the Elasticsearch keystore (keystore).

      Note

      Run the procedure on all hosts of the source cluster.

      Add the following:

      • Key ID:

        $ES_PATH/bin/elasticsearch-keystore add s3.client.default.access_key
        

      • Secret key:

        $ES_PATH/bin/elasticsearch-keystore add s3.client.default.secret_key
        

      Note

      The path to Elasticsearch ($ES_PATH) depends on the selected installation method. To find a path to your Elasticsearch installation, see the installation documentation (for example, for DEB, RPM).

   2. Upload the data from the keystore:

      curl --request POST "https://<IP_or_FQDN_of_source_cluster_DATA_host>:9200/_nodes/reload_secure_settings"
      

   3. Register the repository:

      curl --request PUT \
           "https://<IP_or_FQDN_of_source_cluster_DATA_host>:9200/_snapshot/<repository_name>" \
           --header 'Content-Type: application/json' \
           --data '{
             "type": "s3",
             "settings": {
               "bucket": "<bucket_name>",
               "endpoint": "storage.yandexcloud.net"
             }
           }'
      

   To learn more about adding the repository, see the plugin documentation.

   Alert

   If a bucket is registered in an Elasticsearch cluster as a snapshot repository, do not edit the bucket contents manually as this will disrupt the Elasticsearch snapshot mechanism.

2. Run the snapshot creation in the repository created in the previous step. You can create a snapshot of the entire cluster or some of the data. For more information, see the Elasticsearch documentation.

   Example of creating a snapshot with the snapshot_1 name for the entire cluster:

   curl --request PUT \
        "https://<IP_or_FQDN_of_the_DATA_host_in_the_source_cluster>:9200/_snapshot/<repository_name>/snapshot_1?wait_for_completion=false&pretty"
   

   Creating a snapshot may take a long time. Track the operation progress using the Elasticsearch tools, for example:

   curl --request GET \
        "https://<IP_or_FQDN_of_the_DATA_host_in_the_source_cluster>:9200/_snapshot/<repository_name>/snapshot_1/_status?pretty"
   

Restore a snapshot on the target clusterRestore a snapshot on the target cluster

1. Configure access to the bucket with snapshots for the target cluster. Use the service account you previously created.

2. Attach an Object Storage bucket to the target cluster. This bucket will be used as a read-only snapshot storage:

   curl --request PUT \
        "https://admin:<admin_user_password>@<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>" \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --data '{
          "type": "s3",
          "settings": {
            "bucket": "<bucket_name>",
            "readonly" : "true",
            "endpoint": "storage.yandexcloud.net"
          }
        }'
   

3. Select how to restore an index on the target cluster.

   With the default settings, an attempt to restore an index will fail in a cluster where the same-name index is already open. Even in Managed Service for OpenSearch clusters without user data, there are open system indexes (such as .apm-custom-link or .kibana_*), which may interfere with the restore operation. To avoid this, use one of the following methods:

   • Migrate only your custom indexes. The existing system indexes are not migrated. The import process only affects the user-created indexes on the source cluster.

   • Use the rename_pattern and rename_replacement parameters. Indexes will be renamed as they are restored. To learn more, see the OpenSearch documentation.

   Example of restoring the entire snapshot:

   curl --request POST \
        "https://admin:<admin_user_password>@<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/snapshot_1/_restore" \
        --cacert ~/.opensearch/root.crt
   

4. Start restoring data from the snapshot on the target cluster.

   Example of restoring a snapshot with indication of the custom indexes to be restored on the target cluster:

   curl --request POST \
        "https://admin:<admin_user_password>@<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/snapshot_1/_restore?wait_for_completion=false&pretty" \
        --cacert ~/.opensearch/root.crt \
        --header 'Content-Type: application/json' \
        --data '{
          "indices": "<list_of_indexes>"
        }'
   

   Where indices is a list of comma-separated indexes to be restored, e.g., my_index*, my_index_2.*.

   Restoring a snapshot may take a long time. To check the restoring status, run this command:

   curl --request GET \
        "https://admin:<admin_user_password>@<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/snapshot_1/_status?pretty" \
        --cacert ~/.opensearch/root.crt
   

Complete your migrationComplete your migration

1. Make sure all the indexes you need have been transferred to the target Managed Service for OpenSearch cluster, and the number of documents in them is the same as in the source cluster:

   Bash

   OpenSearch Dashboards

   Run this command:

   curl \
       --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
       --cacert ~/.opensearch/root.crt \
       --request GET 'https://<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_cat/indices?v'
   

   The list should contain the indexes transferred from Elasticsearch with the number of documents specified in the docs.count column.

   1. Connect to the target cluster using OpenSearch Dashboards.
   2. Select the Global tenant.
   3. Open the control panel by clicking .
   4. Under OpenSearch Plugins, select Index Management.
   5. Go to Indexes.

   The list should contain the indexes transferred from Elasticsearch with the number of documents specified in the Total documents column.

2. Disable the snapshot repository on the source and target clusters, if required.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Migration using reindexingMigration using reindexing

To migrate data from a source cluster in Elasticsearch to a target cluster in Managed Service for OpenSearch through reindexing:

1. Configure the target cluster.
2. Start reindexing.
3. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

1. Prepare the infrastructure:

   Manually

   Using Terraform

   Create a target Managed Service for OpenSearch cluster in the relevant configuration with public access to a group of hosts with the DATA role.

   1. If you do not have Terraform yet, install it.

   2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

   3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

   4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

   5. Download the es-mos-migration-reindex.tf configuration file to the same working directory. The file describes:

      • Network.
      • Subnet.
      • Security group and rules required to connect to a Managed Service for OpenSearch cluster.
      • Managed Service for OpenSearch target cluster.

   6. In the es-mos-migration-reindex.tf file, specify these variables:

      • os_admin_password: OpenSearch admin user password.
      • os_version: OpenSearch version.

   7. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   8. Create the required infrastructure:

      1. Run the command to view planned changes:

         terraform plan
         

         If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

         1. Run the command:

            terraform apply
            

         2. Confirm the update of resources.

         3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

2. Install an SSL certificate:

   Linux (Bash)/macOS (Zsh)

   Windows (PowerShell)

   mkdir -p ~/.opensearch && \
   wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
        --output-document ~/.opensearch/root.crt && \
   chmod 0600 ~/.opensearch/root.crt
   

   The certificate will be saved to the ~/.opensearch/root.crt file.

   mkdir $HOME\.opensearch; curl --output $HOME\.opensearch\root.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
   

   The certificate will be saved to the $HOME\.opensearch\root.crt file.

   Corporate policies and antivirus software can block the download of certificates. For more information, see FAQ.

3. Make sure you can connect to the target Managed Service for OpenSearch cluster using the OpenSearch API and Dashboards.

4. Make sure the Elasticsearch source cluster can access the internet.

5. In the source cluster, create a user with the monitoring_user and viewer roles.

Configure the target clusterConfigure the target cluster

1. Create a role with the create_index and write privileges for all indexes (*).

2. Create a user and assign this role to them.

   Tip

   In Managed Service for OpenSearch clusters, you can run re-indexing as the admin user with the superuser role; however, it is more secure to create separate users with limited privileges for each job. For more information, see Managing OpenSearch users.

Start reindexingStart reindexing

1. Retrieve the list of hosts in the target cluster.

2. To start reindexing, run the request against the target cluster's host with the DATA role:

   curl --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
        --cacert ~/.opensearch/root.crt \
        --request POST \
        "https://<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_reindex?wait_for_completion=false&pretty" \
        --header 'Content-Type: application/json' \
        --data '{
          "source": {
            "remote": {
              "host": "https://<source_cluster_DATA_host_IP_or_FQDN>:9200",
              "username": "<username_in_the_source_cluster>",
              "password": "<user_password_in_the_source_cluster>"
            },
            "index": "<source_cluster_index_alias_or_data_stream_name>"
          },
          "dest": {
            "index": "<target_cluster_index_alias_or_data_stream_name>"
          }
        }'
   

   Result:

   {
     "task" : "<reindexing_job_ID>"
   }
   

   To transfer several indexes, use a for loop:

   for index in <space-separated_list_of_index_alias_or_data_stream_names>; do
     curl --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
          --cacert ~/.opensearch/root.crt \
          --request POST \
          "https://<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_reindex?wait_for_completion=false&pretty" \
          --header 'Content-Type: application/json' \
          --data '{
            "source": {
              "remote": {
                "host": "https://<source_cluster_DATA_host_IP_or_FQDN>:9200",
                "username": "<username_in_the_source_cluster>",
                "password": "<user_password_in_the_source_cluster>"
              },
              "index": "'$index'"
            },
            "dest": {
              "index": "'$index'"
            }
          }'
   done
   

   Result:

   {
     "task" : "<ID_of_reindexing_job_1>"
   }
   {
     "task" : "<ID_of_reindexing_job_2>"
   }
   ...
   

   To learn more about reindexing parameters, see the OpenSearch documentation.

   Reindexing may take a long time. To check the operation status, run this command:

   curl --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
        --cacert ~/.opensearch/root.crt \
        --request GET \
        "https://<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_tasks/<reindexing_job_ID>"
   

3. To cancel reindexing, run this command:

   curl --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
        --cacert ~/.opensearch/root.crt \
        --request POST \
        "https://<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_tasks/<reindexing_job_ID>/_cancel"
   

Check the resultCheck the result

Make sure all the indexes you need have been transferred to the target Managed Service for OpenSearch cluster, and the number of documents in them is the same as in the source cluster:

curl \
    --user <username_in_the_target_cluster>:<user_password_in_the_target_cluster> \
    --cacert ~/.opensearch/root.crt \
    --request GET 'https://<OpenSearch_DATA_host_ID>.mdb.yandexcloud.net:9200/_cat/indices?v'

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

• Delete the resources depending on how they were created:

  Manually

  Using Terraform

  Delete the Managed Service for OpenSearch cluster.

  1. In the terminal window, go to the directory containing the infrastructure plan.

  2. Delete the es-mos-migration-reindex.tf configuration file.

  3. Make sure the Terraform configuration files are correct using this command:

     terraform validate
     

     If there are any errors in the configuration files, Terraform will point them out.

  4. Confirm updating the resources.

     1. Run the command to view planned changes:

        terraform plan
        

        If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

     2. If you are happy with the planned changes, apply them:

        1. Run the command:

           terraform apply
           

        2. Confirm the update of resources.

        3. Wait for the operation to complete.

     All the resources described in the es-mos-migration-reindex.tf configuration file will be deleted.

• If you reserved public static IPs for cluster access, release and delete them.