Yandex Cloud
Search
Contact UsGet started
  • Blog
  • Pricing
  • Documentation
  • All Services
  • System Status
    • Featured
    • Infrastructure & Network
    • Data Platform
    • Containers
    • Developer tools
    • Serverless
    • Security
    • Monitoring & Resources
    • ML & AI
    • Business tools
  • All Solutions
    • By industry
    • By use case
    • Economics and Pricing
    • Security
    • Technical Support
    • Customer Stories
    • Cloud credits to scale your IT product
    • Gateway to Russia
    • Cloud for Startups
    • Education and Science
    • Yandex Cloud Partner program
  • Blog
  • Pricing
  • Documentation
© 2025 Direct Cursus Technology L.L.C.
Yandex Object Storage
    • All tutorials
    • Getting statistics on object queries with S3 Select
    • Getting website traffic statistics with S3 Select
    • Getting statistics on object queries using Yandex Query
    • Generating a resource-by-resource cost breakdown report using S3 Select
    • Server-side encryption
    • Integrating an L7 load balancer with CDN and Object Storage
    • Blue-green and canary deployment of service versions
    • Analyzing logs in DataLens
    • Mounting buckets to the file system of Yandex Data Processing hosts
    • Using Object Storage in Yandex Data Processing
    • Importing data from Object Storage, processing and exporting to Managed Service for ClickHouse®
    • Mounting a bucket as a disk in Windows
    • Migrating data from Yandex Data Streams using Yandex Data Transfer
    • Using hybrid storage in Yandex Managed Service for ClickHouse®
    • Loading data from Yandex Managed Service for OpenSearch to Yandex Object Storage using Yandex Data Transfer
    • Automatically copying objects from one bucket to another
    • Recognizing audio files in a bucket on a regular basis
    • Training a model in Yandex DataSphere on data from Object Storage
    • Connecting to Object Storage from VPC
    • Migrating data to Yandex Managed Service for PostgreSQL using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for Greenplum® using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for ClickHouse® using Yandex Data Transfer
    • Uploading data to Yandex Managed Service for YDB using Yandex Data Transfer
    • Exchanging data between Yandex Managed Service for ClickHouse® and Yandex Data Processing
    • Uploading data from Yandex Managed Service for YDB using Yandex Data Transfer
    • Hosting a static Gatsby website in Object Storage
    • Migrating a database from Managed Service for PostgreSQL to Object Storage
    • Exchanging data between Yandex Managed Service for ClickHouse® and Yandex Data Processing
    • Importing data from Yandex Managed Service for PostgreSQL to Yandex Data Processing using Sqoop
    • Importing data from Yandex Managed Service for MySQL® to Yandex Data Processing using Sqoop
    • Migrating data from Yandex Object Storage to Yandex Managed Service for MySQL® using Yandex Data Transfer
    • Migrating a database from Yandex Managed Service for MySQL® to Yandex Object Storage
    • Exporting Greenplum® data to a cold storage in Yandex Object Storage
    • Loading data from Yandex Direct to a Yandex Managed Service for ClickHouse® data mart using Yandex Cloud Functions, Yandex Object Storage, and Yandex Data Transfer
    • Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch
    • Uploading Terraform states to Object Storage
    • Locking Terraform states using Managed Service for YDB
    • Visualizing Yandex Query data
    • Publishing game updates
    • VM backups using Hystax Acura
    • Backing up to Object Storage with CloudBerry Desktop Backup
    • Backing up to Object Storage with Duplicati
    • Backing up to Object Storage with Bacula
    • Backing up to Yandex Object Storage with Veeam Backup
    • Backing up to Object Storage with Veritas Backup Exec
    • Managed Service for Kubernetes cluster backups in Object Storage
    • Developing a custom integration in API Gateway
    • URL shortener
    • Storing application runtime logs
    • Developing a skill for Alice and a website with authorization
    • Creating an interactive serverless application using WebSocket
    • Deploying a web application using the Java Servlet API
    • Developing a Telegram bot
    • Replicating logs to Object Storage using Fluent Bit
    • Replicating logs to Object Storage using Data Streams
    • Uploading audit logs to ArcSight SIEM
    • Exporting audit logs to SIEM Splunk systems
    • Creating an MLFlow server for logging experiments and artifacts
    • Operations with data using Yandex Query
    • Federated data queries using Query
    • Recognizing text in image archives using Vision OCR
    • Converting a video to a GIF in Python
    • Automating tasks using Managed Service for Apache Airflow™
    • Processing files with usage details in Yandex Cloud Billing
    • Deploying a web app with JWT authorization in API Gateway and authentication in Firebase
    • Searching for Yandex Cloud events in Yandex Query
    • Searching for Yandex Cloud events in Object Storage
    • Creating an external table from a bucket table using a configuration file
    • Migrating databases from Google BigQuery to Managed Service for ClickHouse®
  • Terraform reference
  • Monitoring metrics
  • Audit Trails events
  • Bucket logs
  • Release notes
  • FAQ

In this article:

  • Required paid resources
  • Migration using snapshots
  • Getting started
  • Create a snapshot on the source cluster
  • Restore a snapshot on the target cluster
  • Complete your migration
  • Delete the resources you created
  • Migration using reindexing
  • Getting started
  • Configure the target cluster
  • Start reindexing
  • Check the result
  • Delete the resources you created
  1. Tutorials
  2. Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch

Migrating data from Elasticsearch to Yandex Managed Service for OpenSearch

Written by
Yandex Cloud
Updated at May 7, 2025
  • Required paid resources
  • Migration using snapshots
    • Getting started
    • Create a snapshot on the source cluster
    • Restore a snapshot on the target cluster
    • Complete your migration
    • Delete the resources you created
  • Migration using reindexing
    • Getting started
    • Configure the target cluster
    • Start reindexing
    • Check the result
    • Delete the resources you created

Note

Yandex Managed Service for Elasticsearch is unavailable starting April 11, 2024.

There are three ways to migrate data from a source Elasticsearch cluster to a target Yandex Managed Service for OpenSearch cluster:

  • Yandex Data Transfer

    This method is good for any Elasticsearch cluster.

    For an example of this migration type, see Migrating data to OpenSearch using Yandex Data Transfer.

  • Snapshots

    This method is good for Elasticsearch cluster versions 7.11 or lower.

    For more information about snapshots, see the OpenSearch documentation.

  • Remote reindexing (reindex data).

    You can use this method to move your existing indexes, aliases, or data streams. This method is good for all version 7 Elasticsearch clusters.

Required paid resourcesRequired paid resources

The support cost includes:

  • Managed Service for OpenSearch cluster fee: Using computing resources allocated to hosts (including hosts with the MANAGER role) and disk space (see OpenSearch pricing).
  • Fee for public IP addresses for cluster hosts (see Virtual Private Cloud pricing).
  • Object Storage bucket fee: Storing data and performing operations with it (see Object Storage pricing).

Migration using snapshotsMigration using snapshots

To migrate data from a source Elasticsearch cluster to a target Managed Service for OpenSearch cluster using snapshots:

  1. Create a snapshot in the source cluster.
  2. Restore the snapshot in the target cluster.
  3. Complete your migration.

If you no longer need the resources you are using, delete them.

Getting startedGetting started

Set up your infrastructureSet up your infrastructure

Manually
Using Terraform
  1. Create an Object Storage bucket with restricted access. This bucket will serve as a snapshot repository.

  2. Create a service account and assign it the storage.editor role. A service account is required to access the bucket from the source and target clusters.

  3. If you are transferring data from a third-party Elasticsearch cluster, create a static access key for this service account.

    Warning

    Save the key ID and secret key. You will need these at the next steps.

  4. Create a target Managed Service for OpenSearch cluster in your desired configuration with the following settings:

    • Plugin: repository-s3.
    • Public access to a group of DATA hosts.
  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the es-mos-migration-snapshot.tf configuration file to the same working directory. This file describes:

    • Network.
    • Subnet.
    • Security group and rules required to connect to a Managed Service for OpenSearch cluster.
    • Service account for working with the Object Storage bucket.
    • Object Storage bucket.
    • Target Managed Service for OpenSearch cluster.
  6. In the es-mos-migration-snapshot.tf file, specify these variables:

    • folder_id: Cloud folder ID, same as in the provider settings.
    • bucket_name: Bucket name consistent with the naming conventions.
    • os_admin_password: OpenSearch admin password.
    • os_version: OpenSearch version.
  7. Make sure the Terraform configuration files are correct using this command:

    terraform validate
    

    If there are any errors in the configuration files, Terraform will point them out.

  8. Create the required infrastructure:

    1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply
        
      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Complete the configuration and check access to the resourcesComplete the configuration and check access to the resources

  1. Set up the bucket ACL:

    1. In the Select a user drop-down list, specify the created service account.
    2. Select the READ and WRITE permissions for the selected service account.
    3. Click Add.
    4. Click Save.
  2. Set up the source Elasticsearch cluster:

    1. Install the plugin repository-s3 on all cluster hosts.

    2. For the repository-s3 plugin to work, restart the Elasticsearch and Kibana services on all cluster hosts.

    3. Make sure the Elasticsearch source cluster can access the internet.

  3. Install an SSL certificate.

  4. Make sure you can connect to the target Managed Service for OpenSearch cluster using the OpenSearch API and Dashboards.

Create a snapshot on the source clusterCreate a snapshot on the source cluster

  1. Connect the bucket as a snapshot repository on the source cluster:

    1. Add the static access key information to the Elasticsearch keystore (keystore).

      Note

      Run the procedure on all hosts of the source cluster.

      Add the following:

      • Key ID:

        $ES_PATH/bin/elasticsearch-keystore add s3.client.default.access_key
        
      • Secret key:

        $ES_PATH/bin/elasticsearch-keystore add s3.client.default.secret_key
        

      Note

      The path to Elasticsearch ($ES_PATH) depends on the selected installation method. To find a path to your Elasticsearch installation, see the installation documentation (for example, for DEB, RPM).

    2. Upload the data from the keystore:

      curl --request POST "https://<IP_or_FQDN_of_source_cluster_DATA_host>:9200/_nodes/reload_secure_settings"
      
    3. Register the repository:

      curl --request PUT \
           "https://<IP_or_FQDN_of_source_cluster_DATA_host>:9200/_snapshot/<repository_name>" \
           --header 'Content-Type: application/json' \
           --data '{
             "type": "s3",
             "settings": {
               "bucket": "<bucket_name>",
               "endpoint": "storage.yandexcloud.net"
             }
           }'
      

    To learn more about connecting the repository, see the Elastic plugin documentation.

    Alert

    If a bucket is registered in an Elasticsearch cluster as a snapshot repository, do not edit the bucket contents manually as this will disrupt the Elasticsearch snapshot mechanism.

  2. Run snapshot creation in the repository you created at the previous step. You can create a snapshot of the entire cluster or some of the data. For more information, see this Elasticsearch guide on snapshots.

    Example of creating a snapshot named snapshot_1 for the entire cluster:

    curl --request PUT \
         "https://<IP_or_FQDN_of_the_DATA_host_in_the_source_cluster>:9200/_snapshot/<repository_name>/snapshot_1?wait_for_completion=false&pretty"
    

    Creating a snapshot may take a while. Track the operation progress using Elasticsearch tools, such as:

    curl --request GET \
         "https://<IP_or_FQDN_of_the_DATA_host_in_the_source_cluster>:9200/_snapshot/<repository_name>/snapshot_1/_status?pretty"
    

Restore a snapshot on the target clusterRestore a snapshot on the target cluster

  1. Configure access to the bucket with snapshots for the target cluster. Use the service account you created earlier.

  2. Attach an Object Storage bucket to the target cluster. This bucket will serve as a read-only snapshot storage:

    curl --request PUT \
         "https://admin:<admin_user_password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>" \
         --cacert ~/.opensearch/root.crt \
         --header 'Content-Type: application/json' \
         --data '{
           "type": "s3",
           "settings": {
             "bucket": "<bucket_name>",
             "readonly" : "true",
             "endpoint": "storage.yandexcloud.net"
           }
         }'
    
  3. Select the index restoration method on the target cluster.

    With the default settings, an attempt to restore an index will fail in a cluster where a same-name index is already open. Even in Managed Service for OpenSearch clusters without user data, there are open system indexes (such as .apm-custom-link or .kibana_*, etc.), which may interfere with the restore operation. To avoid this, use one of the following methods:

    • Migrate only your custom indexes. The existing system indexes are not migrated. The import process only affects the user-created indexes on the source cluster.

    • Use the rename_pattern and rename_replacement parameters. Indexes will be renamed as they are restored. For more information, see the relevant OpenSearch documentation.

    Example of restoring the entire snapshot:

    curl --request POST \
         "https://admin:<admin_user_password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/snapshot_1/_restore" \
         --cacert ~/.opensearch/root.crt
    
  4. Start restoring data from the snapshot on the target cluster.

    Example of restoring a snapshot with indication of the custom indexes to restore on the target cluster:

    curl --request POST \
         "https://admin:<admin_user_password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/snapshot_1/_restore?wait_for_completion=false&pretty" \
         --cacert ~/.opensearch/root.crt \
         --header 'Content-Type: application/json' \
         --data '{
           "indices": "<list_of_indexes>"
         }'
    

    Where indices is a list of comma-separated indexes to restore, e.g., my_index*, my_index_2.*.

    Restoring a snapshot may take a while. To check the restoring status, run this command:

    curl --request GET \
         "https://admin:<admin_user_password>@<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_snapshot/<repository_name>/snapshot_1/_status?pretty" \
         --cacert ~/.opensearch/root.crt
    

Complete your migrationComplete your migration

  1. Make sure all indexes have been transferred to the target Managed Service for OpenSearch cluster and the number of documents in them is the same as in the source cluster:

    Bash
    OpenSearch Dashboards

    Run this command:

    curl \
        --user <username_in_target_cluster>:<user_password_in_target_cluster> \
        --cacert ~/.opensearch/root.crt \
        --request GET 'https://<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_cat/indices?v'
    

    The list should contain the indexes transferred from Elasticsearch with the number of documents specified in the docs.count column.

    1. Connect to the target cluster using OpenSearch Dashboards.
    2. Select the Global tenant.
    3. Open the control panel by clicking .
    4. Under OpenSearch Plugins, select Index Management.
    5. Go to Indexes.

    The list should contain the indexes transferred from Elasticsearch with the number of documents specified in the Total documents column.

  2. If required, disable the snapshot repository on the source and target cluster end.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

Manually
Using Terraform
  • Delete the service account.
  • Delete the snapshots from the bucket and then, the entire bucket.
  • Delete the Managed Service for OpenSearch cluster.
  1. Delete all objects from the bucket.
  1. In the terminal window, go to the directory containing the infrastructure plan.

    Warning

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

  2. Delete resources:

    1. Run this command:

      terraform destroy
      
    2. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.

Migration using reindexingMigration using reindexing

To migrate data from a source Elasticsearch cluster to a target Managed Service for OpenSearch cluster by reindexing:

  1. Configure the target cluster.
  2. Start reindexing.
  3. Check the result.

If you no longer need the resources you created, delete them.

Getting startedGetting started

  1. Set up your infrastructure:

    Manually
    Using Terraform

    Create a target Managed Service for OpenSearch cluster in your desired configuration with public access to a group of hosts with the DATA role.

    1. If you do not have Terraform yet, install it.

    2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

    3. Configure and initialize a provider. There is no need to create a provider configuration file manually, you can download it.

    4. Place the configuration file in a separate working directory and specify the parameter values. If you did not add the authentication credentials to environment variables, specify them in the configuration file.

    5. Download the es-mos-migration-reindex.tf configuration file to the same working directory. This file describes:

      • Network.
      • Subnet.
      • Security group and rules required to connect to a Managed Service for OpenSearch cluster.
      • Target Managed Service for OpenSearch cluster.
    6. In the es-mos-migration-reindex.tf file, specify these variables:

      • os_admin_password: OpenSearch admin password.
      • os_version: OpenSearch version.
    7. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

    8. Create the required infrastructure:

      1. Run this command to view the planned changes:

        terraform plan
        

        If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

        1. Run this command:

          terraform apply
          
        2. Confirm updating the resources.

        3. Wait for the operation to complete.

      All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

  2. Install an SSL certificate:

    Linux (Bash)/macOS (Zsh)
    Windows (PowerShell)
    mkdir -p ~/.opensearch && \
    wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
         --output-document ~/.opensearch/root.crt && \
    chmod 0600 ~/.opensearch/root.crt
    

    The certificate will be saved to the ~/.opensearch/root.crt file.

    mkdir $HOME\.opensearch; curl.exe -o $HOME\.opensearch\root.crt https://storage.yandexcloud.net/cloud-certs/CA.pem
    

    The certificate will be saved to the $HOME\.opensearch\root.crt file.

    Corporate policies and antivirus software can block the download of certificates. For more information, see FAQ.

  3. Make sure you can connect to the target Managed Service for OpenSearch cluster using the OpenSearch API and Dashboards.

  4. Make sure the Elasticsearch source cluster can access the internet.

  5. Create a user with the monitoring_user and viewer roles in the target cluster.

Configure the target clusterConfigure the target cluster

  1. Create a role with the create_index and write privileges for all indexes (*).

  2. Create a user and assign this role to them.

    Tip

    In Managed Service for OpenSearch clusters, you can run re-indexing as the admin user with the superuser role; however, a more secure strategy is to create dedicated users with limited privileges for each job. For more information, see Managing OpenSearch users.

Start reindexingStart reindexing

  1. Retrieve the list of hosts in the target cluster.

  2. To start reindexing, run a request to the host with the DATA role in the target cluster:

    curl --user <username_in_target_cluster>:<user_password_in_target_cluster> \
         --cacert ~/.opensearch/root.crt \
         --request POST \
         "https://<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_reindex?wait_for_completion=false&pretty" \
         --header 'Content-Type: application/json' \
         --data '{
           "source": {
             "remote": {
               "host": "https://<IP_address_or_FQDN_of_host_with_DATA_role_in_source_cluster>:9200",
               "username": "<username_in_source_cluster>",
               "password": "<user_password_in_source_cluster>"
             },
             "index": "<name_of_index_alias_or_data_stream_in_source_cluster>"
           },
           "dest": {
             "index": "<name_of_index_alias_or_data_stream_in_target_cluster>"
           }
         }'
    

    Result:

    {
      "task" : "<ID_of_reindexing_job>"
    }
    

    To transfer several indexes, use a for loop:

    for index in <names_of_indexes_of_aliases_or_data_streams_separated_by_spaces>; do
      curl --user <username_in_target_cluster>:<user_password_in_target_cluster> \
           --cacert ~/.opensearch/root.crt \
           --request POST \
           "https://<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_reindex?wait_for_completion=false&pretty" \
           --header 'Content-Type: application/json' \
           --data '{
             "source": {
               "remote": {
                 "host": "https://<IP_address_or_FQDN_of_host_with_DATA_role_in_source_cluster>:9200",
                 "username": "<username_in_source_cluster>",
                 "password": "<user_password_in_source_cluster>"
               },
               "index": "'$index'"
             },
             "dest": {
               "index": "'$index'"
             }
           }'
    done
    

    Result:

    {
      "task" : "<ID_of_reindexing_job_1>"
    }
    {
      "task" : "<ID_of_reindexing_job_2>"
    }
    ...
    

    To learn more about reindexing parameters, see the OpenSearch documentation.

    Reindexing may take a while. To check the operation status, run this command:

    curl --user <username_in_target_cluster>:<user_password_in_target_cluster> \
         --cacert ~/.opensearch/root.crt \
         --request GET \
         "https://<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_tasks/<ID_of_reindexing_job>"
    
  3. To cancel reindexing, run this command:

    curl --user <username_in_target_cluster>:<user_password_in_target_cluster> \
         --cacert ~/.opensearch/root.crt \
         --request POST \
         "https://<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_tasks/<reindexing_job_ID>/_cancel"
    

Check the resultCheck the result

Make sure all indexes have been transferred to the target Managed Service for OpenSearch cluster and the number of documents in them is the same as in the source cluster:

Bash
OpenSearch Dashboards

Run this command:

curl \
    --user <username_in_target_cluster>:<user_password_in_target_cluster> \
    --cacert ~/.opensearch/root.crt \
    --request GET 'https://<ID_of_OpenSearch_host_with_DATA_role>.mdb.yandexcloud.net:9200/_cat/indices?v'

The list should contain the indexes transferred from Elasticsearch with the number of documents specified in the docs.count column.

  1. Connect to the target cluster using OpenSearch Dashboards.
  2. Select the Global tenant.
  3. Open the control panel by clicking .
  4. Under OpenSearch Plugins, select Index Management.
  5. Go to Indexes.

The list should contain the indexes transferred from Elasticsearch with the number of documents specified in the Total documents column.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  • Delete the objects from the bucket.

  • Delete the resources depending on how you created them:

    Manually
    Using Terraform

    Delete the Managed Service for OpenSearch cluster.

    1. In the terminal window, go to the directory containing the infrastructure plan.

      Warning

      Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

    2. Delete resources:

      1. Run this command:

        terraform destroy
        
      2. Confirm deleting the resources and wait for the operation to complete.

      All the resources described in the Terraform manifests will be deleted.

  • If you reserved public static IPs for cluster access, release and delete them.

Was the article helpful?

Previous
Loading data from Yandex Direct to a Yandex Managed Service for ClickHouse® data mart using Yandex Cloud Functions, Yandex Object Storage, and Yandex Data Transfer
Next
Uploading Terraform states to Object Storage
© 2025 Direct Cursus Technology L.L.C.