Copying data from Managed Service for OpenSearch to Managed Service for Greenplum® using Yandex Data Transfer

With Data Transfer, you can transfer data from a Managed Service for OpenSearch source cluster to a Managed Service for Greenplum® target cluster.

To transfer data:

1. Get your cloud ready.
2. Set up your infrastructure.
3. Prepare the test data.
4. Prepare and activate your transfer.
5. Test the transfer.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The infrastructure support costs include:

• Fee for Managed Service for OpenSearch cluster computing resources and storage volume (see Managed Service for OpenSearch pricing).
• Fee for Managed Service for Greenplum® cluster computing resources, storage volume, and backups (see Managed Service for Greenplum® pricing).

Set up your infrastructureSet up your infrastructure

Prepare the test dataPrepare the test data

1. In the source cluster, create a test index named people and set its schema:

   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_username>:<user_password_in_source_cluster> \
        --header 'Content-Type: application/json' \
        --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people' && \
   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_username>:<user_password_in_source_cluster> \
        --header 'Content-Type: application/json' \
        --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_mapping?pretty' -d'
        {
              "properties": {
                 "name": {"type": "text"},
                 "age": {"type": "integer"}
              }
        }
        '
   

2. Populate the test index with data:

   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_username>:<user_password_in_source_cluster> \
        --header 'Content-Type: application/json' \
        --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_doc/?pretty' -d'
        {
              "name": "Alice",
              "age": "30"
        }
        ' && \
   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_username>:<user_password_in_source_cluster> \
        --header 'Content-Type: application/json' \
        --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_doc/?pretty' -d'
        {
              "name": "Robert",
              "age": "32"
        }
        '
   

3. Make sure the data is saved in the test index:

   curl --cacert ~/.opensearch/root.crt \
        --user <username_in_target_cluster>:<user_password_in_target_cluster> \
        --header 'Content-Type: application/json' \
        --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_search?pretty'
   

Prepare and activate your transferPrepare and activate your transfer

1. Create a source endpoint for the Managed Service for OpenSearch cluster you created earlier with the following settings:

   • Database type: OpenSearch.
   • Connection settings:
     • Connection type: Managed Service for OpenSearch cluster.
     • Managed Service for OpenSearch cluster: Select a Managed Service for OpenSearch cluster from the list.
     • User: admin.
     • Password: admin user password.

2. Create a target endpoint for the Managed Service for Greenplum® cluster you created earlier, with the following settings:

   • Database type: Greenplum.
   • Endpoint parameters:
     • Connection type: Select Managed Service for Greenplum cluster.
     • Managed Service for Greenplum cluster: Select a Managed Service for Greenplum® cluster from the list.
     • Database: postgres.
     • User: Enter a name for the Managed Service for Greenplum® cluster user.
     • Password: Enter a password for the Managed Service for Greenplum® cluster user.

3. Create a transfer:

   Manually

   Terraform

   1. Create a transfer of the Snapshot type that will use the created endpoints.
   2. Activate the transfer.

   1. In the opensearch-to-greenplum.tf file, specify the following settings:

      • source_endpoint_id: Source endpoint ID.
      • target_endpoint_id: Target endpoint ID.
      • transfer_enabled: 1 to create a transfer.

   2. Make sure the Terraform configuration files are correct using this command:

      terraform validate
      

      If there are any errors in the configuration files, Terraform will point them out.

   3. Create the required infrastructure:

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

      Once created, your transfer will be activated automatically.

Test the transferTest the transfer

1. Wait for the transfer status to change to Completed.

2. Make sure the data from the source Managed Service for OpenSearch cluster has been migrated to the Managed Service for Greenplum® cluster:

   1. Get an SSL certificate to connect to the Managed Service for Greenplum® cluster.

   2. Install the dependencies:

      sudo apt update && sudo apt install --yes postgresql-client
      

   3. Connect to the database in the Managed Service for Greenplum® cluster.

   4. Make sure the database contains the people table with test data:

      SELECT * FROM people;
      

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need: