Copying data from Managed Service for OpenSearch to Yandex MPP Analytics for PostgreSQL using Yandex Data Transfer

With Data Transfer, you can transfer data from a Managed Service for OpenSearch source cluster to a Greenplum® target cluster in Yandex MPP Analytics for PostgreSQL.

To transfer data:

1. Get your cloud ready.
2. Set up your infrastructure.
3. Prepare your test data.
4. Set up and activate the transfer.
5. Test your transfer.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

• Managed Service for OpenSearch cluster: Use of computing resources and storage size (see Managed Service for OpenSearch pricing).
• Yandex MPP Analytics for PostgreSQL cluster: Computing resources allocated to hosts, storage and backup size (see Yandex MPP Analytics for PostgreSQL pricing).
• Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Set up your infrastructureSet up your infrastructure

Prepare your test dataPrepare your test data

1. In the source cluster, create a test index named people and define its schema:

   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_user_name>:<source_cluster_user_password> \
        --header 'Content-Type: application/json' \
        --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people' && \
   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_user_name>:<source_cluster_user_password> \
        --header 'Content-Type: application/json' \
        --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_mapping?pretty' -d'
        {
              "properties": {
                 "name": {"type": "text"},
                 "age": {"type": "integer"}
              }
        }
        '
   

2. Populate the test index with data:

   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_user_name>:<source_cluster_user_password> \
        --header 'Content-Type: application/json' \
        --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_doc/?pretty' -d'
        {
              "name": "Alice",
              "age": "30"
        }
        ' && \
   curl --cacert ~/.opensearch/root.crt \
        --user <source_cluster_user_name>:<source_cluster_user_password> \
        --header 'Content-Type: application/json' \
        --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_doc/?pretty' -d'
        {
              "name": "Robert",
              "age": "32"
        }
        '
   

3. Make sure the data is saved in the test index:

   curl --cacert ~/.opensearch/root.crt \
        --user <username_in_target_cluster>:<user_password_in_target_cluster> \
        --header 'Content-Type: application/json' \
        --request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/people/_search?pretty'
   

Set up and activate the transferSet up and activate the transfer

1. Create a source endpoint for your pre-configured Managed Service for OpenSearch cluster using the following settings:

   • Database type: OpenSearch.
   • Connection settings:
     • Connection type: Managed Service for OpenSearch cluster.
     • Managed Service for OpenSearch cluster: Select the Managed Service for OpenSearch cluster from the list.
     • User: admin.
     • Password: admin password.

2. Create a target endpoint for the Greenplum® cluster you created earlier, with the following settings:

   • Database type: Greenplum.
   • Endpoint parameters:
     • Connection type: Select Managed Service for Greenplum cluster.
     • Managed Service for Greenplum cluster: Select the Greenplum® cluster from the list.
     • Database: postgres.
     • User: Enter the Greenplum® cluster user name.
     • Password: Enter the Greenplum® cluster user password.

3. Create a transfer:

   Manually

   Terraform

   1. Create a Snapshot-type transfer configured to use the new endpoints.
   2. Activate the transfer.

   1. In the opensearch-to-greenplum.tf file, specify the following settings:

      • source_endpoint_id: Source endpoint ID.
      • target_endpoint_id: Target endpoint ID.
      • transfer_enabled: 1 to create a transfer.

   2. Validate your Terraform configuration files using this command:

      terraform validate
      

      Terraform will display any configuration errors detected in your files.

   3. Create the required infrastructure:

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

      The transfer will activate automatically upon creation.

Test your transferTest your transfer

1. Wait for the transfer status to change to Completed.

2. Make sure the data from the source Managed Service for OpenSearch cluster has been migrated to the Yandex MPP Analytics for PostgreSQL cluster:

   1. Get an SSL certificate to connect to the Greenplum® cluster.

   2. Install the dependencies:

      sudo apt update && sudo apt install --yes postgresql-client
      

   3. Connect to the database in the Greenplum® cluster.

   4. Check that the database contains the people table populated with test data:

      SELECT * FROM people;
      

Delete the resources you createdDelete the resources you created

To reduce the consumption of resources you do not need, delete them: