Transferring data to a OpenSearch target endpoint

Yandex Data Transfer enables you to migrate data to a OpenSearch database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

1. Explore possible data transfer scenarios.
2. Configure one of the supported data sources.
3. Prepare the OpenSearch database for the transfer.
4. Configure the target endpoint in Yandex Data Transfer.
5. Create a transfer and start it.
6. Perform required operations with the database and control the transfer.
7. In case of any issues, use ready-made solutions to resolve them.

Scenarios for transferring data to OpenSearchScenarios for transferring data to OpenSearch

1. Data delivery is a process of delivering arbitrary data to target storage. It includes data retrieval from a queue and its deserialization with subsequent transformation to target storage format.

   • Delivering data from Apache Kafka® to OpenSearch.

2. Migration: Moving data from one storage to another. Migration often means migrating a database from obsolete local databases to managed cloud ones.

   • Migrating an OpenSearch cluster.
   • Migration with change of storage from PostgreSQL to OpenSearch.

For a detailed description of possible Yandex Data Transfer scenarios, see Tutorials.

Configuring the data sourceConfiguring the data source

Configure one of the supported data sources:

• PostgreSQL
• YDS
• Apache Kafka®
• OpenSearch

For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.

Preparing the target databasePreparing the target database

Configuring the OpenSearch target endpointConfiguring the OpenSearch target endpoint

When creating or updating an endpoint, you can define:

• Yandex Managed Service for OpenSearch cluster connection or custom installation settings, including those based on Yandex Compute Cloud VMs. These are required parameters.
• Additional parameters.

Managed Service for OpenSearch clusterManaged Service for OpenSearch cluster

Connection with the cluster specified in Yandex Cloud.

Custom installationCustom installation

Connection to nodes with explicitly specified network addresses and ports.

Additional settingsAdditional settings

After configuring the data source and target, create and start the transfer.

Troubleshooting data transfer issuesTroubleshooting data transfer issues

• Transfer interrupted with an error message
• Document duplication on the target
• Exceeding the limit on the maximum number of fields
• Transfer failure with the mapper_parsing_exception error
• SSL is required error
• No tables found.

For more troubleshooting tips, see Troubleshooting.

Transfer failureTransfer failure

Error messages:

object field starting or ending with a [.] makes object resolution ambiguous <field_description>

Index -1 out of bounds for length 0

The transfer is aborted because the keys in the documents being transferred are not valid for the OpenSearch target. Invalid keys are empty keys and keys that:

• Consist of spaces.
• Consist of periods.
• Have a period at the beginning or end.
• Have two or more periods in a row.
• Include periods separated by spaces.

Solution:

In the target endpoint additional settings, enable Sanitize documents keys and reactivate the transfer.

Document duplication on the targetDocument duplication on the target

When repeatedly transferring data, documents get duplicated on the target.

All documents transferred from the same source table end up under the same index named <schemaName.tableName> on the target. In which case the target automatically generates document IDs (_id) by default. As a result, identical documents get different IDs and get duplicated.

There is no duplication if the primary keys are specified in the source table or endpoint conversion rules. Document IDs are then generated at the transfer stage using the primary key values.

Generation is performed as follows:

1. If the key value contains a period (.), it is escaped with \: some.key --> some\.key.
2. All the primary key values are converted into a string: <some_key1>.<some_key2>.<...>.
3. The resulting string is converted by the url.QueryEscape function.
4. If the resulting string does not exceed 512 characters in length, it is used as the _id. If longer than 512 characters, it is hashed with SHA-1, and the resulting hash is used as the _id.

As a result, documents with the same primary keys will receive the same ID when the data is transferred again, and the document transferred last will overwrite the existing one.

Solution:

1. Set the primary key for one or more columns in the source table or in the endpoint conversion rules.
2. Run the transfer.

Exceeding the maximum number of fields limitExceeding the maximum number of fields limit

Error message:

Limit of total fields [<limit_value>] has been exceeded

The transfer will be interrupted if the number of columns in the source database exceeds the maximum number of fields in the target database OpenSearch indexes.

Solution: Increase the maximum field number in the target database using the index.mapping.total_fields.limit parameter.

Transfer failure with the mapper_parsing_exception errorTransfer failure with the mapper_parsing_exception error

Error message:

mapper_parsing_exception failed to parse field [details.tags] of type [text]

The transfer is aborted due to incompatible data types at source and target.

Solution: Move the data to a new OpenSearch index with the details field type changed to flat_object.

1. Deactivate the transfer.

2. Create a new index in OpenSearch:

   curl \
   --user <OpenSearch_username>:<password> \
   --header 'Content-Type: application/json' \
   --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/<new_index_name>/_settings' \
   --data '{"index.mapping.total_fields.limit": 2000}'
   

3. Change the details field type:

   curl \
   --user <OpenSearch_username>:<password> \
   --header 'Content-Type: application/json' \
   --request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/<new_index_name>/_mapping' \
   --data '
       {
           "properties": {
               "details": {
                   "type": "flat_object"
               }
           }
       }'
   

4. Move the data from the source index to the new one:

   curl \
   --user <OpenSearch_username>:<password> \
   --header 'Content-Type: application/json' \
   --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_reindex' \
   --data '
       {
       "source":{
           "index":"<source_index_name>"
       },
       "dest":{
           "index":"<new_index_name>"
       }
       }'
   

5. Delete the source index:

   curl \
   --user <OpenSearch_username>:<password> \
   --header 'Content-Type: application/json' \
   --request DELETE 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/<source_index_name>'
   

6. Assign an alias to the new index:

   curl \
   --user <OpenSearch_username>:<password> \
   --header 'Content-Type: application/json' \
   --request POST 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_aliases' \
   --data '
       {
       "actions": [
           {
           "add": {
               "index": "<new_alias_name>",
               "alias": "<source_alias_name>"
           }
           }
       ]
       }'
   

SSL is required errorSSL is required error

This error occurs when connecting to a Managed Service for OpenSearch cluster as a custom installation via a OpenSearch host's FQDN if SSL is not enabled in the endpoint settings. By default, Managed Service for OpenSearch clusters require SSL encryption for connections via host FQDNs.

This error may also occur if you are connecting to a custom OpenSearch installation that requires SSL.

Solution:

Enable SSL in the endpoint settings.

For MDB clusters and other sources that use certificates issued by public CAs, you do not usually need to upload a CA certificate.

If your source uses a self-signed certificate, upload your CA certificate to the relevant field in the endpoint settings.

No tables foundNo tables found

Error message:

Unable to find any tables

This error may occur if the source has no available indexes or the specified user has no permissions for the indexes.

Solution:

• Check if there is an index. Make sure that the index name was specified correctly and that the source really has the index you want to transfer.

• Make sure the user has the required permissions to use the index.