Transferring data to an OpenSearch target endpoint

Make sure the number of columns in the source does not exceed the maximum number of fields in OpenSearch indexes. The maximum number of fields is provided in the index.mapping.total_fields.limit parameter. Its default value is 1,000.

To increase the parameter value, set up a template that makes the maximum number of fields in new indexes equal to the specified value.

curl \
--user <OpenSearch_username>:<password> \
--header 'Content-Type: application/json' \
--request PUT "https://<URL_of_OpenSearch_host_with_DATA_role>:9200/_template/index_defaults" \
--data '
    {
        "index_patterns": "cdc*",
        "settings": {
            "index": {
                "mapping": {
                    "total_fields": {
                        "limit": "2000"
                    }
                }
            }
        }
    }'

You can also set up templates using the OpenSearch Dashboards interface.

To check the current index.mapping.total_fields.limit parameter value, execute the following request:

curl \
    --user <OpenSearch_username>:<password> \
    --header 'Content-Type: application/json' \
    --request GET 'https://<URL_of_OpenSearch_host_with_DATA_role>:9200/<index name>/_settings/*total_fields.limit?include_defaults=true'

By default, when transferring data to a single index, only one host is used. To distribute the load across hosts when transferring large amounts of data, set up a template to split new indexes into shards in advance.

curl \
--user <OpenSearch_username>:<password> \
--header 'Content-Type: application/json' \
--request PUT 'https://<URL_of_OpenSearch_host_with_DATA_role>:9200/_template/index_defaults' \
--data '
    {
        "index_patterns": "cdc*",
        "settings" : {
            "index" : {
                "number_of_shards" : 15,
                "number_of_replicas" : 1
            }
        }
    }'

With this template setup, all new indexes with the cdc* mask will be split into 15 shards.

You can also set up templates using the OpenSearch Dashboards interface.

To enhance data security and availability, set up a policy that will create a new index if at least one of the following conditions is met (recommended values):

• Index is over 50 GB in size.
• Index is over 30 days old.

You can create and enable a policy using requests. For more information about policies, see the OpenSearch documentation.

curl \
--user <OpenSearch_username>:<password> \
--header 'Content-Type: application/json' \
--request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_plugins/_ism/policies/rollover_policy' \
--data '
    {
        "policy": {
            "description": "Example rollover policy",
            "default_state": "rollover",
            "schema_version": 1,
            "states": [
                {
                    "name": "rollover",
                    "actions": [
                        {
                            "rollover": {
                                "min_index_age": "30d",
                                "min_primary_shard_size": "50gb"
                            }
                        }
                    ],
                    "transitions": []
                }
            ],
            "ism_template": {
                "index_patterns": ["log*"],
                "priority": 100
            }
        }
    }'

curl \
--user <OpenSearch_username>:<password> \
--header 'Content-Type: application/json' \
--request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_index_template/ism_rollover' \
--data '
    {
        "index_patterns": ["log*"],
        "template": {
            "settings": {
                "plugins.index_state_management.rollover_alias": "log"
            }
        }
    }'

curl \
--user <OpenSearch_username>:<password> \
--header 'Content-Type: application/json' \
--request PUT 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/log-000001' \
--data '
    {
        "aliases": {
            "log": {
                "is_write_index": true
            }
        }
    }'

curl \
--user <OpenSearch_username>:<password> \
--header 'Content-Type: application/json' \
--request GET 'https://<address_of_OpenSearch_host_with_DATA_role>:9200/_plugins/_ism/explain/log-000001?pretty'

Managed Service for OpenSearch cluster: Specify ID of the cluster to connect to.

User: Specify the username Data Transfer will use to connect to the cluster.

Password: Enter the user password to the cluster.

Security groups: Select the cloud network to host the endpoint and security groups for network traffic.

This will let you apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.

Data nodes: Click to add a new data node. For each node, specify:

• Host: IP address or FQDN of the host with the DATA role you want to connect to.
• Port: Port number Data Transfer will use for connections to the DATA host.

SSL: Select this option if a secure SSL connection is used.

CA certificate: Upload the certificate file or add its contents as text if transmitted data must be encrypted, for example, to meet PCI DSS requirements.

Subnet ID: Select or create a subnet in the desired availability zone.

If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

User: Specify the username Data Transfer will use to connect to the cluster.

Password: Enter the user password to the cluster.

Security groups: Select the cloud network to host the endpoint and security groups for network traffic.

Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.

Cleanup policy: Select a way to clean up data in the target database before the transfer:

• Don't cleanup: Select this option if you are only going to do replication without copying data.

• Drop: Completely delete the tables included in the transfer (default).

  Use this option to always transfer the latest version of the table schema to the target database from the source whenever the transfer is activated.

Sanitize documents keys: Use this option to automatically replace keys that are not valid for OpenSearch in the target fields.

The autocorrect rules are as follows:

• Empty keys or keys consisting of spaces and periods will be replaced with underscores: "", " ", "." → "_".
• Leading and trailing periods will be removed: "somekey.", ".somekey" → "somekey".
• If there are two periods in a row or there is nothing but spaces between them, the entire fragment will be replaced with a period: " some . . key" → " some . key".

Here is an example of how the autocorrect works: ". s o m e ..incorrect....key. . . " → " s o m e .incorrect.key".