Transferring data to an Elasticsearch target endpoint

If not planning to use Cloud Interconnect or VPN for connections to an external cluster, make such cluster accessible from the Internet from IP addresses used by Data Transfer.

For details on linking your network up with external resources, see this concept.

Make sure the number of columns in the source does not exceed the maximum number of fields in Elasticsearch indexes. The maximum number of fields is provided in the index.mapping.total_fields.limit parameter; the default value is 1000.

To increase the parameter value, set up a template that makes the maximum number of fields in new indexes equal to the specified value.

curl \
    --user <Elasticsearch_username>:<password> \
    --header 'Content-Type: application/json' \
    --request PUT "https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults" \
    --data '
        {
            "index_patterns": "cdc*",
            "settings": {
                "index": {
                    "mapping": {
                        "total_fields": {
                            "limit": "2000"
                        }
                    }
                }
            }
        }'

You can also set up templates using the Kibana interface.

To check the current index.mapping.total_fields.limit parameter value, use the Kibana interface or execute the following request:

curl \
    --user <Elasticsearch_username>:<password> \
    --header 'Content-Type: application/json' \
    --request GET 'https://<Elasticsearch_cluster_FQDN>:9200/<index_name>/_settings/*total_fields.limit?include_defaults=true'

By default, when transferring data to a single index, only one host is used. To distribute the load across hosts when transferring large amounts of data, set up a template to split new indexes into shards in advance.

curl \
    --user <Elasticsearch_username>:<password> \
    --header 'Content-Type: application/json' \
    --request PUT 'https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults' \
    --data '
        {
            "index_patterns": "cdc*",
            "settings" : {
                "index" : {
                    "number_of_shards" : 15,
                    "number_of_replicas" : 1
                }
            }
        }'

With this template setup, all new indexes with the cdc* mask will be split into 15 shards.

You can also set up templates using the Kibana interface.

Data nodes: Click to add a new data node. For each node, specify:

• Host: IP address or FQDN of the host with the DATA role you want to connect to.

• Port: Port number Data Transfer will use for connections to the DATA host.

SSL: Select this option if a secure SSL connection is used.

CA certificate: Upload the certificate file or add its contents as text if you need to encrypt the data to transfer, e.g., to meet the PCI DSS requirements.

Subnet ID: Select or create a subnet in the required availability zone.

If the value in this field is specified for both endpoints, both subnets must be hosted in the same availability zone.

User: Specify the username Data Transfer will use to connect to the cluster.

Password: Enter the user password to the cluster.

Security groups: Select the cloud network to host the endpoint and security groups for network traffic.

Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.

Cleanup policy: Select a way to clean up data in the target database before the transfer:

• Don't cleanup: Select this option if you are only going to do replication without copying data.

• Drop: Completely delete the tables included in the transfer (default).

  Use this option to always transfer the latest version of the table schema to the target database from the source whenever the transfer is activated.

Sanitize documents keys: Use this option to automatically replace keys that are not valid for Elasticsearch in the target fields.

The autocorrect rules are as follows:

• Empty keys or keys consisting of spaces and periods will be replaced with underscores: "", " ", "." → "_".
• Leading and trailing periods will be removed: "somekey.", ".somekey" → "somekey".
• If there are two periods in a row or there is nothing but spaces between them, the entire fragment will be replaced with a period: " some . . key" → " some . key".

Here is an example of how the autocorrect works: ". s o m e ..incorrect....key. . . " → " s o m e .incorrect.key".