Transferring data to an Elasticsearch target endpoint

Yandex Data Transfer enables you to migrate data to a Elasticsearch database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

1. Configure one of the supported data sources.
2. Prepare the Elasticsearch database for the transfer.
3. Configure the target endpoint in Yandex Data Transfer.
4. Create a transfer and start it.
5. Perform required operations with the database and control the transfer.
6. In case of any issues, use ready-made solutions to resolve them.

Configuring the data sourceConfiguring the data source

Configure one of the supported data sources:

• Apache Kafka®
• YDS
• PostgreSQL
• Elasticsearch
• OpenSearch

For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.

Preparing the target databasePreparing the target database

If not planning to use Cloud Interconnect or VPN for connections to an external cluster, make such cluster accessible from the Internet from IP addresses used by Data Transfer.

For details on linking your network up with external resources, see this concept.

• Make sure the number of columns in the source does not exceed the maximum number of fields in Elasticsearch indexes. The maximum number of fields is provided in the index.mapping.total_fields.limit parameter; the default value is 1000.

  To increase the parameter value, set up a template that makes the maximum number of fields in new indexes equal to the specified value.

  Sample template setup request

  curl \
      --user <Elasticsearch_username>:<password> \
      --header 'Content-Type: application/json' \
      --request PUT "https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults" \
      --data '
          {
              "index_patterns": "cdc*",
              "settings": {
                  "index": {
                      "mapping": {
                          "total_fields": {
                              "limit": "2000"
                          }
                      }
                  }
              }
          }'
  

  With this template setup, all new indexes with the cdc* mask may contain up to 2000 fields.

  You can also set up templates using the Kibana interface.

  To check the current index.mapping.total_fields.limit parameter value, use the Kibana interface or execute the following request:

  curl \
      --user <Elasticsearch_username>:<password> \
      --header 'Content-Type: application/json' \
      --request GET 'https://<Elasticsearch_cluster_FQDN>:9200/<index_name>/_settings/*total_fields.limit?include_defaults=true'
  

• By default, when transferring data to a single index, only one host is used. To distribute the load across hosts when transferring large amounts of data, set up a template to split new indexes into shards in advance.

  Sample template setup request

  curl \
      --user <Elasticsearch_username>:<password> \
      --header 'Content-Type: application/json' \
      --request PUT 'https://<Elasticsearch_cluster_FQDN>:9200/_template/index_defaults' \
      --data '
          {
              "index_patterns": "cdc*",
              "settings" : {
                  "index" : {
                      "number_of_shards" : 15,
                      "number_of_replicas" : 1
                  }
              }
          }'
  

  With this template setup, all new indexes with the cdc* mask will be split into 15 shards.

  You can also set up templates using the Kibana interface.

Configuring the Elasticsearch target endpointConfiguring the Elasticsearch target endpoint

When creating or updating an endpoint, you can define:

• Elasticsearch connection settings. These are required parameters.
• Additional parameters.

Custom installationCustom installation

Connection to nodes with explicitly specified network addresses and ports.

Additional settingsAdditional settings

After configuring the data source and target, create and start the transfer.