Transferring data from an Elasticsearch source endpoint
Yandex Data Transfer enables you to migrate search and analytics data from an Elasticsearch database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:
- Explore possible data transfer scenarios.
- Prepare the Elasticsearch database for the transfer.
- Set up a source endpoint in Yandex Data Transfer.
- Set up one of the supported data targets.
- Create a transfer and start it.
- Perform required operations with the database and control the transfer.
- In case of any issues, use ready-made solutions to resolve them.
Scenarios for transferring data from Elasticsearch
Migration: Moving data from one storage to another. Migration often means migrating a database from obsolete local databases to managed cloud ones.
For a detailed description of possible Yandex Data Transfer scenarios, see Tutorials.
Preparing the source database
Note
Elasticsearch versions lower than 7.x are not supported.
If not planning to use Cloud Interconnect or VPN
For details on linking your network up with external resources, see this concept.
Configuring the Elasticsearch source endpoint
When creating or updating an endpoint, you can define:
- Custom installation connection settings. These are required parameters.
- Additional parameters.
Custom installation
Connection to nodes with explicitly specified network addresses and ports.
-
Data nodes: Click
to add a new data node. For each node, specify: -
Host: IP address or FQDN of the host with the
DATArole you need to connect to. -
Port: Port number Data Transfer will use to connect to the host with the
DATArole. -
SSL: Select this option if a secure SSL connection is used.
-
CA certificate: Upload the certificate file or add its contents as text if data encryption is required, e.g., to comply with the PCI DSS requirements.
-
Subnet ID: Select or create a subnet in the required availability zone.
If this field has a value specified for both endpoints, both subnets must be hosted in the same availability zone. -
User: Specify the username Data Transfer will use to connect to the cluster.
-
Password: Enter the user password to the cluster.
-
Security groups: Select the cloud network to host the endpoint and security groups for network traffic.
Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.
Additional settings
- Dump an index with type mapping: Select this option to move data types from a source to a target before a transfer is started. If the option is disabled and no index schema is set on the target, data types on the target will be identified automatically during a transfer.
Warning
If a source index includes data types that are not supported on the target, enabling this option may cause a transfer run error. In this case, disable the option and create an index schema on the target manually.
Configuring the data target
Configure the target endpoint:
- OpenSearch
- ClickHouse®
- Greenplum®
- Yandex Managed Service for YDB
- Yandex Object Storage
- Apache Kafka®
- YDS
For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.
After configuring the data source and target, create and start the transfer.