Getting started with Data Transfer

Copy or replicate data from a source to a target by creating endpoints and configuring transfers between them.
See the full list of endpoints and possible transfers between them.

Getting startedGetting started

1. Go to the management console and log in to Yandex Cloud or register a new account.
2. Go to Yandex Cloud Billing and make sure you have a billing account linked and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account, create one and link a cloud to it.
3. Navigate to the Yandex Cloud console and select a folder you are going to operate in. If there is no such folder, create it.
4. On the Access management page, make sure you have the editor role or higher for the desired folder or the cloud that the folder belongs to.

If you are creating a managed database endpoint for a cluster residing in a different folder, you will need a service or primitive viewer role for that folder.

For more information about the roles required to use Data Transfer, see Required roles.

Configure the source and the targetConfigure the source and the target

Different systems can act as a source and as a target.

Compatibility of sources and targetsCompatibility of sources and targets

Possible source and target combinations:

Target Source	PostgreSQL	MySQL®	MongoDB	ClickHouse®	Greenplum®	YDB	Object Storage	Apache Kafka	Data Streams	Elasticsearch	OpenSearch	Target Source
PostgreSQL	CR	CR	-	CR	CR	CR	C	CR	CR	C	C	PostgreSQL
MySQL®	CR	CR	-	CR	CR	CR	C	CR	CR	-	-	MySQL®
Oracle	CR	-	-	CR	CR	-	-	-	-	-	-	Oracle
MongoDB	-	-	CR	-	-	-	C	-	-	-	-	MongoDB
ClickHouse®	-	-	-	C	-	-	-	-	-	-	-	ClickHouse®
Greenplum®	C	-	-	C	C	-	-	-	-	-	-	Greenplum®
YDB	-	-	-	CR	-	-	C	CR	CR	-	-	YDB
Object Storage	CR	CR	-	CR	CR	CR	-	-	-	-	-	Object Storage
Metrica	-	-	-	R	-	-	-	-	-	-	-	Metrica
Data Streams	R	R	R	R	R	R	R	R	R	R	R	Yandex Data Streams
Apache Kafka®	R	R	R	R	R	R	R	R	R	R	R	Apache Kafka®
Airbyte®	C	C	C	C	C	C	-	C	C	-	-	Airbyte®
Elasticsearch	C	-	-	C	C	C	C	C	C	C	C	Elasticsearch
OpenSearch	C	-	-	C	C	C	C	C	C	C	C	OpenSearch

C: Copy
R: Replicate
CR: Copy and replicate

: Transfer is at the GA stage and charged as per the relevant pricing policy.
The remaining transfers are at the Preview stage; you can activate them through a request to our technical support or your account manager.

Airbyte® endpointsAirbyte® endpoints

You can use Airbyte® to configure the following source endpoints:

• AWS CloudTrail
• BigQuery
• Microsoft SQL Server
• S3

Airbyte® is already built into Data Transfer, so you do not have to create a separate VM and deploy Airbyte®.

Preparing sources and targetsPreparing sources and targets

Prepare the source for sending data:

• Airbyte®:

  • AWS CloudTrail
  • BigQuery
  • Microsoft SQL Server
  • S3

• Apache Kafka®

• ClickHouse®

• Elasticsearch

• Greenplum®

• MongoDB

• MySQL®

• OpenSearch

• Oracle

• PostgreSQL

• Yandex Data Streams

• Yandex Managed Service for YDB

Prepare the target for receiving the data:

• ClickHouse®

• Elasticsearch

• Greenplum®

• MongoDB

• MySQL®

• Object Storage

• OpenSearch

• PostgreSQL

• Managed Service for YDB

Creating an endpoint for the sourceCreating an endpoint for the source

1. Go to the folder page and select Yandex Data Transfer.
2. In the left-hand panel, select Endpoints.
3. Click Create endpoint.
4. Make sure that the Direction field is set to Source.
5. Enter a name for the endpoint.
6. In the Database type field, select the type of the DBMS to transfer data from.
7. Specify endpoint parameters in the corresponding settings section.
8. Click Create.

For more information, see Managing endpoints.

Creating an endpoint for the targetCreating an endpoint for the target

1. Go to the folder page and select Yandex Data Transfer.
2. In the left-hand panel, select Endpoints.
3. Click Create endpoint.
4. Make sure that the Direction field is set to Target.
5. Enter a name for the endpoint.
6. In the Database type field, select the type of DBMS that you want to transfer data to.
7. Specify endpoint parameters in the corresponding settings section.
8. Click Create.

For more information, see Managing endpoints.

Create a transferCreate a transfer

1. Go to the folder page and select Yandex Data Transfer.
2. In the left-hand panel, select Transfers.
3. Click Create transfer.
4. Enter a name for the transfer.
5. (Optional) Add a transfer description.
6. Select the endpoint for the source and the endpoint for the target.
7. Select the type of transfer:
   • Snapshot: To create a full copy of the data without receiving further updates from the source. You can also use this type to replicate constantly changing tables.
     If you need to create a full copy of the data at specific intervals, enable Periodic snapshot.
   • Replication: To continuously receive data updates from the source and apply them to the target (without creating a full copy of the source data).
   • Snapshot and increment: To create a full copy of the source data and keep it up-to-date.
8. Click Create.

For more information, see Types of transfers.

Activate the transferActivate the transfer

1. Go to the folder page and select Yandex Data Transfer.
2. In the left-hand panel, select Transfers.
3. Click next to the transfer name and select Activate.

The data transfer process will begin.

For more information, see Managing transfer process.

What's nextWhat's next

• Read about service concepts.
• Learn more about preparing databases for the service and setting up transfers.
• Learn which roles exist in Data Transfer.
• Check out our tutorials on using Data Transfer.

Greenplum® and Greenplum Database® are registered trademarks or trademarks of VMware, Inc. in the United States and/or other countries.

ClickHouse® is a registered trademark of ClickHouse, Inc.