Resource relationships in Data Transfer

Yandex Data Transfer helps transfer data between DBMS, object storages, and message brokers. This way you can reduce the migration period and minimize downtime when switching to a new database.

Yandex Data Transfer is configurable via Yandex Cloud standard interfaces.

The service is suitable for creating a permanent replica of the database. The transfer of the database schema from the source to the target is automated.

EndpointEndpoint

An endpoint is a configuration used to connect to a service: datasource or target. In addition to connection settings, the endpoint may contain information about which data will be involved in the transfer and how it should be processed during the transfer.

The following can be the data source or target:

Service	Source	Target
Apache Kafka® topic: Your own or as part of Managed Service for Apache Kafka®
AWS CloudTrail message stream
Your own BigQuery database
ClickHouse® database: Your own or as part of Managed Service for ClickHouse®
Your own Elasticsearch database
Greenplum® database: Your own or as part of Managed Service for Greenplum®
MongoDB database: Your own or as part of Managed Service for MongoDB
MySQL® database: Your own or as part of Managed Service for MySQL®
Your own Oracle database
PostgreSQL database: Your own or as part of Managed Service for PostgreSQL
OpenSearch database: Your own or as part of Managed Service for OpenSearch
S3-compatible bucket
Yandex Data Streams data stream
Managed Service for YDB database: as part of Managed Service for YDB.
Yandex Object Storage bucket

Endpoint statusesEndpoint statuses

As part of the Data Transfer transition to asynchronous operations with endpoints, we introduce endpoint statuses as follows:

• Ready: Endpoint is ready to use.
• Creating: Endpoint create operation has started. When the operation is complete, the endpoint status changes to Ready.
• Updating: Endpoint update operation has started. When the operation is complete, the endpoint status changes to Ready.
• Deleting: Endpoint delete operation has started.

TransferTransfer

Transfer is the process of transmitting data between the source and target service. It should be in the same folder as the endpoints used.

If subnets are specified for endpoints, these subnets must be hosted in the same availability zone. Otherwise, activating the transfer with such endpoints will result in an error.

WorkerWorker

Worker is a utility process that starts a data transfer. A separate VM is allocated for each worker. You can specify which computing resources to use for this virtual machine:

• 2 vCPUs and 4 GB RAM. This is the default configuration.
• 4 vCPUs and 8 GB RAM.
• 8 vCPUs and 16 GB RAM.

During parallel copying or parallel replication (for the YDS, YDB, and Apache Kafka® sources), the user selects the number of workers to run at the same time.

vCPU count and RAM size impact the cost of Data Transfer resources. To optimize usage and data transfer costs, we recommend using workers efficiently by reducing their number and increasing the load on each worker. You can also change the worker configuration in the transfer settings for billable source-target pairs at the GA stage.

Transfer typesTransfer types

The following types of transfers are available:

• Snapshot: Transfers a snapshot of the source to the target. Apart from a one-time snapshot transfer, there are copy types, such as Regular and Regular incremental.
• Replication: Continuously receives changes from the source and applies them to the target. Initial data synchronization is not performed.
• Snapshot and increment: Transfers the current state of the source to the target and keeps it up-to-date.

For more information about the differences between transfer types, see Transfer types and lifecycles.

Compatibility of sources and targetsCompatibility of sources and targets

Possible source and target combinations:

Target Source	PostgreSQL	MySQL®	MongoDB	ClickHouse®	Greenplum®	YDB	Object Storage	Apache Kafka	Data Streams	Elasticsearch	OpenSearch	Target Source
PostgreSQL	CR	CR	-	CR	CR	CR	C	CR	CR	C	C	PostgreSQL
MySQL®	CR	CR	-	CR	CR	CR	C	CR	CR	-	-	MySQL®
Oracle	CR	-	-	CR	CR	-	-	-	-	-	-	Oracle
MongoDB	-	-	CR	-	-	-	C	-	-	-	-	MongoDB
ClickHouse®	-	-	-	C	-	-	-	-	-	-	-	ClickHouse®
Greenplum®	C	-	-	C	C	-	-	-	-	-	-	Greenplum®
YDB	-	-	-	CR	-	-	C	CR	CR	-	-	YDB
Object Storage	CR	CR	-	CR	CR	CR	-	-	-	-	-	Object Storage
Metrica	-	-	-	R	-	-	-	-	-	-	-	Metrica
Data Streams	R	R	R	R	R	R	R	R	R	R	R	Yandex Data Streams
Apache Kafka®	R	R	R	R	R	R	R	R	R	R	R	Apache Kafka®
Airbyte®	C	C	C	C	C	C	-	C	C	-	-	Airbyte®
Elasticsearch	C	-	-	C	C	C	C	C	C	C	C	Elasticsearch
OpenSearch	C	-	-	C	C	C	C	C	C	C	C	OpenSearch
Source Target	PostgreSQL	MySQL®	MongoDB	ClickHouse®	Greenplum®	YDB	Object Storage	Apache Kafka	Data Streams	Elasticsearch	OpenSearch	Source Target

C: Copy
R: Replicate
CR: Copy and replicate

: Transfer is at the GA stage and charged as per the relevant pricing policy.
The remaining transfers are at the Preview stage; you can activate them through a request to our technical support or your account manager.

Airbyte® endpointsAirbyte® endpoints

You can use Airbyte® to configure the following source endpoints:

• AWS CloudTrail
• BigQuery
• Microsoft SQL Server
• S3

Airbyte® is already built into Data Transfer, so you do not have to create a separate VM and deploy Airbyte®.

ClickHouse® is a registered trademark of ClickHouse, Inc.