Getting started with Data Transfer
Copy or replicate data from a source to a target by creating endpoints and configuring transfers between them.
See the full list of endpoints and possible transfers between them.
Getting started
-
Go to the Yandex Cloud console
and select the folder where you want to perform the operations. If that folder does not exist, create it:Management consoleCLIAPI-
In the management console
, select the appropriate cloud in the list on the left. -
At the top right, click
Create folder. -
Enter the folder name. The naming requirements are as follows:
- The name must be from 3 to 63 characters long.
- It may contain lowercase Latin letters, numbers, and hyphens.
- The first character must be a letter and the last character cannot be a hyphen.
-
(Optional) Enter a description of the folder.
-
Select Create a default network. This will create a network with subnets in each availability zone. Within this network, a default security group will be created, inside which all network traffic is allowed.
-
Click Create.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
-
View the description of the create folder command:
yc resource-manager folder create --help
-
Create a new folder:
-
with a name and without a description:
yc resource-manager folder create \ --name new-folder
The folder naming requirements are as follows:
- The name must be from 3 to 63 characters long.
- It may contain lowercase Latin letters, numbers, and hyphens.
- The first character must be a letter and the last character cannot be a hyphen.
-
with a name and description:
yc resource-manager folder create \ --name new-folder \ --description "my first folder with description"
-
-
-
Go to Yandex Cloud Billing
and make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not yet have a billing account, create one. -
On the Access management
page, make sure you have theeditor
role or higher for the desired folder or the cloud that the folder belongs to.
Configure the source and the target
Different systems can act as a source and as a target.
Compatibility of sources and targets
Possible source and target combinations:
Target Source |
PostgreSQL |
MySQL® |
MongoDB |
ClickHouse® |
Greenplum® |
YDB |
Object Storage |
Apache Kafka |
Data Streams |
Elasticsearch |
OpenSearch |
Target Source |
---|---|---|---|---|---|---|---|---|---|---|---|---|
PostgreSQL |
CR |
CR | - | CR |
CR | CR | C | CR |
CR | C | C | PostgreSQL |
MySQL® |
CR | CR |
- | CR |
CR | CR | C | CR |
CR | - | - | MySQL® |
Oracle |
CR | - | - | CR | CR | - | - | - | - | - | - | Oracle |
MongoDB |
- | - | CR |
- | - | - | C | - | - | - | - | MongoDB |
ClickHouse® |
- | - | - | C |
- | - | - | - | - | - | - | ClickHouse® |
Greenplum® |
C | - | - | C |
C | - | - | - | - | - | - | Greenplum® |
YDB |
- | - | - | CR | - | - | C | CR | CR | - | - | YDB |
Object Storage |
CR | CR | - | CR | CR | CR | - | - | - | - | - | Object Storage |
Metrica |
- | - | - | R | - | - | - | - | - | - | - | Metrica |
Data Streams |
R | R | R | R |
R | R |
R | R |
R | R | R | Yandex Data Streams |
Apache Kafka® |
R | R | R | R | R | R |
R | R |
R | R | R | Apache Kafka® |
Airbyte® |
C | C | C | C | C | C | - | C | C | - | - | Airbyte® |
Elasticsearch |
C | - | - | C | C | C | C | C | C | C | C | Elasticsearch |
OpenSearch |
C | - | - | C | C | C | C | C | C | C | C | OpenSearch |
C: Copy
R: Replicate
CR: Copy and replicate
The remaining transfers are at the Preview stage; you can activate them through a request to our technical support
Airbyte® endpoints
You can use Airbyte®
Airbyte® is already built into Data Transfer, so you do not have to create a separate VM and deploy Airbyte®.
Preparing sources and targets
Prepare the source for sending data:
Prepare the target for receiving the data:
Creating an endpoint for the source
- Go to the folder page and select Yandex Data Transfer.
- In the left-hand panel, select
Endpoints. - Click Create endpoint.
- Make sure that the Direction field is set to
Source
. - Enter a name for the endpoint.
- In the Database type field, select the type of the DBMS to transfer data from.
- Specify endpoint parameters in the corresponding settings section.
- Click Create.
For more information, see Managing endpoints.
Creating an endpoint for the target
- Go to the folder page and select Yandex Data Transfer.
- In the left-hand panel, select
Endpoints. - Click Create endpoint.
- Make sure that the Direction field is set to
Target
. - Enter a name for the endpoint.
- In the Database type field, select the type of DBMS that you want to transfer data to.
- Specify endpoint parameters in the corresponding settings section.
- Click Create.
For more information, see Managing endpoints.
Create a transfer
- Go to the folder page and select Yandex Data Transfer.
- On the left-hand panel, select
Transfers. - Click Create transfer.
- Select the endpoint for the source and the endpoint for the target.
- Enter a name for the transfer.
- Select the type of transfer:
- Snapshot: To create a full copy of the data without receiving further updates from the source. You can also use this type to replicate constantly changing tables.
- Regular snapshot: To create a full copy of data at certain intervals of time.
- Replication: To continuously receive data updates from the source and apply them to the target (without creating a full copy of the source data).
- Snapshot and increment: To create a full copy of the source data and keep it up-to-date.
- (Optional) Add a transfer description.
- Click Create.
For more information, see Types of transfers.
Activate the transfer
- Go to the folder page and select Yandex Data Transfer.
- On the left-hand panel, select
Transfers. - Click
next to the name of the desired transfer and select Activate.
The data transfer process will begin.
For more information, see Managing transfer process.
What's next
- Read about service concepts.
- Learn more about preparing databases for the service and setting up transfers.
- Check out our tutorials on using Data Transfer.
Greenplum® and Greenplum Database® are registered trademarks or trademarks of VMware, Inc. in the United States and/or other countries.
ClickHouse® is a registered trademark of ClickHouse, Inc