Data Transfer tutorials

Using Data Transfer, you can implement a variety of scenarios for migrating data, capturing data changes, delivering data from queues, loading data to data marts, and mirroring and uploading data to scalable storage.

Data migrationData migration

Migration is a task that involves transferring data from one data storage to another. In Data Transfer, migration often means migrating a database from obsolete local databases to managed cloud ones.

• Migrating PostgreSQL clusters
• Migrating MySQL® clusters
• Migrating ClickHouse® clusters
• Data resharding in a Managed Service for ClickHouse® cluster
• Migrating Greenplum® clusters
• Migrating MongoDB clusters
• Migrating MongoDB cluster from 4.4 to 6.0
• Migrating data from AWS RDS for PostgreSQL
• Migrating OpenSearch clusters

Mirroring data across queues is a separate migration task.

• Apache Kafka® mirroring

Syncing data from a third-party cluster.

• Syncing data from MySQL®

Migration with data storage type changed.

• MySQL® to YDB
• PostgreSQL to YDB
• MySQL® to PostgreSQL
• PostgreSQL to MySQL®
• MySQL® to Greenplum®
• Elasticsearch to OpenSearch
• OpenSearch to YDB
• OpenSearch to Greenplum®
• PostgreSQL to OpenSearch

Change data captureChange data capture

Change Data Capture (CDC) is a process of tracking changes in a database and delivering them to consumers with public serialization format supported when writing data to a Debezium queue.
CDC is used for creating apps that are sensitive to real-time data changes, exchanging data between different services, including those with different levels of availability, and collecting and delivering data from the production environment to internal data storage for processing and analysis.

• MySQL® change data capture and delivery to:
  • Apache Kafka®
  • YDS
• PostgreSQL change data capture and delivery to:
  • Apache Kafka®
  • YDS
• YDB change data capture and delivery to:
  • Apache Kafka®
  • YDS

Delivering data from queuesDelivering data from queues

Data delivery is a process of delivering arbitrary data to target storage. It includes data retrieval from a queue and its deserialization with subsequent transformation to target storage format.

• Apache Kafka® to ClickHouse®
• Apache Kafka® to PostgreSQL
• Apache Kafka® to Greenplum®
• Apache Kafka® to MongoDB
• Apache Kafka® to MySQL®
• Apache Kafka® to OpenSearch
• Apache Kafka® to YDB
• Apache Kafka® to YDS
• YDS to ClickHouse®
• YDS to Object Storage
• YDS to Managed Service for YDB
• YDS to Apache Kafka®
• Entering data into storage systems

Uploading data to data martsUploading data to data marts

Loading data to data marts is a process of transferring prepared data to storage for subsequent visualization. Sources are data storages used for massively parallel processing of data. Targets are data storages that can quickly deliver data.

• Greenplum® to ClickHouse®
• MySQL® to ClickHouse®
• Yandex Metrica to ClickHouse®
• PostgreSQL to ClickHouse®
• Greenplum® to PostgreSQL
• Yandex Direct to ClickHouse®
• Object Storage to PostgreSQL
• Object Storage to Greenplum®
• Object Storage to MySQL®
• Object Storage to ClickHouse®
• Object Storage to YDB
• YDB to ClickHouse®
• OpenSearch to ClickHouse®

Importing data to Object StorageImporting data to Object Storage

Uploading data to scalable Object Storage storage allows you to save on data storage and simplifies the exchange with contractors.

• MySQL® to Object Storage
• PostgreSQL to Object Storage
• OpenSearch to Object Storage
• YDB to Object Storage
• Replicating logs to Object Storage using Data Streams
• Replicating logs to Object Storage using Fluent Bit

Greenplum® and Greenplum Database® are registered trademarks or trademarks of VMware, Inc. in the United States and/or other countries.

ClickHouse® is a registered trademark of ClickHouse, Inc.