Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Transferring data from a Managed Service for YDB source endpoint

Written by
Updated at November 18, 2024

Yandex Data Transfer enables you to migrate data from a Managed Service for YDB database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

  1. Explore possible data transfer scenarios.
  2. Prepare the Managed Service for YDB database for the transfer.
  3. Set up a source endpoint in Yandex Data Transfer.
  4. Set up one of the supported data targets.
  5. Create a transfer and start it.
  6. Perform required operations with the database and control the transfer.
  7. In case of any issues, use ready-made solutions to resolve them.

Scenarios for transferring data from Managed Service for YDB

  1. Data change capture means tracking changes to a database and delivering those changes to consumers. It is used for applications that are sensitive to real-time data changes.

  2. Uploading data to data marts is a process of transferring prepared data to storage for subsequent visualization.

  3. Uploading data to scalable Object Storage storage allows you to save on data storage and simplifies the exchange with contractors.

For a detailed description of possible Yandex Data Transfer scenarios, see Tutorials.

Preparing the source database

If you selected Dedicated database mode, create and configure a security group in the network hosting the DB.

Configuring the Managed Service for YDB source endpoint

When creating or updating an endpoint, you can define:

  • Yandex Managed Service for YDB DB connection settings. These are required parameters.
  • Transfer path list (for tables and directories).

Warning

To create or edit an endpoint of a managed database, you need to have the ydb.viewer role or the viewer primitive role assigned for the folder where this managed database cluster resides.

  • Database: Select a Managed Service for YDB database from the list.

  • Service account ID: Select or create a service account with the ydb.editor role that Data Transfer will use to connect to the database.

    • Security groups: Select the cloud network to host the endpoint and security groups for network traffic.

      Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.

  • Included paths list: Add paths to the Managed Service for YDB tables and directories to transfer.

    Table and directory names must meet the ID naming rules in YDB. Escaping double quotes is not required. The / character is used as a path delimiter. For more information, see the YDB documentation.

    Adding new paths when editing an endpoint used in Snapshot and increment transfers in the Replicating status will not result in uploading the data history of these tables. To add a table with its historical data, use the List of objects for transfer field in the transfer settings.

    For Snapshot transfers, you may skip specifying paths. In this case, all tables will be transferred.

    For Replication or Snapshot and increment transfers, specifying paths is required, including when you replicate all tables.

  • Name of pre-created changefeed: Here, specify the name of the update stream if it was already created. Otherwise, leave the field blank.

Configuring the data target

Configure one of the supported data targets:

For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.

After configuring the data source and target, create and start the transfer.

Troubleshooting data transfer issues

Known issues when using a Managed Service for YDB endpoint:

Transfer failure

A Replication or Snapshot and increment transfer is interrupted with an error.

Error message:

/Ydb.PersQueue.V1.PersQueueService/AddReadRule failed: OVERLOADED

Transfers are aborted due to the cloud quota on the number of operations with Managed Service for YDB.

Solution:

  1. Increase the Number of schema transactions per minute property in the Managed Service for YDB quotas for the cloud with the required database and reactivate the transfer.

For more troubleshooting tips, see Troubleshooting.

Previous
Target
Next
Target