Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Transferring data to a Managed Service for YDB target endpoint

Written by
Improved by
Updated at November 18, 2024

Yandex Data Transfer enables you to migrate data to a Managed Service for YDB database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

  1. Explore possible data transfer scenarios.
  2. Configure one of the supported data sources.
  3. Prepare the Managed Service for YDB database for the transfer.
  4. Configure the target endpoint in Yandex Data Transfer.
  5. Create a transfer and start it.
  6. Perform required operations with the database and control the transfer.
  7. In case of any issues, use ready-made solutions to resolve them.

Scenarios for transferring data to Managed Service for YDB

  1. Migration: Moving data from one storage to another. Migration often means migrating a database from obsolete local databases to managed cloud ones.

  2. Data delivery is a process of delivering arbitrary data to target storage. It includes data retrieval from a queue and its deserialization with subsequent transformation to target storage format.

  3. Uploading data to data marts is a process of transferring prepared data to storage for subsequent visualization.

For a detailed description of possible Yandex Data Transfer scenarios, see Tutorials.

Configuring the data source

Configure one of the supported data sources:

For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.

Preparing the target database

  1. Create a service account with the ydb.editor role.
  2. For the database running in Dedicated mode, create and configure a security group in the network hosting the DB.

Configuring the Managed Service for YDB target endpoint

When creating or updating an endpoint, you can define:

  • Yandex Managed Service for YDB DB connection settings. These are required parameters.
  • Additional parameters.

Yandex Managed Service for YDB cluster

Warning

To create or edit an endpoint of a managed database, you need to have the ydb.viewer role or the viewer primitive role assigned for the folder where this managed database cluster resides.

Connecting to the database with the cluster ID specified in Yandex Cloud.

  • Database: Select a Managed Service for YDB database from the list.

  • Service account ID: Select or create a service account with the ydb.editor role that Data Transfer will use to connect to the database.

    • Security groups: Select the cloud network to host the endpoint and security groups for network traffic.

      Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.

  • Cleanup policy: Select a way to clean up data in the target database before the transfer:

    • Drop: Fully delete tables included in the transfer (used by default).

      Use this option so that the latest version of the table schema is always transferred to the target database from the source whenever the transfer is activated.

    • Don't cleanup: Do not clean up.

      Select this option if only replication without copying data is performed.

Additional settings

  • Number of shards: Specify the required number of N shards.

    If the setting is specified, the _shard_col column is added to tables. The values in it are calculated as the remainder of H/N, where H is the result of the hash function at the current time and N is the number of shards specified by the setting.

  • Compression for default column group: Set the COMPRESSION setting for the default column group (FAMILY default).

  • Sub directory for tables: Specify the subdirectory to host your tables.

  • Table rotation:

    • Size unit: Hour, day, or month.

    • Table size: In the selected units of measurement.

      When the time interval equal to the selected unit of measurement ends, the oldest database table will be deleted and a new one will be created.

    • Number of tables: Required number of tables in the target database.

    • Partition by column: Split (partition) a table by the specified column's values. The column must be of the time type.

      For more information about partitioning tables, see the Yandex Managed Service for YDB documentation.

    If this setting is used, the specified number of data tables for different intervals is created in the target database. The name of each table is selected automatically by the date and time of the start of the interval. Depending on the values in the specified column of the source table, the original rows are distributed across the respective tables in the target database.

  • Renaming tables: Fill in if you need to rename tables in the source database when transferring data to the target database.

  • Create OLAP tables: Select this option to create column-oriented OLAP tables. By default, row-oriented OLTP tables are used.

After configuring the data source and target, create and start the transfer.

Troubleshooting data transfer issues

Known issues when using a Managed Service for YDB endpoint:

Transfer failure

A Replication or Snapshot and increment transfer is interrupted with an error.

Error message:

/Ydb.PersQueue.V1.PersQueueService/AddReadRule failed: OVERLOADED

Transfers are aborted due to the cloud quota on the number of operations with Managed Service for YDB.

Solution:

  1. Increase the Number of schema transactions per minute property in the Managed Service for YDB quotas for the cloud with the required database and reactivate the transfer.

For more troubleshooting tips, see Troubleshooting.

Previous
Source
Next
Managing transfer process