Contact UsGet started

Transferring data to a Managed Service for YDB target endpoint

Written by
Improved by
Updated at March 28, 2025

Yandex Data Transfer enables you to migrate data to a Managed Service for YDB database and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

  1. Explore possible data transfer scenarios.
  2. Configure one of the supported data sources.
  3. Prepare the Managed Service for YDB database for the transfer.
  4. Configure the target endpoint in Yandex Data Transfer.
  5. Create a transfer and start it.
  6. Perform required operations with the database and control the transfer.
  7. In case of any issues, use ready-made solutions to resolve them.

Scenarios for transferring data to Managed Service for YDB

  1. Migration: Moving data from one storage to another. Migration often means migrating a database from obsolete local databases to managed cloud ones.

  2. Data delivery is a process of delivering arbitrary data to target storage. It includes data retrieval from a queue and its deserialization with subsequent transformation to target storage format.

  3. Uploading data to data marts is a process of transferring prepared data to storage for subsequent visualization.

For a detailed description of possible Yandex Data Transfer scenarios, see Tutorials.

Configuring the data source

Configure one of the supported data sources:

For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.

Preparing the target database

  1. Create a service account with the ydb.editor role.
  2. For the database running in Dedicated mode, create and configure a security group in the network hosting the DB.

Configuring the Managed Service for YDB target endpoint

When creating or updating an endpoint, you can define:

  • Yandex Managed Service for YDB DB connection settings. These are required parameters.
  • Additional parameters.

Yandex Managed Service for YDB cluster

Warning

To create or edit an endpoint of a managed database, you will need the ydb.viewer role or the primitive viewer role for the folder the cluster of this managed database resides in.

Connecting to the database with the cluster ID specified in Yandex Cloud.

  • Database: Select a Managed Service for YDB database from the list.

  • Service account ID: Select or create a service account with the ydb.editor role that Data Transfer will use to connect to the database.

  • Security groups: Select the cloud network to host the endpoint and security groups for network traffic.

    Thus, you will be able to apply the specified security group rules to the VMs and clusters in the selected network without changing the settings of these VMs and clusters. For more information, see Networking in Yandex Data Transfer.

  • Cleanup policy: Select a way to clean up data in the target database before the transfer:

    • Drop: Fully delete tables included in the transfer (used by default).

      Use this option so that the latest version of the table schema is always transferred to the target database from the source whenever the transfer is activated.

    • Don't cleanup: Do not clean up.

      Select this option if only replication without copying data is performed.

  • Endpoint type: ydb_target.

  • name: Managed Service for YDB endpoint name.

  • database: Path to the Managed Service for YDB database the data is transferred to.

    You can view the DB path in the management console on the YDB database information page.

    Example of a DB path: /ru-central1/b1gia87mbaomkfvs6rgl/etnudu2n9ri35luqe4h1.

  • cleanup_policy: Select a way to clean up data in the target database before the transfer:

    • YDB_CLEANUP_POLICY_DROP: Completely delete the tables included in the transfer (default). This will delete all tables in the target before transferring new tables.

      Use this option to always transfer the latest version of the table schema to the target database from the source whenever the transfer is activated.

    • YDB_CLEANUP_POLICY_DISABLED: Do not clean up.

      Select this option if you are only going to do replication without copying data.

  • service_account_id: Service account with the ydb.editor role that Data Transfer will use to connect to the database.

  • security_groups: Security groups for the network traffic.

    Security group rules apply to a transfer. They allow opening up network access from the transfer VM to the cluster. For more information, see Networking in Yandex Data Transfer.

    Security groups must belong to the same network as the cluster.

    Note

    In Terraform, it is not required to specify a network for security groups.

Here is an example of the configuration file structure (not all parameters are given):

resource "yandex_datatransfer_endpoint" "ydb-target" {
name        = "<target_endpoint_name>"
settings {
    ydb_target {
      database       = "<target_database_path>"
      cleanup_policy = "<data_cleanup_policy>"
      service_account_id = "<service_account_ID>"          
    }
  }
}

For more information, see the Terraform provider documentation.

Additional settings

  • Number of shards: Specify the required number of N shards.

    If the setting is specified, the _shard_col column is added to tables. Its values are calculated as the remainder of H/N, where H is the result of the hash function at the current time, and N is the number of shards according to the setting.

  • Compression for default column group: Set the COMPRESSION setting for the default column group (FAMILY default).

  • Sub directory for tables: Specify the subdirectory to host your tables.

  • Table rotation:

    • Size unit: Hour, day, or month.

    • Table size: In the selected units of measurement.

      When the time interval equal to the selected unit of measurement ends, the oldest database table will be deleted and a new one will be created.

    • Number of tables: Required number of tables in the target database.

    • Partition by column: Column to split (partition) the table based on. The column must be of the time type.

      For more information about partitioning tables, see the Yandex Managed Service for YDB documentation.

    If this setting is used, the specified number of data tables for different intervals is created in the target database. The name of each table is selected automatically by the date and time of the start of the interval. Depending on the values in the specified column of the source table, the original rows are distributed across the respective tables in the target database.

  • Renaming tables: Fill if you need to rename the source database tables when transferring them to the target database.

  • Create OLAP tables: Select this option to create column-oriented OLAP tables. By default, row-oriented OLTP tables are used.

  • path: Subdirectory to host your tables.

  • default_compression: Compression setting for the default column group.

  • is_table_column_oriented: Set to true to create column-oriented OLAP tables. By default, row-oriented OLTP tables are used.

After configuring the data source and target, create and start the transfer.

Troubleshooting data transfer issues

Known issues when using a Managed Service for YDB endpoint:

Transfer failure

A Replication or Snapshot and increment transfer is interrupted with an error.

Error message:

/Ydb.PersQueue.V1.PersQueueService/AddReadRule failed: OVERLOADED

Transfers are aborted due to the cloud quota on the number of operations with Managed Service for YDB.

Solution:

  1. Increase the Number of schema transactions per minute property in the Managed Service for YDB quotas for the cloud with the required database and reactivate the transfer.

For more troubleshooting tips, see Troubleshooting.

Previous
Source
Next
Managing transfer process