Transferring data to a Yandex Object Storage target endpoint

Yandex Data Transfer enables you to migrate data to the Yandex Object Storage storage and implement various data transfer, processing, and transformation scenarios. To implement a transfer:

1. Explore possible data transfer scenarios.
2. Configure one of the supported data sources.
3. Configure the target endpoint in Yandex Data Transfer.
4. Create a transfer and start it.
5. Perform required operations with the storage and control the transfer.
6. In case of any issues, use ready-made solutions to resolve them.

Scenarios for transferring data to Yandex Object StorageScenarios for transferring data to Yandex Object Storage

1. Data delivery is a process of delivering arbitrary data to target storage. It includes data retrieval from a queue and its deserialization with subsequent transformation to target storage format.

   • Delivering data from YDS to Object Storage.

2. Uploading data to scalable Object Storage storage allows you to save on data storage and simplifies the exchange with contractors.

   • Loading data from MySQL® to Object Storage.
   • Loading data from PostgreSQL to Object Storage.
   • Loading data from OpenSearch to Object Storage.
   • Loading data from Managed Service for YDB to Object Storage.

For a detailed description of possible Yandex Data Transfer scenarios, see Tutorials.

Configuring the data sourceConfiguring the data source

Configure one of the supported data sources:

• PostgreSQL
• MySQL®
• MongoDB
• Apache Kafka®
• Airbyte®
• YDS
• Oracle
• Managed Service for YDB
• OpenSearch

For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.

Configuring the Object Storage target endpointConfiguring the Object Storage target endpoint

When creating or updating an endpoint, you can define:

• Connection settings for an Yandex Object Storage bucket or custom S3-compatible storage.
• Additional parameters.

Connection settingsConnection settings

If the new endpoint and the Object Storage bucket reside in different folders, then when configuring the connection, you must select manual setup and specify the following values:

• ru-central1 region.
• storage.yandexcloud.net endpoint.
• Enable both settings related to SSL usage.

Additional settingsAdditional settings

• Path prefix: The path in the bucket used for data export.

• Data format: Format in which the data will be written to the bucket: JSON, CSV, or Parquet. For more information, see Serialization on delivery to Object Storage.

  JSON

  CSV

  Parquet

  • Compression codec: Output data compression algorithm: Gzip or Uncompressed.

  • Compression codec: Output data compression algorithm: Gzip or Uncompressed.

  • Compression codec for columns: Compression algorithm of data parts: Gzip, Snappy, or Zstd.

  • Row group size: Parameters for splitting data into groups. You can specify the maximum number of rows in a group or the maximum group size in bytes. If you use these settings together, a new group will be created when at least one of the limits is exceeded.

• Advanced writer settings: Additional settings for writing output files:

  Snapshot writer

  Replication writer

  • Maximum number of records per file: If left empty, the data will be written to a single file.
  • Maximum number of bytes per file: If left empty, the data will be written to a single file.

  • Rotation interval: Value in seconds after which a new file will be written. The timestamp data is taken from the parameters of the received messages. The default value is one hour.

After configuring the data source and target, create and start the transfer.

Troubleshooting data transfer issuesTroubleshooting data transfer issues

For the full list of recommendations, see Troubleshooting.

Source data update errorSource data update error

Error message:

Push failed: kind: update not supported

Object Storage only supports inserting new data but does not support updating it. If data is updated at the source, the transfer will fail with the above error.

Solution: Use sources supporting data insertion only or select a target other than Object Storage.