Transferring data to a Apache Iceberg™ target endpoint
Yandex Data Transfer enables you to migrate data to Apache Iceberg™ tables in a Apache Hive™ Metastore cluster and implement various data transfer, processing, and transformation scenarios. To set up a transfer:
- Review possible data transfer scenarios.
- Configure one of the supported data sources.
- Configure the target endpoint in Yandex Data Transfer.
- Create and launch the transfer.
- Perform the required operations with the tables and see how the transfer is going.
Scenarios for transferring data to Apache Iceberg™
For a detailed description of possible Yandex Data Transfer scenarios, see Tutorials.
Configuring the data source
Configure one of the supported data sources:
- ClickHouse®
- Greenplum®
- MongoDB
- MySQL®
- PostgreSQL
- Elasticsearch
- Yandex Object Storage
- Oracle
- Managed Service for YDB
- YTsaurus
For a complete list of supported sources and targets in Yandex Data Transfer, see Available transfers.
Configuring the Apache Iceberg™ target endpoint
When creating or editing an endpoint, you can configure:
- Settings for connecting to a Apache Hive™ Metastore cluster.
- Configuration settings for an Yandex Object Storage bucket or custom S3-compatible storage.
- Optional settings.
Apache Hive™ Metastore cluster
Warning
To create or edit a managed database endpoint, you will need the managed-metastore.viewer role or the primitive viewer role for the folder where its cluster resides.
Note
To configure the Apache Iceberg™ target endpoint, use the Apache Hive™ Metastore cluster version 4.2.0 or higher. Versions below 4.2.0 do not support the REST protocol required for Apache Iceberg™ tables. For more information on connection protocols, see Apache Hive™ Metastore clusters.
Connection with the cluster specified in Yandex Cloud.
-
Apache Hive™ Metastore cluster: ID of the cluster whose folder is used for Apache Iceberg™ tables.
-
Security groups: Select the cloud network to host the endpoint and security groups for network traffic. This will allow you to apply the specified security group rules to the VMs and clusters in the selected network without changing their settings. For more information, see Networking in Yandex Data Transfer.
Make sure the selected security groups are configured.
File storage
- Bucket: Name of the bucket to upload source data to.
- Service account: Select or create a service account with the
storage.uploaderrole that Data Transfer will use to connect to the bucket.
- (Optional) Endpoint: Endpoint for an Amazon S3-compatible service. Leave this field empty to use Amazon.
- Region: Region to send requests.
- Bucket: Bucket name.
- Access Key ID and Secret Access Key: ID and contents of the AWS key used to access a private bucket.
- Path prefix: Path prefix for writing objects to the bucket. This is optional.
Additional settings
-
Cleanup policy: Select the pre-transfer cleanup policy for the target database:
-
DISABLED: Use the existing tables to write new data. -
DROP: Remove all tables involved in the transfer.
Use this option to always transfer the latest table schemas on every transfer activation.
-
-
Writer settings: Settings for data writes to Apache Iceberg™ tables:
- Max rows per file: Maximum number of rows per data file. If set to
0, the number of rows is unlimited. - Max bytes per file: Maximum data file size, in bytes. If set to
0, the file size is unlimited. - Parquet settings: Parquet data format settings:
- Row group max bytes: Maximum number of rows per group. If set to
0, the number of rows per group is unlimited. - Row group max rows: Maximum row group size, in bytes. If set to
0, the row group size is unlimited.
- Row group max bytes: Maximum number of rows per group. If set to
- Max rows per file: Maximum number of rows per data file. If set to
-
Replication settings → Commit interval: Interval between commits to an Apache Iceberg™ table during replication, in seconds.
-
Default namespace: Apache Iceberg™ namespace to transfer tables to if the source database has no data schema.
Once you have configured the source and target, create and launch the transfer.