Data Transfer API, gRPC: TransferService.List
- gRPC request
- ListTransfersRequest
- ListTransfersResponse
- Transfer
- Endpoint
- EndpointSettings
- MysqlSource
- MysqlConnection
- OnPremiseMysql
- TLSMode
- TLSConfig
- ConnectionManagerConnection
- Secret
- MysqlObjectTransferSettings
- PostgresSource
- PostgresConnection
- OnPremisePostgres
- PostgresObjectTransferSettings
- YdbSource
- YDSSource
- Parser
- GenericParserCommon
- DataSchema
- FieldList
- ColSchema
- AuditTrailsV1Parser
- CloudLoggingParser
- KafkaSource
- KafkaConnectionOptions
- OnPremiseKafka
- KafkaAuth
- KafkaSaslSecurity
- NoAuth
- DataTransformationOptions
- MongoSource
- MongoConnection
- MongoConnectionOptions
- OnPremiseMongo
- MongoConnectionManagerConnection
- MongoCollection
- ClickhouseSource
- ClickhouseConnection
- ClickhouseConnectionOptions
- OnPremiseClickhouse
- ClickhouseShard
- MysqlTarget
- PostgresTarget
- ClickhouseTarget
- AltName
- ClickhouseSharding
- ColumnValueHash
- ColumnValueMapping
- ValueToShard
- ColumnValue
- YdbTarget
- KafkaTarget
- KafkaTargetTopicSettings
- KafkaTargetTopic
- Serializer
- SerializerAuto
- SerializerJSON
- SerializerDebezium
- DebeziumSerializerParameter
- MongoTarget
- MetrikaSource
- MetrikaStream
- YDSTarget
- Runtime
- YcRuntime
- ShardingUploadParams
- Transformation
- Transformer
- MaskFieldTransformer
- TablesFilter
- MaskFunction
- MaskFunctionHash
- FilterColumnsTransformer
- ColumnsFilter
- RenameTablesTransformer
- RenameTable
- Table
- ReplacePrimaryKeyTransformer
- ToStringTransformer
- SharderTransformer
- SharderTransformerTypeRandom
- TableSplitterTransformer
- FilterRowsTransformer
- DataObjects
Lists transfers in the specified folder.
gRPC request
rpc List (ListTransfersRequest) returns (ListTransfersResponse)
ListTransfersRequest
{
"folder_id": "string",
"page_size": "int64",
"page_token": "string"
}
|
Field |
Description |
|
folder_id |
string Identifier of the folder containing the transfers to be listed. To get the folder ID, make a |
|
page_size |
int64 The maximum number of transfers to be sent in the response message. If the |
|
page_token |
string Opaque value identifying the transfers page to be fetched. Should be empty in |
ListTransfersResponse
{
"transfers": [
{
"id": "string",
"folder_id": "string",
"name": "string",
"description": "string",
"labels": "map<string, string>",
"source": {
"id": "string",
"folder_id": "string",
"name": "string",
"description": "string",
"labels": "map<string, string>",
"settings": {
// Includes only one of the fields `mysql_source`, `postgres_source`, `ydb_source`, `yds_source`, `kafka_source`, `mongo_source`, `clickhouse_source`, `mysql_target`, `postgres_target`, `clickhouse_target`, `ydb_target`, `kafka_target`, `mongo_target`, `metrika_source`, `yds_target`
"mysql_source": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"timezone": "string",
"object_transfer_settings": {
"view": "ObjectTransferStage",
"routine": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"tables": "ObjectTransferStage"
},
"include_tables_regex": [
"string"
],
"exclude_tables_regex": [
"string"
],
"security_groups": [
"string"
],
"service_database": "string"
},
"postgres_source": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"include_tables": [
"string"
],
"exclude_tables": [
"string"
],
"slot_byte_lag_limit": "int64",
"service_schema": "string",
"object_transfer_settings": {
"sequence": "ObjectTransferStage",
"sequence_owned_by": "ObjectTransferStage",
"table": "ObjectTransferStage",
"primary_key": "ObjectTransferStage",
"fk_constraint": "ObjectTransferStage",
"default_values": "ObjectTransferStage",
"constraint": "ObjectTransferStage",
"index": "ObjectTransferStage",
"view": "ObjectTransferStage",
"function": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"type": "ObjectTransferStage",
"rule": "ObjectTransferStage",
"collation": "ObjectTransferStage",
"policy": "ObjectTransferStage",
"cast": "ObjectTransferStage",
"materialized_view": "ObjectTransferStage",
"sequence_set": "ObjectTransferStage"
},
"security_groups": [
"string"
]
},
"ydb_source": {
"database": "string",
"instance": "string",
"paths": [
"string"
],
"service_account_id": "string",
"subnet_id": "string",
"sa_key_content": "string",
"security_groups": [
"string"
],
"changefeed_custom_name": "string",
"changefeed_custom_consumer_name": "string"
},
"yds_source": {
"database": "string",
"stream": "string",
"service_account_id": "string",
"supported_codecs": [
"YdsCompressionCodec"
],
"parser": {
// Includes only one of the fields `json_parser`, `audit_trails_v1_parser`, `cloud_logging_parser`, `tskv_parser`
"json_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
},
"audit_trails_v1_parser": "AuditTrailsV1Parser",
"cloud_logging_parser": "CloudLoggingParser",
"tskv_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
}
// end of the list of possible fields
},
"allow_ttl_rewind": "bool",
"endpoint": "string",
"subnet_id": "string",
"security_groups": [
"string"
],
"consumer": "string"
},
"kafka_source": {
"connection": {
// Includes only one of the fields `cluster_id`, `on_premise`, `connection_manager_connection`
"cluster_id": "string",
"on_premise": {
"broker_urls": [
"string"
],
"subnet_id": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `no_auth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"no_auth": "NoAuth"
// end of the list of possible fields
},
"security_groups": [
"string"
],
"topic_name": "string",
"transformer": {
"cloud_function": "string",
"number_of_retries": "int64",
"buffer_size": "string",
"buffer_flush_interval": "string",
"invocation_timeout": "string",
"service_account_id": "string"
},
"parser": {
// Includes only one of the fields `json_parser`, `audit_trails_v1_parser`, `cloud_logging_parser`, `tskv_parser`
"json_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
},
"audit_trails_v1_parser": "AuditTrailsV1Parser",
"cloud_logging_parser": "CloudLoggingParser",
"tskv_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
}
// end of the list of possible fields
},
"topic_names": [
"string"
]
},
"mongo_source": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"hosts": [
"string"
],
"port": "int64",
"replica_set": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"replica_set": "string"
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"auth_source": "string"
}
// end of the list of possible fields
},
"subnet_id": "string",
"collections": [
{
"database_name": "string",
"collection_name": "string"
}
],
"excluded_collections": [
{
"database_name": "string",
"collection_name": "string"
}
],
"secondary_preferred_mode": "bool",
"security_groups": [
"string"
]
},
"clickhouse_source": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `on_premise`, `connection_manager_connection`, `mdb_cluster_id`
"on_premise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"http_port": "int64",
"native_port": "int64",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
},
"mdb_cluster_id": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"include_tables": [
"string"
],
"exclude_tables": [
"string"
],
"subnet_id": "string",
"security_groups": [
"string"
],
"clickhouse_cluster_name": "string"
},
"mysql_target": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"sql_mode": "string",
"skip_constraint_checks": "bool",
"timezone": "string",
"cleanup_policy": "CleanupPolicy",
"service_database": "string",
"security_groups": [
"string"
],
"is_schema_migration_disabled": "bool"
},
"postgres_target": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"cleanup_policy": "CleanupPolicy",
"security_groups": [
"string"
],
"is_schema_migration_disabled": "bool"
},
"clickhouse_target": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `on_premise`, `connection_manager_connection`, `mdb_cluster_id`
"on_premise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"http_port": "int64",
"native_port": "int64",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
},
"mdb_cluster_id": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"subnet_id": "string",
"alt_names": [
{
"from_name": "string",
"to_name": "string"
}
],
"cleanup_policy": "ClickhouseCleanupPolicy",
"sharding": {
// Includes only one of the fields `column_value_hash`, `custom_mapping`, `transfer_id`, `round_robin`
"column_value_hash": {
"column_name": "string"
},
"custom_mapping": {
"column_name": "string",
"mapping": [
{
"column_value": {
// Includes only one of the fields `string_value`
"string_value": "string"
// end of the list of possible fields
},
"shard_name": "string"
}
]
},
"transfer_id": "google.protobuf.Empty",
"round_robin": "google.protobuf.Empty"
// end of the list of possible fields
},
"is_schema_migration_disabled": "bool",
"clickhouse_cluster_name": "string",
"security_groups": [
"string"
]
},
"ydb_target": {
"database": "string",
"instance": "string",
"path": "string",
"service_account_id": "string",
"cleanup_policy": "YdbCleanupPolicy",
"subnet_id": "string",
"sa_key_content": "string",
"security_groups": [
"string"
],
"is_table_column_oriented": "bool",
"default_compression": "YdbDefaultCompression",
"is_schema_migration_disabled": "bool"
},
"kafka_target": {
"connection": {
// Includes only one of the fields `cluster_id`, `on_premise`, `connection_manager_connection`
"cluster_id": "string",
"on_premise": {
"broker_urls": [
"string"
],
"subnet_id": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `no_auth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"no_auth": "NoAuth"
// end of the list of possible fields
},
"security_groups": [
"string"
],
"topic_settings": {
// Includes only one of the fields `topic`, `topic_prefix`
"topic": {
"topic_name": "string",
"save_tx_order": "bool"
},
"topic_prefix": "string"
// end of the list of possible fields
},
"serializer": {
// Includes only one of the fields `serializer_auto`, `serializer_json`, `serializer_debezium`
"serializer_auto": "SerializerAuto",
"serializer_json": "SerializerJSON",
"serializer_debezium": {
"serializer_parameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
}
},
"mongo_target": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"hosts": [
"string"
],
"port": "int64",
"replica_set": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"replica_set": "string"
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"auth_source": "string"
}
// end of the list of possible fields
},
"database": "string",
"cleanup_policy": "CleanupPolicy",
"subnet_id": "string",
"security_groups": [
"string"
]
},
"metrika_source": {
"counter_ids": [
"int64"
],
"token": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"streams": [
{
"type": "MetrikaStreamType",
"columns": [
"string"
]
}
]
},
"yds_target": {
"database": "string",
"stream": "string",
"service_account_id": "string",
"save_tx_order": "bool",
"compression_codec": "YdsCompressionCodec",
"serializer": {
// Includes only one of the fields `serializer_auto`, `serializer_json`, `serializer_debezium`
"serializer_auto": "SerializerAuto",
"serializer_json": "SerializerJSON",
"serializer_debezium": {
"serializer_parameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
},
"endpoint": "string",
"subnet_id": "string",
"security_groups": [
"string"
]
}
// end of the list of possible fields
}
},
"target": {
"id": "string",
"folder_id": "string",
"name": "string",
"description": "string",
"labels": "map<string, string>",
"settings": {
// Includes only one of the fields `mysql_source`, `postgres_source`, `ydb_source`, `yds_source`, `kafka_source`, `mongo_source`, `clickhouse_source`, `mysql_target`, `postgres_target`, `clickhouse_target`, `ydb_target`, `kafka_target`, `mongo_target`, `metrika_source`, `yds_target`
"mysql_source": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"timezone": "string",
"object_transfer_settings": {
"view": "ObjectTransferStage",
"routine": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"tables": "ObjectTransferStage"
},
"include_tables_regex": [
"string"
],
"exclude_tables_regex": [
"string"
],
"security_groups": [
"string"
],
"service_database": "string"
},
"postgres_source": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"include_tables": [
"string"
],
"exclude_tables": [
"string"
],
"slot_byte_lag_limit": "int64",
"service_schema": "string",
"object_transfer_settings": {
"sequence": "ObjectTransferStage",
"sequence_owned_by": "ObjectTransferStage",
"table": "ObjectTransferStage",
"primary_key": "ObjectTransferStage",
"fk_constraint": "ObjectTransferStage",
"default_values": "ObjectTransferStage",
"constraint": "ObjectTransferStage",
"index": "ObjectTransferStage",
"view": "ObjectTransferStage",
"function": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"type": "ObjectTransferStage",
"rule": "ObjectTransferStage",
"collation": "ObjectTransferStage",
"policy": "ObjectTransferStage",
"cast": "ObjectTransferStage",
"materialized_view": "ObjectTransferStage",
"sequence_set": "ObjectTransferStage"
},
"security_groups": [
"string"
]
},
"ydb_source": {
"database": "string",
"instance": "string",
"paths": [
"string"
],
"service_account_id": "string",
"subnet_id": "string",
"sa_key_content": "string",
"security_groups": [
"string"
],
"changefeed_custom_name": "string",
"changefeed_custom_consumer_name": "string"
},
"yds_source": {
"database": "string",
"stream": "string",
"service_account_id": "string",
"supported_codecs": [
"YdsCompressionCodec"
],
"parser": {
// Includes only one of the fields `json_parser`, `audit_trails_v1_parser`, `cloud_logging_parser`, `tskv_parser`
"json_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
},
"audit_trails_v1_parser": "AuditTrailsV1Parser",
"cloud_logging_parser": "CloudLoggingParser",
"tskv_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
}
// end of the list of possible fields
},
"allow_ttl_rewind": "bool",
"endpoint": "string",
"subnet_id": "string",
"security_groups": [
"string"
],
"consumer": "string"
},
"kafka_source": {
"connection": {
// Includes only one of the fields `cluster_id`, `on_premise`, `connection_manager_connection`
"cluster_id": "string",
"on_premise": {
"broker_urls": [
"string"
],
"subnet_id": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `no_auth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"no_auth": "NoAuth"
// end of the list of possible fields
},
"security_groups": [
"string"
],
"topic_name": "string",
"transformer": {
"cloud_function": "string",
"number_of_retries": "int64",
"buffer_size": "string",
"buffer_flush_interval": "string",
"invocation_timeout": "string",
"service_account_id": "string"
},
"parser": {
// Includes only one of the fields `json_parser`, `audit_trails_v1_parser`, `cloud_logging_parser`, `tskv_parser`
"json_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
},
"audit_trails_v1_parser": "AuditTrailsV1Parser",
"cloud_logging_parser": "CloudLoggingParser",
"tskv_parser": {
"data_schema": {
// Includes only one of the fields `json_fields`, `fields`
"json_fields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"null_keys_allowed": "bool",
"add_rest_column": "bool",
"unescape_string_values": "bool"
}
// end of the list of possible fields
},
"topic_names": [
"string"
]
},
"mongo_source": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"hosts": [
"string"
],
"port": "int64",
"replica_set": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"replica_set": "string"
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"auth_source": "string"
}
// end of the list of possible fields
},
"subnet_id": "string",
"collections": [
{
"database_name": "string",
"collection_name": "string"
}
],
"excluded_collections": [
{
"database_name": "string",
"collection_name": "string"
}
],
"secondary_preferred_mode": "bool",
"security_groups": [
"string"
]
},
"clickhouse_source": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `on_premise`, `connection_manager_connection`, `mdb_cluster_id`
"on_premise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"http_port": "int64",
"native_port": "int64",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
},
"mdb_cluster_id": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"include_tables": [
"string"
],
"exclude_tables": [
"string"
],
"subnet_id": "string",
"security_groups": [
"string"
],
"clickhouse_cluster_name": "string"
},
"mysql_target": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"sql_mode": "string",
"skip_constraint_checks": "bool",
"timezone": "string",
"cleanup_policy": "CleanupPolicy",
"service_database": "string",
"security_groups": [
"string"
],
"is_schema_migration_disabled": "bool"
},
"postgres_target": {
"connection": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"port": "int64",
"subnet_id": "string",
"hosts": [
"string"
],
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"cleanup_policy": "CleanupPolicy",
"security_groups": [
"string"
],
"is_schema_migration_disabled": "bool"
},
"clickhouse_target": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `on_premise`, `connection_manager_connection`, `mdb_cluster_id`
"on_premise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"http_port": "int64",
"native_port": "int64",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
},
"mdb_cluster_id": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"subnet_id": "string",
"alt_names": [
{
"from_name": "string",
"to_name": "string"
}
],
"cleanup_policy": "ClickhouseCleanupPolicy",
"sharding": {
// Includes only one of the fields `column_value_hash`, `custom_mapping`, `transfer_id`, `round_robin`
"column_value_hash": {
"column_name": "string"
},
"custom_mapping": {
"column_name": "string",
"mapping": [
{
"column_value": {
// Includes only one of the fields `string_value`
"string_value": "string"
// end of the list of possible fields
},
"shard_name": "string"
}
]
},
"transfer_id": "google.protobuf.Empty",
"round_robin": "google.protobuf.Empty"
// end of the list of possible fields
},
"is_schema_migration_disabled": "bool",
"clickhouse_cluster_name": "string",
"security_groups": [
"string"
]
},
"ydb_target": {
"database": "string",
"instance": "string",
"path": "string",
"service_account_id": "string",
"cleanup_policy": "YdbCleanupPolicy",
"subnet_id": "string",
"sa_key_content": "string",
"security_groups": [
"string"
],
"is_table_column_oriented": "bool",
"default_compression": "YdbDefaultCompression",
"is_schema_migration_disabled": "bool"
},
"kafka_target": {
"connection": {
// Includes only one of the fields `cluster_id`, `on_premise`, `connection_manager_connection`
"cluster_id": "string",
"on_premise": {
"broker_urls": [
"string"
],
"subnet_id": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"subnet_id": "string"
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `no_auth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"no_auth": "NoAuth"
// end of the list of possible fields
},
"security_groups": [
"string"
],
"topic_settings": {
// Includes only one of the fields `topic`, `topic_prefix`
"topic": {
"topic_name": "string",
"save_tx_order": "bool"
},
"topic_prefix": "string"
// end of the list of possible fields
},
"serializer": {
// Includes only one of the fields `serializer_auto`, `serializer_json`, `serializer_debezium`
"serializer_auto": "SerializerAuto",
"serializer_json": "SerializerJSON",
"serializer_debezium": {
"serializer_parameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
}
},
"mongo_target": {
"connection": {
// Includes only one of the fields `connection_options`
"connection_options": {
// Includes only one of the fields `mdb_cluster_id`, `on_premise`, `connection_manager_connection`
"mdb_cluster_id": "string",
"on_premise": {
"hosts": [
"string"
],
"port": "int64",
"replica_set": "string",
"tls_mode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"ca_certificate": "string"
}
// end of the list of possible fields
}
},
"connection_manager_connection": {
"connection_id": "string",
"replica_set": "string"
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"auth_source": "string"
}
// end of the list of possible fields
},
"database": "string",
"cleanup_policy": "CleanupPolicy",
"subnet_id": "string",
"security_groups": [
"string"
]
},
"metrika_source": {
"counter_ids": [
"int64"
],
"token": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"streams": [
{
"type": "MetrikaStreamType",
"columns": [
"string"
]
}
]
},
"yds_target": {
"database": "string",
"stream": "string",
"service_account_id": "string",
"save_tx_order": "bool",
"compression_codec": "YdsCompressionCodec",
"serializer": {
// Includes only one of the fields `serializer_auto`, `serializer_json`, `serializer_debezium`
"serializer_auto": "SerializerAuto",
"serializer_json": "SerializerJSON",
"serializer_debezium": {
"serializer_parameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
},
"endpoint": "string",
"subnet_id": "string",
"security_groups": [
"string"
]
}
// end of the list of possible fields
}
},
"runtime": {
// Includes only one of the fields `yc_runtime`
"yc_runtime": {
"job_count": "int64",
"upload_shard_params": {
"job_count": "int64",
"process_count": "int64"
}
}
// end of the list of possible fields
},
"status": "TransferStatus",
"type": "TransferType",
"warning": "string",
"transformation": {
"transformers": [
{
// Includes only one of the fields `mask_field`, `filter_columns`, `rename_tables`, `replace_primary_key`, `convert_to_string`, `sharder_transformer`, `table_splitter_transformer`, `filter_rows`
"mask_field": {
"tables": {
"include_tables": [
"string"
],
"exclude_tables": [
"string"
]
},
"columns": [
"string"
],
"function": {
// Includes only one of the fields `mask_function_hash`
"mask_function_hash": {
"user_defined_salt": "string"
}
// end of the list of possible fields
}
},
"filter_columns": {
"tables": {
"include_tables": [
"string"
],
"exclude_tables": [
"string"
]
},
"columns": {
"include_columns": [
"string"
],
"exclude_columns": [
"string"
]
}
},
"rename_tables": {
"rename_tables": [
{
"original_name": {
"name_space": "string",
"name": "string"
},
"new_name": {
"name_space": "string",
"name": "string"
}
}
]
},
"replace_primary_key": {
"tables": {
"include_tables": [
"string"
],
"exclude_tables": [
"string"
]
},
"keys": [
"string"
]
},
"convert_to_string": {
"tables": {
"include_tables": [
"string"
],
"exclude_tables": [
"string"
]
},
"columns": {
"include_columns": [
"string"
],
"exclude_columns": [
"string"
]
}
},
"sharder_transformer": {
"tables": {
"include_tables": [
"string"
],
"exclude_tables": [
"string"
]
},
// Includes only one of the fields `columns`, `random`
"columns": {
"include_columns": [
"string"
],
"exclude_columns": [
"string"
]
},
"random": "SharderTransformerTypeRandom",
// end of the list of possible fields
"shards_count": "int64"
},
"table_splitter_transformer": {
"tables": {
"include_tables": [
"string"
],
"exclude_tables": [
"string"
]
},
"columns": [
"string"
],
"splitter": "string"
},
"filter_rows": {
"tables": {
"include_tables": [
"string"
],
"exclude_tables": [
"string"
]
},
"filter": "string",
"filters": [
"string"
]
}
// end of the list of possible fields
}
]
},
"data_objects": {
"include_objects": [
"string"
]
},
"prestable": "bool",
"replication_runtime": {
// Includes only one of the fields `yc_runtime`
"yc_runtime": {
"job_count": "int64",
"upload_shard_params": {
"job_count": "int64",
"process_count": "int64"
}
}
// end of the list of possible fields
}
}
],
"next_page_token": "string"
}
|
Field |
Description |
|
transfers[] |
The list of transfers. If there are more transfers in the folder, then |
|
next_page_token |
string Opaque value identifying the next transfers page. This field is empty if there |
Transfer
Transfer core entity
|
Field |
Description |
|
id |
string |
|
folder_id |
string |
|
name |
string |
|
description |
string |
|
labels |
object (map<string, string>) |
|
source |
|
|
target |
|
|
runtime |
Runtime parameters for the transfer |
|
status |
enum TransferStatus
|
|
type |
enum TransferType Type of the transfer. One of SNAPSHOT_ONLY, INCREMENT_ONLY,
|
|
warning |
string Error description if transfer has any errors. |
|
transformation |
Transformation for the transfer. |
|
data_objects |
|
|
prestable |
bool |
|
replication_runtime |
Replication runtime parameters for the transfer |
Endpoint
Data Transfer endpoint. For more information, see the official
documentation
|
Field |
Description |
|
id |
string |
|
folder_id |
string |
|
name |
string |
|
description |
string |
|
labels |
object (map<string, string>) |
|
settings |
DataTransfer Endpoint Settings block |
EndpointSettings
|
Field |
Description |
|
mysql_source |
Includes only one of the fields |
|
postgres_source |
Includes only one of the fields |
|
ydb_source |
Includes only one of the fields |
|
yds_source |
Includes only one of the fields |
|
kafka_source |
Includes only one of the fields |
|
mongo_source |
Includes only one of the fields |
|
clickhouse_source |
Includes only one of the fields |
|
mysql_target |
Includes only one of the fields |
|
postgres_target |
Includes only one of the fields |
|
clickhouse_target |
Includes only one of the fields |
|
ydb_target |
Includes only one of the fields |
|
kafka_target |
Includes only one of the fields |
|
mongo_target |
Includes only one of the fields |
|
metrika_source |
Includes only one of the fields |
|
yds_target |
Includes only one of the fields |
MysqlSource
Settings specific to the MySQL source endpoint
|
Field |
Description |
|
connection |
Database connection settings |
|
database |
string Name of the database to transfer You can leave it empty, then it will be possible to transfer tables from several |
|
user |
string User for database access. Required unless connection manager connection is used |
|
password |
Password for database access. |
|
timezone |
string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from |
|
object_transfer_settings |
Schema migration |
|
include_tables_regex[] |
string List of regular expressions of table names which should be transferred. A table |
|
exclude_tables_regex[] |
string Opposite of |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
service_database |
string Database for service tables |
MysqlConnection
|
Field |
Description |
|
mdb_cluster_id |
string Managed Service for MySQL cluster ID Includes only one of the fields |
|
on_premise |
Connection options for on-premise MySQL Includes only one of the fields |
|
connection_manager_connection |
Get Mysql installation params and credentials from Connection Manager Includes only one of the fields |
OnPremiseMysql
|
Field |
Description |
|
port |
int64 Port for the database connection |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
hosts[] |
string List of host names of the MySQL server. Exactly one host is expected |
|
tls_mode |
TLS settings for server connection. Disabled by default. |
TLSMode
TLS configuration
|
Field |
Description |
|
disabled |
Empty block designating that the connection is not secured, i.e. plaintext Includes only one of the fields |
|
enabled |
TLS is used for the server connection Includes only one of the fields |
TLSConfig
|
Field |
Description |
|
ca_certificate |
string CA certificate X.509 certificate of the certificate authority which issued the server's |
ConnectionManagerConnection
Use Connection Manager connection
|
Field |
Description |
|
connection_id |
string ID of connection in Connection Manager with installation params and credetials |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
Secret
|
Field |
Description |
|
raw |
string Raw secret value Includes only one of the fields |
MysqlObjectTransferSettings
|
Field |
Description |
|
view |
enum ObjectTransferStage Views CREATE VIEW ...
|
|
routine |
enum ObjectTransferStage Routines CREATE PROCEDURE ... ; CREATE FUNCTION ... ;
|
|
trigger |
enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
|
tables |
enum ObjectTransferStage
|
PostgresSource
Settings specific to the PostgreSQL source endpoint.
|
Field |
Description |
|
connection |
Database connection settings |
|
database |
string Name of the database to transfer |
|
user |
string User for database access. Required unless Connection Manager connection is used. |
|
password |
Password for database access. |
|
include_tables[] |
string List of tables to transfer, formatted as |
|
exclude_tables[] |
string List of tables which will not be transfered, formatted as |
|
slot_byte_lag_limit |
int64 Maximum WAL size held by the replication slot (API - in bytes, terraform - in |
|
service_schema |
string Name of the database schema in which auxiliary tables needed for the transfer |
|
object_transfer_settings |
PostgresObjectTransferSettings Defines which database schema objects should be transferred, e.g. views, |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
PostgresConnection
|
Field |
Description |
|
mdb_cluster_id |
string Managed Service for PostgreSQL cluster ID Includes only one of the fields |
|
on_premise |
Connection options for on-premise PostgreSQL Includes only one of the fields |
|
connection_manager_connection |
Get Postgres installation params and credentials from Connection Manager Includes only one of the fields |
OnPremisePostgres
|
Field |
Description |
|
port |
int64 PG port. Will be used if the cluster ID is not specified. |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
hosts[] |
string PG installation hosts |
|
tls_mode |
TLS settings for server connection. Disabled by default. |
PostgresObjectTransferSettings
|
Field |
Description |
|
sequence |
enum ObjectTransferStage Sequences CREATE SEQUENCE ...
|
|
sequence_owned_by |
enum ObjectTransferStage Owned sequences CREATE SEQUENCE ... OWNED BY ...
|
|
table |
enum ObjectTransferStage Tables CREATE TABLE ...
|
|
primary_key |
enum ObjectTransferStage Primary keys ALTER TABLE ... ADD PRIMARY KEY ...
|
|
fk_constraint |
enum ObjectTransferStage Foreign keys ALTER TABLE ... ADD FOREIGN KEY ...
|
|
default_values |
enum ObjectTransferStage Default values ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...
|
|
constraint |
enum ObjectTransferStage Constraints ALTER TABLE ... ADD CONSTRAINT ...
|
|
index |
enum ObjectTransferStage Indexes CREATE INDEX ...
|
|
view |
enum ObjectTransferStage Views CREATE VIEW ...
|
|
function |
enum ObjectTransferStage Functions CREATE FUNCTION ...
|
|
trigger |
enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
|
type |
enum ObjectTransferStage Types CREATE TYPE ...
|
|
rule |
enum ObjectTransferStage Rules CREATE RULE ...
|
|
collation |
enum ObjectTransferStage Collations CREATE COLLATION ...
|
|
policy |
enum ObjectTransferStage Policies CREATE POLICY ...
|
|
cast |
enum ObjectTransferStage Casts CREATE CAST ...
|
|
materialized_view |
enum ObjectTransferStage Materialized views CREATE MATERIALIZED VIEW ...
|
|
sequence_set |
enum ObjectTransferStage
|
YdbSource
Settings specific to the YDB source endpoint
|
Field |
Description |
|
database |
string Database path in YDB where tables are stored. |
|
instance |
string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135. |
|
paths[] |
string A list of paths which should be uploaded. When not specified, all available |
|
service_account_id |
string Service account ID for interaction with database |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
sa_key_content |
string Authorization Key |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
changefeed_custom_name |
string Pre-created change feed if any |
|
changefeed_custom_consumer_name |
string Consumer for pre-created change feed if any |
YDSSource
Settings specific to the YDS source endpoint
|
Field |
Description |
|
database |
string Database path in YDB for streams |
|
stream |
string Stream to read |
|
service_account_id |
string Service account ID which has read access to the stream. |
|
supported_codecs[] |
enum YdsCompressionCodec List of supported compression codecs
|
|
parser |
Data parsing rules |
|
allow_ttl_rewind |
bool Should continue working, if consumer read lag exceed TTL of topic |
|
endpoint |
string YDS Endpoint for dedicated db |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
consumer |
string Custom consumer - for important streams |
Parser
|
Field |
Description |
|
json_parser |
Parse data in json format Includes only one of the fields |
|
audit_trails_v1_parser |
Parse Audit Trails data. Empty struct Includes only one of the fields |
|
cloud_logging_parser |
Parse Cloud Logging data. Empty struct Includes only one of the fields |
|
tskv_parser |
Parse data in tskv format Includes only one of the fields |
GenericParserCommon
|
Field |
Description |
|
data_schema |
Data parsing scheme |
|
null_keys_allowed |
bool Allow null keys, if no - null keys will be putted to unparsed data |
|
add_rest_column |
bool Will add _rest column for all unknown fields |
|
unescape_string_values |
bool Unescape string values |
DataSchema
|
Field |
Description |
|
json_fields |
string Description of the data schema as JSON specification Includes only one of the fields |
|
fields |
Description of the data schema in the array of Includes only one of the fields |
FieldList
|
Field |
Description |
|
fields[] |
Description of the column schema in the array of |
ColSchema
|
Field |
Description |
|
name |
string Field name |
|
type |
enum ColumnType Field type, one of:
|
|
key |
bool Mark field as Primary Key |
|
required |
bool Mark field as required |
|
path |
string Path to the field |
AuditTrailsV1Parser
|
Field |
Description |
|
Empty |
|
CloudLoggingParser
|
Field |
Description |
|
Empty |
|
KafkaSource
Settings specific to the Kafka source endpoint
|
Field |
Description |
|
connection |
Connection settings |
|
auth |
Authentication settings |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
topic_name |
string Deprecated*. Please use |
|
transformer |
Transform data with a custom Cloud Function |
|
parser |
Data parsing parameters. If not set, the source messages are read in raw |
|
topic_names[] |
string List of full source topic names to read |
KafkaConnectionOptions
|
Field |
Description |
|
cluster_id |
string Managed Service for Kafka cluster ID. Includes only one of the fields |
|
on_premise |
Connection options for on-premise Kafka Includes only one of the fields |
|
connection_manager_connection |
Get Kafka installation params and credentials from Connection Manager Includes only one of the fields |
OnPremiseKafka
On-premise Kafka installation options
|
Field |
Description |
|
broker_urls[] |
string Kafka broker URLs |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
tls_mode |
TLS settings for broker connection. Disabled by default. |
KafkaAuth
|
Field |
Description |
|
sasl |
Authentication with SASL Includes only one of the fields |
|
no_auth |
No authentication Includes only one of the fields |
KafkaSaslSecurity
|
Field |
Description |
|
user |
string User name |
|
mechanism |
enum KafkaMechanism SASL mechanism for authentication, use one of: KAFKA_MECHANISM_SHA256,
|
|
password |
Password for user |
NoAuth
No authentication
|
Field |
Description |
|
Empty |
|
DataTransformationOptions
|
Field |
Description |
|
cloud_function |
string Cloud function |
|
number_of_retries |
int64 Number of retries |
|
buffer_size |
string Buffer size for function. Maximum 4 GB. Use value with units, i.e. 10 B, 20 kB, |
|
buffer_flush_interval |
string Flush interval |
|
invocation_timeout |
string Invocation timeout |
|
service_account_id |
string Service account |
MongoSource
Settings specific to the MongoDB source endpoint
|
Field |
Description |
|
connection |
Connection settings |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
collections[] |
List of collections for replication. Empty list implies replication of all |
|
excluded_collections[] |
List of forbidden collections for replication. Allowed to use * as collection |
|
secondary_preferred_mode |
bool Read mode for mongo client: whether the secondary server should be preferred to |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
MongoConnection
|
Field |
Description |
|
connection_options |
Includes only one of the fields |
MongoConnectionOptions
|
Field |
Description |
|
mdb_cluster_id |
string Identifier of the Yandex StoreDoc cluster Includes only one of the fields |
|
on_premise |
Connection settings of the on-premise MongoDB server Includes only one of the fields |
|
connection_manager_connection |
MongoConnectionManagerConnection Get StoreDoc/MongoDB installation params and credentials from Connection Manager Includes only one of the fields |
|
user |
string User name, required unless connection_manager_connection is used |
|
password |
Password for user |
|
auth_source |
string Database name associated with the credentials |
OnPremiseMongo
|
Field |
Description |
|
hosts[] |
string Host names of the replica set |
|
port |
int64 TCP Port number |
|
replica_set |
string Replica set name |
|
tls_mode |
TLS settings for the server connection. Empty implies plaintext connection |
MongoConnectionManagerConnection
|
Field |
Description |
|
connection_id |
string ID of connectionmanager connection with mongodb/Yandex Storedoc installation |
|
replica_set |
string Replica set name, used only for on-premise mongodb installations |
MongoCollection
|
Field |
Description |
|
database_name |
string |
|
collection_name |
string |
ClickhouseSource
Settings specific to the ClickHouse source endpoint
|
Field |
Description |
|
connection |
Connection settings |
|
include_tables[] |
string White list of tables for replication. If none or empty list is presented - will |
|
exclude_tables[] |
string Exclude list of tables for replication. If none or empty list is presented - |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
clickhouse_cluster_name |
string Name of the ClickHouse cluster. For Managed ClickHouse that is name of |
ClickhouseConnection
|
Field |
Description |
|
connection_options |
Includes only one of the fields |
ClickhouseConnectionOptions
|
Field |
Description |
|
on_premise |
Connection settings of the on-premise ClickHouse server Includes only one of the fields |
|
connection_manager_connection |
Get ClickHouse installation params and credentials from Connection Manager Includes only one of the fields |
|
mdb_cluster_id |
string Identifier of the Managed ClickHouse cluster Includes only one of the fields |
|
user |
string User for database access. Required unless connection_manager_connection is used |
|
password |
Password for the database access |
|
database |
string Database name |
OnPremiseClickhouse
|
Field |
Description |
|
shards[] |
|
|
http_port |
int64 |
|
native_port |
int64 |
|
tls_mode |
TLS settings for server connection. Disabled by default |
ClickhouseShard
|
Field |
Description |
|
name |
string |
|
hosts[] |
string |
MysqlTarget
Settings specific to the MySQL target endpoint
|
Field |
Description |
|
connection |
Database connection settings |
|
database |
string Database name Allowed to leave it empty, then the tables will be created in databases with the |
|
user |
string User for database access. Required unless connection manager connection is used |
|
password |
Password for database access. |
|
sql_mode |
string sql_mode to use when |
|
skip_constraint_checks |
bool Disable constraints checks |
|
timezone |
string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from |
|
cleanup_policy |
enum CleanupPolicy Cleanup policy for activate, reactivate and reupload processes.
|
|
service_database |
string Database schema for service table |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
is_schema_migration_disabled |
bool Whether can change table schema if schema changed on source |
PostgresTarget
Settings specific to the PostgreSQL target endpoint
|
Field |
Description |
|
connection |
Database connection settings |
|
database |
string Target database name |
|
user |
string User for database access. Required unless Connection Manager connection is used |
|
password |
Password for database access. |
|
cleanup_policy |
enum CleanupPolicy Cleanup policy for activate, reactivate and reupload processes.
|
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
is_schema_migration_disabled |
bool Whether can change table schema if schema changed on source |
ClickhouseTarget
Settings specific to the ClickHouse target endpoint
|
Field |
Description |
|
connection |
Connection settings |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
alt_names[] |
Table renaming rules in target |
|
cleanup_policy |
enum ClickhouseCleanupPolicy How to clean collections when activating the transfer. One of
|
|
sharding |
Shard selection rules for the data being transferred |
|
is_schema_migration_disabled |
bool Whether can change table schema if schema changed on source |
|
clickhouse_cluster_name |
string Name of the ClickHouse cluster. For Managed ClickHouse that is name of |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
AltName
|
Field |
Description |
|
from_name |
string Source table name |
|
to_name |
string Target table name |
ClickhouseSharding
|
Field |
Description |
|
column_value_hash |
Shard data by the hash value of the specified column Includes only one of the fields |
|
custom_mapping |
A custom shard mapping by the value of the specified column Includes only one of the fields |
|
transfer_id |
Shard data by ID of the transfer Includes only one of the fields |
|
round_robin |
Distribute incoming rows between ClickHouse shards in a round-robin manner. Includes only one of the fields |
ColumnValueHash
|
Field |
Description |
|
column_name |
string The name of the column to calculate hash from |
ColumnValueMapping
|
Field |
Description |
|
column_name |
string The name of the column to inspect when deciding the shard to chose for an |
|
mapping[] |
The mapping of the specified column values to the shard names |
ValueToShard
|
Field |
Description |
|
column_value |
The value of the column. Currently only the string columns are supported |
|
shard_name |
string The name of the shard into which all the rows with the specified |
ColumnValue
|
Field |
Description |
|
string_value |
string Includes only one of the fields |
YdbTarget
Settings specific to the YDB target endpoint
|
Field |
Description |
|
database |
string Database path in YDB where tables are stored. |
|
instance |
string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135. |
|
path |
string Path extension for database, each table will be layouted into this path |
|
service_account_id |
string Service account ID for interaction with database |
|
cleanup_policy |
enum YdbCleanupPolicy Cleanup policy determine how to clean collections when activating the transfer.
|
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
sa_key_content |
string Authentication key |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
is_table_column_oriented |
bool Whether a column-oriented (i.e. OLAP) tables should be created. |
|
default_compression |
enum YdbDefaultCompression Compression that will be used for default columns family on YDB table creation.
|
|
is_schema_migration_disabled |
bool Whether can change table schema if schema changed on source |
KafkaTarget
Settings specific to the Kafka target endpoint
|
Field |
Description |
|
connection |
Connection settings |
|
auth |
Authentication settings |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
|
topic_settings |
Target topic settings |
|
serializer |
Data serialization format settings |
KafkaTargetTopicSettings
|
Field |
Description |
|
topic |
All messages will be sent to one topic Includes only one of the fields |
|
topic_prefix |
string Topic prefix Includes only one of the fields |
KafkaTargetTopic
|
Field |
Description |
|
topic_name |
string Full topic name |
|
save_tx_order |
bool Save transactions order |
Serializer
Data serialization format
|
Field |
Description |
|
serializer_auto |
Empty block. Select the serialization format automatically Includes only one of the fields |
|
serializer_json |
Empty block. Serialize data in json format Includes only one of the fields |
|
serializer_debezium |
Serialize data in debezium json format Includes only one of the fields |
SerializerAuto
Empty block. Select data serialization format automatically
|
Field |
Description |
|
Empty |
|
SerializerJSON
Empty block. Serialize data in json format
|
Field |
Description |
|
Empty |
|
SerializerDebezium
Serialize data in json format
|
Field |
Description |
|
serializer_parameters[] |
A list of Debezium parameters set by the structure of the |
DebeziumSerializerParameter
|
Field |
Description |
|
key |
string Name of the serializer parameter |
|
value |
string Value of the serializer parameter |
MongoTarget
Settings specific to the MongoDB target endpoint
|
Field |
Description |
|
connection |
Connection settings |
|
database |
string Database name. If not empty, then all the data will be written to the database |
|
cleanup_policy |
enum CleanupPolicy How to clean collections when activating the transfer. One of
|
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
MetrikaSource
Settings specific to the Yandex Metrika source endpoint
|
Field |
Description |
|
counter_ids[] |
int64 Counter IDs |
|
token |
Authentication token |
|
streams[] |
Streams |
MetrikaStream
|
Field |
Description |
|
type |
enum MetrikaStreamType Stream type, one of: METRIKA_STREAM_TYPE_HITS, METRIKA_STREAM_TYPE_VISITS,
|
|
columns[] |
string Column names |
YDSTarget
Settings specific to the YDS target endpoint
|
Field |
Description |
|
database |
string Database path in YDB for streams |
|
stream |
string Stream to write to |
|
service_account_id |
string Service account ID which has read access to the stream |
|
save_tx_order |
bool Save transaction order |
|
compression_codec |
enum YdsCompressionCodec Codec to use for output data compression. If not specified, no compression will
|
|
serializer |
Data serialization format |
|
endpoint |
string YDS Endpoint for dedicated db |
|
subnet_id |
string Identifier of the Yandex Cloud VPC subnetwork to user for accessing the |
|
security_groups[] |
string List of security groups that the transfer associated with this endpoint should |
Runtime
|
Field |
Description |
|
yc_runtime |
Includes only one of the fields |
YcRuntime
YC Runtime parameters for the transfer
|
Field |
Description |
|
job_count |
int64 Number of workers in parallel replication. |
|
upload_shard_params |
Parallel snapshot parameters |
ShardingUploadParams
Parallel snapshot parameters
|
Field |
Description |
|
job_count |
int64 Number of workers. |
|
process_count |
int64 Number of threads. |
Transformation
Transformation is converting data using special transformer functions.
These functions are executed on a data stream, applied to each data change item,
and transform them.
A transformer can be run at both the metadata and data levels.
Data can only be transformed if the source and target are of different types.
|
Field |
Description |
|
transformers[] |
A list of transformers. You can specify exactly 1 transformer in each element of |
Transformer
Some transformers may have limitations and only apply to some source-target
pairs.
|
Field |
Description |
|
mask_field |
Mask field transformer allows you to hash data Includes only one of the fields |
|
filter_columns |
Set up a list of table columns to transfer Includes only one of the fields |
|
rename_tables |
Set rules for renaming tables by specifying the current names of the tables in Includes only one of the fields |
|
replace_primary_key |
Override primary keys. Includes only one of the fields |
|
convert_to_string |
Convert column values to strings Includes only one of the fields |
|
sharder_transformer |
Set the number of shards for particular tables and a list of columns whose Includes only one of the fields |
|
table_splitter_transformer |
Splits the X table into multiple tables (X_1, X_2, ..., X_n) based on data. Includes only one of the fields |
|
filter_rows |
This filter only applies to transfers with queues (Logbroker or Apache Kafka®) Includes only one of the fields |
MaskFieldTransformer
Mask field transformer allows you to hash data
|
Field |
Description |
|
tables |
List of included and excluded tables |
|
columns[] |
string Specify the name of the column for data masking (a regular expression). |
|
function |
Mask function |
TablesFilter
Filter tables using lists of included and excluded tables.
|
Field |
Description |
|
include_tables[] |
string List of tables that will be included to transfer |
|
exclude_tables[] |
string List of tables that will be excluded to transfer |
MaskFunction
Mask function
|
Field |
Description |
|
mask_function_hash |
Hash mask function Includes only one of the fields |
MaskFunctionHash
Hash data using HMAC
|
Field |
Description |
|
user_defined_salt |
string This string will be used in the HMAC(sha256, salt) function applied to the |
FilterColumnsTransformer
Set up a list of table columns to transfer
|
Field |
Description |
|
tables |
List of the tables to filter using lists of included and excluded tables. |
|
columns |
List of the columns to transfer to the target tables using lists of included and |
ColumnsFilter
Filter columns using lists of included and excluded columns.
|
Field |
Description |
|
include_columns[] |
string List of columns that will be included to transfer |
|
exclude_columns[] |
string List of columns that will be excluded to transfer |
RenameTablesTransformer
Set rules for renaming tables by specifying the current names of the tables in
the source and new names for these tables in the target.
|
Field |
Description |
|
rename_tables[] |
List of renaming rules |
RenameTable
Specify rule for renaming table
|
Field |
Description |
|
original_name |
Specify the current names of the table in the source |
|
new_name |
Specify the new names for this table in the target |
Table
|
Field |
Description |
|
name_space |
string |
|
name |
string |
ReplacePrimaryKeyTransformer
Override primary keys
|
Field |
Description |
|
tables |
List of included and excluded tables |
|
keys[] |
string List of columns to be used as primary keys |
ToStringTransformer
Convert column values to strings
The values will be converted depending on the source type
Conversion rules are described here:
https://cloud.yandex.com/en/docs/data-transfer/concepts/data-transformation#convert-to-string
|
Field |
Description |
|
tables |
List of included and excluded tables |
|
columns |
List of included and excluded columns |
SharderTransformer
Set the number of shards for particular tables and a list of columns whose
values will be used for calculating a hash to determine a shard.
|
Field |
Description |
|
tables |
List of included and excluded tables |
|
columns |
List of included and excluded columns Includes only one of the fields |
|
random |
Includes only one of the fields |
|
shards_count |
int64 Number of shards |
SharderTransformerTypeRandom
|
Field |
Description |
|
Empty |
|
TableSplitterTransformer
A transfer splits the X table into multiple tables (X_1, X_2, ..., X_n) based on
data.
If a row was located in the X table before it was split, it is now in the X_i
table,
where i is determined by the column list and split string parameters.
Example:
If the column list has two columns, month of birth and gender, specified and the
split string states @,
information about an employee whose name is John and who was born on February
11, 1984,
from the Employees table will get to a new table named Employees@February@male.
|
Field |
Description |
|
tables |
List of included and excluded tables |
|
columns[] |
string Specify the columns in the tables to be partitioned. |
|
splitter |
string Specify the split string to be used for merging components in a new table name. |
FilterRowsTransformer
This filter only applies to transfers with queues (Logbroker or Apache Kafka®)
as a data source.
When running a transfer, only the strings meeting the specified criteria remain
in a changefeed.
|
Field |
Description |
|
tables |
List of included and excluded tables. |
|
filter |
string Filtering criterion. This can be comparison operators for numeric, string, and |
|
filters[] |
string Data is transported if it satisfies at least one of filters. Consider that there |
DataObjects
|
Field |
Description |
|
include_objects[] |
string |