Data Transfer API, gRPC: TransferService
Written by
Updated at September 4, 2024
- Calls TransferService
- Create
- CreateTransferRequest
- Runtime
- YcRuntime
- ShardingUploadParams
- Transformation
- Transformer
- MaskFieldTransformer
- TablesFilter
- MaskFunction
- MaskFunctionHash
- FilterColumnsTransformer
- ColumnsFilter
- RenameTablesTransformer
- RenameTable
- Table
- ReplacePrimaryKeyTransformer
- ToStringTransformer
- SharderTransformer
- TableSplitterTransformer
- FilterRowsTransformer
- Operation
- Update
- UpdateTransferRequest
- Runtime
- YcRuntime
- ShardingUploadParams
- Transformation
- Transformer
- MaskFieldTransformer
- TablesFilter
- MaskFunction
- MaskFunctionHash
- FilterColumnsTransformer
- ColumnsFilter
- RenameTablesTransformer
- RenameTable
- Table
- ReplacePrimaryKeyTransformer
- ToStringTransformer
- SharderTransformer
- TableSplitterTransformer
- FilterRowsTransformer
- Operation
- Delete
- List
- ListTransfersRequest
- ListTransfersResponse
- Transfer
- Endpoint
- EndpointSettings
- MysqlSource
- MysqlConnection
- OnPremiseMysql
- TLSMode
- TLSConfig
- Secret
- MysqlObjectTransferSettings
- PostgresSource
- PostgresConnection
- OnPremisePostgres
- PostgresObjectTransferSettings
- YdbSource
- YDSSource
- Parser
- GenericParserCommon
- DataSchema
- FieldList
- ColSchema
- AuditTrailsV1Parser
- CloudLoggingParser
- KafkaSource
- KafkaConnectionOptions
- OnPremiseKafka
- KafkaAuth
- KafkaSaslSecurity
- NoAuth
- DataTransformationOptions
- MongoSource
- MongoConnection
- MongoConnectionOptions
- OnPremiseMongo
- MongoCollection
- ClickhouseSource
- ClickhouseConnection
- ClickhouseConnectionOptions
- OnPremiseClickhouse
- ClickhouseShard
- MysqlTarget
- PostgresTarget
- ClickhouseTarget
- AltName
- ClickhouseSharding
- ColumnValueHash
- ColumnValueMapping
- ValueToShard
- ColumnValue
- YdbTarget
- KafkaTarget
- KafkaTargetTopicSettings
- KafkaTargetTopic
- Serializer
- SerializerAuto
- SerializerJSON
- SerializerDebezium
- DebeziumSerializerParameter
- MongoTarget
- MetrikaSource
- MetrikaStream
- YDSTarget
- Runtime
- YcRuntime
- ShardingUploadParams
- Transformation
- Transformer
- MaskFieldTransformer
- TablesFilter
- MaskFunction
- MaskFunctionHash
- FilterColumnsTransformer
- ColumnsFilter
- RenameTablesTransformer
- RenameTable
- Table
- ReplacePrimaryKeyTransformer
- ToStringTransformer
- SharderTransformer
- TableSplitterTransformer
- FilterRowsTransformer
- Get
- GetTransferRequest
- Transfer
- Endpoint
- EndpointSettings
- MysqlSource
- MysqlConnection
- OnPremiseMysql
- TLSMode
- TLSConfig
- Secret
- MysqlObjectTransferSettings
- PostgresSource
- PostgresConnection
- OnPremisePostgres
- PostgresObjectTransferSettings
- YdbSource
- YDSSource
- Parser
- GenericParserCommon
- DataSchema
- FieldList
- ColSchema
- AuditTrailsV1Parser
- CloudLoggingParser
- KafkaSource
- KafkaConnectionOptions
- OnPremiseKafka
- KafkaAuth
- KafkaSaslSecurity
- NoAuth
- DataTransformationOptions
- MongoSource
- MongoConnection
- MongoConnectionOptions
- OnPremiseMongo
- MongoCollection
- ClickhouseSource
- ClickhouseConnection
- ClickhouseConnectionOptions
- OnPremiseClickhouse
- ClickhouseShard
- MysqlTarget
- PostgresTarget
- ClickhouseTarget
- AltName
- ClickhouseSharding
- ColumnValueHash
- ColumnValueMapping
- ValueToShard
- ColumnValue
- YdbTarget
- KafkaTarget
- KafkaTargetTopicSettings
- KafkaTargetTopic
- Serializer
- SerializerAuto
- SerializerJSON
- SerializerDebezium
- DebeziumSerializerParameter
- MongoTarget
- MetrikaSource
- MetrikaStream
- YDSTarget
- Runtime
- YcRuntime
- ShardingUploadParams
- Transformation
- Transformer
- MaskFieldTransformer
- TablesFilter
- MaskFunction
- MaskFunctionHash
- FilterColumnsTransformer
- ColumnsFilter
- RenameTablesTransformer
- RenameTable
- Table
- ReplacePrimaryKeyTransformer
- ToStringTransformer
- SharderTransformer
- TableSplitterTransformer
- FilterRowsTransformer
- Deactivate
- Activate
Call | Description |
---|---|
Create | |
Update | |
Delete | |
List | |
Get | |
Deactivate | |
Activate |
Calls TransferService
Create
rpc Create (CreateTransferRequest) returns (operation.Operation)
CreateTransferRequest
Field | Description |
---|---|
source_id | string |
target_id | string |
description | string |
folder_id | string |
runtime | Runtime |
type | enum TransferType
|
name | string |
labels | map<string,string> |
transformation | Transformation |
Runtime
Field | Description |
---|---|
runtime | oneof: yc_runtime |
yc_runtime | YcRuntime |
YcRuntime
Field | Description |
---|---|
job_count | int64 |
upload_shard_params | ShardingUploadParams |
ShardingUploadParams
Field | Description |
---|---|
job_count | int64 |
process_count | int64 |
Transformation
Field | Description |
---|---|
transformers[] | Transformer Transformers are set as a list. When activating a transfer, a transformation plan is made for the tables that match the specified criteria. Transformers are applied to the tables in the sequence specified in the list. |
Transformer
Field | Description |
---|---|
transformer | oneof: mask_field , filter_columns , rename_tables , replace_primary_key , convert_to_string , sharder_transformer , table_splitter_transformer or filter_rows |
mask_field | MaskFieldTransformer |
filter_columns | FilterColumnsTransformer |
rename_tables | RenameTablesTransformer |
replace_primary_key | ReplacePrimaryKeyTransformer |
convert_to_string | ToStringTransformer |
sharder_transformer | SharderTransformer |
table_splitter_transformer | TableSplitterTransformer |
filter_rows | FilterRowsTransformer |
MaskFieldTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the name of the column for data masking (a regular expression). |
function | MaskFunction Mask function |
TablesFilter
Field | Description |
---|---|
include_tables[] | string List of tables that will be included to transfer |
exclude_tables[] | string List of tables that will be excluded to transfer |
MaskFunction
Field | Description |
---|---|
mask_function | oneof: mask_function_hash |
mask_function_hash | MaskFunctionHash Hash mask function |
MaskFunctionHash
Field | Description |
---|---|
user_defined_salt | string This string will be used in the HMAC(sha256, salt) function applied to the column data. |
FilterColumnsTransformer
Field | Description |
---|---|
tables | TablesFilter List of the tables to filter using lists of included and excluded tables. |
columns | ColumnsFilter List of the columns to transfer to the target tables using lists of included and excluded columns. |
ColumnsFilter
Field | Description |
---|---|
include_columns[] | string List of columns that will be included to transfer |
exclude_columns[] | string List of columns that will be excluded to transfer |
RenameTablesTransformer
Field | Description |
---|---|
rename_tables[] | RenameTable List of renaming rules |
RenameTable
Field | Description |
---|---|
original_name | Table Specify the current names of the table in the source |
new_name | Table Specify the new names for this table in the target |
Table
Field | Description |
---|---|
name_space | string |
name | string |
ReplacePrimaryKeyTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
keys[] | string List of columns to be used as primary keys |
ToStringTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
SharderTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
shards_count | int64 Number of shards |
TableSplitterTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the columns in the tables to be partitioned. |
splitter | string Specify the split string to be used for merging components in a new table name. |
FilterRowsTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables. |
filter | string Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. Deprecated: Use filters instead. |
filters[] | string Data is transported if it satisfies at least one of filters. Consider that there is OR statement between filters. Each filter can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details in docs: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty |
Update
rpc Update (UpdateTransferRequest) returns (operation.Operation)
UpdateTransferRequest
Field | Description |
---|---|
transfer_id | string Identifier of the transfer to be updated. |
description | string The new description for the transfer. |
runtime | Runtime |
name | string The new transfer name. Must be unique within the folder. |
update_mask | google.protobuf.FieldMask Field mask specifying transfer fields to be updated. Semantics for this field is described here: https://pkg.go.dev/google.golang.org/protobuf/types/known/fieldmaskpb#FieldMask |
labels | map<string,string> |
transformation | Transformation |
Runtime
Field | Description |
---|---|
runtime | oneof: yc_runtime |
yc_runtime | YcRuntime |
YcRuntime
Field | Description |
---|---|
job_count | int64 |
upload_shard_params | ShardingUploadParams |
ShardingUploadParams
Field | Description |
---|---|
job_count | int64 |
process_count | int64 |
Transformation
Field | Description |
---|---|
transformers[] | Transformer Transformers are set as a list. When activating a transfer, a transformation plan is made for the tables that match the specified criteria. Transformers are applied to the tables in the sequence specified in the list. |
Transformer
Field | Description |
---|---|
transformer | oneof: mask_field , filter_columns , rename_tables , replace_primary_key , convert_to_string , sharder_transformer , table_splitter_transformer or filter_rows |
mask_field | MaskFieldTransformer |
filter_columns | FilterColumnsTransformer |
rename_tables | RenameTablesTransformer |
replace_primary_key | ReplacePrimaryKeyTransformer |
convert_to_string | ToStringTransformer |
sharder_transformer | SharderTransformer |
table_splitter_transformer | TableSplitterTransformer |
filter_rows | FilterRowsTransformer |
MaskFieldTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the name of the column for data masking (a regular expression). |
function | MaskFunction Mask function |
TablesFilter
Field | Description |
---|---|
include_tables[] | string List of tables that will be included to transfer |
exclude_tables[] | string List of tables that will be excluded to transfer |
MaskFunction
Field | Description |
---|---|
mask_function | oneof: mask_function_hash |
mask_function_hash | MaskFunctionHash Hash mask function |
MaskFunctionHash
Field | Description |
---|---|
user_defined_salt | string This string will be used in the HMAC(sha256, salt) function applied to the column data. |
FilterColumnsTransformer
Field | Description |
---|---|
tables | TablesFilter List of the tables to filter using lists of included and excluded tables. |
columns | ColumnsFilter List of the columns to transfer to the target tables using lists of included and excluded columns. |
ColumnsFilter
Field | Description |
---|---|
include_columns[] | string List of columns that will be included to transfer |
exclude_columns[] | string List of columns that will be excluded to transfer |
RenameTablesTransformer
Field | Description |
---|---|
rename_tables[] | RenameTable List of renaming rules |
RenameTable
Field | Description |
---|---|
original_name | Table Specify the current names of the table in the source |
new_name | Table Specify the new names for this table in the target |
Table
Field | Description |
---|---|
name_space | string |
name | string |
ReplacePrimaryKeyTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
keys[] | string List of columns to be used as primary keys |
ToStringTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
SharderTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
shards_count | int64 Number of shards |
TableSplitterTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the columns in the tables to be partitioned. |
splitter | string Specify the split string to be used for merging components in a new table name. |
FilterRowsTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables. |
filter | string Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. Deprecated: Use filters instead. |
filters[] | string Data is transported if it satisfies at least one of filters. Consider that there is OR statement between filters. Each filter can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details in docs: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty |
Delete
rpc Delete (DeleteTransferRequest) returns (operation.Operation)
DeleteTransferRequest
Field | Description |
---|---|
transfer_id | string |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty |
List
rpc List (ListTransfersRequest) returns (ListTransfersResponse)
ListTransfersRequest
Field | Description |
---|---|
folder_id | string Identifier of the folder containing the transfers to be listed. |
page_size | int64 The maximum number of transfers to be sent in the response message. If the folder contains more transfers than page_size , next_page_token will be included in the response message. Include it into the subsequent ListTransfersRequest to fetch the next page. Defaults to 100 if not specified. The maximum allowed value for this field is 500 . |
page_token | string Opaque value identifying the transfers page to be fetched. Should be empty in the first ListTransfersRequest . Subsequent requests should have this field filled with the next_page_token from the previous ListTransfersResponse . |
ListTransfersResponse
Field | Description |
---|---|
transfers[] | Transfer The list of transfers. If there are more transfers in the folder, then next_page_token is a non-empty string to be included into the subsequent ListTransfersRequest to fetch the next transfers page. |
next_page_token | string Opaque value identifying the next transfers page. This field is empty if there are no more transfers in the folder. Otherwise it is non-empty and should be included in the subsequent ListTransfersRequest to fetch the next transfers page. |
Transfer
Field | Description |
---|---|
id | string |
folder_id | string |
name | string |
description | string |
labels | map<string,string> |
source | Endpoint |
target | Endpoint |
runtime | Runtime |
status | enum TransferStatus
|
type | enum TransferType
|
warning | string |
transformation | Transformation |
prestable | bool |
Endpoint
Field | Description |
---|---|
id | string |
folder_id | string |
name | string |
description | string |
labels | map<string,string> |
settings | EndpointSettings |
EndpointSettings
Field | Description |
---|---|
settings | oneof: mysql_source , postgres_source , ydb_source , yds_source , kafka_source , mongo_source , clickhouse_source , mysql_target , postgres_target , clickhouse_target , ydb_target , kafka_target , mongo_target , metrika_source or yds_target |
mysql_source | endpoint.MysqlSource |
postgres_source | endpoint.PostgresSource |
ydb_source | endpoint.YdbSource |
yds_source | endpoint.YDSSource |
kafka_source | endpoint.KafkaSource |
mongo_source | endpoint.MongoSource |
clickhouse_source | endpoint.ClickhouseSource |
mysql_target | endpoint.MysqlTarget |
postgres_target | endpoint.PostgresTarget |
clickhouse_target | endpoint.ClickhouseTarget |
ydb_target | endpoint.YdbTarget |
kafka_target | endpoint.KafkaTarget |
mongo_target | endpoint.MongoTarget |
metrika_source | endpoint.MetrikaSource |
yds_target | endpoint.YDSTarget |
MysqlSource
Field | Description |
---|---|
connection | MysqlConnection Database connection settings |
database | string Database name You can leave it empty, then it will be possible to transfer tables from several databases at the same time from this source. |
user | string User for database access. |
password | Secret Password for database access. |
timezone | string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone. |
object_transfer_settings | MysqlObjectTransferSettings Schema migration Select database objects to be transferred during activation or deactivation. |
include_tables_regex[] | string |
exclude_tables_regex[] | string |
security_groups[] | string Security groups |
service_database | string Database for service tables Default: data source database. Here created technical tables (__tm_keeper, __tm_gtid_keeper). |
MysqlConnection
Field | Description |
---|---|
connection | oneof: mdb_cluster_id or on_premise |
mdb_cluster_id | string Managed Service for MySQL cluster ID |
on_premise | OnPremiseMysql Connection options for on-premise MySQL |
OnPremiseMysql
Field | Description |
---|---|
port | int64 Database port |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
hosts[] | string |
tls_mode | TLSMode TLS settings for server connection. Disabled by default. |
TLSMode
Field | Description |
---|---|
tls_mode | oneof: disabled or enabled |
disabled | google.protobuf.Empty |
enabled | TLSConfig |
TLSConfig
Field | Description |
---|---|
ca_certificate | string CA certificate X.509 certificate of the certificate authority which issued the server's certificate, in PEM format. When CA certificate is specified TLS is used to connect to the server. |
Secret
Field | Description |
---|---|
value | oneof: raw |
raw | string Raw secret value |
MysqlObjectTransferSettings
Field | Description |
---|---|
view | enum ObjectTransferStage Views CREATE VIEW ...
|
routine | enum ObjectTransferStage Routines CREATE PROCEDURE ...; CREATE FUNCTION ...;
|
trigger | enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
tables | enum ObjectTransferStage
|
PostgresSource
Field | Description |
---|---|
connection | PostgresConnection Database connection settings |
database | string Database name |
user | string User for database access. |
password | Secret Password for database access. |
include_tables[] | string Included tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns. |
exclude_tables[] | string Excluded tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns. |
slot_byte_lag_limit | int64 Maximum lag of replication slot (in bytes); after exceeding this limit replication will be aborted. |
service_schema | string Database schema for service tables (__consumer_keeper, __data_transfer_mole_finder). Default is public |
object_transfer_settings | PostgresObjectTransferSettings Select database objects to be transferred during activation or deactivation. |
security_groups[] | string Security groups |
PostgresConnection
Field | Description |
---|---|
connection | oneof: mdb_cluster_id or on_premise |
mdb_cluster_id | string Managed Service for PostgreSQL cluster ID |
on_premise | OnPremisePostgres Connection options for on-premise PostgreSQL |
OnPremisePostgres
Field | Description |
---|---|
port | int64 Will be used if the cluster ID is not specified. |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
hosts[] | string |
tls_mode | TLSMode TLS settings for server connection. Disabled by default. |
PostgresObjectTransferSettings
Field | Description |
---|---|
sequence | enum ObjectTransferStage Sequences CREATE SEQUENCE ...
|
sequence_owned_by | enum ObjectTransferStage Owned sequences CREATE SEQUENCE ... OWNED BY ...
|
table | enum ObjectTransferStage Tables CREATE TABLE ...
|
primary_key | enum ObjectTransferStage Primary keys ALTER TABLE ... ADD PRIMARY KEY ...
|
fk_constraint | enum ObjectTransferStage Foreign keys ALTER TABLE ... ADD FOREIGN KEY ...
|
default_values | enum ObjectTransferStage Default values ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...
|
constraint | enum ObjectTransferStage Constraints ALTER TABLE ... ADD CONSTRAINT ...
|
index | enum ObjectTransferStage Indexes CREATE INDEX ...
|
view | enum ObjectTransferStage Views CREATE VIEW ...
|
function | enum ObjectTransferStage Functions CREATE FUNCTION ...
|
trigger | enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
type | enum ObjectTransferStage Types CREATE TYPE ...
|
rule | enum ObjectTransferStage Rules CREATE RULE ...
|
collation | enum ObjectTransferStage Collations CREATE COLLATION ...
|
policy | enum ObjectTransferStage Policies CREATE POLICY ...
|
cast | enum ObjectTransferStage Casts CREATE CAST ...
|
materialized_view | enum ObjectTransferStage Materialized views CREATE MATERIALIZED VIEW ...
|
sequence_set | enum ObjectTransferStage
|
YdbSource
Field | Description |
---|---|
database | string Path in YDB where to store tables |
instance | string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135 |
paths[] | string |
service_account_id | string |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
sa_key_content | string Authorization Key |
security_groups[] | string Security groups |
changefeed_custom_name | string Pre-created change feed |
YDSSource
Field | Description |
---|---|
database | string Database |
stream | string Stream |
service_account_id | string SA which has read access to the stream. |
supported_codecs[] | enum YdsCompressionCodec Compression codec |
parser | Parser Data parsing rules |
allow_ttl_rewind | bool Should continue working, if consumer read lag exceed TTL of topic False: stop the transfer in error state, if detected lost data. True: continue working with losing part of data |
endpoint | string for dedicated db |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
security_groups[] | string Security groups |
consumer | string for important streams |
Parser
Field | Description |
---|---|
parser | oneof: json_parser , audit_trails_v1_parser , cloud_logging_parser or tskv_parser |
json_parser | GenericParserCommon |
audit_trails_v1_parser | AuditTrailsV1Parser |
cloud_logging_parser | CloudLoggingParser |
tskv_parser | GenericParserCommon |
GenericParserCommon
Field | Description |
---|---|
data_schema | DataSchema |
null_keys_allowed | bool Allow null keys, if no - null keys will be putted to unparsed data |
add_rest_column | bool Will add _rest column for all unknown fields |
unescape_string_values | bool Unescape string values |
DataSchema
Field | Description |
---|---|
schema | oneof: json_fields or fields |
json_fields | string |
fields | FieldList |
FieldList
Field | Description |
---|---|
fields[] | ColSchema Column schema |
ColSchema
Field | Description |
---|---|
name | string |
type | enum ColumnType |
key | bool |
required | bool |
path | string |
AuditTrailsV1Parser
Empty.
CloudLoggingParser
Empty.
KafkaSource
Field | Description |
---|---|
connection | KafkaConnectionOptions Connection settings |
auth | KafkaAuth Authentication settings |
security_groups[] | string Security groups |
topic_name | string Full source topic name Deprecated in favor of topic names |
transformer | DataTransformationOptions Data transformation rules |
parser | Parser Data parsing rules |
topic_names[] | string List of topic names to read |
KafkaConnectionOptions
Field | Description |
---|---|
connection | oneof: cluster_id or on_premise |
cluster_id | string Managed Service for Kafka cluster ID |
on_premise | OnPremiseKafka Connection options for on-premise Kafka |
OnPremiseKafka
Field | Description |
---|---|
broker_urls[] | string Kafka broker URLs |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
tls_mode | TLSMode TLS settings for broker connection. Disabled by default. |
KafkaAuth
Field | Description |
---|---|
security | oneof: sasl or no_auth |
sasl | KafkaSaslSecurity Authentication with SASL |
no_auth | NoAuth No authentication |
KafkaSaslSecurity
Field | Description |
---|---|
user | string User name |
mechanism | enum KafkaMechanism SASL mechanism for authentication |
password | Secret Password for user |
NoAuth
Empty.
DataTransformationOptions
Field | Description |
---|---|
cloud_function | string Cloud function |
number_of_retries | int64 Number of retries |
buffer_size | string Buffer size for function |
buffer_flush_interval | string Flush interval |
invocation_timeout | string Invocation timeout |
service_account_id | string Service account |
MongoSource
Field | Description |
---|---|
connection | MongoConnection |
subnet_id | string |
collections[] | MongoCollection List of collections for replication. Empty list implies replication of all tables on the deployment. Allowed to use * as collection name. |
excluded_collections[] | MongoCollection List of forbidden collections for replication. Allowed to use * as collection name for forbid all collections of concrete schema. |
secondary_preferred_mode | bool Read mode for mongo client |
security_groups[] | string Security groups |
MongoConnection
Field | Description |
---|---|
connection | oneof: connection_options |
connection_options | MongoConnectionOptions |
MongoConnectionOptions
Field | Description |
---|---|
address | oneof: mdb_cluster_id or on_premise |
mdb_cluster_id | string |
on_premise | OnPremiseMongo |
user | string User name |
password | Secret Password for user |
auth_source | string Database name associated with the credentials |
OnPremiseMongo
Field | Description |
---|---|
hosts[] | string |
port | int64 |
replica_set | string |
tls_mode | TLSMode |
MongoCollection
Field | Description |
---|---|
database_name | string |
collection_name | string |
ClickhouseSource
Field | Description |
---|---|
connection | ClickhouseConnection |
include_tables[] | string While list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns. |
exclude_tables[] | string Exclude list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns. |
subnet_id | string |
security_groups[] | string |
clickhouse_cluster_name | string Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup. |
ClickhouseConnection
Field | Description |
---|---|
connection | oneof: connection_options |
connection_options | ClickhouseConnectionOptions |
ClickhouseConnectionOptions
Field | Description |
---|---|
address | oneof: on_premise or mdb_cluster_id |
on_premise | OnPremiseClickhouse |
mdb_cluster_id | string |
user | string |
password | Secret |
database | string Database |
OnPremiseClickhouse
Field | Description |
---|---|
shards[] | ClickhouseShard |
http_port | int64 |
native_port | int64 |
tls_mode | TLSMode |
ClickhouseShard
Field | Description |
---|---|
name | string |
hosts[] | string |
MysqlTarget
Field | Description |
---|---|
connection | MysqlConnection Database connection settings |
database | string Database name Allowed to leave it empty, then the tables will be created in databases with the same names as on the source. If this field is empty, then you must fill below db schema for service table. |
user | string User for database access. |
password | Secret Password for database access. |
sql_mode | string Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION. |
skip_constraint_checks | bool Disable constraints checks Recommend to disable for increase replication speed, but if schema contain cascading operations we don't recommend to disable. This option set FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0. |
timezone | string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone. |
cleanup_policy | enum CleanupPolicy Cleanup policy Cleanup policy for activate, reactivate and reupload processes. Default is DISABLED.
|
service_database | string Database schema for service table Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper). |
security_groups[] | string Security groups |
PostgresTarget
Field | Description |
---|---|
connection | PostgresConnection Database connection settings |
database | string Database name |
user | string User for database access. |
password | Secret Password for database access. |
cleanup_policy | enum CleanupPolicy Cleanup policy for activate, reactivate and reupload processes. Default is truncate.
|
security_groups[] | string Security groups |
ClickhouseTarget
Field | Description |
---|---|
connection | ClickhouseConnection |
subnet_id | string |
alt_names[] | AltName Alternative table names in target |
cleanup_policy | enum ClickhouseCleanupPolicy |
sharding | ClickhouseSharding |
clickhouse_cluster_name | string Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup. |
security_groups[] | string |
AltName
Field | Description |
---|---|
from_name | string Source table name |
to_name | string Target table name |
ClickhouseSharding
Field | Description |
---|---|
sharding | oneof: column_value_hash , custom_mapping , transfer_id or round_robin |
column_value_hash | ColumnValueHash |
custom_mapping | ColumnValueMapping |
transfer_id | google.protobuf.Empty |
round_robin | google.protobuf.Empty |
ColumnValueHash
Field | Description |
---|---|
column_name | string |
ColumnValueMapping
Field | Description |
---|---|
column_name | string |
mapping[] | ValueToShard |
ValueToShard
Field | Description |
---|---|
column_value | ColumnValue |
shard_name | string |
ColumnValue
Field | Description |
---|---|
value | oneof: string_value |
string_value | string |
YdbTarget
Field | Description |
---|---|
database | string Path in YDB where to store tables |
instance | string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135 |
path | string Path extension for database, each table will be layouted into this path |
service_account_id | string |
cleanup_policy | enum YdbCleanupPolicy Cleanup policy |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
sa_key_content | string SA content |
security_groups[] | string Security groups |
is_table_column_oriented | bool Should create column-oriented table (OLAP). By default it creates row-oriented (OLTP) |
default_compression | enum YdbDefaultCompression Compression that will be used for default columns family on YDB table creation |
KafkaTarget
Field | Description |
---|---|
connection | KafkaConnectionOptions Connection settings |
auth | KafkaAuth Authentication settings |
security_groups[] | string Security groups |
topic_settings | KafkaTargetTopicSettings Target topic settings |
serializer | Serializer Data serialization format settings |
KafkaTargetTopicSettings
Field | Description |
---|---|
topic_settings | oneof: topic or topic_prefix |
topic | KafkaTargetTopic Full topic name |
topic_prefix | string Topic prefix Analogue of the Debezium setting database.server.name. Messages will be sent to topic with name <topic_prefix>. |
KafkaTargetTopic
Field | Description |
---|---|
topic_name | string Topic name |
save_tx_order | bool Save transactions order Not to split events queue into separate per-table queues. |
Serializer
Field | Description |
---|---|
serializer | oneof: serializer_auto , serializer_json or serializer_debezium |
serializer_auto | SerializerAuto Select the serialization format automatically |
serializer_json | SerializerJSON Serialize data in json format |
serializer_debezium | SerializerDebezium Serialize data in debezium format |
SerializerAuto
Empty.
SerializerJSON
Empty.
SerializerDebezium
Field | Description |
---|---|
serializer_parameters[] | DebeziumSerializerParameter Settings of sterilization parameters as key-value pairs |
DebeziumSerializerParameter
Field | Description |
---|---|
key | string Name of the serializer parameter |
value | string Value of the serializer parameter |
MongoTarget
Field | Description |
---|---|
connection | MongoConnection |
database | string Database name |
cleanup_policy | enum CleanupPolicy
|
subnet_id | string |
security_groups[] | string Security groups |
MetrikaSource
Field | Description |
---|---|
counter_ids[] | int64 |
token | Secret |
streams[] | MetrikaStream |
MetrikaStream
Field | Description |
---|---|
type | enum MetrikaStreamType |
columns[] | string |
YDSTarget
Field | Description |
---|---|
database | string Database |
stream | string Stream |
service_account_id | string SA which has read access to the stream. |
save_tx_order | bool Save transaction order Not to split events queue into separate per-table queues. Incompatible with setting Topic prefix, only with Topic full name. |
serializer | Serializer Data serialization format |
endpoint | string for dedicated db |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
security_groups[] | string Security groups |
Runtime
Field | Description |
---|---|
runtime | oneof: yc_runtime |
yc_runtime | YcRuntime |
YcRuntime
Field | Description |
---|---|
job_count | int64 |
upload_shard_params | ShardingUploadParams |
ShardingUploadParams
Field | Description |
---|---|
job_count | int64 |
process_count | int64 |
Transformation
Field | Description |
---|---|
transformers[] | Transformer Transformers are set as a list. When activating a transfer, a transformation plan is made for the tables that match the specified criteria. Transformers are applied to the tables in the sequence specified in the list. |
Transformer
Field | Description |
---|---|
transformer | oneof: mask_field , filter_columns , rename_tables , replace_primary_key , convert_to_string , sharder_transformer , table_splitter_transformer or filter_rows |
mask_field | MaskFieldTransformer |
filter_columns | FilterColumnsTransformer |
rename_tables | RenameTablesTransformer |
replace_primary_key | ReplacePrimaryKeyTransformer |
convert_to_string | ToStringTransformer |
sharder_transformer | SharderTransformer |
table_splitter_transformer | TableSplitterTransformer |
filter_rows | FilterRowsTransformer |
MaskFieldTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the name of the column for data masking (a regular expression). |
function | MaskFunction Mask function |
TablesFilter
Field | Description |
---|---|
include_tables[] | string List of tables that will be included to transfer |
exclude_tables[] | string List of tables that will be excluded to transfer |
MaskFunction
Field | Description |
---|---|
mask_function | oneof: mask_function_hash |
mask_function_hash | MaskFunctionHash Hash mask function |
MaskFunctionHash
Field | Description |
---|---|
user_defined_salt | string This string will be used in the HMAC(sha256, salt) function applied to the column data. |
FilterColumnsTransformer
Field | Description |
---|---|
tables | TablesFilter List of the tables to filter using lists of included and excluded tables. |
columns | ColumnsFilter List of the columns to transfer to the target tables using lists of included and excluded columns. |
ColumnsFilter
Field | Description |
---|---|
include_columns[] | string List of columns that will be included to transfer |
exclude_columns[] | string List of columns that will be excluded to transfer |
RenameTablesTransformer
Field | Description |
---|---|
rename_tables[] | RenameTable List of renaming rules |
RenameTable
Field | Description |
---|---|
original_name | Table Specify the current names of the table in the source |
new_name | Table Specify the new names for this table in the target |
Table
Field | Description |
---|---|
name_space | string |
name | string |
ReplacePrimaryKeyTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
keys[] | string List of columns to be used as primary keys |
ToStringTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
SharderTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
shards_count | int64 Number of shards |
TableSplitterTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the columns in the tables to be partitioned. |
splitter | string Specify the split string to be used for merging components in a new table name. |
FilterRowsTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables. |
filter | string Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. Deprecated: Use filters instead. |
filters[] | string Data is transported if it satisfies at least one of filters. Consider that there is OR statement between filters. Each filter can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details in docs: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. |
Get
rpc Get (GetTransferRequest) returns (Transfer)
GetTransferRequest
Field | Description |
---|---|
transfer_id | string |
Transfer
Field | Description |
---|---|
id | string |
folder_id | string |
name | string |
description | string |
labels | map<string,string> |
source | Endpoint |
target | Endpoint |
runtime | Runtime |
status | enum TransferStatus
|
type | enum TransferType
|
warning | string |
transformation | Transformation |
prestable | bool |
Endpoint
Field | Description |
---|---|
id | string |
folder_id | string |
name | string |
description | string |
labels | map<string,string> |
settings | EndpointSettings |
EndpointSettings
Field | Description |
---|---|
settings | oneof: mysql_source , postgres_source , ydb_source , yds_source , kafka_source , mongo_source , clickhouse_source , mysql_target , postgres_target , clickhouse_target , ydb_target , kafka_target , mongo_target , metrika_source or yds_target |
mysql_source | endpoint.MysqlSource |
postgres_source | endpoint.PostgresSource |
ydb_source | endpoint.YdbSource |
yds_source | endpoint.YDSSource |
kafka_source | endpoint.KafkaSource |
mongo_source | endpoint.MongoSource |
clickhouse_source | endpoint.ClickhouseSource |
mysql_target | endpoint.MysqlTarget |
postgres_target | endpoint.PostgresTarget |
clickhouse_target | endpoint.ClickhouseTarget |
ydb_target | endpoint.YdbTarget |
kafka_target | endpoint.KafkaTarget |
mongo_target | endpoint.MongoTarget |
metrika_source | endpoint.MetrikaSource |
yds_target | endpoint.YDSTarget |
MysqlSource
Field | Description |
---|---|
connection | MysqlConnection Database connection settings |
database | string Database name You can leave it empty, then it will be possible to transfer tables from several databases at the same time from this source. |
user | string User for database access. |
password | Secret Password for database access. |
timezone | string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone. |
object_transfer_settings | MysqlObjectTransferSettings Schema migration Select database objects to be transferred during activation or deactivation. |
include_tables_regex[] | string |
exclude_tables_regex[] | string |
security_groups[] | string Security groups |
service_database | string Database for service tables Default: data source database. Here created technical tables (__tm_keeper, __tm_gtid_keeper). |
MysqlConnection
Field | Description |
---|---|
connection | oneof: mdb_cluster_id or on_premise |
mdb_cluster_id | string Managed Service for MySQL cluster ID |
on_premise | OnPremiseMysql Connection options for on-premise MySQL |
OnPremiseMysql
Field | Description |
---|---|
port | int64 Database port |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
hosts[] | string |
tls_mode | TLSMode TLS settings for server connection. Disabled by default. |
TLSMode
Field | Description |
---|---|
tls_mode | oneof: disabled or enabled |
disabled | google.protobuf.Empty |
enabled | TLSConfig |
TLSConfig
Field | Description |
---|---|
ca_certificate | string CA certificate X.509 certificate of the certificate authority which issued the server's certificate, in PEM format. When CA certificate is specified TLS is used to connect to the server. |
Secret
Field | Description |
---|---|
value | oneof: raw |
raw | string Raw secret value |
MysqlObjectTransferSettings
Field | Description |
---|---|
view | enum ObjectTransferStage Views CREATE VIEW ...
|
routine | enum ObjectTransferStage Routines CREATE PROCEDURE ...; CREATE FUNCTION ...;
|
trigger | enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
tables | enum ObjectTransferStage
|
PostgresSource
Field | Description |
---|---|
connection | PostgresConnection Database connection settings |
database | string Database name |
user | string User for database access. |
password | Secret Password for database access. |
include_tables[] | string Included tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns. |
exclude_tables[] | string Excluded tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns. |
slot_byte_lag_limit | int64 Maximum lag of replication slot (in bytes); after exceeding this limit replication will be aborted. |
service_schema | string Database schema for service tables (__consumer_keeper, __data_transfer_mole_finder). Default is public |
object_transfer_settings | PostgresObjectTransferSettings Select database objects to be transferred during activation or deactivation. |
security_groups[] | string Security groups |
PostgresConnection
Field | Description |
---|---|
connection | oneof: mdb_cluster_id or on_premise |
mdb_cluster_id | string Managed Service for PostgreSQL cluster ID |
on_premise | OnPremisePostgres Connection options for on-premise PostgreSQL |
OnPremisePostgres
Field | Description |
---|---|
port | int64 Will be used if the cluster ID is not specified. |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
hosts[] | string |
tls_mode | TLSMode TLS settings for server connection. Disabled by default. |
PostgresObjectTransferSettings
Field | Description |
---|---|
sequence | enum ObjectTransferStage Sequences CREATE SEQUENCE ...
|
sequence_owned_by | enum ObjectTransferStage Owned sequences CREATE SEQUENCE ... OWNED BY ...
|
table | enum ObjectTransferStage Tables CREATE TABLE ...
|
primary_key | enum ObjectTransferStage Primary keys ALTER TABLE ... ADD PRIMARY KEY ...
|
fk_constraint | enum ObjectTransferStage Foreign keys ALTER TABLE ... ADD FOREIGN KEY ...
|
default_values | enum ObjectTransferStage Default values ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...
|
constraint | enum ObjectTransferStage Constraints ALTER TABLE ... ADD CONSTRAINT ...
|
index | enum ObjectTransferStage Indexes CREATE INDEX ...
|
view | enum ObjectTransferStage Views CREATE VIEW ...
|
function | enum ObjectTransferStage Functions CREATE FUNCTION ...
|
trigger | enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
type | enum ObjectTransferStage Types CREATE TYPE ...
|
rule | enum ObjectTransferStage Rules CREATE RULE ...
|
collation | enum ObjectTransferStage Collations CREATE COLLATION ...
|
policy | enum ObjectTransferStage Policies CREATE POLICY ...
|
cast | enum ObjectTransferStage Casts CREATE CAST ...
|
materialized_view | enum ObjectTransferStage Materialized views CREATE MATERIALIZED VIEW ...
|
sequence_set | enum ObjectTransferStage
|
YdbSource
Field | Description |
---|---|
database | string Path in YDB where to store tables |
instance | string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135 |
paths[] | string |
service_account_id | string |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
sa_key_content | string Authorization Key |
security_groups[] | string Security groups |
changefeed_custom_name | string Pre-created change feed |
YDSSource
Field | Description |
---|---|
database | string Database |
stream | string Stream |
service_account_id | string SA which has read access to the stream. |
supported_codecs[] | enum YdsCompressionCodec Compression codec |
parser | Parser Data parsing rules |
allow_ttl_rewind | bool Should continue working, if consumer read lag exceed TTL of topic False: stop the transfer in error state, if detected lost data. True: continue working with losing part of data |
endpoint | string for dedicated db |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
security_groups[] | string Security groups |
consumer | string for important streams |
Parser
Field | Description |
---|---|
parser | oneof: json_parser , audit_trails_v1_parser , cloud_logging_parser or tskv_parser |
json_parser | GenericParserCommon |
audit_trails_v1_parser | AuditTrailsV1Parser |
cloud_logging_parser | CloudLoggingParser |
tskv_parser | GenericParserCommon |
GenericParserCommon
Field | Description |
---|---|
data_schema | DataSchema |
null_keys_allowed | bool Allow null keys, if no - null keys will be putted to unparsed data |
add_rest_column | bool Will add _rest column for all unknown fields |
unescape_string_values | bool Unescape string values |
DataSchema
Field | Description |
---|---|
schema | oneof: json_fields or fields |
json_fields | string |
fields | FieldList |
FieldList
Field | Description |
---|---|
fields[] | ColSchema Column schema |
ColSchema
Field | Description |
---|---|
name | string |
type | enum ColumnType |
key | bool |
required | bool |
path | string |
AuditTrailsV1Parser
Empty.
CloudLoggingParser
Empty.
KafkaSource
Field | Description |
---|---|
connection | KafkaConnectionOptions Connection settings |
auth | KafkaAuth Authentication settings |
security_groups[] | string Security groups |
topic_name | string Full source topic name Deprecated in favor of topic names |
transformer | DataTransformationOptions Data transformation rules |
parser | Parser Data parsing rules |
topic_names[] | string List of topic names to read |
KafkaConnectionOptions
Field | Description |
---|---|
connection | oneof: cluster_id or on_premise |
cluster_id | string Managed Service for Kafka cluster ID |
on_premise | OnPremiseKafka Connection options for on-premise Kafka |
OnPremiseKafka
Field | Description |
---|---|
broker_urls[] | string Kafka broker URLs |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
tls_mode | TLSMode TLS settings for broker connection. Disabled by default. |
KafkaAuth
Field | Description |
---|---|
security | oneof: sasl or no_auth |
sasl | KafkaSaslSecurity Authentication with SASL |
no_auth | NoAuth No authentication |
KafkaSaslSecurity
Field | Description |
---|---|
user | string User name |
mechanism | enum KafkaMechanism SASL mechanism for authentication |
password | Secret Password for user |
NoAuth
Empty.
DataTransformationOptions
Field | Description |
---|---|
cloud_function | string Cloud function |
number_of_retries | int64 Number of retries |
buffer_size | string Buffer size for function |
buffer_flush_interval | string Flush interval |
invocation_timeout | string Invocation timeout |
service_account_id | string Service account |
MongoSource
Field | Description |
---|---|
connection | MongoConnection |
subnet_id | string |
collections[] | MongoCollection List of collections for replication. Empty list implies replication of all tables on the deployment. Allowed to use * as collection name. |
excluded_collections[] | MongoCollection List of forbidden collections for replication. Allowed to use * as collection name for forbid all collections of concrete schema. |
secondary_preferred_mode | bool Read mode for mongo client |
security_groups[] | string Security groups |
MongoConnection
Field | Description |
---|---|
connection | oneof: connection_options |
connection_options | MongoConnectionOptions |
MongoConnectionOptions
Field | Description |
---|---|
address | oneof: mdb_cluster_id or on_premise |
mdb_cluster_id | string |
on_premise | OnPremiseMongo |
user | string User name |
password | Secret Password for user |
auth_source | string Database name associated with the credentials |
OnPremiseMongo
Field | Description |
---|---|
hosts[] | string |
port | int64 |
replica_set | string |
tls_mode | TLSMode |
MongoCollection
Field | Description |
---|---|
database_name | string |
collection_name | string |
ClickhouseSource
Field | Description |
---|---|
connection | ClickhouseConnection |
include_tables[] | string While list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns. |
exclude_tables[] | string Exclude list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns. |
subnet_id | string |
security_groups[] | string |
clickhouse_cluster_name | string Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup. |
ClickhouseConnection
Field | Description |
---|---|
connection | oneof: connection_options |
connection_options | ClickhouseConnectionOptions |
ClickhouseConnectionOptions
Field | Description |
---|---|
address | oneof: on_premise or mdb_cluster_id |
on_premise | OnPremiseClickhouse |
mdb_cluster_id | string |
user | string |
password | Secret |
database | string Database |
OnPremiseClickhouse
Field | Description |
---|---|
shards[] | ClickhouseShard |
http_port | int64 |
native_port | int64 |
tls_mode | TLSMode |
ClickhouseShard
Field | Description |
---|---|
name | string |
hosts[] | string |
MysqlTarget
Field | Description |
---|---|
connection | MysqlConnection Database connection settings |
database | string Database name Allowed to leave it empty, then the tables will be created in databases with the same names as on the source. If this field is empty, then you must fill below db schema for service table. |
user | string User for database access. |
password | Secret Password for database access. |
sql_mode | string Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION. |
skip_constraint_checks | bool Disable constraints checks Recommend to disable for increase replication speed, but if schema contain cascading operations we don't recommend to disable. This option set FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0. |
timezone | string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone. |
cleanup_policy | enum CleanupPolicy Cleanup policy Cleanup policy for activate, reactivate and reupload processes. Default is DISABLED.
|
service_database | string Database schema for service table Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper). |
security_groups[] | string Security groups |
PostgresTarget
Field | Description |
---|---|
connection | PostgresConnection Database connection settings |
database | string Database name |
user | string User for database access. |
password | Secret Password for database access. |
cleanup_policy | enum CleanupPolicy Cleanup policy for activate, reactivate and reupload processes. Default is truncate.
|
security_groups[] | string Security groups |
ClickhouseTarget
Field | Description |
---|---|
connection | ClickhouseConnection |
subnet_id | string |
alt_names[] | AltName Alternative table names in target |
cleanup_policy | enum ClickhouseCleanupPolicy |
sharding | ClickhouseSharding |
clickhouse_cluster_name | string Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup. |
security_groups[] | string |
AltName
Field | Description |
---|---|
from_name | string Source table name |
to_name | string Target table name |
ClickhouseSharding
Field | Description |
---|---|
sharding | oneof: column_value_hash , custom_mapping , transfer_id or round_robin |
column_value_hash | ColumnValueHash |
custom_mapping | ColumnValueMapping |
transfer_id | google.protobuf.Empty |
round_robin | google.protobuf.Empty |
ColumnValueHash
Field | Description |
---|---|
column_name | string |
ColumnValueMapping
Field | Description |
---|---|
column_name | string |
mapping[] | ValueToShard |
ValueToShard
Field | Description |
---|---|
column_value | ColumnValue |
shard_name | string |
ColumnValue
Field | Description |
---|---|
value | oneof: string_value |
string_value | string |
YdbTarget
Field | Description |
---|---|
database | string Path in YDB where to store tables |
instance | string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135 |
path | string Path extension for database, each table will be layouted into this path |
service_account_id | string |
cleanup_policy | enum YdbCleanupPolicy Cleanup policy |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
sa_key_content | string SA content |
security_groups[] | string Security groups |
is_table_column_oriented | bool Should create column-oriented table (OLAP). By default it creates row-oriented (OLTP) |
default_compression | enum YdbDefaultCompression Compression that will be used for default columns family on YDB table creation |
KafkaTarget
Field | Description |
---|---|
connection | KafkaConnectionOptions Connection settings |
auth | KafkaAuth Authentication settings |
security_groups[] | string Security groups |
topic_settings | KafkaTargetTopicSettings Target topic settings |
serializer | Serializer Data serialization format settings |
KafkaTargetTopicSettings
Field | Description |
---|---|
topic_settings | oneof: topic or topic_prefix |
topic | KafkaTargetTopic Full topic name |
topic_prefix | string Topic prefix Analogue of the Debezium setting database.server.name. Messages will be sent to topic with name <topic_prefix>. |
KafkaTargetTopic
Field | Description |
---|---|
topic_name | string Topic name |
save_tx_order | bool Save transactions order Not to split events queue into separate per-table queues. |
Serializer
Field | Description |
---|---|
serializer | oneof: serializer_auto , serializer_json or serializer_debezium |
serializer_auto | SerializerAuto Select the serialization format automatically |
serializer_json | SerializerJSON Serialize data in json format |
serializer_debezium | SerializerDebezium Serialize data in debezium format |
SerializerAuto
Empty.
SerializerJSON
Empty.
SerializerDebezium
Field | Description |
---|---|
serializer_parameters[] | DebeziumSerializerParameter Settings of sterilization parameters as key-value pairs |
DebeziumSerializerParameter
Field | Description |
---|---|
key | string Name of the serializer parameter |
value | string Value of the serializer parameter |
MongoTarget
Field | Description |
---|---|
connection | MongoConnection |
database | string Database name |
cleanup_policy | enum CleanupPolicy
|
subnet_id | string |
security_groups[] | string Security groups |
MetrikaSource
Field | Description |
---|---|
counter_ids[] | int64 |
token | Secret |
streams[] | MetrikaStream |
MetrikaStream
Field | Description |
---|---|
type | enum MetrikaStreamType |
columns[] | string |
YDSTarget
Field | Description |
---|---|
database | string Database |
stream | string Stream |
service_account_id | string SA which has read access to the stream. |
save_tx_order | bool Save transaction order Not to split events queue into separate per-table queues. Incompatible with setting Topic prefix, only with Topic full name. |
serializer | Serializer Data serialization format |
endpoint | string for dedicated db |
subnet_id | string Network interface for endpoint. If none will assume public ipv4 |
security_groups[] | string Security groups |
Runtime
Field | Description |
---|---|
runtime | oneof: yc_runtime |
yc_runtime | YcRuntime |
YcRuntime
Field | Description |
---|---|
job_count | int64 |
upload_shard_params | ShardingUploadParams |
ShardingUploadParams
Field | Description |
---|---|
job_count | int64 |
process_count | int64 |
Transformation
Field | Description |
---|---|
transformers[] | Transformer Transformers are set as a list. When activating a transfer, a transformation plan is made for the tables that match the specified criteria. Transformers are applied to the tables in the sequence specified in the list. |
Transformer
Field | Description |
---|---|
transformer | oneof: mask_field , filter_columns , rename_tables , replace_primary_key , convert_to_string , sharder_transformer , table_splitter_transformer or filter_rows |
mask_field | MaskFieldTransformer |
filter_columns | FilterColumnsTransformer |
rename_tables | RenameTablesTransformer |
replace_primary_key | ReplacePrimaryKeyTransformer |
convert_to_string | ToStringTransformer |
sharder_transformer | SharderTransformer |
table_splitter_transformer | TableSplitterTransformer |
filter_rows | FilterRowsTransformer |
MaskFieldTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the name of the column for data masking (a regular expression). |
function | MaskFunction Mask function |
TablesFilter
Field | Description |
---|---|
include_tables[] | string List of tables that will be included to transfer |
exclude_tables[] | string List of tables that will be excluded to transfer |
MaskFunction
Field | Description |
---|---|
mask_function | oneof: mask_function_hash |
mask_function_hash | MaskFunctionHash Hash mask function |
MaskFunctionHash
Field | Description |
---|---|
user_defined_salt | string This string will be used in the HMAC(sha256, salt) function applied to the column data. |
FilterColumnsTransformer
Field | Description |
---|---|
tables | TablesFilter List of the tables to filter using lists of included and excluded tables. |
columns | ColumnsFilter List of the columns to transfer to the target tables using lists of included and excluded columns. |
ColumnsFilter
Field | Description |
---|---|
include_columns[] | string List of columns that will be included to transfer |
exclude_columns[] | string List of columns that will be excluded to transfer |
RenameTablesTransformer
Field | Description |
---|---|
rename_tables[] | RenameTable List of renaming rules |
RenameTable
Field | Description |
---|---|
original_name | Table Specify the current names of the table in the source |
new_name | Table Specify the new names for this table in the target |
Table
Field | Description |
---|---|
name_space | string |
name | string |
ReplacePrimaryKeyTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
keys[] | string List of columns to be used as primary keys |
ToStringTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
SharderTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns | ColumnsFilter List of included and excluded columns |
shards_count | int64 Number of shards |
TableSplitterTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables |
columns[] | string Specify the columns in the tables to be partitioned. |
splitter | string Specify the split string to be used for merging components in a new table name. |
FilterRowsTransformer
Field | Description |
---|---|
tables | TablesFilter List of included and excluded tables. |
filter | string Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. Deprecated: Use filters instead. |
filters[] | string Data is transported if it satisfies at least one of filters. Consider that there is OR statement between filters. Each filter can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details in docs: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. |
Deactivate
rpc Deactivate (DeactivateTransferRequest) returns (operation.Operation)
DeactivateTransferRequest
Field | Description |
---|---|
transfer_id | string |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty |
Activate
rpc Activate (ActivateTransferRequest) returns (operation.Operation)
ActivateTransferRequest
Field | Description |
---|---|
transfer_id | string |
Operation
Field | Description |
---|---|
id | string ID of the operation. |
description | string Description of the operation. 0-256 characters long. |
created_at | google.protobuf.Timestamp Creation timestamp. |
created_by | string ID of the user or service account who initiated the operation. |
modified_at | google.protobuf.Timestamp The time when the Operation resource was last modified. |
done | bool If the value is false , it means the operation is still in progress. If true , the operation is completed, and either error or response is available. |
metadata | google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any. |
result | oneof: error or response The operation result. If done == false and there was no failure detected, neither error nor response is set. If done == false and there was a failure detected, error is set. If done == true , exactly one of error or response is set. |
error | google.rpc.Status The error result of the operation in case of failure or cancellation. |
response | google.protobuf.Any The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty |