Data Transfer API, gRPC: TransferService.Get
- gRPC request
- GetTransferRequest
- Transfer
- Endpoint
- EndpointSettings
- MysqlSource
- MysqlConnection
- OnPremiseMysql
- TLSMode
- TLSConfig
- Secret
- MysqlObjectTransferSettings
- PostgresSource
- PostgresConnection
- OnPremisePostgres
- PostgresObjectTransferSettings
- YdbSource
- YDSSource
- Parser
- GenericParserCommon
- DataSchema
- FieldList
- ColSchema
- AuditTrailsV1Parser
- CloudLoggingParser
- KafkaSource
- KafkaConnectionOptions
- OnPremiseKafka
- KafkaAuth
- KafkaSaslSecurity
- NoAuth
- DataTransformationOptions
- MongoSource
- MongoConnection
- MongoConnectionOptions
- OnPremiseMongo
- MongoCollection
- ClickhouseSource
- ClickhouseConnection
- ClickhouseConnectionOptions
- OnPremiseClickhouse
- ClickhouseShard
- MysqlTarget
- PostgresTarget
- ClickhouseTarget
- AltName
- ClickhouseSharding
- ColumnValueHash
- ColumnValueMapping
- ValueToShard
- ColumnValue
- YdbTarget
- KafkaTarget
- KafkaTargetTopicSettings
- KafkaTargetTopic
- Serializer
- SerializerAuto
- SerializerJSON
- SerializerDebezium
- DebeziumSerializerParameter
- MongoTarget
- MetrikaSource
- MetrikaStream
- YDSTarget
- Runtime
- YcRuntime
- ShardingUploadParams
- Transformation
- Transformer
- MaskFieldTransformer
- TablesFilter
- MaskFunction
- MaskFunctionHash
- FilterColumnsTransformer
- ColumnsFilter
- RenameTablesTransformer
- RenameTable
- Table
- ReplacePrimaryKeyTransformer
- ToStringTransformer
- SharderTransformer
- TableSplitterTransformer
- FilterRowsTransformer
gRPC request
rpc Get (GetTransferRequest) returns (Transfer)
GetTransferRequest
{
"transferId": "string"
}
Field |
Description |
transferId |
string |
Transfer
{
"id": "string",
"folderId": "string",
"name": "string",
"description": "string",
"labels": "string",
"source": {
"id": "string",
"folderId": "string",
"name": "string",
"description": "string",
"labels": "string",
"settings": {
// Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`
"mysqlSource": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"timezone": "string",
"objectTransferSettings": {
"view": "ObjectTransferStage",
"routine": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"tables": "ObjectTransferStage"
},
"includeTablesRegex": [
"string"
],
"excludeTablesRegex": [
"string"
],
"securityGroups": [
"string"
],
"serviceDatabase": "string"
},
"postgresSource": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"includeTables": [
"string"
],
"excludeTables": [
"string"
],
"slotByteLagLimit": "int64",
"serviceSchema": "string",
"objectTransferSettings": {
"sequence": "ObjectTransferStage",
"sequenceOwnedBy": "ObjectTransferStage",
"table": "ObjectTransferStage",
"primaryKey": "ObjectTransferStage",
"fkConstraint": "ObjectTransferStage",
"defaultValues": "ObjectTransferStage",
"constraint": "ObjectTransferStage",
"index": "ObjectTransferStage",
"view": "ObjectTransferStage",
"function": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"type": "ObjectTransferStage",
"rule": "ObjectTransferStage",
"collation": "ObjectTransferStage",
"policy": "ObjectTransferStage",
"cast": "ObjectTransferStage",
"materializedView": "ObjectTransferStage",
"sequenceSet": "ObjectTransferStage"
},
"securityGroups": [
"string"
]
},
"ydbSource": {
"database": "string",
"instance": "string",
"paths": [
"string"
],
"serviceAccountId": "string",
"subnetId": "string",
"saKeyContent": "string",
"securityGroups": [
"string"
],
"changefeedCustomName": "string"
},
"ydsSource": {
"database": "string",
"stream": "string",
"serviceAccountId": "string",
"supportedCodecs": [
"YdsCompressionCodec"
],
"parser": {
// Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
"jsonParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
},
"auditTrailsV1Parser": "AuditTrailsV1Parser",
"cloudLoggingParser": "CloudLoggingParser",
"tskvParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
}
// end of the list of possible fields
},
"allowTtlRewind": "bool",
"endpoint": "string",
"subnetId": "string",
"securityGroups": [
"string"
],
"consumer": "string"
},
"kafkaSource": {
"connection": {
// Includes only one of the fields `clusterId`, `onPremise`
"clusterId": "string",
"onPremise": {
"brokerUrls": [
"string"
],
"subnetId": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `noAuth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"noAuth": "NoAuth"
// end of the list of possible fields
},
"securityGroups": [
"string"
],
"topicName": "string",
"transformer": {
"cloudFunction": "string",
"numberOfRetries": "int64",
"bufferSize": "string",
"bufferFlushInterval": "string",
"invocationTimeout": "string",
"serviceAccountId": "string"
},
"parser": {
// Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
"jsonParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
},
"auditTrailsV1Parser": "AuditTrailsV1Parser",
"cloudLoggingParser": "CloudLoggingParser",
"tskvParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
}
// end of the list of possible fields
},
"topicNames": [
"string"
]
},
"mongoSource": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"hosts": [
"string"
],
"port": "int64",
"replicaSet": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"authSource": "string"
}
// end of the list of possible fields
},
"subnetId": "string",
"collections": [
{
"databaseName": "string",
"collectionName": "string"
}
],
"excludedCollections": [
{
"databaseName": "string",
"collectionName": "string"
}
],
"secondaryPreferredMode": "bool",
"securityGroups": [
"string"
]
},
"clickhouseSource": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `onPremise`, `mdbClusterId`
"onPremise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"httpPort": "int64",
"nativePort": "int64",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
"mdbClusterId": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"includeTables": [
"string"
],
"excludeTables": [
"string"
],
"subnetId": "string",
"securityGroups": [
"string"
],
"clickhouseClusterName": "string"
},
"mysqlTarget": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"sqlMode": "string",
"skipConstraintChecks": "bool",
"timezone": "string",
"cleanupPolicy": "CleanupPolicy",
"serviceDatabase": "string",
"securityGroups": [
"string"
]
},
"postgresTarget": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"cleanupPolicy": "CleanupPolicy",
"securityGroups": [
"string"
]
},
"clickhouseTarget": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `onPremise`, `mdbClusterId`
"onPremise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"httpPort": "int64",
"nativePort": "int64",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
"mdbClusterId": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"subnetId": "string",
"altNames": [
{
"fromName": "string",
"toName": "string"
}
],
"cleanupPolicy": "ClickhouseCleanupPolicy",
"sharding": {
// Includes only one of the fields `columnValueHash`, `customMapping`, `transferId`, `roundRobin`
"columnValueHash": {
"columnName": "string"
},
"customMapping": {
"columnName": "string",
"mapping": [
{
"columnValue": {
// Includes only one of the fields `stringValue`
"stringValue": "string"
// end of the list of possible fields
},
"shardName": "string"
}
]
},
"transferId": "google.protobuf.Empty",
"roundRobin": "google.protobuf.Empty"
// end of the list of possible fields
},
"clickhouseClusterName": "string",
"securityGroups": [
"string"
]
},
"ydbTarget": {
"database": "string",
"instance": "string",
"path": "string",
"serviceAccountId": "string",
"cleanupPolicy": "YdbCleanupPolicy",
"subnetId": "string",
"saKeyContent": "string",
"securityGroups": [
"string"
],
"isTableColumnOriented": "bool",
"defaultCompression": "YdbDefaultCompression"
},
"kafkaTarget": {
"connection": {
// Includes only one of the fields `clusterId`, `onPremise`
"clusterId": "string",
"onPremise": {
"brokerUrls": [
"string"
],
"subnetId": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `noAuth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"noAuth": "NoAuth"
// end of the list of possible fields
},
"securityGroups": [
"string"
],
"topicSettings": {
// Includes only one of the fields `topic`, `topicPrefix`
"topic": {
"topicName": "string",
"saveTxOrder": "bool"
},
"topicPrefix": "string"
// end of the list of possible fields
},
"serializer": {
// Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
"serializerAuto": "SerializerAuto",
"serializerJson": "SerializerJSON",
"serializerDebezium": {
"serializerParameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
}
},
"mongoTarget": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"hosts": [
"string"
],
"port": "int64",
"replicaSet": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"authSource": "string"
}
// end of the list of possible fields
},
"database": "string",
"cleanupPolicy": "CleanupPolicy",
"subnetId": "string",
"securityGroups": [
"string"
]
},
"metrikaSource": {
"counterIds": [
"int64"
],
"token": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"streams": [
{
"type": "MetrikaStreamType",
"columns": [
"string"
]
}
]
},
"ydsTarget": {
"database": "string",
"stream": "string",
"serviceAccountId": "string",
"saveTxOrder": "bool",
"serializer": {
// Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
"serializerAuto": "SerializerAuto",
"serializerJson": "SerializerJSON",
"serializerDebezium": {
"serializerParameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
},
"endpoint": "string",
"subnetId": "string",
"securityGroups": [
"string"
]
}
// end of the list of possible fields
}
},
"target": {
"id": "string",
"folderId": "string",
"name": "string",
"description": "string",
"labels": "string",
"settings": {
// Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`
"mysqlSource": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"timezone": "string",
"objectTransferSettings": {
"view": "ObjectTransferStage",
"routine": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"tables": "ObjectTransferStage"
},
"includeTablesRegex": [
"string"
],
"excludeTablesRegex": [
"string"
],
"securityGroups": [
"string"
],
"serviceDatabase": "string"
},
"postgresSource": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"includeTables": [
"string"
],
"excludeTables": [
"string"
],
"slotByteLagLimit": "int64",
"serviceSchema": "string",
"objectTransferSettings": {
"sequence": "ObjectTransferStage",
"sequenceOwnedBy": "ObjectTransferStage",
"table": "ObjectTransferStage",
"primaryKey": "ObjectTransferStage",
"fkConstraint": "ObjectTransferStage",
"defaultValues": "ObjectTransferStage",
"constraint": "ObjectTransferStage",
"index": "ObjectTransferStage",
"view": "ObjectTransferStage",
"function": "ObjectTransferStage",
"trigger": "ObjectTransferStage",
"type": "ObjectTransferStage",
"rule": "ObjectTransferStage",
"collation": "ObjectTransferStage",
"policy": "ObjectTransferStage",
"cast": "ObjectTransferStage",
"materializedView": "ObjectTransferStage",
"sequenceSet": "ObjectTransferStage"
},
"securityGroups": [
"string"
]
},
"ydbSource": {
"database": "string",
"instance": "string",
"paths": [
"string"
],
"serviceAccountId": "string",
"subnetId": "string",
"saKeyContent": "string",
"securityGroups": [
"string"
],
"changefeedCustomName": "string"
},
"ydsSource": {
"database": "string",
"stream": "string",
"serviceAccountId": "string",
"supportedCodecs": [
"YdsCompressionCodec"
],
"parser": {
// Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
"jsonParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
},
"auditTrailsV1Parser": "AuditTrailsV1Parser",
"cloudLoggingParser": "CloudLoggingParser",
"tskvParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
}
// end of the list of possible fields
},
"allowTtlRewind": "bool",
"endpoint": "string",
"subnetId": "string",
"securityGroups": [
"string"
],
"consumer": "string"
},
"kafkaSource": {
"connection": {
// Includes only one of the fields `clusterId`, `onPremise`
"clusterId": "string",
"onPremise": {
"brokerUrls": [
"string"
],
"subnetId": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `noAuth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"noAuth": "NoAuth"
// end of the list of possible fields
},
"securityGroups": [
"string"
],
"topicName": "string",
"transformer": {
"cloudFunction": "string",
"numberOfRetries": "int64",
"bufferSize": "string",
"bufferFlushInterval": "string",
"invocationTimeout": "string",
"serviceAccountId": "string"
},
"parser": {
// Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
"jsonParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
},
"auditTrailsV1Parser": "AuditTrailsV1Parser",
"cloudLoggingParser": "CloudLoggingParser",
"tskvParser": {
"dataSchema": {
// Includes only one of the fields `jsonFields`, `fields`
"jsonFields": "string",
"fields": {
"fields": [
{
"name": "string",
"type": "ColumnType",
"key": "bool",
"required": "bool",
"path": "string"
}
]
}
// end of the list of possible fields
},
"nullKeysAllowed": "bool",
"addRestColumn": "bool",
"unescapeStringValues": "bool"
}
// end of the list of possible fields
},
"topicNames": [
"string"
]
},
"mongoSource": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"hosts": [
"string"
],
"port": "int64",
"replicaSet": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"authSource": "string"
}
// end of the list of possible fields
},
"subnetId": "string",
"collections": [
{
"databaseName": "string",
"collectionName": "string"
}
],
"excludedCollections": [
{
"databaseName": "string",
"collectionName": "string"
}
],
"secondaryPreferredMode": "bool",
"securityGroups": [
"string"
]
},
"clickhouseSource": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `onPremise`, `mdbClusterId`
"onPremise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"httpPort": "int64",
"nativePort": "int64",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
"mdbClusterId": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"includeTables": [
"string"
],
"excludeTables": [
"string"
],
"subnetId": "string",
"securityGroups": [
"string"
],
"clickhouseClusterName": "string"
},
"mysqlTarget": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"sqlMode": "string",
"skipConstraintChecks": "bool",
"timezone": "string",
"cleanupPolicy": "CleanupPolicy",
"serviceDatabase": "string",
"securityGroups": [
"string"
]
},
"postgresTarget": {
"connection": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"port": "int64",
"subnetId": "string",
"hosts": [
"string"
],
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"database": "string",
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"cleanupPolicy": "CleanupPolicy",
"securityGroups": [
"string"
]
},
"clickhouseTarget": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `onPremise`, `mdbClusterId`
"onPremise": {
"shards": [
{
"name": "string",
"hosts": [
"string"
]
}
],
"httpPort": "int64",
"nativePort": "int64",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
"mdbClusterId": "string",
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"database": "string"
}
// end of the list of possible fields
},
"subnetId": "string",
"altNames": [
{
"fromName": "string",
"toName": "string"
}
],
"cleanupPolicy": "ClickhouseCleanupPolicy",
"sharding": {
// Includes only one of the fields `columnValueHash`, `customMapping`, `transferId`, `roundRobin`
"columnValueHash": {
"columnName": "string"
},
"customMapping": {
"columnName": "string",
"mapping": [
{
"columnValue": {
// Includes only one of the fields `stringValue`
"stringValue": "string"
// end of the list of possible fields
},
"shardName": "string"
}
]
},
"transferId": "google.protobuf.Empty",
"roundRobin": "google.protobuf.Empty"
// end of the list of possible fields
},
"clickhouseClusterName": "string",
"securityGroups": [
"string"
]
},
"ydbTarget": {
"database": "string",
"instance": "string",
"path": "string",
"serviceAccountId": "string",
"cleanupPolicy": "YdbCleanupPolicy",
"subnetId": "string",
"saKeyContent": "string",
"securityGroups": [
"string"
],
"isTableColumnOriented": "bool",
"defaultCompression": "YdbDefaultCompression"
},
"kafkaTarget": {
"connection": {
// Includes only one of the fields `clusterId`, `onPremise`
"clusterId": "string",
"onPremise": {
"brokerUrls": [
"string"
],
"subnetId": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
}
// end of the list of possible fields
},
"auth": {
// Includes only one of the fields `sasl`, `noAuth`
"sasl": {
"user": "string",
"mechanism": "KafkaMechanism",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
}
},
"noAuth": "NoAuth"
// end of the list of possible fields
},
"securityGroups": [
"string"
],
"topicSettings": {
// Includes only one of the fields `topic`, `topicPrefix`
"topic": {
"topicName": "string",
"saveTxOrder": "bool"
},
"topicPrefix": "string"
// end of the list of possible fields
},
"serializer": {
// Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
"serializerAuto": "SerializerAuto",
"serializerJson": "SerializerJSON",
"serializerDebezium": {
"serializerParameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
}
},
"mongoTarget": {
"connection": {
// Includes only one of the fields `connectionOptions`
"connectionOptions": {
// Includes only one of the fields `mdbClusterId`, `onPremise`
"mdbClusterId": "string",
"onPremise": {
"hosts": [
"string"
],
"port": "int64",
"replicaSet": "string",
"tlsMode": {
// Includes only one of the fields `disabled`, `enabled`
"disabled": "google.protobuf.Empty",
"enabled": {
"caCertificate": "string"
}
// end of the list of possible fields
}
},
// end of the list of possible fields
"user": "string",
"password": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"authSource": "string"
}
// end of the list of possible fields
},
"database": "string",
"cleanupPolicy": "CleanupPolicy",
"subnetId": "string",
"securityGroups": [
"string"
]
},
"metrikaSource": {
"counterIds": [
"int64"
],
"token": {
// Includes only one of the fields `raw`
"raw": "string"
// end of the list of possible fields
},
"streams": [
{
"type": "MetrikaStreamType",
"columns": [
"string"
]
}
]
},
"ydsTarget": {
"database": "string",
"stream": "string",
"serviceAccountId": "string",
"saveTxOrder": "bool",
"serializer": {
// Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
"serializerAuto": "SerializerAuto",
"serializerJson": "SerializerJSON",
"serializerDebezium": {
"serializerParameters": [
{
"key": "string",
"value": "string"
}
]
}
// end of the list of possible fields
},
"endpoint": "string",
"subnetId": "string",
"securityGroups": [
"string"
]
}
// end of the list of possible fields
}
},
"runtime": {
// Includes only one of the fields `ycRuntime`
"ycRuntime": {
"jobCount": "int64",
"uploadShardParams": {
"jobCount": "int64",
"processCount": "int64"
}
}
// end of the list of possible fields
},
"status": "TransferStatus",
"type": "TransferType",
"warning": "string",
"transformation": {
"transformers": [
{
// Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`
"maskField": {
"tables": {
"includeTables": [
"string"
],
"excludeTables": [
"string"
]
},
"columns": [
"string"
],
"function": {
// Includes only one of the fields `maskFunctionHash`
"maskFunctionHash": {
"userDefinedSalt": "string"
}
// end of the list of possible fields
}
},
"filterColumns": {
"tables": {
"includeTables": [
"string"
],
"excludeTables": [
"string"
]
},
"columns": {
"includeColumns": [
"string"
],
"excludeColumns": [
"string"
]
}
},
"renameTables": {
"renameTables": [
{
"originalName": {
"nameSpace": "string",
"name": "string"
},
"newName": {
"nameSpace": "string",
"name": "string"
}
}
]
},
"replacePrimaryKey": {
"tables": {
"includeTables": [
"string"
],
"excludeTables": [
"string"
]
},
"keys": [
"string"
]
},
"convertToString": {
"tables": {
"includeTables": [
"string"
],
"excludeTables": [
"string"
]
},
"columns": {
"includeColumns": [
"string"
],
"excludeColumns": [
"string"
]
}
},
"sharderTransformer": {
"tables": {
"includeTables": [
"string"
],
"excludeTables": [
"string"
]
},
"columns": {
"includeColumns": [
"string"
],
"excludeColumns": [
"string"
]
},
"shardsCount": "int64"
},
"tableSplitterTransformer": {
"tables": {
"includeTables": [
"string"
],
"excludeTables": [
"string"
]
},
"columns": [
"string"
],
"splitter": "string"
},
"filterRows": {
"tables": {
"includeTables": [
"string"
],
"excludeTables": [
"string"
]
},
"filter": "string",
"filters": [
"string"
]
}
// end of the list of possible fields
}
]
},
"prestable": "bool"
}
Transfer core entity
Field |
Description |
id |
string |
folderId |
string |
name |
string |
description |
string |
labels |
string |
source |
|
target |
|
runtime |
|
status |
enum TransferStatus
|
type |
enum TransferType
|
warning |
string |
transformation |
|
prestable |
bool |
Endpoint
Field |
Description |
id |
string |
folderId |
string |
name |
string |
description |
string |
labels |
string |
settings |
EndpointSettings
Field |
Description |
mysqlSource |
Includes only one of the fields |
postgresSource |
Includes only one of the fields |
ydbSource |
Includes only one of the fields |
ydsSource |
Includes only one of the fields |
kafkaSource |
Includes only one of the fields |
mongoSource |
Includes only one of the fields |
clickhouseSource |
Includes only one of the fields |
mysqlTarget |
Includes only one of the fields |
postgresTarget |
Includes only one of the fields |
clickhouseTarget |
Includes only one of the fields |
ydbTarget |
Includes only one of the fields |
kafkaTarget |
Includes only one of the fields |
mongoTarget |
Includes only one of the fields |
metrikaSource |
Includes only one of the fields |
ydsTarget |
Includes only one of the fields |
MysqlSource
Field |
Description |
connection |
Database connection settings |
database |
string Database name You can leave it empty, then it will be possible to transfer tables from several |
user |
string User for database access. |
password |
Password for database access. |
timezone |
string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from |
objectTransferSettings |
Schema migration Select database objects to be transferred during activation or deactivation. |
includeTablesRegex[] |
string |
excludeTablesRegex[] |
string |
securityGroups[] |
string Security groups |
serviceDatabase |
string Database for service tables Default: data source database. Here created technical tables (__tm_keeper, |
MysqlConnection
Field |
Description |
mdbClusterId |
string Managed Service for MySQL cluster ID Includes only one of the fields |
onPremise |
Connection options for on-premise MySQL Includes only one of the fields |
OnPremiseMysql
Field |
Description |
port |
int64 Database port |
subnetId |
string Network interface for endpoint. If none will assume public ipv4 |
hosts[] |
string |
tlsMode |
TLS settings for server connection. Disabled by default. |
TLSMode
Field |
Description |
disabled |
Includes only one of the fields |
enabled |
Includes only one of the fields |
TLSConfig
Field |
Description |
caCertificate |
string CA certificate X.509 certificate of the certificate authority which issued the server's |
Secret
Field |
Description |
raw |
string Raw secret value Includes only one of the fields |
MysqlObjectTransferSettings
Field |
Description |
view |
enum ObjectTransferStage Views CREATE VIEW ...
|
routine |
enum ObjectTransferStage Routines CREATE PROCEDURE ... ; CREATE FUNCTION ... ;
|
trigger |
enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
tables |
enum ObjectTransferStage
|
PostgresSource
Field |
Description |
connection |
Database connection settings |
database |
string Database name |
user |
string User for database access. not required as may be in connection |
password |
Password for database access. |
includeTables[] |
string Included tables If none or empty list is presented, all tables are replicated. Full table name |
excludeTables[] |
string Excluded tables If none or empty list is presented, all tables are replicated. Full table name |
slotByteLagLimit |
int64 Maximum lag of replication slot (in bytes); after exceeding this limit |
serviceSchema |
string Database schema for service tables (__consumer_keeper, |
objectTransferSettings |
PostgresObjectTransferSettings Select database objects to be transferred during activation or deactivation. |
securityGroups[] |
string Security groups |
PostgresConnection
Field |
Description |
mdbClusterId |
string Managed Service for PostgreSQL cluster ID Includes only one of the fields |
onPremise |
Connection options for on-premise PostgreSQL Includes only one of the fields |
OnPremisePostgres
Field |
Description |
port |
int64 Will be used if the cluster ID is not specified. |
subnetId |
string Network interface for endpoint. If none will assume public ipv4 |
hosts[] |
string |
tlsMode |
TLS settings for server connection. Disabled by default. |
PostgresObjectTransferSettings
Field |
Description |
sequence |
enum ObjectTransferStage Sequences CREATE SEQUENCE ...
|
sequenceOwnedBy |
enum ObjectTransferStage Owned sequences CREATE SEQUENCE ... OWNED BY ...
|
table |
enum ObjectTransferStage Tables CREATE TABLE ...
|
primaryKey |
enum ObjectTransferStage Primary keys ALTER TABLE ... ADD PRIMARY KEY ...
|
fkConstraint |
enum ObjectTransferStage Foreign keys ALTER TABLE ... ADD FOREIGN KEY ...
|
defaultValues |
enum ObjectTransferStage Default values ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...
|
constraint |
enum ObjectTransferStage Constraints ALTER TABLE ... ADD CONSTRAINT ...
|
index |
enum ObjectTransferStage Indexes CREATE INDEX ...
|
view |
enum ObjectTransferStage Views CREATE VIEW ...
|
function |
enum ObjectTransferStage Functions CREATE FUNCTION ...
|
trigger |
enum ObjectTransferStage Triggers CREATE TRIGGER ...
|
type |
enum ObjectTransferStage Types CREATE TYPE ...
|
rule |
enum ObjectTransferStage Rules CREATE RULE ...
|
collation |
enum ObjectTransferStage Collations CREATE COLLATION ...
|
policy |
enum ObjectTransferStage Policies CREATE POLICY ...
|
cast |
enum ObjectTransferStage Casts CREATE CAST ...
|
materializedView |
enum ObjectTransferStage Materialized views CREATE MATERIALIZED VIEW ...
|
sequenceSet |
enum ObjectTransferStage
|
YdbSource
Field |
Description |
database |
string Path in YDB where to store tables |
instance |
string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135 |
paths[] |
string |
serviceAccountId |
string |
subnetId |
string Network interface for endpoint. If none will assume public ipv4 |
saKeyContent |
string Authorization Key |
securityGroups[] |
string Security groups |
changefeedCustomName |
string Pre-created change feed |
YDSSource
Field |
Description |
database |
string Database |
stream |
string Stream |
serviceAccountId |
string SA which has read access to the stream. |
supportedCodecs[] |
enum YdsCompressionCodec Compression codec
|
parser |
Data parsing rules |
allowTtlRewind |
bool Should continue working, if consumer read lag exceed TTL of topic |
endpoint |
string for dedicated db |
subnetId |
string Network interface for endpoint. If none will assume public ipv4 |
securityGroups[] |
string Security groups |
consumer |
string for important streams |
Parser
Field |
Description |
jsonParser |
Includes only one of the fields |
auditTrailsV1Parser |
Includes only one of the fields |
cloudLoggingParser |
Includes only one of the fields |
tskvParser |
Includes only one of the fields |
GenericParserCommon
Field |
Description |
dataSchema |
|
nullKeysAllowed |
bool Allow null keys, if no - null keys will be putted to unparsed data |
addRestColumn |
bool Will add _rest column for all unknown fields |
unescapeStringValues |
bool Unescape string values |
DataSchema
Field |
Description |
jsonFields |
string Includes only one of the fields |
fields |
Includes only one of the fields |
FieldList
Field |
Description |
fields[] |
Column schema |
ColSchema
Field |
Description |
name |
string |
type |
enum ColumnType
|
key |
bool |
required |
bool |
path |
string |
AuditTrailsV1Parser
Field |
Description |
Empty |
CloudLoggingParser
Field |
Description |
Empty |
KafkaSource
Field |
Description |
connection |
Connection settings |
auth |
Authentication settings |
securityGroups[] |
string Security groups |
topicName |
string Full source topic name |
transformer |
Data transformation rules |
parser |
Data parsing rules |
topicNames[] |
string List of topic names to read |
KafkaConnectionOptions
Field |
Description |
clusterId |
string Managed Service for Kafka cluster ID Includes only one of the fields |
onPremise |
Connection options for on-premise Kafka Includes only one of the fields |
OnPremiseKafka
Field |
Description |
brokerUrls[] |
string Kafka broker URLs |
subnetId |
string Network interface for endpoint. If none will assume public ipv4 |
tlsMode |
TLS settings for broker connection. Disabled by default. |
KafkaAuth
Field |
Description |
sasl |
Authentication with SASL Includes only one of the fields |
noAuth |
No authentication Includes only one of the fields |
KafkaSaslSecurity
Field |
Description |
user |
string User name |
mechanism |
enum KafkaMechanism SASL mechanism for authentication
|
password |
Password for user |
NoAuth
No authentication
Field |
Description |
Empty |
DataTransformationOptions
Field |
Description |
cloudFunction |
string Cloud function |
numberOfRetries |
int64 Number of retries |
bufferSize |
string Buffer size for function |
bufferFlushInterval |
string Flush interval |
invocationTimeout |
string Invocation timeout |
serviceAccountId |
string Service account |
MongoSource
Field |
Description |
connection |
|
subnetId |
string |
collections[] |
List of collections for replication. Empty list implies replication of all |
excludedCollections[] |
List of forbidden collections for replication. Allowed to use * as collection |
secondaryPreferredMode |
bool Read mode for mongo client |
securityGroups[] |
string Security groups |
MongoConnection
Field |
Description |
connectionOptions |
Includes only one of the fields |
MongoConnectionOptions
Field |
Description |
mdbClusterId |
string Includes only one of the fields |
onPremise |
Includes only one of the fields |
user |
string User name |
password |
Password for user |
authSource |
string Database name associated with the credentials |
OnPremiseMongo
Field |
Description |
hosts[] |
string |
port |
int64 |
replicaSet |
string |
tlsMode |
MongoCollection
Field |
Description |
databaseName |
string |
collectionName |
string |
ClickhouseSource
Field |
Description |
connection |
|
includeTables[] |
string While list of tables for replication. If none or empty list is presented - will |
excludeTables[] |
string Exclude list of tables for replication. If none or empty list is presented - |
subnetId |
string |
securityGroups[] |
string |
clickhouseClusterName |
string Name of the ClickHouse cluster. For Managed ClickHouse that is name of |
ClickhouseConnection
Field |
Description |
connectionOptions |
Includes only one of the fields |
ClickhouseConnectionOptions
Field |
Description |
onPremise |
Includes only one of the fields |
mdbClusterId |
string Includes only one of the fields |
user |
string |
password |
|
database |
string Database |
OnPremiseClickhouse
Field |
Description |
shards[] |
|
httpPort |
int64 |
nativePort |
int64 |
tlsMode |
ClickhouseShard
Field |
Description |
name |
string |
hosts[] |
string |
MysqlTarget
Field |
Description |
connection |
Database connection settings |
database |
string Database name Allowed to leave it empty, then the tables will be created in databases with the |
user |
string User for database access. |
password |
Password for database access. |
sqlMode |
string Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION. |
skipConstraintChecks |
bool Disable constraints checks Recommend to disable for increase replication speed, but if schema contain |
timezone |
string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from |
cleanupPolicy |
enum CleanupPolicy Cleanup policy Cleanup policy for activate, reactivate and reupload processes. Default is
|
serviceDatabase |
string Database schema for service table Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper). |
securityGroups[] |
string Security groups |
PostgresTarget
Field |
Description |
connection |
Database connection settings |
database |
string Database name |
user |
string User for database access. not required as may be in connection |
password |
Password for database access. |
cleanupPolicy |
enum CleanupPolicy Cleanup policy for activate, reactivate and reupload processes. Default is
|
securityGroups[] |
string Security groups |
ClickhouseTarget
Field |
Description |
connection |
|
subnetId |
string |
altNames[] |
Alternative table names in target |
cleanupPolicy |
enum ClickhouseCleanupPolicy
|
sharding |
|
clickhouseClusterName |
string Name of the ClickHouse cluster. For Managed ClickHouse that is name of |
securityGroups[] |
string |
AltName
Field |
Description |
fromName |
string Source table name |
toName |
string Target table name |
ClickhouseSharding
Field |
Description |
columnValueHash |
Includes only one of the fields |
customMapping |
Includes only one of the fields |
transferId |
Includes only one of the fields |
roundRobin |
Includes only one of the fields |
ColumnValueHash
Field |
Description |
columnName |
string |
ColumnValueMapping
Field |
Description |
columnName |
string |
mapping[] |
ValueToShard
Field |
Description |
columnValue |
|
shardName |
string |
ColumnValue
Field |
Description |
stringValue |
string Includes only one of the fields |
YdbTarget
Field |
Description |
database |
string Path in YDB where to store tables |
instance |
string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135 |
path |
string Path extension for database, each table will be layouted into this path |
serviceAccountId |
string |
cleanupPolicy |
enum YdbCleanupPolicy Cleanup policy
|
subnetId |
string Network interface for endpoint. If none will assume public ipv4 |
saKeyContent |
string SA content |
securityGroups[] |
string Security groups |
isTableColumnOriented |
bool Should create column-oriented table (OLAP). By default it creates row-oriented |
defaultCompression |
enum YdbDefaultCompression Compression that will be used for default columns family on YDB table creation
|
KafkaTarget
Field |
Description |
connection |
Connection settings |
auth |
Authentication settings |
securityGroups[] |
string Security groups |
topicSettings |
Target topic settings |
serializer |
Data serialization format settings |
KafkaTargetTopicSettings
Field |
Description |
topic |
Full topic name Includes only one of the fields |
topicPrefix |
string Topic prefix Analogue of the Debezium setting database.server.name. Includes only one of the fields |
KafkaTargetTopic
Field |
Description |
topicName |
string Topic name |
saveTxOrder |
bool Save transactions order |
Serializer
Data serialization format
Field |
Description |
serializerAuto |
Select the serialization format automatically Includes only one of the fields |
serializerJson |
Serialize data in json format Includes only one of the fields |
serializerDebezium |
Serialize data in debezium format Includes only one of the fields |
SerializerAuto
Field |
Description |
Empty |
SerializerJSON
Field |
Description |
Empty |
SerializerDebezium
Field |
Description |
serializerParameters[] |
Settings of sterilization parameters as key-value pairs |
DebeziumSerializerParameter
Field |
Description |
key |
string Name of the serializer parameter |
value |
string Value of the serializer parameter |
MongoTarget
Field |
Description |
connection |
|
database |
string Database name |
cleanupPolicy |
enum CleanupPolicy
|
subnetId |
string |
securityGroups[] |
string Security groups |
MetrikaSource
Field |
Description |
counterIds[] |
int64 |
token |
|
streams[] |
MetrikaStream
Field |
Description |
type |
enum MetrikaStreamType
|
columns[] |
string |
YDSTarget
Field |
Description |
database |
string Database |
stream |
string Stream |
serviceAccountId |
string SA which has read access to the stream. |
saveTxOrder |
bool Save transaction order |
serializer |
Data serialization format |
endpoint |
string for dedicated db |
subnetId |
string Network interface for endpoint. If none will assume public ipv4 |
securityGroups[] |
string Security groups |
Runtime
Field |
Description |
ycRuntime |
Includes only one of the fields |
YcRuntime
Field |
Description |
jobCount |
int64 |
uploadShardParams |
ShardingUploadParams
Field |
Description |
jobCount |
int64 |
processCount |
int64 |
Transformation
Transformation is converting data using special transformer functions.
These functions are executed on a data stream, applied to each data change item,
and transform them.
A transformer can be run at both the metadata and data levels.
Data can only be transformed if the source and target are of different types.
Field |
Description |
transformers[] |
Transformers are set as a list. |
Transformer
Some transformers may have limitations and only apply to some source-target
pairs.
Field |
Description |
maskField |
Includes only one of the fields |
filterColumns |
Includes only one of the fields |
renameTables |
Includes only one of the fields |
replacePrimaryKey |
Includes only one of the fields |
convertToString |
Includes only one of the fields |
sharderTransformer |
Includes only one of the fields |
tableSplitterTransformer |
Includes only one of the fields |
filterRows |
Includes only one of the fields |
MaskFieldTransformer
Mask field transformer allows you to hash data
Field |
Description |
tables |
List of included and excluded tables |
columns[] |
string Specify the name of the column for data masking (a regular expression). |
function |
Mask function |
TablesFilter
Filter tables using lists of included and excluded tables.
Field |
Description |
includeTables[] |
string List of tables that will be included to transfer |
excludeTables[] |
string List of tables that will be excluded to transfer |
MaskFunction
Mask function
Field |
Description |
maskFunctionHash |
Hash mask function Includes only one of the fields |
MaskFunctionHash
Hash data using HMAC
Field |
Description |
userDefinedSalt |
string This string will be used in the HMAC(sha256, salt) function applied to the |
FilterColumnsTransformer
Set up a list of table columns to transfer
Field |
Description |
tables |
List of the tables to filter using lists of included and excluded tables. |
columns |
List of the columns to transfer to the target tables using lists of included and |
ColumnsFilter
Filter columns using lists of included and excluded columns.
Field |
Description |
includeColumns[] |
string List of columns that will be included to transfer |
excludeColumns[] |
string List of columns that will be excluded to transfer |
RenameTablesTransformer
Set rules for renaming tables by specifying the current names of the tables in
the source and new names for these tables in the target.
Field |
Description |
renameTables[] |
List of renaming rules |
RenameTable
Specify rule for renaming table
Field |
Description |
originalName |
Specify the current names of the table in the source |
newName |
Specify the new names for this table in the target |
Table
Field |
Description |
nameSpace |
string |
name |
string |
ReplacePrimaryKeyTransformer
Override primary keys
Field |
Description |
tables |
List of included and excluded tables |
keys[] |
string List of columns to be used as primary keys |
ToStringTransformer
Convert column values to strings
The values will be converted depending on the source type
Conversion rules are described here:
https://cloud.yandex.com/en/docs/data-transfer/concepts/data-transformation#convert-to-string
Field |
Description |
tables |
List of included and excluded tables |
columns |
List of included and excluded columns |
SharderTransformer
Set the number of shards for particular tables and a list of columns whose
values will be used for calculating a hash to determine a shard.
Field |
Description |
tables |
List of included and excluded tables |
columns |
List of included and excluded columns |
shardsCount |
int64 Number of shards |
TableSplitterTransformer
A transfer splits the X table into multiple tables (X_1, X_2, ..., X_n) based on
data.
If a row was located in the X table before it was split, it is now in the X_i
table,
where i is determined by the column list and split string parameters.
Example:
If the column list has two columns, month of birth and gender, specified and the
split string states @,
information about an employee whose name is John and who was born on February
11, 1984,
from the Employees table will get to a new table named Employees@February@male.
Field |
Description |
tables |
List of included and excluded tables |
columns[] |
string Specify the columns in the tables to be partitioned. |
splitter |
string Specify the split string to be used for merging components in a new table name. |
FilterRowsTransformer
This filter only applies to transfers with queues (Logbroker or Apache Kafka®)
as a data source.
When running a transfer, only the strings meeting the specified criteria remain
in a changefeed.
Field |
Description |
tables |
List of included and excluded tables. |
filter |
string Filtering criterion. This can be comparison operators for numeric, string, and |
filters[] |
string Data is transported if it satisfies at least one of filters. Consider that there |