Yandex Cloud
Поиск
Связаться с намиПодключиться
  • Документация
  • Блог
  • Все сервисы
  • Статус работы сервисов
    • Популярные
    • Инфраструктура и сеть
    • Платформа данных
    • Контейнеры
    • Инструменты разработчика
    • Бессерверные вычисления
    • Безопасность
    • Мониторинг и управление ресурсами
    • Машинное обучение
    • Бизнес-инструменты
  • Все решения
    • По отраслям
    • По типу задач
    • Экономика платформы
    • Безопасность
    • Техническая поддержка
    • Каталог партнёров
    • Обучение и сертификация
    • Облако для стартапов
    • Облако для крупного бизнеса
    • Центр технологий для общества
    • Облако для интеграторов
    • Поддержка IT-бизнеса
    • Облако для фрилансеров
    • Обучение и сертификация
    • Блог
    • Документация
    • Контент-программа
    • Мероприятия и вебинары
    • Контакты, чаты и сообщества
    • Идеи
    • Истории успеха
    • Тарифы Yandex Cloud
    • Промоакции и free tier
    • Правила тарификации
  • Документация
  • Блог
Проект Яндекса
© 2025 ООО «Яндекс.Облако»
Yandex Data Transfer
  • Доступные трансферы
  • Начало работы
  • Решение проблем
  • Управление доступом
  • Справочник Terraform
    • Аутентификация в API
      • Overview
        • Overview
        • Create
        • Update
        • Delete
        • List
        • Get
        • Deactivate
        • Activate
  • Метрики Monitoring
  • Аудитные логи Audit Trails
  • Публичные материалы
  • Обучающие курсы

В этой статье:

  • HTTP request
  • Path parameters
  • Query parameters
  • Response
  • Transfer
  • Endpoint
  • EndpointSettings
  • MysqlSource
  • MysqlConnection
  • OnPremiseMysql
  • TLSMode
  • TLSConfig
  • ConnectionManagerConnection
  • Secret
  • MysqlObjectTransferSettings
  • PostgresSource
  • PostgresConnection
  • OnPremisePostgres
  • PostgresObjectTransferSettings
  • YdbSource
  • YDSSource
  • Parser
  • GenericParserCommon
  • DataSchema
  • FieldList
  • ColSchema
  • KafkaSource
  • KafkaConnectionOptions
  • OnPremiseKafka
  • KafkaAuth
  • KafkaSaslSecurity
  • DataTransformationOptions
  • MongoSource
  • MongoConnection
  • MongoConnectionOptions
  • OnPremiseMongo
  • MongoCollection
  • ClickhouseSource
  • ClickhouseConnection
  • ClickhouseConnectionOptions
  • OnPremiseClickhouse
  • ClickhouseShard
  • MysqlTarget
  • PostgresTarget
  • ClickhouseTarget
  • AltName
  • ClickhouseSharding
  • ColumnValueHash
  • ColumnValueMapping
  • ValueToShard
  • ColumnValue
  • YdbTarget
  • KafkaTarget
  • KafkaTargetTopicSettings
  • KafkaTargetTopic
  • Serializer
  • SerializerDebezium
  • DebeziumSerializerParameter
  • MongoTarget
  • MetrikaSource
  • MetrikaStream
  • YDSTarget
  • Runtime
  • YcRuntime
  • ShardingUploadParams
  • Transformation
  • Transformer
  • MaskFieldTransformer
  • TablesFilter
  • MaskFunction
  • MaskFunctionHash
  • FilterColumnsTransformer
  • ColumnsFilter
  • RenameTablesTransformer
  • RenameTable
  • Table
  • ReplacePrimaryKeyTransformer
  • ToStringTransformer
  • SharderTransformer
  • TableSplitterTransformer
  • FilterRowsTransformer
  • DataObjects
  1. Справочник API
  2. REST (англ.)
  3. Transfer
  4. List

Data Transfer API, REST: Transfer.List

Статья создана
Yandex Cloud
Улучшена
Tania L.
Обновлена 21 февраля 2025 г.
  • HTTP request
  • Path parameters
  • Query parameters
  • Response
  • Transfer
  • Endpoint
  • EndpointSettings
  • MysqlSource
  • MysqlConnection
  • OnPremiseMysql
  • TLSMode
  • TLSConfig
  • ConnectionManagerConnection
  • Secret
  • MysqlObjectTransferSettings
  • PostgresSource
  • PostgresConnection
  • OnPremisePostgres
  • PostgresObjectTransferSettings
  • YdbSource
  • YDSSource
  • Parser
  • GenericParserCommon
  • DataSchema
  • FieldList
  • ColSchema
  • KafkaSource
  • KafkaConnectionOptions
  • OnPremiseKafka
  • KafkaAuth
  • KafkaSaslSecurity
  • DataTransformationOptions
  • MongoSource
  • MongoConnection
  • MongoConnectionOptions
  • OnPremiseMongo
  • MongoCollection
  • ClickhouseSource
  • ClickhouseConnection
  • ClickhouseConnectionOptions
  • OnPremiseClickhouse
  • ClickhouseShard
  • MysqlTarget
  • PostgresTarget
  • ClickhouseTarget
  • AltName
  • ClickhouseSharding
  • ColumnValueHash
  • ColumnValueMapping
  • ValueToShard
  • ColumnValue
  • YdbTarget
  • KafkaTarget
  • KafkaTargetTopicSettings
  • KafkaTargetTopic
  • Serializer
  • SerializerDebezium
  • DebeziumSerializerParameter
  • MongoTarget
  • MetrikaSource
  • MetrikaStream
  • YDSTarget
  • Runtime
  • YcRuntime
  • ShardingUploadParams
  • Transformation
  • Transformer
  • MaskFieldTransformer
  • TablesFilter
  • MaskFunction
  • MaskFunctionHash
  • FilterColumnsTransformer
  • ColumnsFilter
  • RenameTablesTransformer
  • RenameTable
  • Table
  • ReplacePrimaryKeyTransformer
  • ToStringTransformer
  • SharderTransformer
  • TableSplitterTransformer
  • FilterRowsTransformer
  • DataObjects

Lists transfers in the specified folder.

HTTP requestHTTP request

GET https://datatransfer.api.cloud.yandex.net/v1/transfers/list/{folderId}

Path parametersPath parameters

Field

Description

folderId

string

Required field. Identifier of the folder containing the transfers to be listed.

To get the folder ID, make a
yandex.cloud.resourcemanager.v1.FolderService.List request.

Query parametersQuery parameters

Field

Description

pageSize

string (int64)

The maximum number of transfers to be sent in the response message. If the
folder contains more transfers than page_size, next_page_token will be
included
in the response message. Include it into the subsequent ListTransfersRequest
to
fetch the next page. Defaults to 100 if not specified. The maximum allowed
value
for this field is 500.

pageToken

string

Opaque value identifying the transfers page to be fetched. Should be empty in
the first ListTransfersRequest. Subsequent requests should have this field
filled
with the next_page_token from the previous ListTransfersResponse.

ResponseResponse

HTTP Code: 200 - OK

{
  "transfers": [
    {
      "id": "string",
      "folderId": "string",
      "name": "string",
      "description": "string",
      "labels": "object",
      "source": {
        "id": "string",
        "folderId": "string",
        "name": "string",
        "description": "string",
        "labels": "object",
        "settings": {
          // Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`
          "mysqlSource": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "timezone": "string",
            "objectTransferSettings": {
              "view": "string",
              "routine": "string",
              "trigger": "string",
              "tables": "string"
            },
            "includeTablesRegex": [
              "string"
            ],
            "excludeTablesRegex": [
              "string"
            ],
            "securityGroups": [
              "string"
            ],
            "serviceDatabase": "string"
          },
          "postgresSource": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ],
            "slotByteLagLimit": "string",
            "serviceSchema": "string",
            "objectTransferSettings": {
              "sequence": "string",
              "sequenceOwnedBy": "string",
              "table": "string",
              "primaryKey": "string",
              "fkConstraint": "string",
              "defaultValues": "string",
              "constraint": "string",
              "index": "string",
              "view": "string",
              "function": "string",
              "trigger": "string",
              "type": "string",
              "rule": "string",
              "collation": "string",
              "policy": "string",
              "cast": "string",
              "materializedView": "string",
              "sequenceSet": "string"
            },
            "securityGroups": [
              "string"
            ]
          },
          "ydbSource": {
            "database": "string",
            "instance": "string",
            "paths": [
              "string"
            ],
            "serviceAccountId": "string",
            "subnetId": "string",
            "saKeyContent": "string",
            "securityGroups": [
              "string"
            ],
            "changefeedCustomName": "string",
            "changefeedCustomConsumerName": "string"
          },
          "ydsSource": {
            "database": "string",
            "stream": "string",
            "serviceAccountId": "string",
            "supportedCodecs": [
              "string"
            ],
            "parser": {
              // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
              "jsonParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              },
              "auditTrailsV1Parser": "object",
              "cloudLoggingParser": "object",
              "tskvParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              }
              // end of the list of possible fields
            },
            "allowTtlRewind": "boolean",
            "endpoint": "string",
            "subnetId": "string",
            "securityGroups": [
              "string"
            ],
            "consumer": "string"
          },
          "kafkaSource": {
            "connection": {
              // Includes only one of the fields `clusterId`, `onPremise`
              "clusterId": "string",
              "onPremise": {
                "brokerUrls": [
                  "string"
                ],
                "subnetId": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              }
              // end of the list of possible fields
            },
            "auth": {
              // Includes only one of the fields `sasl`, `noAuth`
              "sasl": {
                "user": "string",
                "mechanism": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                }
              },
              "noAuth": "object"
              // end of the list of possible fields
            },
            "securityGroups": [
              "string"
            ],
            "topicName": "string",
            "transformer": {
              "cloudFunction": "string",
              "numberOfRetries": "string",
              "bufferSize": "string",
              "bufferFlushInterval": "string",
              "invocationTimeout": "string",
              "serviceAccountId": "string"
            },
            "parser": {
              // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
              "jsonParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              },
              "auditTrailsV1Parser": "object",
              "cloudLoggingParser": "object",
              "tskvParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              }
              // end of the list of possible fields
            },
            "topicNames": [
              "string"
            ]
          },
          "mongoSource": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `mdbClusterId`, `onPremise`
                "mdbClusterId": "string",
                "onPremise": {
                  "hosts": [
                    "string"
                  ],
                  "port": "string",
                  "replicaSet": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "authSource": "string"
              }
              // end of the list of possible fields
            },
            "subnetId": "string",
            "collections": [
              {
                "databaseName": "string",
                "collectionName": "string"
              }
            ],
            "excludedCollections": [
              {
                "databaseName": "string",
                "collectionName": "string"
              }
            ],
            "secondaryPreferredMode": "boolean",
            "securityGroups": [
              "string"
            ]
          },
          "clickhouseSource": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `onPremise`, `mdbClusterId`
                "onPremise": {
                  "shards": [
                    {
                      "name": "string",
                      "hosts": [
                        "string"
                      ]
                    }
                  ],
                  "httpPort": "string",
                  "nativePort": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                "mdbClusterId": "string",
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "database": "string"
              }
              // end of the list of possible fields
            },
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ],
            "subnetId": "string",
            "securityGroups": [
              "string"
            ],
            "clickhouseClusterName": "string"
          },
          "mysqlTarget": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "sqlMode": "string",
            "skipConstraintChecks": "boolean",
            "timezone": "string",
            "cleanupPolicy": "string",
            "serviceDatabase": "string",
            "securityGroups": [
              "string"
            ]
          },
          "postgresTarget": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "cleanupPolicy": "string",
            "securityGroups": [
              "string"
            ]
          },
          "clickhouseTarget": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `onPremise`, `mdbClusterId`
                "onPremise": {
                  "shards": [
                    {
                      "name": "string",
                      "hosts": [
                        "string"
                      ]
                    }
                  ],
                  "httpPort": "string",
                  "nativePort": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                "mdbClusterId": "string",
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "database": "string"
              }
              // end of the list of possible fields
            },
            "subnetId": "string",
            "altNames": [
              {
                "fromName": "string",
                "toName": "string"
              }
            ],
            "cleanupPolicy": "string",
            "sharding": {
              // Includes only one of the fields `columnValueHash`, `customMapping`
              "columnValueHash": {
                "columnName": "string"
              },
              "customMapping": {
                "columnName": "string",
                "mapping": [
                  {
                    "columnValue": {
                      // Includes only one of the fields `stringValue`
                      "stringValue": "string"
                      // end of the list of possible fields
                    },
                    "shardName": "string"
                  }
                ]
              }
              // end of the list of possible fields
            },
            "clickhouseClusterName": "string",
            "securityGroups": [
              "string"
            ]
          },
          "ydbTarget": {
            "database": "string",
            "instance": "string",
            "path": "string",
            "serviceAccountId": "string",
            "cleanupPolicy": "string",
            "subnetId": "string",
            "saKeyContent": "string",
            "securityGroups": [
              "string"
            ],
            "isTableColumnOriented": "boolean",
            "defaultCompression": "string"
          },
          "kafkaTarget": {
            "connection": {
              // Includes only one of the fields `clusterId`, `onPremise`
              "clusterId": "string",
              "onPremise": {
                "brokerUrls": [
                  "string"
                ],
                "subnetId": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              }
              // end of the list of possible fields
            },
            "auth": {
              // Includes only one of the fields `sasl`, `noAuth`
              "sasl": {
                "user": "string",
                "mechanism": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                }
              },
              "noAuth": "object"
              // end of the list of possible fields
            },
            "securityGroups": [
              "string"
            ],
            "topicSettings": {
              // Includes only one of the fields `topic`, `topicPrefix`
              "topic": {
                "topicName": "string",
                "saveTxOrder": "boolean"
              },
              "topicPrefix": "string"
              // end of the list of possible fields
            },
            "serializer": {
              // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
              "serializerAuto": "object",
              "serializerJson": "object",
              "serializerDebezium": {
                "serializerParameters": [
                  {
                    "key": "string",
                    "value": "string"
                  }
                ]
              }
              // end of the list of possible fields
            }
          },
          "mongoTarget": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `mdbClusterId`, `onPremise`
                "mdbClusterId": "string",
                "onPremise": {
                  "hosts": [
                    "string"
                  ],
                  "port": "string",
                  "replicaSet": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "authSource": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "cleanupPolicy": "string",
            "subnetId": "string",
            "securityGroups": [
              "string"
            ]
          },
          "metrikaSource": {
            "counterIds": [
              "string"
            ],
            "token": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "streams": [
              {
                "type": "string",
                "columns": [
                  "string"
                ]
              }
            ]
          },
          "ydsTarget": {
            "database": "string",
            "stream": "string",
            "serviceAccountId": "string",
            "saveTxOrder": "boolean",
            "compressionCodec": "string",
            "serializer": {
              // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
              "serializerAuto": "object",
              "serializerJson": "object",
              "serializerDebezium": {
                "serializerParameters": [
                  {
                    "key": "string",
                    "value": "string"
                  }
                ]
              }
              // end of the list of possible fields
            },
            "endpoint": "string",
            "subnetId": "string",
            "securityGroups": [
              "string"
            ]
          }
          // end of the list of possible fields
        }
      },
      "target": {
        "id": "string",
        "folderId": "string",
        "name": "string",
        "description": "string",
        "labels": "object",
        "settings": {
          // Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`
          "mysqlSource": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "timezone": "string",
            "objectTransferSettings": {
              "view": "string",
              "routine": "string",
              "trigger": "string",
              "tables": "string"
            },
            "includeTablesRegex": [
              "string"
            ],
            "excludeTablesRegex": [
              "string"
            ],
            "securityGroups": [
              "string"
            ],
            "serviceDatabase": "string"
          },
          "postgresSource": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ],
            "slotByteLagLimit": "string",
            "serviceSchema": "string",
            "objectTransferSettings": {
              "sequence": "string",
              "sequenceOwnedBy": "string",
              "table": "string",
              "primaryKey": "string",
              "fkConstraint": "string",
              "defaultValues": "string",
              "constraint": "string",
              "index": "string",
              "view": "string",
              "function": "string",
              "trigger": "string",
              "type": "string",
              "rule": "string",
              "collation": "string",
              "policy": "string",
              "cast": "string",
              "materializedView": "string",
              "sequenceSet": "string"
            },
            "securityGroups": [
              "string"
            ]
          },
          "ydbSource": {
            "database": "string",
            "instance": "string",
            "paths": [
              "string"
            ],
            "serviceAccountId": "string",
            "subnetId": "string",
            "saKeyContent": "string",
            "securityGroups": [
              "string"
            ],
            "changefeedCustomName": "string",
            "changefeedCustomConsumerName": "string"
          },
          "ydsSource": {
            "database": "string",
            "stream": "string",
            "serviceAccountId": "string",
            "supportedCodecs": [
              "string"
            ],
            "parser": {
              // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
              "jsonParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              },
              "auditTrailsV1Parser": "object",
              "cloudLoggingParser": "object",
              "tskvParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              }
              // end of the list of possible fields
            },
            "allowTtlRewind": "boolean",
            "endpoint": "string",
            "subnetId": "string",
            "securityGroups": [
              "string"
            ],
            "consumer": "string"
          },
          "kafkaSource": {
            "connection": {
              // Includes only one of the fields `clusterId`, `onPremise`
              "clusterId": "string",
              "onPremise": {
                "brokerUrls": [
                  "string"
                ],
                "subnetId": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              }
              // end of the list of possible fields
            },
            "auth": {
              // Includes only one of the fields `sasl`, `noAuth`
              "sasl": {
                "user": "string",
                "mechanism": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                }
              },
              "noAuth": "object"
              // end of the list of possible fields
            },
            "securityGroups": [
              "string"
            ],
            "topicName": "string",
            "transformer": {
              "cloudFunction": "string",
              "numberOfRetries": "string",
              "bufferSize": "string",
              "bufferFlushInterval": "string",
              "invocationTimeout": "string",
              "serviceAccountId": "string"
            },
            "parser": {
              // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
              "jsonParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              },
              "auditTrailsV1Parser": "object",
              "cloudLoggingParser": "object",
              "tskvParser": {
                "dataSchema": {
                  // Includes only one of the fields `jsonFields`, `fields`
                  "jsonFields": "string",
                  "fields": {
                    "fields": [
                      {
                        "name": "string",
                        "type": "string",
                        "key": "boolean",
                        "required": "boolean",
                        "path": "string"
                      }
                    ]
                  }
                  // end of the list of possible fields
                },
                "nullKeysAllowed": "boolean",
                "addRestColumn": "boolean",
                "unescapeStringValues": "boolean"
              }
              // end of the list of possible fields
            },
            "topicNames": [
              "string"
            ]
          },
          "mongoSource": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `mdbClusterId`, `onPremise`
                "mdbClusterId": "string",
                "onPremise": {
                  "hosts": [
                    "string"
                  ],
                  "port": "string",
                  "replicaSet": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "authSource": "string"
              }
              // end of the list of possible fields
            },
            "subnetId": "string",
            "collections": [
              {
                "databaseName": "string",
                "collectionName": "string"
              }
            ],
            "excludedCollections": [
              {
                "databaseName": "string",
                "collectionName": "string"
              }
            ],
            "secondaryPreferredMode": "boolean",
            "securityGroups": [
              "string"
            ]
          },
          "clickhouseSource": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `onPremise`, `mdbClusterId`
                "onPremise": {
                  "shards": [
                    {
                      "name": "string",
                      "hosts": [
                        "string"
                      ]
                    }
                  ],
                  "httpPort": "string",
                  "nativePort": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                "mdbClusterId": "string",
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "database": "string"
              }
              // end of the list of possible fields
            },
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ],
            "subnetId": "string",
            "securityGroups": [
              "string"
            ],
            "clickhouseClusterName": "string"
          },
          "mysqlTarget": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "sqlMode": "string",
            "skipConstraintChecks": "boolean",
            "timezone": "string",
            "cleanupPolicy": "string",
            "serviceDatabase": "string",
            "securityGroups": [
              "string"
            ]
          },
          "postgresTarget": {
            "connection": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`, `connectionManagerConnection`
              "mdbClusterId": "string",
              "onPremise": {
                "port": "string",
                "subnetId": "string",
                "hosts": [
                  "string"
                ],
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "connectionManagerConnection": {
                "connectionId": "string",
                "subnetId": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "user": "string",
            "password": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "cleanupPolicy": "string",
            "securityGroups": [
              "string"
            ]
          },
          "clickhouseTarget": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `onPremise`, `mdbClusterId`
                "onPremise": {
                  "shards": [
                    {
                      "name": "string",
                      "hosts": [
                        "string"
                      ]
                    }
                  ],
                  "httpPort": "string",
                  "nativePort": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                "mdbClusterId": "string",
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "database": "string"
              }
              // end of the list of possible fields
            },
            "subnetId": "string",
            "altNames": [
              {
                "fromName": "string",
                "toName": "string"
              }
            ],
            "cleanupPolicy": "string",
            "sharding": {
              // Includes only one of the fields `columnValueHash`, `customMapping`
              "columnValueHash": {
                "columnName": "string"
              },
              "customMapping": {
                "columnName": "string",
                "mapping": [
                  {
                    "columnValue": {
                      // Includes only one of the fields `stringValue`
                      "stringValue": "string"
                      // end of the list of possible fields
                    },
                    "shardName": "string"
                  }
                ]
              }
              // end of the list of possible fields
            },
            "clickhouseClusterName": "string",
            "securityGroups": [
              "string"
            ]
          },
          "ydbTarget": {
            "database": "string",
            "instance": "string",
            "path": "string",
            "serviceAccountId": "string",
            "cleanupPolicy": "string",
            "subnetId": "string",
            "saKeyContent": "string",
            "securityGroups": [
              "string"
            ],
            "isTableColumnOriented": "boolean",
            "defaultCompression": "string"
          },
          "kafkaTarget": {
            "connection": {
              // Includes only one of the fields `clusterId`, `onPremise`
              "clusterId": "string",
              "onPremise": {
                "brokerUrls": [
                  "string"
                ],
                "subnetId": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              }
              // end of the list of possible fields
            },
            "auth": {
              // Includes only one of the fields `sasl`, `noAuth`
              "sasl": {
                "user": "string",
                "mechanism": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                }
              },
              "noAuth": "object"
              // end of the list of possible fields
            },
            "securityGroups": [
              "string"
            ],
            "topicSettings": {
              // Includes only one of the fields `topic`, `topicPrefix`
              "topic": {
                "topicName": "string",
                "saveTxOrder": "boolean"
              },
              "topicPrefix": "string"
              // end of the list of possible fields
            },
            "serializer": {
              // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
              "serializerAuto": "object",
              "serializerJson": "object",
              "serializerDebezium": {
                "serializerParameters": [
                  {
                    "key": "string",
                    "value": "string"
                  }
                ]
              }
              // end of the list of possible fields
            }
          },
          "mongoTarget": {
            "connection": {
              // Includes only one of the fields `connectionOptions`
              "connectionOptions": {
                // Includes only one of the fields `mdbClusterId`, `onPremise`
                "mdbClusterId": "string",
                "onPremise": {
                  "hosts": [
                    "string"
                  ],
                  "port": "string",
                  "replicaSet": "string",
                  "tlsMode": {
                    // Includes only one of the fields `enabled`
                    "enabled": {
                      "caCertificate": "string"
                    }
                    // end of the list of possible fields
                  }
                },
                // end of the list of possible fields
                "user": "string",
                "password": {
                  // Includes only one of the fields `raw`
                  "raw": "string"
                  // end of the list of possible fields
                },
                "authSource": "string"
              }
              // end of the list of possible fields
            },
            "database": "string",
            "cleanupPolicy": "string",
            "subnetId": "string",
            "securityGroups": [
              "string"
            ]
          },
          "metrikaSource": {
            "counterIds": [
              "string"
            ],
            "token": {
              // Includes only one of the fields `raw`
              "raw": "string"
              // end of the list of possible fields
            },
            "streams": [
              {
                "type": "string",
                "columns": [
                  "string"
                ]
              }
            ]
          },
          "ydsTarget": {
            "database": "string",
            "stream": "string",
            "serviceAccountId": "string",
            "saveTxOrder": "boolean",
            "compressionCodec": "string",
            "serializer": {
              // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
              "serializerAuto": "object",
              "serializerJson": "object",
              "serializerDebezium": {
                "serializerParameters": [
                  {
                    "key": "string",
                    "value": "string"
                  }
                ]
              }
              // end of the list of possible fields
            },
            "endpoint": "string",
            "subnetId": "string",
            "securityGroups": [
              "string"
            ]
          }
          // end of the list of possible fields
        }
      },
      "runtime": {
        // Includes only one of the fields `ycRuntime`
        "ycRuntime": {
          "jobCount": "string",
          "uploadShardParams": {
            "jobCount": "string",
            "processCount": "string"
          }
        }
        // end of the list of possible fields
      },
      "status": "string",
      "type": "string",
      "warning": "string",
      "transformation": {
        "transformers": [
          {
            // Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`
            "maskField": {
              "tables": {
                "includeTables": [
                  "string"
                ],
                "excludeTables": [
                  "string"
                ]
              },
              "columns": [
                "string"
              ],
              "function": {
                // Includes only one of the fields `maskFunctionHash`
                "maskFunctionHash": {
                  "userDefinedSalt": "string"
                }
                // end of the list of possible fields
              }
            },
            "filterColumns": {
              "tables": {
                "includeTables": [
                  "string"
                ],
                "excludeTables": [
                  "string"
                ]
              },
              "columns": {
                "includeColumns": [
                  "string"
                ],
                "excludeColumns": [
                  "string"
                ]
              }
            },
            "renameTables": {
              "renameTables": [
                {
                  "originalName": {
                    "nameSpace": "string",
                    "name": "string"
                  },
                  "newName": {
                    "nameSpace": "string",
                    "name": "string"
                  }
                }
              ]
            },
            "replacePrimaryKey": {
              "tables": {
                "includeTables": [
                  "string"
                ],
                "excludeTables": [
                  "string"
                ]
              },
              "keys": [
                "string"
              ]
            },
            "convertToString": {
              "tables": {
                "includeTables": [
                  "string"
                ],
                "excludeTables": [
                  "string"
                ]
              },
              "columns": {
                "includeColumns": [
                  "string"
                ],
                "excludeColumns": [
                  "string"
                ]
              }
            },
            "sharderTransformer": {
              "tables": {
                "includeTables": [
                  "string"
                ],
                "excludeTables": [
                  "string"
                ]
              },
              "columns": {
                "includeColumns": [
                  "string"
                ],
                "excludeColumns": [
                  "string"
                ]
              },
              "shardsCount": "string"
            },
            "tableSplitterTransformer": {
              "tables": {
                "includeTables": [
                  "string"
                ],
                "excludeTables": [
                  "string"
                ]
              },
              "columns": [
                "string"
              ],
              "splitter": "string"
            },
            "filterRows": {
              "tables": {
                "includeTables": [
                  "string"
                ],
                "excludeTables": [
                  "string"
                ]
              },
              "filter": "string",
              "filters": [
                "string"
              ]
            }
            // end of the list of possible fields
          }
        ]
      },
      "dataObjects": {
        "includeObjects": [
          "string"
        ]
      },
      "prestable": "boolean"
    }
  ],
  "nextPageToken": "string"
}

Field

Description

transfers[]

Transfer

The list of transfers. If there are more transfers in the folder, then
next_page_token is a non-empty string to be included into the subsequent
ListTransfersRequest to fetch the next transfers page.

nextPageToken

string

Opaque value identifying the next transfers page. This field is empty if there
are no more transfers in the folder. Otherwise it is non-empty and should be
included in the subsequent ListTransfersRequest to fetch the next transfers
page.

TransferTransfer

Transfer core entity

Field

Description

id

string

folderId

string

name

string

description

string

labels

object (map<string, string>)

source

Endpoint

target

Endpoint

runtime

Runtime

status

enum (TransferStatus)

  • TRANSFER_STATUS_UNSPECIFIED
  • CREATING: Transfer does some work before running
  • CREATED: Transfer created but not started by user
  • RUNNING: Transfer currently doing replication work
  • STOPPING: Transfer shutdown
  • STOPPED: Transfer stopped by user
  • ERROR: Transfer stopped by system
  • SNAPSHOTTING: Transfer copy snapshot
  • DONE: Transfer reach terminal phase

type

enum (TransferType)

  • TRANSFER_TYPE_UNSPECIFIED
  • SNAPSHOT_AND_INCREMENT: Snapshot and increment
  • SNAPSHOT_ONLY: Snapshot
  • INCREMENT_ONLY: Increment

warning

string

transformation

Transformation

dataObjects

DataObjects

prestable

boolean

EndpointEndpoint

Field

Description

id

string

folderId

string

name

string

description

string

labels

object (map<string, string>)

settings

EndpointSettings

EndpointSettingsEndpointSettings

Field

Description

mysqlSource

MysqlSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

postgresSource

PostgresSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydbSource

YdbSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydsSource

YDSSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

kafkaSource

KafkaSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

mongoSource

MongoSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

clickhouseSource

ClickhouseSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

mysqlTarget

MysqlTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

postgresTarget

PostgresTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

clickhouseTarget

ClickhouseTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydbTarget

YdbTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

kafkaTarget

KafkaTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

mongoTarget

MongoTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

metrikaSource

MetrikaSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydsTarget

YDSTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

MysqlSourceMysqlSource

Field

Description

connection

MysqlConnection

Database connection settings

database

string

Database name

You can leave it empty, then it will be possible to transfer tables from several
databases at the same time from this source.

user

string

User for database access. not required as may be in connection

password

Secret

Password for database access.

timezone

string

Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from
IANA timezone database. Default: local timezone.

objectTransferSettings

MysqlObjectTransferSettings

Schema migration

Select database objects to be transferred during activation or deactivation.

includeTablesRegex[]

string

excludeTablesRegex[]

string

securityGroups[]

string

Security groups

serviceDatabase

string

Database for service tables

Default: data source database. Here created technical tables (__tm_keeper,
__tm_gtid_keeper).

MysqlConnectionMysqlConnection

Field

Description

mdbClusterId

string

Managed Service for MySQL cluster ID

Includes only one of the fields mdbClusterId, onPremise, connectionManagerConnection.

onPremise

OnPremiseMysql

Connection options for on-premise MySQL

Includes only one of the fields mdbClusterId, onPremise, connectionManagerConnection.

connectionManagerConnection

ConnectionManagerConnection

Includes only one of the fields mdbClusterId, onPremise, connectionManagerConnection.

OnPremiseMysqlOnPremiseMysql

Field

Description

port

string (int64)

Database port

subnetId

string

Network interface for endpoint. If none will assume public ipv4

hosts[]

string

tlsMode

TLSMode

TLS settings for server connection. Disabled by default.

TLSModeTLSMode

Field

Description

enabled

TLSConfig

Includes only one of the fields enabled.

TLSConfigTLSConfig

Field

Description

caCertificate

string

CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified TLS is used to
connect to the server.

ConnectionManagerConnectionConnectionManagerConnection

Field

Description

connectionId

string

subnetId

string

Network interface for endpoint. If none will assume public ipv4

SecretSecret

Field

Description

raw

string

Raw secret value

Includes only one of the fields raw.

MysqlObjectTransferSettingsMysqlObjectTransferSettings

Field

Description

view

enum (ObjectTransferStage)

Views

CREATE VIEW ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

routine

enum (ObjectTransferStage)

Routines

CREATE PROCEDURE ... ; CREATE FUNCTION ... ;

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

trigger

enum (ObjectTransferStage)

Triggers

CREATE TRIGGER ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

tables

enum (ObjectTransferStage)

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

PostgresSourcePostgresSource

Field

Description

connection

PostgresConnection

Database connection settings

database

string

Database name

user

string

User for database access. not required as may be in connection

password

Secret

Password for database access.

includeTables[]

string

Included tables

If none or empty list is presented, all tables are replicated. Full table name
with schema. Can contain schema_name.* patterns.

excludeTables[]

string

Excluded tables

If none or empty list is presented, all tables are replicated. Full table name
with schema. Can contain schema_name.* patterns.

slotByteLagLimit

string (int64)

Maximum lag of replication slot (in bytes); after exceeding this limit
replication will be aborted.

serviceSchema

string

Database schema for service tables (__consumer_keeper,
__data_transfer_mole_finder). Default is public

objectTransferSettings

PostgresObjectTransferSettings

Select database objects to be transferred during activation or deactivation.

securityGroups[]

string

Security groups

PostgresConnectionPostgresConnection

Field

Description

mdbClusterId

string

Managed Service for PostgreSQL cluster ID

Includes only one of the fields mdbClusterId, onPremise, connectionManagerConnection.

onPremise

OnPremisePostgres

Connection options for on-premise PostgreSQL

Includes only one of the fields mdbClusterId, onPremise, connectionManagerConnection.

connectionManagerConnection

ConnectionManagerConnection

Includes only one of the fields mdbClusterId, onPremise, connectionManagerConnection.

OnPremisePostgresOnPremisePostgres

Field

Description

port

string (int64)

Will be used if the cluster ID is not specified.

subnetId

string

Network interface for endpoint. If none will assume public ipv4

hosts[]

string

tlsMode

TLSMode

TLS settings for server connection. Disabled by default.

PostgresObjectTransferSettingsPostgresObjectTransferSettings

Field

Description

sequence

enum (ObjectTransferStage)

Sequences

CREATE SEQUENCE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

sequenceOwnedBy

enum (ObjectTransferStage)

Owned sequences

CREATE SEQUENCE ... OWNED BY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

table

enum (ObjectTransferStage)

Tables

CREATE TABLE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

primaryKey

enum (ObjectTransferStage)

Primary keys

ALTER TABLE ... ADD PRIMARY KEY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

fkConstraint

enum (ObjectTransferStage)

Foreign keys

ALTER TABLE ... ADD FOREIGN KEY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

defaultValues

enum (ObjectTransferStage)

Default values

ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

constraint

enum (ObjectTransferStage)

Constraints

ALTER TABLE ... ADD CONSTRAINT ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

index

enum (ObjectTransferStage)

Indexes

CREATE INDEX ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

view

enum (ObjectTransferStage)

Views

CREATE VIEW ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

function

enum (ObjectTransferStage)

Functions

CREATE FUNCTION ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

trigger

enum (ObjectTransferStage)

Triggers

CREATE TRIGGER ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

type

enum (ObjectTransferStage)

Types

CREATE TYPE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

rule

enum (ObjectTransferStage)

Rules

CREATE RULE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

collation

enum (ObjectTransferStage)

Collations

CREATE COLLATION ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

policy

enum (ObjectTransferStage)

Policies

CREATE POLICY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

cast

enum (ObjectTransferStage)

Casts

CREATE CAST ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

materializedView

enum (ObjectTransferStage)

Materialized views

CREATE MATERIALIZED VIEW ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

sequenceSet

enum (ObjectTransferStage)

Sequence sets

CREATE SEQUENCE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

YdbSourceYdbSource

Field

Description

database

string

Path in YDB where to store tables

instance

string

Instance of YDB. example: ydb-ru-prestable.yandex.net:2135

paths[]

string

serviceAccountId

string

subnetId

string

Network interface for endpoint. If none will assume public ipv4

saKeyContent

string

Authorization Key

securityGroups[]

string

Security groups

changefeedCustomName

string

Pre-created change feed

changefeedCustomConsumerName

string

YDSSourceYDSSource

Field

Description

database

string

Database

stream

string

Stream

serviceAccountId

string

SA which has read access to the stream.

supportedCodecs[]

enum (YdsCompressionCodec)

Compression codec

  • YDS_COMPRESSION_CODEC_UNSPECIFIED
  • YDS_COMPRESSION_CODEC_RAW
  • YDS_COMPRESSION_CODEC_GZIP
  • YDS_COMPRESSION_CODEC_ZSTD

parser

Parser

Data parsing rules

allowTtlRewind

boolean

Should continue working, if consumer read lag exceed TTL of topic
False: stop the transfer in error state, if detected lost data. True: continue
working with losing part of data

endpoint

string

for dedicated db

subnetId

string

Network interface for endpoint. If none will assume public ipv4

securityGroups[]

string

Security groups

consumer

string

for important streams

ParserParser

Field

Description

jsonParser

GenericParserCommon

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

auditTrailsV1Parser

object

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

cloudLoggingParser

object

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

tskvParser

GenericParserCommon

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

GenericParserCommonGenericParserCommon

Field

Description

dataSchema

DataSchema

nullKeysAllowed

boolean

Allow null keys, if no - null keys will be putted to unparsed data

addRestColumn

boolean

Will add _rest column for all unknown fields

unescapeStringValues

boolean

Unescape string values

DataSchemaDataSchema

Field

Description

jsonFields

string

Includes only one of the fields jsonFields, fields.

fields

FieldList

Includes only one of the fields jsonFields, fields.

FieldListFieldList

Field

Description

fields[]

ColSchema

Column schema

ColSchemaColSchema

Field

Description

name

string

type

enum (ColumnType)

  • COLUMN_TYPE_UNSPECIFIED
  • INT32
  • INT16
  • INT8
  • UINT64
  • UINT32
  • UINT16
  • UINT8
  • DOUBLE
  • BOOLEAN
  • STRING
  • UTF8
  • ANY
  • DATETIME
  • INT64

key

boolean

required

boolean

path

string

KafkaSourceKafkaSource

Field

Description

connection

KafkaConnectionOptions

Connection settings

auth

KafkaAuth

Authentication settings

securityGroups[]

string

Security groups

topicName

string

Full source topic name
Deprecated in favor of topic names

transformer

DataTransformationOptions

Data transformation rules

parser

Parser

Data parsing rules

topicNames[]

string

List of topic names to read

KafkaConnectionOptionsKafkaConnectionOptions

Field

Description

clusterId

string

Managed Service for Kafka cluster ID

Includes only one of the fields clusterId, onPremise.

onPremise

OnPremiseKafka

Connection options for on-premise Kafka

Includes only one of the fields clusterId, onPremise.

OnPremiseKafkaOnPremiseKafka

Field

Description

brokerUrls[]

string

Kafka broker URLs

subnetId

string

Network interface for endpoint. If none will assume public ipv4

tlsMode

TLSMode

TLS settings for broker connection. Disabled by default.

KafkaAuthKafkaAuth

Field

Description

sasl

KafkaSaslSecurity

Authentication with SASL

Includes only one of the fields sasl, noAuth.

noAuth

object

No authentication

Includes only one of the fields sasl, noAuth.

KafkaSaslSecurityKafkaSaslSecurity

Field

Description

user

string

User name

mechanism

enum (KafkaMechanism)

SASL mechanism for authentication

  • KAFKA_MECHANISM_UNSPECIFIED
  • KAFKA_MECHANISM_SHA256
  • KAFKA_MECHANISM_SHA512

password

Secret

Password for user

DataTransformationOptionsDataTransformationOptions

Field

Description

cloudFunction

string

Cloud function

numberOfRetries

string (int64)

Number of retries

bufferSize

string

Buffer size for function

bufferFlushInterval

string

Flush interval

invocationTimeout

string

Invocation timeout

serviceAccountId

string

Service account

MongoSourceMongoSource

Field

Description

connection

MongoConnection

subnetId

string

collections[]

MongoCollection

List of collections for replication. Empty list implies replication of all
tables on the deployment. Allowed to use * as collection name.

excludedCollections[]

MongoCollection

List of forbidden collections for replication. Allowed to use * as collection
name for forbid all collections of concrete schema.

secondaryPreferredMode

boolean

Read mode for mongo client

securityGroups[]

string

Security groups

MongoConnectionMongoConnection

Field

Description

connectionOptions

MongoConnectionOptions

Includes only one of the fields connectionOptions.

MongoConnectionOptionsMongoConnectionOptions

Field

Description

mdbClusterId

string

Includes only one of the fields mdbClusterId, onPremise.

onPremise

OnPremiseMongo

Includes only one of the fields mdbClusterId, onPremise.

user

string

User name

password

Secret

Password for user

authSource

string

Database name associated with the credentials

OnPremiseMongoOnPremiseMongo

Field

Description

hosts[]

string

port

string (int64)

replicaSet

string

tlsMode

TLSMode

MongoCollectionMongoCollection

Field

Description

databaseName

string

collectionName

string

ClickhouseSourceClickhouseSource

Field

Description

connection

ClickhouseConnection

includeTables[]

string

White list of tables for replication. If none or empty list is presented - will
replicate all tables. Can contain * patterns.

excludeTables[]

string

Exclude list of tables for replication. If none or empty list is presented -
will replicate all tables. Can contain * patterns.

subnetId

string

securityGroups[]

string

clickhouseClusterName

string

Name of the ClickHouse cluster. For Managed ClickHouse that is name of
ShardGroup.

ClickhouseConnectionClickhouseConnection

Field

Description

connectionOptions

ClickhouseConnectionOptions

Includes only one of the fields connectionOptions.

ClickhouseConnectionOptionsClickhouseConnectionOptions

Field

Description

onPremise

OnPremiseClickhouse

Includes only one of the fields onPremise, mdbClusterId.

mdbClusterId

string

Includes only one of the fields onPremise, mdbClusterId.

user

string

password

Secret

database

string

Database

OnPremiseClickhouseOnPremiseClickhouse

Field

Description

shards[]

ClickhouseShard

httpPort

string (int64)

nativePort

string (int64)

tlsMode

TLSMode

ClickhouseShardClickhouseShard

Field

Description

name

string

hosts[]

string

MysqlTargetMysqlTarget

Field

Description

connection

MysqlConnection

Database connection settings

database

string

Database name

Allowed to leave it empty, then the tables will be created in databases with the
same names as on the source. If this field is empty, then you must fill below db
schema for service table.

user

string

User for database access. not required as may be in connection

password

Secret

Password for database access.

sqlMode

string

Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION.

skipConstraintChecks

boolean

Disable constraints checks

Recommend to disable for increase replication speed, but if schema contain
cascading operations we don't recommend to disable. This option set
FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0.

timezone

string

Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from
IANA timezone database. Default: local timezone.

cleanupPolicy

enum (CleanupPolicy)

Cleanup policy

Cleanup policy for activate, reactivate and reupload processes. Default is
DISABLED.

  • CLEANUP_POLICY_UNSPECIFIED
  • DISABLED: Don't cleanup
  • DROP: Drop
  • TRUNCATE: Truncate

serviceDatabase

string

Database schema for service table

Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper).

securityGroups[]

string

Security groups

PostgresTargetPostgresTarget

Field

Description

connection

PostgresConnection

Database connection settings

database

string

Database name

user

string

User for database access. not required as may be in connection

password

Secret

Password for database access.

cleanupPolicy

enum (CleanupPolicy)

Cleanup policy for activate, reactivate and reupload processes. Default is
truncate.

  • CLEANUP_POLICY_UNSPECIFIED
  • DISABLED: Don't cleanup
  • DROP: Drop
  • TRUNCATE: Truncate

securityGroups[]

string

Security groups

ClickhouseTargetClickhouseTarget

Field

Description

connection

ClickhouseConnection

subnetId

string

altNames[]

AltName

Alternative table names in target

cleanupPolicy

enum (ClickhouseCleanupPolicy)

  • CLICKHOUSE_CLEANUP_POLICY_UNSPECIFIED
  • CLICKHOUSE_CLEANUP_POLICY_DISABLED
  • CLICKHOUSE_CLEANUP_POLICY_DROP
  • CLICKHOUSE_CLEANUP_POLICY_TRUNCATE

sharding

ClickhouseSharding

clickhouseClusterName

string

Name of the ClickHouse cluster. For Managed ClickHouse that is name of
ShardGroup.

securityGroups[]

string

AltNameAltName

Field

Description

fromName

string

Source table name

toName

string

Target table name

ClickhouseShardingClickhouseSharding

Field

Description

columnValueHash

ColumnValueHash

Includes only one of the fields columnValueHash, customMapping.

customMapping

ColumnValueMapping

Includes only one of the fields columnValueHash, customMapping.

ColumnValueHashColumnValueHash

Field

Description

columnName

string

ColumnValueMappingColumnValueMapping

Field

Description

columnName

string

mapping[]

ValueToShard

ValueToShardValueToShard

Field

Description

columnValue

ColumnValue

shardName

string

ColumnValueColumnValue

Field

Description

stringValue

string

Includes only one of the fields stringValue.

YdbTargetYdbTarget

Field

Description

database

string

Path in YDB where to store tables

instance

string

Instance of YDB. example: ydb-ru-prestable.yandex.net:2135

path

string

Path extension for database, each table will be layouted into this path

serviceAccountId

string

cleanupPolicy

enum (YdbCleanupPolicy)

Cleanup policy

  • YDB_CLEANUP_POLICY_UNSPECIFIED
  • YDB_CLEANUP_POLICY_DISABLED
  • YDB_CLEANUP_POLICY_DROP

subnetId

string

Network interface for endpoint. If none will assume public ipv4

saKeyContent

string

SA content

securityGroups[]

string

Security groups

isTableColumnOriented

boolean

Should create column-oriented table (OLAP). By default it creates row-oriented
(OLTP)

defaultCompression

enum (YdbDefaultCompression)

Compression that will be used for default columns family on YDB table creation

  • YDB_DEFAULT_COMPRESSION_UNSPECIFIED
  • YDB_DEFAULT_COMPRESSION_DISABLED
  • YDB_DEFAULT_COMPRESSION_LZ4

KafkaTargetKafkaTarget

Field

Description

connection

KafkaConnectionOptions

Connection settings

auth

KafkaAuth

Authentication settings

securityGroups[]

string

Security groups

topicSettings

KafkaTargetTopicSettings

Target topic settings

serializer

Serializer

Data serialization format settings

KafkaTargetTopicSettingsKafkaTargetTopicSettings

Field

Description

topic

KafkaTargetTopic

Full topic name

Includes only one of the fields topic, topicPrefix.

topicPrefix

string

Topic prefix

Analogue of the Debezium setting database.server.name.
Messages will be sent to topic with name <topic_prefix>..<table_name>.

Includes only one of the fields topic, topicPrefix.

KafkaTargetTopicKafkaTargetTopic

Field

Description

topicName

string

Topic name

saveTxOrder

boolean

Save transactions order
Not to split events queue into separate per-table queues.

SerializerSerializer

Data serialization format

Field

Description

serializerAuto

object

Select the serialization format automatically

Includes only one of the fields serializerAuto, serializerJson, serializerDebezium.

serializerJson

object

Serialize data in json format

Includes only one of the fields serializerAuto, serializerJson, serializerDebezium.

serializerDebezium

SerializerDebezium

Serialize data in debezium format

Includes only one of the fields serializerAuto, serializerJson, serializerDebezium.

SerializerDebeziumSerializerDebezium

Field

Description

serializerParameters[]

DebeziumSerializerParameter

Settings of sterilization parameters as key-value pairs

DebeziumSerializerParameterDebeziumSerializerParameter

Field

Description

key

string

Name of the serializer parameter

value

string

Value of the serializer parameter

MongoTargetMongoTarget

Field

Description

connection

MongoConnection

database

string

Database name

cleanupPolicy

enum (CleanupPolicy)

  • CLEANUP_POLICY_UNSPECIFIED
  • DISABLED: Don't cleanup
  • DROP: Drop
  • TRUNCATE: Truncate

subnetId

string

securityGroups[]

string

Security groups

MetrikaSourceMetrikaSource

Field

Description

counterIds[]

string (int64)

token

Secret

streams[]

MetrikaStream

MetrikaStreamMetrikaStream

Field

Description

type

enum (MetrikaStreamType)

  • METRIKA_STREAM_TYPE_UNSPECIFIED
  • METRIKA_STREAM_TYPE_HITS
  • METRIKA_STREAM_TYPE_VISITS
  • METRIKA_STREAM_TYPE_HITS_V2

columns[]

string

YDSTargetYDSTarget

Field

Description

database

string

Database

stream

string

Stream

serviceAccountId

string

SA which has read access to the stream.

saveTxOrder

boolean

Save transaction order
Not to split events queue into separate per-table queues.
Incompatible with setting Topic prefix, only with Topic full name.

compressionCodec

enum (YdsCompressionCodec)

  • YDS_COMPRESSION_CODEC_UNSPECIFIED
  • YDS_COMPRESSION_CODEC_RAW
  • YDS_COMPRESSION_CODEC_GZIP
  • YDS_COMPRESSION_CODEC_ZSTD

serializer

Serializer

Data serialization format

endpoint

string

for dedicated db

subnetId

string

Network interface for endpoint. If none will assume public ipv4

securityGroups[]

string

Security groups

RuntimeRuntime

Field

Description

ycRuntime

YcRuntime

Includes only one of the fields ycRuntime.

YcRuntimeYcRuntime

Field

Description

jobCount

string (int64)

uploadShardParams

ShardingUploadParams

ShardingUploadParamsShardingUploadParams

Field

Description

jobCount

string (int64)

processCount

string (int64)

TransformationTransformation

Transformation is converting data using special transformer functions.
These functions are executed on a data stream, applied to each data change item,
and transform them.
A transformer can be run at both the metadata and data levels.
Data can only be transformed if the source and target are of different types.

Field

Description

transformers[]

Transformer

Transformers are set as a list.
When activating a transfer, a transformation plan is made for the tables that
match the specified criteria.
Transformers are applied to the tables in the sequence specified in the list.

TransformerTransformer

Some transformers may have limitations and only apply to some source-target
pairs.

Field

Description

maskField

MaskFieldTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

filterColumns

FilterColumnsTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

renameTables

RenameTablesTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

replacePrimaryKey

ReplacePrimaryKeyTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

convertToString

ToStringTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

sharderTransformer

SharderTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

tableSplitterTransformer

TableSplitterTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

filterRows

FilterRowsTransformer

Includes only one of the fields maskField, filterColumns, renameTables, replacePrimaryKey, convertToString, sharderTransformer, tableSplitterTransformer, filterRows.

MaskFieldTransformerMaskFieldTransformer

Mask field transformer allows you to hash data

Field

Description

tables

TablesFilter

List of included and excluded tables

columns[]

string

Specify the name of the column for data masking (a regular expression).

function

MaskFunction

Mask function

TablesFilterTablesFilter

Filter tables using lists of included and excluded tables.

Field

Description

includeTables[]

string

List of tables that will be included to transfer

excludeTables[]

string

List of tables that will be excluded to transfer

MaskFunctionMaskFunction

Mask function

Field

Description

maskFunctionHash

MaskFunctionHash

Hash mask function

Includes only one of the fields maskFunctionHash.

MaskFunctionHashMaskFunctionHash

Hash data using HMAC

Field

Description

userDefinedSalt

string

This string will be used in the HMAC(sha256, salt) function applied to the
column data.

FilterColumnsTransformerFilterColumnsTransformer

Set up a list of table columns to transfer

Field

Description

tables

TablesFilter

List of the tables to filter using lists of included and excluded tables.

columns

ColumnsFilter

List of the columns to transfer to the target tables using lists of included and
excluded columns.

ColumnsFilterColumnsFilter

Filter columns using lists of included and excluded columns.

Field

Description

includeColumns[]

string

List of columns that will be included to transfer

excludeColumns[]

string

List of columns that will be excluded to transfer

RenameTablesTransformerRenameTablesTransformer

Set rules for renaming tables by specifying the current names of the tables in
the source and new names for these tables in the target.

Field

Description

renameTables[]

RenameTable

List of renaming rules

RenameTableRenameTable

Specify rule for renaming table

Field

Description

originalName

Table

Specify the current names of the table in the source

newName

Table

Specify the new names for this table in the target

TableTable

Field

Description

nameSpace

string

name

string

ReplacePrimaryKeyTransformerReplacePrimaryKeyTransformer

Override primary keys

Field

Description

tables

TablesFilter

List of included and excluded tables

keys[]

string

List of columns to be used as primary keys

ToStringTransformerToStringTransformer

Convert column values to strings
The values will be converted depending on the source type
Conversion rules are described here:
https://cloud.yandex.com/en/docs/data-transfer/concepts/data-transformation#convert-to-string

Field

Description

tables

TablesFilter

List of included and excluded tables

columns

ColumnsFilter

List of included and excluded columns

SharderTransformerSharderTransformer

Set the number of shards for particular tables and a list of columns whose
values will be used for calculating a hash to determine a shard.

Field

Description

tables

TablesFilter

List of included and excluded tables

columns

ColumnsFilter

List of included and excluded columns

shardsCount

string (int64)

Number of shards

TableSplitterTransformerTableSplitterTransformer

A transfer splits the X table into multiple tables (X_1, X_2, ..., X_n) based on
data.
If a row was located in the X table before it was split, it is now in the X_i
table,
where i is determined by the column list and split string parameters.
Example:
If the column list has two columns, month of birth and gender, specified and the
split string states @,
information about an employee whose name is John and who was born on February
11, 1984,
from the Employees table will get to a new table named Employees@February@male.

Field

Description

tables

TablesFilter

List of included and excluded tables

columns[]

string

Specify the columns in the tables to be partitioned.

splitter

string

Specify the split string to be used for merging components in a new table name.

FilterRowsTransformerFilterRowsTransformer

This filter only applies to transfers with queues (Logbroker or Apache Kafka®)
as a data source.
When running a transfer, only the strings meeting the specified criteria remain
in a changefeed.

Field

Description

tables

TablesFilter

List of included and excluded tables.

filter

string

Filtering criterion. This can be comparison operators for numeric, string, and
Boolean values,
comparison to NULL, and checking whether a substring is part of a string.
Details here:
https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources.
Deprecated: Use filters instead.

filters[]

string

Data is transported if it satisfies at least one of filters. Consider that there
is OR statement between filters.
Each filter can be comparison operators for numeric, string, and Boolean values,
comparison to NULL, and
checking whether a substring is part of a string.
Details in docs:
https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources.

DataObjectsDataObjects

Field

Description

includeObjects[]

string

Была ли статья полезна?

Предыдущая
Delete
Следующая
Get
Проект Яндекса
© 2025 ООО «Яндекс.Облако»