Data Transfer API, REST: Endpoint.List

Written by

Updated at October 17, 2024

HTTP request
Path parameters
Query parameters
Response
Endpoint
EndpointSettings
MysqlSource
MysqlConnection
OnPremiseMysql
TLSMode
TLSConfig
Secret
MysqlObjectTransferSettings
PostgresSource
PostgresConnection
OnPremisePostgres
PostgresObjectTransferSettings
YdbSource
YDSSource
Parser
GenericParserCommon
DataSchema
FieldList
ColSchema
KafkaSource
KafkaConnectionOptions
OnPremiseKafka
KafkaAuth
KafkaSaslSecurity
DataTransformationOptions
MongoSource
MongoConnection
MongoConnectionOptions
OnPremiseMongo
MongoCollection
ClickhouseSource
ClickhouseConnection
ClickhouseConnectionOptions
OnPremiseClickhouse
ClickhouseShard
MysqlTarget
PostgresTarget
ClickhouseTarget
AltName
ClickhouseSharding
ColumnValueHash
ColumnValueMapping
ValueToShard
ColumnValue
YdbTarget
KafkaTarget
KafkaTargetTopicSettings
KafkaTargetTopic
Serializer
SerializerDebezium
DebeziumSerializerParameter
MongoTarget
MetrikaSource
MetrikaStream
YDSTarget

HTTP request

GET https://datatransfer.api.cloud.yandex.net/v1/endpoints/list/{folderId}

Path parameters

Field

Description

folderId

string

Required field. Identifier of the folder containing the endpoints to be listed.

Query parameters

Field

Description

pageSize

string (int64)

The maximum number of endpoints to be sent in the response message. If the
folder contains more endpoints than page_size, next_page_token will be
included
in the response message. Include it into the subsequent ListEndpointRequest to
fetch the next page. Defaults to 100 if not specified. The maximum allowed
value
for this field is 500.

pageToken

string

Opaque value identifying the endpoints page to be fetched. Should be empty in
the first ListEndpointsRequest. Subsequent requests should have this field
filled
with the next_page_token from the previous ListEndpointsResponse.

Response

HTTP Code: 200 - OK

{
  "endpoints": [
    {
      "id": "string",
      "folderId": "string",
      "name": "string",
      "description": "string",
      "labels": "string",
      "settings": {
        // Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`
        "mysqlSource": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "string",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `enabled`
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "timezone": "string",
          "objectTransferSettings": {
            "view": "string",
            "routine": "string",
            "trigger": "string",
            "tables": "string"
          },
          "includeTablesRegex": [
            "string"
          ],
          "excludeTablesRegex": [
            "string"
          ],
          "securityGroups": [
            "string"
          ],
          "serviceDatabase": "string"
        },
        "postgresSource": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "string",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `enabled`
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "includeTables": [
            "string"
          ],
          "excludeTables": [
            "string"
          ],
          "slotByteLagLimit": "string",
          "serviceSchema": "string",
          "objectTransferSettings": {
            "sequence": "string",
            "sequenceOwnedBy": "string",
            "table": "string",
            "primaryKey": "string",
            "fkConstraint": "string",
            "defaultValues": "string",
            "constraint": "string",
            "index": "string",
            "view": "string",
            "function": "string",
            "trigger": "string",
            "type": "string",
            "rule": "string",
            "collation": "string",
            "policy": "string",
            "cast": "string",
            "materializedView": "string",
            "sequenceSet": "string"
          },
          "securityGroups": [
            "string"
          ]
        },
        "ydbSource": {
          "database": "string",
          "instance": "string",
          "paths": [
            "string"
          ],
          "serviceAccountId": "string",
          "subnetId": "string",
          "saKeyContent": "string",
          "securityGroups": [
            "string"
          ],
          "changefeedCustomName": "string"
        },
        "ydsSource": {
          "database": "string",
          "stream": "string",
          "serviceAccountId": "string",
          "supportedCodecs": [
            "string"
          ],
          "parser": {
            // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
            "jsonParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "string",
                      "key": "boolean",
                      "required": "boolean",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "boolean",
              "addRestColumn": "boolean",
              "unescapeStringValues": "boolean"
            },
            "auditTrailsV1Parser": "object",
            "cloudLoggingParser": "object",
            "tskvParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "string",
                      "key": "boolean",
                      "required": "boolean",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "boolean",
              "addRestColumn": "boolean",
              "unescapeStringValues": "boolean"
            }
            // end of the list of possible fields
          },
          "allowTtlRewind": "boolean",
          "endpoint": "string",
          "subnetId": "string",
          "securityGroups": [
            "string"
          ],
          "consumer": "string"
        },
        "kafkaSource": {
          "connection": {
            // Includes only one of the fields `clusterId`, `onPremise`
            "clusterId": "string",
            "onPremise": {
              "brokerUrls": [
                "string"
              ],
              "subnetId": "string",
              "tlsMode": {
                // Includes only one of the fields `enabled`
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "auth": {
            // Includes only one of the fields `sasl`, `noAuth`
            "sasl": {
              "user": "string",
              "mechanism": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              }
            },
            "noAuth": "object"
            // end of the list of possible fields
          },
          "securityGroups": [
            "string"
          ],
          "topicName": "string",
          "transformer": {
            "cloudFunction": "string",
            "numberOfRetries": "string",
            "bufferSize": "string",
            "bufferFlushInterval": "string",
            "invocationTimeout": "string",
            "serviceAccountId": "string"
          },
          "parser": {
            // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
            "jsonParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "string",
                      "key": "boolean",
                      "required": "boolean",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "boolean",
              "addRestColumn": "boolean",
              "unescapeStringValues": "boolean"
            },
            "auditTrailsV1Parser": "object",
            "cloudLoggingParser": "object",
            "tskvParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "string",
                      "key": "boolean",
                      "required": "boolean",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "boolean",
              "addRestColumn": "boolean",
              "unescapeStringValues": "boolean"
            }
            // end of the list of possible fields
          },
          "topicNames": [
            "string"
          ]
        },
        "mongoSource": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`
              "mdbClusterId": "string",
              "onPremise": {
                "hosts": [
                  "string"
                ],
                "port": "string",
                "replicaSet": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "authSource": "string"
            }
            // end of the list of possible fields
          },
          "subnetId": "string",
          "collections": [
            {
              "databaseName": "string",
              "collectionName": "string"
            }
          ],
          "excludedCollections": [
            {
              "databaseName": "string",
              "collectionName": "string"
            }
          ],
          "secondaryPreferredMode": "boolean",
          "securityGroups": [
            "string"
          ]
        },
        "clickhouseSource": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `onPremise`, `mdbClusterId`
              "onPremise": {
                "shards": [
                  {
                    "name": "string",
                    "hosts": [
                      "string"
                    ]
                  }
                ],
                "httpPort": "string",
                "nativePort": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "mdbClusterId": "string",
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "database": "string"
            }
            // end of the list of possible fields
          },
          "includeTables": [
            "string"
          ],
          "excludeTables": [
            "string"
          ],
          "subnetId": "string",
          "securityGroups": [
            "string"
          ],
          "clickhouseClusterName": "string"
        },
        "mysqlTarget": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "string",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `enabled`
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "sqlMode": "string",
          "skipConstraintChecks": "boolean",
          "timezone": "string",
          "cleanupPolicy": "string",
          "serviceDatabase": "string",
          "securityGroups": [
            "string"
          ]
        },
        "postgresTarget": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "string",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `enabled`
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "cleanupPolicy": "string",
          "securityGroups": [
            "string"
          ]
        },
        "clickhouseTarget": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `onPremise`, `mdbClusterId`
              "onPremise": {
                "shards": [
                  {
                    "name": "string",
                    "hosts": [
                      "string"
                    ]
                  }
                ],
                "httpPort": "string",
                "nativePort": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "mdbClusterId": "string",
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "database": "string"
            }
            // end of the list of possible fields
          },
          "subnetId": "string",
          "altNames": [
            {
              "fromName": "string",
              "toName": "string"
            }
          ],
          "cleanupPolicy": "string",
          "sharding": {
            // Includes only one of the fields `columnValueHash`, `customMapping`
            "columnValueHash": {
              "columnName": "string"
            },
            "customMapping": {
              "columnName": "string",
              "mapping": [
                {
                  "columnValue": {
                    // Includes only one of the fields `stringValue`
                    "stringValue": "string"
                    // end of the list of possible fields
                  },
                  "shardName": "string"
                }
              ]
            }
            // end of the list of possible fields
          },
          "clickhouseClusterName": "string",
          "securityGroups": [
            "string"
          ]
        },
        "ydbTarget": {
          "database": "string",
          "instance": "string",
          "path": "string",
          "serviceAccountId": "string",
          "cleanupPolicy": "string",
          "subnetId": "string",
          "saKeyContent": "string",
          "securityGroups": [
            "string"
          ],
          "isTableColumnOriented": "boolean",
          "defaultCompression": "string"
        },
        "kafkaTarget": {
          "connection": {
            // Includes only one of the fields `clusterId`, `onPremise`
            "clusterId": "string",
            "onPremise": {
              "brokerUrls": [
                "string"
              ],
              "subnetId": "string",
              "tlsMode": {
                // Includes only one of the fields `enabled`
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "auth": {
            // Includes only one of the fields `sasl`, `noAuth`
            "sasl": {
              "user": "string",
              "mechanism": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              }
            },
            "noAuth": "object"
            // end of the list of possible fields
          },
          "securityGroups": [
            "string"
          ],
          "topicSettings": {
            // Includes only one of the fields `topic`, `topicPrefix`
            "topic": {
              "topicName": "string",
              "saveTxOrder": "boolean"
            },
            "topicPrefix": "string"
            // end of the list of possible fields
          },
          "serializer": {
            // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
            "serializerAuto": "object",
            "serializerJson": "object",
            "serializerDebezium": {
              "serializerParameters": [
                {
                  "key": "string",
                  "value": "string"
                }
              ]
            }
            // end of the list of possible fields
          }
        },
        "mongoTarget": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`
              "mdbClusterId": "string",
              "onPremise": {
                "hosts": [
                  "string"
                ],
                "port": "string",
                "replicaSet": "string",
                "tlsMode": {
                  // Includes only one of the fields `enabled`
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "authSource": "string"
            }
            // end of the list of possible fields
          },
          "database": "string",
          "cleanupPolicy": "string",
          "subnetId": "string",
          "securityGroups": [
            "string"
          ]
        },
        "metrikaSource": {
          "counterIds": [
            "string"
          ],
          "token": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "streams": [
            {
              "type": "string",
              "columns": [
                "string"
              ]
            }
          ]
        },
        "ydsTarget": {
          "database": "string",
          "stream": "string",
          "serviceAccountId": "string",
          "saveTxOrder": "boolean",
          "serializer": {
            // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
            "serializerAuto": "object",
            "serializerJson": "object",
            "serializerDebezium": {
              "serializerParameters": [
                {
                  "key": "string",
                  "value": "string"
                }
              ]
            }
            // end of the list of possible fields
          },
          "endpoint": "string",
          "subnetId": "string",
          "securityGroups": [
            "string"
          ]
        }
        // end of the list of possible fields
      }
    }
  ],
  "nextPageToken": "string"
}

Field

Description

endpoints[]

Endpoint

The list of endpoints. If there are more endpoints in the folder, then
next_page_token is a non-empty string to be included into the subsequent
ListEndpointsRequest to fetch the next endpoints page.

nextPageToken

string

Opaque value identifying the next endpoints page. This field is empty if there
are no more endpoints in the folder. Otherwise, it is non-empty and should be
included in the subsequent ListEndpointsRequest to fetch the next endpoints
page.

Endpoint

Field	Description
id	string
folderId	string
name	string
description	string
labels	string
settings	EndpointSettings

EndpointSettings

Field	Description
mysqlSource	MysqlSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
postgresSource	PostgresSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
ydbSource	YdbSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
ydsSource	YDSSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
kafkaSource	KafkaSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
mongoSource	MongoSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
clickhouseSource	ClickhouseSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
mysqlTarget	MysqlTarget Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
postgresTarget	PostgresTarget Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
clickhouseTarget	ClickhouseTarget Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
ydbTarget	YdbTarget Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
kafkaTarget	KafkaTarget Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
mongoTarget	MongoTarget Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
metrikaSource	MetrikaSource Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.
ydsTarget	YDSTarget Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`.

MysqlSource

Field	Description
connection	MysqlConnection Database connection settings
database	string Database name You can leave it empty, then it will be possible to transfer tables from several databases at the same time from this source.
user	string User for database access.
password	Secret Password for database access.
timezone	string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.
objectTransferSettings	MysqlObjectTransferSettings Schema migration Select database objects to be transferred during activation or deactivation.
includeTablesRegex[]	string
excludeTablesRegex[]	string
securityGroups[]	string Security groups
serviceDatabase	string Database for service tables Default: data source database. Here created technical tables (__tm_keeper, __tm_gtid_keeper).

MysqlConnection

Field

Description

mdbClusterId

string

Managed Service for MySQL cluster ID

Includes only one of the fields mdbClusterId, onPremise.

onPremise

OnPremiseMysql

Connection options for on-premise MySQL

Includes only one of the fields mdbClusterId, onPremise.

OnPremiseMysql

Field	Description
port	string (int64) Database port
subnetId	string Network interface for endpoint. If none will assume public ipv4
hosts[]	string
tlsMode	TLSMode TLS settings for server connection. Disabled by default.

TLSMode

Field

Description

enabled

TLSConfig

Includes only one of the fields enabled.

TLSConfig

Field

Description

caCertificate

string

CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified TLS is used to
connect to the server.

Secret

Field

Description

raw

string

Raw secret value

Includes only one of the fields raw.

MysqlObjectTransferSettings

Field	Description
view	enum (ObjectTransferStage) Views CREATE VIEW ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
routine	enum (ObjectTransferStage) Routines CREATE PROCEDURE ... ; CREATE FUNCTION ... ; `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
trigger	enum (ObjectTransferStage) Triggers CREATE TRIGGER ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
tables	enum (ObjectTransferStage) `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy

PostgresSource

Field	Description
connection	PostgresConnection Database connection settings
database	string Database name
user	string User for database access. not required as may be in connection
password	Secret Password for database access.
includeTables[]	string Included tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.
excludeTables[]	string Excluded tables If none or empty list is presented, all tables are replicated. Full table name with schema. Can contain schema_name.* patterns.
slotByteLagLimit	string (int64) Maximum lag of replication slot (in bytes); after exceeding this limit replication will be aborted.
serviceSchema	string Database schema for service tables (__consumer_keeper, __data_transfer_mole_finder). Default is public
objectTransferSettings	PostgresObjectTransferSettings Select database objects to be transferred during activation or deactivation.
securityGroups[]	string Security groups

PostgresConnection

Field

Description

mdbClusterId

string

Managed Service for PostgreSQL cluster ID

Includes only one of the fields mdbClusterId, onPremise.

onPremise

OnPremisePostgres

Connection options for on-premise PostgreSQL

Includes only one of the fields mdbClusterId, onPremise.

OnPremisePostgres

Field	Description
port	string (int64) Will be used if the cluster ID is not specified.
subnetId	string Network interface for endpoint. If none will assume public ipv4
hosts[]	string
tlsMode	TLSMode TLS settings for server connection. Disabled by default.

PostgresObjectTransferSettings

Field	Description
sequence	enum (ObjectTransferStage) Sequences CREATE SEQUENCE ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
sequenceOwnedBy	enum (ObjectTransferStage) Owned sequences CREATE SEQUENCE ... OWNED BY ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
table	enum (ObjectTransferStage) Tables CREATE TABLE ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
primaryKey	enum (ObjectTransferStage) Primary keys ALTER TABLE ... ADD PRIMARY KEY ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
fkConstraint	enum (ObjectTransferStage) Foreign keys ALTER TABLE ... ADD FOREIGN KEY ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
defaultValues	enum (ObjectTransferStage) Default values ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
constraint	enum (ObjectTransferStage) Constraints ALTER TABLE ... ADD CONSTRAINT ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
index	enum (ObjectTransferStage) Indexes CREATE INDEX ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
view	enum (ObjectTransferStage) Views CREATE VIEW ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
function	enum (ObjectTransferStage) Functions CREATE FUNCTION ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
trigger	enum (ObjectTransferStage) Triggers CREATE TRIGGER ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
type	enum (ObjectTransferStage) Types CREATE TYPE ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
rule	enum (ObjectTransferStage) Rules CREATE RULE ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
collation	enum (ObjectTransferStage) Collations CREATE COLLATION ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
policy	enum (ObjectTransferStage) Policies CREATE POLICY ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
cast	enum (ObjectTransferStage) Casts CREATE CAST ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
materializedView	enum (ObjectTransferStage) Materialized views CREATE MATERIALIZED VIEW ... `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy
sequenceSet	enum (ObjectTransferStage) `OBJECT_TRANSFER_STAGE_UNSPECIFIED` `BEFORE_DATA`: Before data transfer `AFTER_DATA`: After data transfer `NEVER`: Don't copy

YdbSource

Field	Description
database	string Path in YDB where to store tables
instance	string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135
paths[]	string
serviceAccountId	string
subnetId	string Network interface for endpoint. If none will assume public ipv4
saKeyContent	string Authorization Key
securityGroups[]	string Security groups
changefeedCustomName	string Pre-created change feed

YDSSource

Field	Description
database	string Database
stream	string Stream
serviceAccountId	string SA which has read access to the stream.
supportedCodecs[]	enum (YdsCompressionCodec) Compression codec `YDS_COMPRESSION_CODEC_UNSPECIFIED` `YDS_COMPRESSION_CODEC_RAW` `YDS_COMPRESSION_CODEC_GZIP` `YDS_COMPRESSION_CODEC_ZSTD`
parser	Parser Data parsing rules
allowTtlRewind	boolean Should continue working, if consumer read lag exceed TTL of topic False: stop the transfer in error state, if detected lost data. True: continue working with losing part of data
endpoint	string for dedicated db
subnetId	string Network interface for endpoint. If none will assume public ipv4
securityGroups[]	string Security groups
consumer	string for important streams

Parser

Field	Description
jsonParser	GenericParserCommon Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`.
auditTrailsV1Parser	object Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`.
cloudLoggingParser	object Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`.
tskvParser	GenericParserCommon Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`.

GenericParserCommon

Field	Description
dataSchema	DataSchema
nullKeysAllowed	boolean Allow null keys, if no - null keys will be putted to unparsed data
addRestColumn	boolean Will add _rest column for all unknown fields
unescapeStringValues	boolean Unescape string values

DataSchema

Field

Description

jsonFields

string

Includes only one of the fields jsonFields, fields.

fields

FieldList

Includes only one of the fields jsonFields, fields.

FieldList

Field

Description

fields[]

ColSchema

Column schema

ColSchema

Field	Description
name	string
type	enum (ColumnType) `COLUMN_TYPE_UNSPECIFIED` `INT32` `INT16` `INT8` `UINT64` `UINT32` `UINT16` `UINT8` `DOUBLE` `BOOLEAN` `STRING` `UTF8` `ANY` `DATETIME` `INT64`
key	boolean
required	boolean
path	string

KafkaSource

Field	Description
connection	KafkaConnectionOptions Connection settings
auth	KafkaAuth Authentication settings
securityGroups[]	string Security groups
topicName	string Full source topic name Deprecated in favor of topic names
transformer	DataTransformationOptions Data transformation rules
parser	Parser Data parsing rules
topicNames[]	string List of topic names to read

KafkaConnectionOptions

Field

Description

clusterId

string

Managed Service for Kafka cluster ID

Includes only one of the fields clusterId, onPremise.

onPremise

OnPremiseKafka

Connection options for on-premise Kafka

Includes only one of the fields clusterId, onPremise.

OnPremiseKafka

Field	Description
brokerUrls[]	string Kafka broker URLs
subnetId	string Network interface for endpoint. If none will assume public ipv4
tlsMode	TLSMode TLS settings for broker connection. Disabled by default.

KafkaAuth

Field

Description

sasl

KafkaSaslSecurity

Authentication with SASL

Includes only one of the fields sasl, noAuth.

noAuth

object

No authentication

Includes only one of the fields sasl, noAuth.

KafkaSaslSecurity

Field	Description
user	string User name
mechanism	enum (KafkaMechanism) SASL mechanism for authentication `KAFKA_MECHANISM_UNSPECIFIED` `KAFKA_MECHANISM_SHA256` `KAFKA_MECHANISM_SHA512`
password	Secret Password for user

DataTransformationOptions

Field	Description
cloudFunction	string Cloud function
numberOfRetries	string (int64) Number of retries
bufferSize	string Buffer size for function
bufferFlushInterval	string Flush interval
invocationTimeout	string Invocation timeout
serviceAccountId	string Service account

MongoSource

Field	Description
connection	MongoConnection
subnetId	string
collections[]	MongoCollection List of collections for replication. Empty list implies replication of all tables on the deployment. Allowed to use * as collection name.
excludedCollections[]	MongoCollection List of forbidden collections for replication. Allowed to use * as collection name for forbid all collections of concrete schema.
secondaryPreferredMode	boolean Read mode for mongo client
securityGroups[]	string Security groups

MongoConnection

Field

Description

connectionOptions

MongoConnectionOptions

Includes only one of the fields connectionOptions.

MongoConnectionOptions

Field	Description
mdbClusterId	string Includes only one of the fields `mdbClusterId`, `onPremise`.
onPremise	OnPremiseMongo Includes only one of the fields `mdbClusterId`, `onPremise`.
user	string User name
password	Secret Password for user
authSource	string Database name associated with the credentials

OnPremiseMongo

Field	Description
hosts[]	string
port	string (int64)
replicaSet	string
tlsMode	TLSMode

MongoCollection

Field	Description
databaseName	string
collectionName	string

ClickhouseSource

Field	Description
connection	ClickhouseConnection
includeTables[]	string While list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.
excludeTables[]	string Exclude list of tables for replication. If none or empty list is presented - will replicate all tables. Can contain * patterns.
subnetId	string
securityGroups[]	string
clickhouseClusterName	string Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup.

ClickhouseConnection

Field

Description

connectionOptions

ClickhouseConnectionOptions

Includes only one of the fields connectionOptions.

ClickhouseConnectionOptions

Field	Description
onPremise	OnPremiseClickhouse Includes only one of the fields `onPremise`, `mdbClusterId`.
mdbClusterId	string Includes only one of the fields `onPremise`, `mdbClusterId`.
user	string
password	Secret
database	string Database

OnPremiseClickhouse

Field	Description
shards[]	ClickhouseShard
httpPort	string (int64)
nativePort	string (int64)
tlsMode	TLSMode

ClickhouseShard

Field	Description
name	string
hosts[]	string

MysqlTarget

Field	Description
connection	MysqlConnection Database connection settings
database	string Database name Allowed to leave it empty, then the tables will be created in databases with the same names as on the source. If this field is empty, then you must fill below db schema for service table.
user	string User for database access.
password	Secret Password for database access.
sqlMode	string Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION.
skipConstraintChecks	boolean Disable constraints checks Recommend to disable for increase replication speed, but if schema contain cascading operations we don't recommend to disable. This option set FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0.
timezone	string Database timezone Is used for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.
cleanupPolicy	enum (CleanupPolicy) Cleanup policy Cleanup policy for activate, reactivate and reupload processes. Default is DISABLED. `CLEANUP_POLICY_UNSPECIFIED` `DISABLED`: Don't cleanup `DROP`: Drop `TRUNCATE`: Truncate
serviceDatabase	string Database schema for service table Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper).
securityGroups[]	string Security groups

PostgresTarget

Field	Description
connection	PostgresConnection Database connection settings
database	string Database name
user	string User for database access. not required as may be in connection
password	Secret Password for database access.
cleanupPolicy	enum (CleanupPolicy) Cleanup policy for activate, reactivate and reupload processes. Default is truncate. `CLEANUP_POLICY_UNSPECIFIED` `DISABLED`: Don't cleanup `DROP`: Drop `TRUNCATE`: Truncate
securityGroups[]	string Security groups

ClickhouseTarget

Field	Description
connection	ClickhouseConnection
subnetId	string
altNames[]	AltName Alternative table names in target
cleanupPolicy	enum (ClickhouseCleanupPolicy) `CLICKHOUSE_CLEANUP_POLICY_UNSPECIFIED` `CLICKHOUSE_CLEANUP_POLICY_DISABLED` `CLICKHOUSE_CLEANUP_POLICY_DROP` `CLICKHOUSE_CLEANUP_POLICY_TRUNCATE`
sharding	ClickhouseSharding
clickhouseClusterName	string Name of the ClickHouse cluster. For Managed ClickHouse that is name of ShardGroup.
securityGroups[]	string

AltName

Field

Description

fromName

string

Source table name

toName

string

Target table name

ClickhouseSharding

Field

Description

columnValueHash

ColumnValueHash

Includes only one of the fields columnValueHash, customMapping.

customMapping

ColumnValueMapping

Includes only one of the fields columnValueHash, customMapping.

ColumnValueHash

Field	Description
columnName	string

ColumnValueMapping

Field	Description
columnName	string
mapping[]	ValueToShard

ValueToShard

Field	Description
columnValue	ColumnValue
shardName	string

ColumnValue

Field

Description

stringValue

string

Includes only one of the fields stringValue.

YdbTarget

Field	Description
database	string Path in YDB where to store tables
instance	string Instance of YDB. example: ydb-ru-prestable.yandex.net:2135
path	string Path extension for database, each table will be layouted into this path
serviceAccountId	string
cleanupPolicy	enum (YdbCleanupPolicy) Cleanup policy `YDB_CLEANUP_POLICY_UNSPECIFIED` `YDB_CLEANUP_POLICY_DISABLED` `YDB_CLEANUP_POLICY_DROP`
subnetId	string Network interface for endpoint. If none will assume public ipv4
saKeyContent	string SA content
securityGroups[]	string Security groups
isTableColumnOriented	boolean Should create column-oriented table (OLAP). By default it creates row-oriented (OLTP)
defaultCompression	enum (YdbDefaultCompression) Compression that will be used for default columns family on YDB table creation `YDB_DEFAULT_COMPRESSION_UNSPECIFIED` `YDB_DEFAULT_COMPRESSION_DISABLED` `YDB_DEFAULT_COMPRESSION_LZ4`

KafkaTarget

Field	Description
connection	KafkaConnectionOptions Connection settings
auth	KafkaAuth Authentication settings
securityGroups[]	string Security groups
topicSettings	KafkaTargetTopicSettings Target topic settings
serializer	Serializer Data serialization format settings

KafkaTargetTopicSettings

Field

Description

topic

KafkaTargetTopic

Full topic name

Includes only one of the fields topic, topicPrefix.

topicPrefix

string

Topic prefix

Analogue of the Debezium setting database.server.name.
Messages will be sent to topic with name <topic_prefix>..<table_name>.

Includes only one of the fields topic, topicPrefix.

KafkaTargetTopic

Field

Description

topicName

string

Topic name

saveTxOrder

boolean

Save transactions order
Not to split events queue into separate per-table queues.

Serializer

Data serialization format

Field	Description
serializerAuto	object Select the serialization format automatically Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`.
serializerJson	object Serialize data in json format Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`.
serializerDebezium	SerializerDebezium Serialize data in debezium format Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`.

SerializerDebezium

Field

Description

serializerParameters[]

DebeziumSerializerParameter

Settings of sterilization parameters as key-value pairs

DebeziumSerializerParameter

Field

Description

key

string

Name of the serializer parameter

value

string

Value of the serializer parameter

MongoTarget

Field	Description
connection	MongoConnection
database	string Database name
cleanupPolicy	enum (CleanupPolicy) `CLEANUP_POLICY_UNSPECIFIED` `DISABLED`: Don't cleanup `DROP`: Drop `TRUNCATE`: Truncate
subnetId	string
securityGroups[]	string Security groups

MetrikaSource

Field	Description
counterIds[]	string (int64)
token	Secret
streams[]	MetrikaStream

MetrikaStream

Field

Description

type

enum (MetrikaStreamType)

METRIKA_STREAM_TYPE_UNSPECIFIED
METRIKA_STREAM_TYPE_HITS
METRIKA_STREAM_TYPE_VISITS
METRIKA_STREAM_TYPE_HITS_V2

columns[]

string

YDSTarget

Field	Description
database	string Database
stream	string Stream
serviceAccountId	string SA which has read access to the stream.
saveTxOrder	boolean Save transaction order Not to split events queue into separate per-table queues. Incompatible with setting Topic prefix, only with Topic full name.
serializer	Serializer Data serialization format
endpoint	string for dedicated db
subnetId	string Network interface for endpoint. If none will assume public ipv4
securityGroups[]	string Security groups

Data Transfer API, REST: Endpoint.List

HTTP requestHTTP request

Path parametersPath parameters

Query parametersQuery parameters

ResponseResponse

EndpointEndpoint

EndpointSettingsEndpointSettings

MysqlSourceMysqlSource

MysqlConnectionMysqlConnection

OnPremiseMysqlOnPremiseMysql

TLSModeTLSMode

TLSConfigTLSConfig

SecretSecret

MysqlObjectTransferSettingsMysqlObjectTransferSettings

PostgresSourcePostgresSource

PostgresConnectionPostgresConnection

OnPremisePostgresOnPremisePostgres

PostgresObjectTransferSettingsPostgresObjectTransferSettings

YdbSourceYdbSource

YDSSourceYDSSource

ParserParser

GenericParserCommonGenericParserCommon

DataSchemaDataSchema

FieldListFieldList

ColSchemaColSchema

KafkaSourceKafkaSource

KafkaConnectionOptionsKafkaConnectionOptions

OnPremiseKafkaOnPremiseKafka

KafkaAuthKafkaAuth

KafkaSaslSecurityKafkaSaslSecurity

DataTransformationOptionsDataTransformationOptions

MongoSourceMongoSource

MongoConnectionMongoConnection

MongoConnectionOptionsMongoConnectionOptions

OnPremiseMongoOnPremiseMongo

MongoCollectionMongoCollection

ClickhouseSourceClickhouseSource

ClickhouseConnectionClickhouseConnection

ClickhouseConnectionOptionsClickhouseConnectionOptions

OnPremiseClickhouseOnPremiseClickhouse

ClickhouseShardClickhouseShard

MysqlTargetMysqlTarget

PostgresTargetPostgresTarget

ClickhouseTargetClickhouseTarget

AltNameAltName

ClickhouseShardingClickhouseSharding

ColumnValueHashColumnValueHash

ColumnValueMappingColumnValueMapping

ValueToShardValueToShard

ColumnValueColumnValue

YdbTargetYdbTarget

KafkaTargetKafkaTarget

KafkaTargetTopicSettingsKafkaTargetTopicSettings

KafkaTargetTopicKafkaTargetTopic

SerializerSerializer

SerializerDebeziumSerializerDebezium

DebeziumSerializerParameterDebeziumSerializerParameter

MongoTargetMongoTarget

MetrikaSourceMetrikaSource

MetrikaStreamMetrikaStream

YDSTargetYDSTarget

Was the article helpful?

HTTP request

Path parameters

Query parameters

Response

Endpoint

EndpointSettings

MysqlSource

MysqlConnection

OnPremiseMysql

TLSMode

TLSConfig

Secret

MysqlObjectTransferSettings

PostgresSource

PostgresConnection

OnPremisePostgres

PostgresObjectTransferSettings

YdbSource

YDSSource

Parser

GenericParserCommon

DataSchema

FieldList

ColSchema

KafkaSource

KafkaConnectionOptions

OnPremiseKafka

KafkaAuth

KafkaSaslSecurity

DataTransformationOptions

MongoSource

MongoConnection

MongoConnectionOptions

OnPremiseMongo

MongoCollection

ClickhouseSource

ClickhouseConnection

ClickhouseConnectionOptions

OnPremiseClickhouse

ClickhouseShard

MysqlTarget

PostgresTarget

ClickhouseTarget

AltName

ClickhouseSharding

ColumnValueHash

ColumnValueMapping

ValueToShard

ColumnValue

YdbTarget

KafkaTarget

KafkaTargetTopicSettings

KafkaTargetTopic

Serializer

SerializerDebezium

DebeziumSerializerParameter

MongoTarget

MetrikaSource

MetrikaStream

YDSTarget