Связаться с намиПодключиться

Data Transfer API, gRPC: EndpointService.List

Статья создана
Обновлена 17 октября 2024 г.

gRPC request

rpc List (ListEndpointsRequest) returns (ListEndpointsResponse)

ListEndpointsRequest

{
  "folderId": "string",
  "pageSize": "int64",
  "pageToken": "string"
}

Field

Description

folderId

string

Identifier of the folder containing the endpoints to be listed.

pageSize

int64

The maximum number of endpoints to be sent in the response message. If the
folder contains more endpoints than page_size, next_page_token will be
included
in the response message. Include it into the subsequent ListEndpointRequest to
fetch the next page. Defaults to 100 if not specified. The maximum allowed
value
for this field is 500.

pageToken

string

Opaque value identifying the endpoints page to be fetched. Should be empty in
the first ListEndpointsRequest. Subsequent requests should have this field
filled
with the next_page_token from the previous ListEndpointsResponse.

ListEndpointsResponse

{
  "endpoints": [
    {
      "id": "string",
      "folderId": "string",
      "name": "string",
      "description": "string",
      "labels": "string",
      "settings": {
        // Includes only one of the fields `mysqlSource`, `postgresSource`, `ydbSource`, `ydsSource`, `kafkaSource`, `mongoSource`, `clickhouseSource`, `mysqlTarget`, `postgresTarget`, `clickhouseTarget`, `ydbTarget`, `kafkaTarget`, `mongoTarget`, `metrikaSource`, `ydsTarget`
        "mysqlSource": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "int64",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `disabled`, `enabled`
                "disabled": "google.protobuf.Empty",
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "timezone": "string",
          "objectTransferSettings": {
            "view": "ObjectTransferStage",
            "routine": "ObjectTransferStage",
            "trigger": "ObjectTransferStage",
            "tables": "ObjectTransferStage"
          },
          "includeTablesRegex": [
            "string"
          ],
          "excludeTablesRegex": [
            "string"
          ],
          "securityGroups": [
            "string"
          ],
          "serviceDatabase": "string"
        },
        "postgresSource": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "int64",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `disabled`, `enabled`
                "disabled": "google.protobuf.Empty",
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "includeTables": [
            "string"
          ],
          "excludeTables": [
            "string"
          ],
          "slotByteLagLimit": "int64",
          "serviceSchema": "string",
          "objectTransferSettings": {
            "sequence": "ObjectTransferStage",
            "sequenceOwnedBy": "ObjectTransferStage",
            "table": "ObjectTransferStage",
            "primaryKey": "ObjectTransferStage",
            "fkConstraint": "ObjectTransferStage",
            "defaultValues": "ObjectTransferStage",
            "constraint": "ObjectTransferStage",
            "index": "ObjectTransferStage",
            "view": "ObjectTransferStage",
            "function": "ObjectTransferStage",
            "trigger": "ObjectTransferStage",
            "type": "ObjectTransferStage",
            "rule": "ObjectTransferStage",
            "collation": "ObjectTransferStage",
            "policy": "ObjectTransferStage",
            "cast": "ObjectTransferStage",
            "materializedView": "ObjectTransferStage",
            "sequenceSet": "ObjectTransferStage"
          },
          "securityGroups": [
            "string"
          ]
        },
        "ydbSource": {
          "database": "string",
          "instance": "string",
          "paths": [
            "string"
          ],
          "serviceAccountId": "string",
          "subnetId": "string",
          "saKeyContent": "string",
          "securityGroups": [
            "string"
          ],
          "changefeedCustomName": "string"
        },
        "ydsSource": {
          "database": "string",
          "stream": "string",
          "serviceAccountId": "string",
          "supportedCodecs": [
            "YdsCompressionCodec"
          ],
          "parser": {
            // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
            "jsonParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "ColumnType",
                      "key": "bool",
                      "required": "bool",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "bool",
              "addRestColumn": "bool",
              "unescapeStringValues": "bool"
            },
            "auditTrailsV1Parser": "AuditTrailsV1Parser",
            "cloudLoggingParser": "CloudLoggingParser",
            "tskvParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "ColumnType",
                      "key": "bool",
                      "required": "bool",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "bool",
              "addRestColumn": "bool",
              "unescapeStringValues": "bool"
            }
            // end of the list of possible fields
          },
          "allowTtlRewind": "bool",
          "endpoint": "string",
          "subnetId": "string",
          "securityGroups": [
            "string"
          ],
          "consumer": "string"
        },
        "kafkaSource": {
          "connection": {
            // Includes only one of the fields `clusterId`, `onPremise`
            "clusterId": "string",
            "onPremise": {
              "brokerUrls": [
                "string"
              ],
              "subnetId": "string",
              "tlsMode": {
                // Includes only one of the fields `disabled`, `enabled`
                "disabled": "google.protobuf.Empty",
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "auth": {
            // Includes only one of the fields `sasl`, `noAuth`
            "sasl": {
              "user": "string",
              "mechanism": "KafkaMechanism",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              }
            },
            "noAuth": "NoAuth"
            // end of the list of possible fields
          },
          "securityGroups": [
            "string"
          ],
          "topicName": "string",
          "transformer": {
            "cloudFunction": "string",
            "numberOfRetries": "int64",
            "bufferSize": "string",
            "bufferFlushInterval": "string",
            "invocationTimeout": "string",
            "serviceAccountId": "string"
          },
          "parser": {
            // Includes only one of the fields `jsonParser`, `auditTrailsV1Parser`, `cloudLoggingParser`, `tskvParser`
            "jsonParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "ColumnType",
                      "key": "bool",
                      "required": "bool",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "bool",
              "addRestColumn": "bool",
              "unescapeStringValues": "bool"
            },
            "auditTrailsV1Parser": "AuditTrailsV1Parser",
            "cloudLoggingParser": "CloudLoggingParser",
            "tskvParser": {
              "dataSchema": {
                // Includes only one of the fields `jsonFields`, `fields`
                "jsonFields": "string",
                "fields": {
                  "fields": [
                    {
                      "name": "string",
                      "type": "ColumnType",
                      "key": "bool",
                      "required": "bool",
                      "path": "string"
                    }
                  ]
                }
                // end of the list of possible fields
              },
              "nullKeysAllowed": "bool",
              "addRestColumn": "bool",
              "unescapeStringValues": "bool"
            }
            // end of the list of possible fields
          },
          "topicNames": [
            "string"
          ]
        },
        "mongoSource": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`
              "mdbClusterId": "string",
              "onPremise": {
                "hosts": [
                  "string"
                ],
                "port": "int64",
                "replicaSet": "string",
                "tlsMode": {
                  // Includes only one of the fields `disabled`, `enabled`
                  "disabled": "google.protobuf.Empty",
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "authSource": "string"
            }
            // end of the list of possible fields
          },
          "subnetId": "string",
          "collections": [
            {
              "databaseName": "string",
              "collectionName": "string"
            }
          ],
          "excludedCollections": [
            {
              "databaseName": "string",
              "collectionName": "string"
            }
          ],
          "secondaryPreferredMode": "bool",
          "securityGroups": [
            "string"
          ]
        },
        "clickhouseSource": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `onPremise`, `mdbClusterId`
              "onPremise": {
                "shards": [
                  {
                    "name": "string",
                    "hosts": [
                      "string"
                    ]
                  }
                ],
                "httpPort": "int64",
                "nativePort": "int64",
                "tlsMode": {
                  // Includes only one of the fields `disabled`, `enabled`
                  "disabled": "google.protobuf.Empty",
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "mdbClusterId": "string",
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "database": "string"
            }
            // end of the list of possible fields
          },
          "includeTables": [
            "string"
          ],
          "excludeTables": [
            "string"
          ],
          "subnetId": "string",
          "securityGroups": [
            "string"
          ],
          "clickhouseClusterName": "string"
        },
        "mysqlTarget": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "int64",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `disabled`, `enabled`
                "disabled": "google.protobuf.Empty",
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "sqlMode": "string",
          "skipConstraintChecks": "bool",
          "timezone": "string",
          "cleanupPolicy": "CleanupPolicy",
          "serviceDatabase": "string",
          "securityGroups": [
            "string"
          ]
        },
        "postgresTarget": {
          "connection": {
            // Includes only one of the fields `mdbClusterId`, `onPremise`
            "mdbClusterId": "string",
            "onPremise": {
              "port": "int64",
              "subnetId": "string",
              "hosts": [
                "string"
              ],
              "tlsMode": {
                // Includes only one of the fields `disabled`, `enabled`
                "disabled": "google.protobuf.Empty",
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "database": "string",
          "user": "string",
          "password": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "cleanupPolicy": "CleanupPolicy",
          "securityGroups": [
            "string"
          ]
        },
        "clickhouseTarget": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `onPremise`, `mdbClusterId`
              "onPremise": {
                "shards": [
                  {
                    "name": "string",
                    "hosts": [
                      "string"
                    ]
                  }
                ],
                "httpPort": "int64",
                "nativePort": "int64",
                "tlsMode": {
                  // Includes only one of the fields `disabled`, `enabled`
                  "disabled": "google.protobuf.Empty",
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              "mdbClusterId": "string",
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "database": "string"
            }
            // end of the list of possible fields
          },
          "subnetId": "string",
          "altNames": [
            {
              "fromName": "string",
              "toName": "string"
            }
          ],
          "cleanupPolicy": "ClickhouseCleanupPolicy",
          "sharding": {
            // Includes only one of the fields `columnValueHash`, `customMapping`, `transferId`, `roundRobin`
            "columnValueHash": {
              "columnName": "string"
            },
            "customMapping": {
              "columnName": "string",
              "mapping": [
                {
                  "columnValue": {
                    // Includes only one of the fields `stringValue`
                    "stringValue": "string"
                    // end of the list of possible fields
                  },
                  "shardName": "string"
                }
              ]
            },
            "transferId": "google.protobuf.Empty",
            "roundRobin": "google.protobuf.Empty"
            // end of the list of possible fields
          },
          "clickhouseClusterName": "string",
          "securityGroups": [
            "string"
          ]
        },
        "ydbTarget": {
          "database": "string",
          "instance": "string",
          "path": "string",
          "serviceAccountId": "string",
          "cleanupPolicy": "YdbCleanupPolicy",
          "subnetId": "string",
          "saKeyContent": "string",
          "securityGroups": [
            "string"
          ],
          "isTableColumnOriented": "bool",
          "defaultCompression": "YdbDefaultCompression"
        },
        "kafkaTarget": {
          "connection": {
            // Includes only one of the fields `clusterId`, `onPremise`
            "clusterId": "string",
            "onPremise": {
              "brokerUrls": [
                "string"
              ],
              "subnetId": "string",
              "tlsMode": {
                // Includes only one of the fields `disabled`, `enabled`
                "disabled": "google.protobuf.Empty",
                "enabled": {
                  "caCertificate": "string"
                }
                // end of the list of possible fields
              }
            }
            // end of the list of possible fields
          },
          "auth": {
            // Includes only one of the fields `sasl`, `noAuth`
            "sasl": {
              "user": "string",
              "mechanism": "KafkaMechanism",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              }
            },
            "noAuth": "NoAuth"
            // end of the list of possible fields
          },
          "securityGroups": [
            "string"
          ],
          "topicSettings": {
            // Includes only one of the fields `topic`, `topicPrefix`
            "topic": {
              "topicName": "string",
              "saveTxOrder": "bool"
            },
            "topicPrefix": "string"
            // end of the list of possible fields
          },
          "serializer": {
            // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
            "serializerAuto": "SerializerAuto",
            "serializerJson": "SerializerJSON",
            "serializerDebezium": {
              "serializerParameters": [
                {
                  "key": "string",
                  "value": "string"
                }
              ]
            }
            // end of the list of possible fields
          }
        },
        "mongoTarget": {
          "connection": {
            // Includes only one of the fields `connectionOptions`
            "connectionOptions": {
              // Includes only one of the fields `mdbClusterId`, `onPremise`
              "mdbClusterId": "string",
              "onPremise": {
                "hosts": [
                  "string"
                ],
                "port": "int64",
                "replicaSet": "string",
                "tlsMode": {
                  // Includes only one of the fields `disabled`, `enabled`
                  "disabled": "google.protobuf.Empty",
                  "enabled": {
                    "caCertificate": "string"
                  }
                  // end of the list of possible fields
                }
              },
              // end of the list of possible fields
              "user": "string",
              "password": {
                // Includes only one of the fields `raw`
                "raw": "string"
                // end of the list of possible fields
              },
              "authSource": "string"
            }
            // end of the list of possible fields
          },
          "database": "string",
          "cleanupPolicy": "CleanupPolicy",
          "subnetId": "string",
          "securityGroups": [
            "string"
          ]
        },
        "metrikaSource": {
          "counterIds": [
            "int64"
          ],
          "token": {
            // Includes only one of the fields `raw`
            "raw": "string"
            // end of the list of possible fields
          },
          "streams": [
            {
              "type": "MetrikaStreamType",
              "columns": [
                "string"
              ]
            }
          ]
        },
        "ydsTarget": {
          "database": "string",
          "stream": "string",
          "serviceAccountId": "string",
          "saveTxOrder": "bool",
          "serializer": {
            // Includes only one of the fields `serializerAuto`, `serializerJson`, `serializerDebezium`
            "serializerAuto": "SerializerAuto",
            "serializerJson": "SerializerJSON",
            "serializerDebezium": {
              "serializerParameters": [
                {
                  "key": "string",
                  "value": "string"
                }
              ]
            }
            // end of the list of possible fields
          },
          "endpoint": "string",
          "subnetId": "string",
          "securityGroups": [
            "string"
          ]
        }
        // end of the list of possible fields
      }
    }
  ],
  "nextPageToken": "string"
}

Field

Description

endpoints[]

Endpoint

The list of endpoints. If there are more endpoints in the folder, then
next_page_token is a non-empty string to be included into the subsequent
ListEndpointsRequest to fetch the next endpoints page.

nextPageToken

string

Opaque value identifying the next endpoints page. This field is empty if there
are no more endpoints in the folder. Otherwise, it is non-empty and should be
included in the subsequent ListEndpointsRequest to fetch the next endpoints
page.

Endpoint

Field

Description

id

string

folderId

string

name

string

description

string

labels

string

settings

EndpointSettings

EndpointSettings

Field

Description

mysqlSource

MysqlSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

postgresSource

PostgresSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydbSource

YdbSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydsSource

YDSSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

kafkaSource

KafkaSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

mongoSource

MongoSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

clickhouseSource

ClickhouseSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

mysqlTarget

MysqlTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

postgresTarget

PostgresTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

clickhouseTarget

ClickhouseTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydbTarget

YdbTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

kafkaTarget

KafkaTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

mongoTarget

MongoTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

metrikaSource

MetrikaSource

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

ydsTarget

YDSTarget

Includes only one of the fields mysqlSource, postgresSource, ydbSource, ydsSource, kafkaSource, mongoSource, clickhouseSource, mysqlTarget, postgresTarget, clickhouseTarget, ydbTarget, kafkaTarget, mongoTarget, metrikaSource, ydsTarget.

MysqlSource

Field

Description

connection

MysqlConnection

Database connection settings

database

string

Database name

You can leave it empty, then it will be possible to transfer tables from several
databases at the same time from this source.

user

string

User for database access.

password

Secret

Password for database access.

timezone

string

Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from
IANA timezone database. Default: local timezone.

objectTransferSettings

MysqlObjectTransferSettings

Schema migration

Select database objects to be transferred during activation or deactivation.

includeTablesRegex[]

string

excludeTablesRegex[]

string

securityGroups[]

string

Security groups

serviceDatabase

string

Database for service tables

Default: data source database. Here created technical tables (__tm_keeper,
__tm_gtid_keeper).

MysqlConnection

Field

Description

mdbClusterId

string

Managed Service for MySQL cluster ID

Includes only one of the fields mdbClusterId, onPremise.

onPremise

OnPremiseMysql

Connection options for on-premise MySQL

Includes only one of the fields mdbClusterId, onPremise.

OnPremiseMysql

Field

Description

port

int64

Database port

subnetId

string

Network interface for endpoint. If none will assume public ipv4

hosts[]

string

tlsMode

TLSMode

TLS settings for server connection. Disabled by default.

TLSMode

Field

Description

disabled

google.protobuf.Empty

Includes only one of the fields disabled, enabled.

enabled

TLSConfig

Includes only one of the fields disabled, enabled.

TLSConfig

Field

Description

caCertificate

string

CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified TLS is used to
connect to the server.

Secret

Field

Description

raw

string

Raw secret value

Includes only one of the fields raw.

MysqlObjectTransferSettings

Field

Description

view

enum ObjectTransferStage

Views

CREATE VIEW ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

routine

enum ObjectTransferStage

Routines

CREATE PROCEDURE ... ; CREATE FUNCTION ... ;

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

trigger

enum ObjectTransferStage

Triggers

CREATE TRIGGER ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

tables

enum ObjectTransferStage

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

PostgresSource

Field

Description

connection

PostgresConnection

Database connection settings

database

string

Database name

user

string

User for database access. not required as may be in connection

password

Secret

Password for database access.

includeTables[]

string

Included tables

If none or empty list is presented, all tables are replicated. Full table name
with schema. Can contain schema_name.* patterns.

excludeTables[]

string

Excluded tables

If none or empty list is presented, all tables are replicated. Full table name
with schema. Can contain schema_name.* patterns.

slotByteLagLimit

int64

Maximum lag of replication slot (in bytes); after exceeding this limit
replication will be aborted.

serviceSchema

string

Database schema for service tables (__consumer_keeper,
__data_transfer_mole_finder). Default is public

objectTransferSettings

PostgresObjectTransferSettings

Select database objects to be transferred during activation or deactivation.

securityGroups[]

string

Security groups

PostgresConnection

Field

Description

mdbClusterId

string

Managed Service for PostgreSQL cluster ID

Includes only one of the fields mdbClusterId, onPremise.

onPremise

OnPremisePostgres

Connection options for on-premise PostgreSQL

Includes only one of the fields mdbClusterId, onPremise.

OnPremisePostgres

Field

Description

port

int64

Will be used if the cluster ID is not specified.

subnetId

string

Network interface for endpoint. If none will assume public ipv4

hosts[]

string

tlsMode

TLSMode

TLS settings for server connection. Disabled by default.

PostgresObjectTransferSettings

Field

Description

sequence

enum ObjectTransferStage

Sequences

CREATE SEQUENCE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

sequenceOwnedBy

enum ObjectTransferStage

Owned sequences

CREATE SEQUENCE ... OWNED BY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

table

enum ObjectTransferStage

Tables

CREATE TABLE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

primaryKey

enum ObjectTransferStage

Primary keys

ALTER TABLE ... ADD PRIMARY KEY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

fkConstraint

enum ObjectTransferStage

Foreign keys

ALTER TABLE ... ADD FOREIGN KEY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

defaultValues

enum ObjectTransferStage

Default values

ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

constraint

enum ObjectTransferStage

Constraints

ALTER TABLE ... ADD CONSTRAINT ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

index

enum ObjectTransferStage

Indexes

CREATE INDEX ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

view

enum ObjectTransferStage

Views

CREATE VIEW ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

function

enum ObjectTransferStage

Functions

CREATE FUNCTION ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

trigger

enum ObjectTransferStage

Triggers

CREATE TRIGGER ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

type

enum ObjectTransferStage

Types

CREATE TYPE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

rule

enum ObjectTransferStage

Rules

CREATE RULE ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

collation

enum ObjectTransferStage

Collations

CREATE COLLATION ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

policy

enum ObjectTransferStage

Policies

CREATE POLICY ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

cast

enum ObjectTransferStage

Casts

CREATE CAST ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

materializedView

enum ObjectTransferStage

Materialized views

CREATE MATERIALIZED VIEW ...

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

sequenceSet

enum ObjectTransferStage

  • OBJECT_TRANSFER_STAGE_UNSPECIFIED
  • BEFORE_DATA: Before data transfer
  • AFTER_DATA: After data transfer
  • NEVER: Don't copy

YdbSource

Field

Description

database

string

Path in YDB where to store tables

instance

string

Instance of YDB. example: ydb-ru-prestable.yandex.net:2135

paths[]

string

serviceAccountId

string

subnetId

string

Network interface for endpoint. If none will assume public ipv4

saKeyContent

string

Authorization Key

securityGroups[]

string

Security groups

changefeedCustomName

string

Pre-created change feed

YDSSource

Field

Description

database

string

Database

stream

string

Stream

serviceAccountId

string

SA which has read access to the stream.

supportedCodecs[]

enum YdsCompressionCodec

Compression codec

  • YDS_COMPRESSION_CODEC_UNSPECIFIED
  • YDS_COMPRESSION_CODEC_RAW
  • YDS_COMPRESSION_CODEC_GZIP
  • YDS_COMPRESSION_CODEC_ZSTD

parser

Parser

Data parsing rules

allowTtlRewind

bool

Should continue working, if consumer read lag exceed TTL of topic
False: stop the transfer in error state, if detected lost data. True: continue
working with losing part of data

endpoint

string

for dedicated db

subnetId

string

Network interface for endpoint. If none will assume public ipv4

securityGroups[]

string

Security groups

consumer

string

for important streams

Parser

Field

Description

jsonParser

GenericParserCommon

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

auditTrailsV1Parser

AuditTrailsV1Parser

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

cloudLoggingParser

CloudLoggingParser

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

tskvParser

GenericParserCommon

Includes only one of the fields jsonParser, auditTrailsV1Parser, cloudLoggingParser, tskvParser.

GenericParserCommon

Field

Description

dataSchema

DataSchema

nullKeysAllowed

bool

Allow null keys, if no - null keys will be putted to unparsed data

addRestColumn

bool

Will add _rest column for all unknown fields

unescapeStringValues

bool

Unescape string values

DataSchema

Field

Description

jsonFields

string

Includes only one of the fields jsonFields, fields.

fields

FieldList

Includes only one of the fields jsonFields, fields.

FieldList

Field

Description

fields[]

ColSchema

Column schema

ColSchema

Field

Description

name

string

type

enum ColumnType

  • COLUMN_TYPE_UNSPECIFIED
  • INT32
  • INT16
  • INT8
  • UINT64
  • UINT32
  • UINT16
  • UINT8
  • DOUBLE
  • BOOLEAN
  • STRING
  • UTF8
  • ANY
  • DATETIME
  • INT64

key

bool

required

bool

path

string

AuditTrailsV1Parser

Field

Description

Empty

CloudLoggingParser

Field

Description

Empty

KafkaSource

Field

Description

connection

KafkaConnectionOptions

Connection settings

auth

KafkaAuth

Authentication settings

securityGroups[]

string

Security groups

topicName

string

Full source topic name
Deprecated in favor of topic names

transformer

DataTransformationOptions

Data transformation rules

parser

Parser

Data parsing rules

topicNames[]

string

List of topic names to read

KafkaConnectionOptions

Field

Description

clusterId

string

Managed Service for Kafka cluster ID

Includes only one of the fields clusterId, onPremise.

onPremise

OnPremiseKafka

Connection options for on-premise Kafka

Includes only one of the fields clusterId, onPremise.

OnPremiseKafka

Field

Description

brokerUrls[]

string

Kafka broker URLs

subnetId

string

Network interface for endpoint. If none will assume public ipv4

tlsMode

TLSMode

TLS settings for broker connection. Disabled by default.

KafkaAuth

Field

Description

sasl

KafkaSaslSecurity

Authentication with SASL

Includes only one of the fields sasl, noAuth.

noAuth

NoAuth

No authentication

Includes only one of the fields sasl, noAuth.

KafkaSaslSecurity

Field

Description

user

string

User name

mechanism

enum KafkaMechanism

SASL mechanism for authentication

  • KAFKA_MECHANISM_UNSPECIFIED
  • KAFKA_MECHANISM_SHA256
  • KAFKA_MECHANISM_SHA512

password

Secret

Password for user

NoAuth

No authentication

Field

Description

Empty

DataTransformationOptions

Field

Description

cloudFunction

string

Cloud function

numberOfRetries

int64

Number of retries

bufferSize

string

Buffer size for function

bufferFlushInterval

string

Flush interval

invocationTimeout

string

Invocation timeout

serviceAccountId

string

Service account

MongoSource

Field

Description

connection

MongoConnection

subnetId

string

collections[]

MongoCollection

List of collections for replication. Empty list implies replication of all
tables on the deployment. Allowed to use * as collection name.

excludedCollections[]

MongoCollection

List of forbidden collections for replication. Allowed to use * as collection
name for forbid all collections of concrete schema.

secondaryPreferredMode

bool

Read mode for mongo client

securityGroups[]

string

Security groups

MongoConnection

Field

Description

connectionOptions

MongoConnectionOptions

Includes only one of the fields connectionOptions.

MongoConnectionOptions

Field

Description

mdbClusterId

string

Includes only one of the fields mdbClusterId, onPremise.

onPremise

OnPremiseMongo

Includes only one of the fields mdbClusterId, onPremise.

user

string

User name

password

Secret

Password for user

authSource

string

Database name associated with the credentials

OnPremiseMongo

Field

Description

hosts[]

string

port

int64

replicaSet

string

tlsMode

TLSMode

MongoCollection

Field

Description

databaseName

string

collectionName

string

ClickhouseSource

Field

Description

connection

ClickhouseConnection

includeTables[]

string

While list of tables for replication. If none or empty list is presented - will
replicate all tables. Can contain * patterns.

excludeTables[]

string

Exclude list of tables for replication. If none or empty list is presented -
will replicate all tables. Can contain * patterns.

subnetId

string

securityGroups[]

string

clickhouseClusterName

string

Name of the ClickHouse cluster. For Managed ClickHouse that is name of
ShardGroup.

ClickhouseConnection

Field

Description

connectionOptions

ClickhouseConnectionOptions

Includes only one of the fields connectionOptions.

ClickhouseConnectionOptions

Field

Description

onPremise

OnPremiseClickhouse

Includes only one of the fields onPremise, mdbClusterId.

mdbClusterId

string

Includes only one of the fields onPremise, mdbClusterId.

user

string

password

Secret

database

string

Database

OnPremiseClickhouse

Field

Description

shards[]

ClickhouseShard

httpPort

int64

nativePort

int64

tlsMode

TLSMode

ClickhouseShard

Field

Description

name

string

hosts[]

string

MysqlTarget

Field

Description

connection

MysqlConnection

Database connection settings

database

string

Database name

Allowed to leave it empty, then the tables will be created in databases with the
same names as on the source. If this field is empty, then you must fill below db
schema for service table.

user

string

User for database access.

password

Secret

Password for database access.

sqlMode

string

Default: NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION.

skipConstraintChecks

bool

Disable constraints checks

Recommend to disable for increase replication speed, but if schema contain
cascading operations we don't recommend to disable. This option set
FOREIGN_KEY_CHECKS=0 and UNIQUE_CHECKS=0.

timezone

string

Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from
IANA timezone database. Default: local timezone.

cleanupPolicy

enum CleanupPolicy

Cleanup policy

Cleanup policy for activate, reactivate and reupload processes. Default is
DISABLED.

  • CLEANUP_POLICY_UNSPECIFIED
  • DISABLED: Don't cleanup
  • DROP: Drop
  • TRUNCATE: Truncate

serviceDatabase

string

Database schema for service table

Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper).

securityGroups[]

string

Security groups

PostgresTarget

Field

Description

connection

PostgresConnection

Database connection settings

database

string

Database name

user

string

User for database access. not required as may be in connection

password

Secret

Password for database access.

cleanupPolicy

enum CleanupPolicy

Cleanup policy for activate, reactivate and reupload processes. Default is
truncate.

  • CLEANUP_POLICY_UNSPECIFIED
  • DISABLED: Don't cleanup
  • DROP: Drop
  • TRUNCATE: Truncate

securityGroups[]

string

Security groups

ClickhouseTarget

Field

Description

connection

ClickhouseConnection

subnetId

string

altNames[]

AltName

Alternative table names in target

cleanupPolicy

enum ClickhouseCleanupPolicy

  • CLICKHOUSE_CLEANUP_POLICY_UNSPECIFIED
  • CLICKHOUSE_CLEANUP_POLICY_DISABLED
  • CLICKHOUSE_CLEANUP_POLICY_DROP
  • CLICKHOUSE_CLEANUP_POLICY_TRUNCATE

sharding

ClickhouseSharding

clickhouseClusterName

string

Name of the ClickHouse cluster. For Managed ClickHouse that is name of
ShardGroup.

securityGroups[]

string

AltName

Field

Description

fromName

string

Source table name

toName

string

Target table name

ClickhouseSharding

Field

Description

columnValueHash

ColumnValueHash

Includes only one of the fields columnValueHash, customMapping, transferId, roundRobin.

customMapping

ColumnValueMapping

Includes only one of the fields columnValueHash, customMapping, transferId, roundRobin.

transferId

google.protobuf.Empty

Includes only one of the fields columnValueHash, customMapping, transferId, roundRobin.

roundRobin

google.protobuf.Empty

Includes only one of the fields columnValueHash, customMapping, transferId, roundRobin.

ColumnValueHash

Field

Description

columnName

string

ColumnValueMapping

Field

Description

columnName

string

mapping[]

ValueToShard

ValueToShard

Field

Description

columnValue

ColumnValue

shardName

string

ColumnValue

Field

Description

stringValue

string

Includes only one of the fields stringValue.

YdbTarget

Field

Description

database

string

Path in YDB where to store tables

instance

string

Instance of YDB. example: ydb-ru-prestable.yandex.net:2135

path

string

Path extension for database, each table will be layouted into this path

serviceAccountId

string

cleanupPolicy

enum YdbCleanupPolicy

Cleanup policy

  • YDB_CLEANUP_POLICY_UNSPECIFIED
  • YDB_CLEANUP_POLICY_DISABLED
  • YDB_CLEANUP_POLICY_DROP

subnetId

string

Network interface for endpoint. If none will assume public ipv4

saKeyContent

string

SA content

securityGroups[]

string

Security groups

isTableColumnOriented

bool

Should create column-oriented table (OLAP). By default it creates row-oriented
(OLTP)

defaultCompression

enum YdbDefaultCompression

Compression that will be used for default columns family on YDB table creation

  • YDB_DEFAULT_COMPRESSION_UNSPECIFIED
  • YDB_DEFAULT_COMPRESSION_DISABLED
  • YDB_DEFAULT_COMPRESSION_LZ4

KafkaTarget

Field

Description

connection

KafkaConnectionOptions

Connection settings

auth

KafkaAuth

Authentication settings

securityGroups[]

string

Security groups

topicSettings

KafkaTargetTopicSettings

Target topic settings

serializer

Serializer

Data serialization format settings

KafkaTargetTopicSettings

Field

Description

topic

KafkaTargetTopic

Full topic name

Includes only one of the fields topic, topicPrefix.

topicPrefix

string

Topic prefix

Analogue of the Debezium setting database.server.name.
Messages will be sent to topic with name <topic_prefix>..<table_name>.

Includes only one of the fields topic, topicPrefix.

KafkaTargetTopic

Field

Description

topicName

string

Topic name

saveTxOrder

bool

Save transactions order
Not to split events queue into separate per-table queues.

Serializer

Data serialization format

Field

Description

serializerAuto

SerializerAuto

Select the serialization format automatically

Includes only one of the fields serializerAuto, serializerJson, serializerDebezium.

serializerJson

SerializerJSON

Serialize data in json format

Includes only one of the fields serializerAuto, serializerJson, serializerDebezium.

serializerDebezium

SerializerDebezium

Serialize data in debezium format

Includes only one of the fields serializerAuto, serializerJson, serializerDebezium.

SerializerAuto

Field

Description

Empty

SerializerJSON

Field

Description

Empty

SerializerDebezium

Field

Description

serializerParameters[]

DebeziumSerializerParameter

Settings of sterilization parameters as key-value pairs

DebeziumSerializerParameter

Field

Description

key

string

Name of the serializer parameter

value

string

Value of the serializer parameter

MongoTarget

Field

Description

connection

MongoConnection

database

string

Database name

cleanupPolicy

enum CleanupPolicy

  • CLEANUP_POLICY_UNSPECIFIED
  • DISABLED: Don't cleanup
  • DROP: Drop
  • TRUNCATE: Truncate

subnetId

string

securityGroups[]

string

Security groups

MetrikaSource

Field

Description

counterIds[]

int64

token

Secret

streams[]

MetrikaStream

MetrikaStream

Field

Description

type

enum MetrikaStreamType

  • METRIKA_STREAM_TYPE_UNSPECIFIED
  • METRIKA_STREAM_TYPE_HITS
  • METRIKA_STREAM_TYPE_VISITS
  • METRIKA_STREAM_TYPE_HITS_V2

columns[]

string

YDSTarget

Field

Description

database

string

Database

stream

string

Stream

serviceAccountId

string

SA which has read access to the stream.

saveTxOrder

bool

Save transaction order
Not to split events queue into separate per-table queues.
Incompatible with setting Topic prefix, only with Topic full name.

serializer

Serializer

Data serialization format

endpoint

string

for dedicated db

subnetId

string

Network interface for endpoint. If none will assume public ipv4

securityGroups[]

string

Security groups
Предыдущая
Get
Следующая
Create