yandex_datatransfer_endpoint (Resource)

Manages a Data Transfer endpoint. For more information, see the official documentation.

Example usageExample usage

//
// Create a new pair of Data Transfer Endpoints - Source & Target.
//
resource "yandex_datatransfer_endpoint" "pg_source" {
  name = "pg-test-source"
  settings {
    postgres_source {
      connection {
        on_premise {
          hosts = [
            "example.org"
          ]
          port = 5432
        }
      }
      slot_gigabyte_lag_limit = 100
      database                = "db1"
      user                    = "user1"
      password {
        raw = "123"
      }
    }
  }
}

resource "yandex_datatransfer_endpoint" "pg_target" {
  folder_id = "some_folder_id"
  name      = "pg-test-target2"
  settings {
    postgres_target {
      connection {
        mdb_cluster_id = "some_cluster_id"
      }
      security_groups = [/* list of security group ids */]
      database        = "db2"
      user            = "user2"
      password {
        raw = "321"
      }
    }
  }
}

SchemaSchema

OptionalOptional

• description (String) The resource description.
• folder_id (String) The folder identifier that resource belongs to. If it is not provided, the default provider folder-id is used.
• labels (Map of String) A set of key/value label pairs which assigned to resource.
• name (String) The resource name.
• settings (Block List, Max: 1) DataTransfer Endpoint Settings block. (see below for nested schema)

Read-OnlyRead-Only

• id (String) The ID of this resource.

Nested Schema for Nested Schema for settings

Optional:

• clickhouse_source (Block List, Max: 1) Settings specific to the ClickHouse source endpoint. (see below for nested schema)
• clickhouse_target (Block List, Max: 1) Settings specific to the ClickHouse target endpoint. (see below for nested schema)
• kafka_source (Block List, Max: 1) Settings specific to the Kafka source endpoint. (see below for nested schema)
• kafka_target (Block List, Max: 1) Settings specific to the Kafka target endpoint. (see below for nested schema)
• metrika_source (Block List, Max: 1) Settings specific to the Yandex Metrika source endpoint. (see below for nested schema)
• mongo_source (Block List, Max: 1) Settings specific to the MongoDB source endpoint. (see below for nested schema)
• mongo_target (Block List, Max: 1) Settings specific to the MongoDB target endpoint. (see below for nested schema)
• mysql_source (Block List, Max: 1) Settings specific to the MySQL source endpoint. (see below for nested schema)
• mysql_target (Block List, Max: 1) Settings specific to the MySQL target endpoint. (see below for nested schema)
• postgres_source (Block List, Max: 1) Settings specific to the PostgreSQL source endpoint. (see below for nested schema)
• postgres_target (Block List, Max: 1) Settings specific to the PostgreSQL target endpoint. (see below for nested schema)
• ydb_source (Block List, Max: 1) Settings specific to the YDB source endpoint. (see below for nested schema)
• ydb_target (Block List, Max: 1) Settings specific to the YDB target endpoint. (see below for nested schema)
• yds_source (Block List, Max: 1) Settings specific to the YDS source endpoint. (see below for nested schema)
• yds_target (Block List, Max: 1) Settings specific to the YDS target endpoint. (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source

Optional:

• clickhouse_cluster_name (String)
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• exclude_tables (List of String) The list of tables that should not be transferred.
• include_tables (List of String) The list of tables that should be transferred. Leave empty if all tables should be transferred.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.

Nested Schema for Nested Schema for settings.clickhouse_source.connection

Optional:

• connection_options (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options

Optional:

• database (String)
• mdb_cluster_id (String)
• on_premise (Block List, Max: 1) (see below for nested schema)
• password (Block List, Max: 1) (see below for nested schema)
• user (String)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise

Optional:

• http_port (Number)
• native_port (Number)
• shards (Block List) (see below for nested schema)
• tls_mode (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.shards

Optional:

• hosts (List of String)
• name (String)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.password

Optional:

• raw (String, Sensitive)

Nested Schema for Nested Schema for settings.clickhouse_target

Optional:

• alt_names (Block List) Table renaming rules. (see below for nested schema)
• cleanup_policy (String) How to clean collections when activating the transfer. One of CLICKHOUSE_CLEANUP_POLICY_DISABLED or CLICKHOUSE_CLEANUP_POLICY_DROP.
• clickhouse_cluster_name (String) Name of the ClickHouse cluster. For managed ClickHouse clusters defaults to managed cluster ID.
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• sharding (Block List, Max: 1) Shard selection rules for the data being transferred. (see below for nested schema)
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.

Nested Schema for Nested Schema for settings.clickhouse_target.alt_names

Optional:

• from_name (String)
• to_name (String)

Nested Schema for Nested Schema for settings.clickhouse_target.connection

Optional:

• connection_options (Block List, Max: 1) Connection options. (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options

Optional:

• database (String) Database name.
• mdb_cluster_id (String) Identifier of the Managed ClickHouse cluster.
• on_premise (Block List, Max: 1) Connection settings of the on-premise ClickHouse server. (see below for nested schema)
• password (Block List, Max: 1) Password for the database access. (see below for nested schema)
• user (String) User for database access.

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise

Optional:

• http_port (Number) TCP port number for the HTTP interface of the ClickHouse server.
• native_port (Number) TCP port number for the native interface of the ClickHouse server.
• shards (Block List) The list of ClickHouse shards. (see below for nested schema)
• tls_mode (Block List, Max: 1) TLS settings for the server connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.shards

Optional:

• hosts (List of String) List of ClickHouse server host names.
• name (String) Arbitrary shard name. This name may be used in sharding block to specify custom sharding rules.

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.password

Optional:

• raw (String, Sensitive) Password for the database access.

Nested Schema for Nested Schema for settings.clickhouse_target.sharding

Optional:

• column_value_hash (Block List, Max: 1) Shard data by the hash value of the specified column. (see below for nested schema)
• custom_mapping (Block List, Max: 1) A custom shard mapping by the value of the specified column. (see below for nested schema)
• round_robin (Block List, Max: 1) Distribute incoming rows between ClickHouse shards in a round-robin manner. Specify as an empty block to enable. (see below for nested schema)
• transfer_id (Block List, Max: 1) Shard data by ID of the transfer. (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.column_value_hash

Optional:

• column_name (String) The name of the column to calculate hash from.

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.custom_mapping

Optional:

• column_name (String) The name of the column to inspect when deciding the shard to chose for an incoming row.
• mapping (Block List) The mapping of the specified column values to the shard names. (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.custom_mapping.mapping

Optional:

• column_value (Block List, Max: 1) The value of the column. Currently only the string columns are supported. (see below for nested schema)
• shard_name (String) The name of the shard into which all the rows with the specified column_value will be written.

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.custom_mapping.mapping.column_value

Optional:

• string_value (String) The string value of the column.

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.round_robin

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.transfer_id

Nested Schema for Nested Schema for settings.kafka_source

Optional:

• auth (Block List, Max: 1) Authentication data. (see below for nested schema)
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• parser (Block List, Max: 1) Data parsing parameters. If not set, the source messages are read in raw. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• topic_name (String) Deprecated. Please use topic_names instead.
• topic_names (List of String) The list of full source topic names.
• transformer (Block List, Max: 1) Transform data with a custom Cloud Function. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.auth

Optional:

• no_auth (Block List, Max: 1) (see below for nested schema)
• sasl (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.auth.no_auth

Nested Schema for Nested Schema for settings.kafka_source.auth.sasl

Optional:

• mechanism (String)
• password (Block List, Max: 1) (see below for nested schema)
• user (String)

Nested Schema for Nested Schema for settings.kafka_source.auth.sasl.password

Optional:

• raw (String, Sensitive)

Nested Schema for Nested Schema for settings.kafka_source.connection

Optional:

• cluster_id (String)
• on_premise (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise

Optional:

• broker_urls (List of String)
• subnet_id (String)
• tls_mode (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) Empty block designating that the connection is not secured, i.e. plaintext connection. (see below for nested schema)
• enabled (Block List, Max: 1) If this attribute is not an empty block, then TLS is used for the server connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) X.509 certificate of the certificate authority which issued the server's certificate, in PEM format. If empty, the server's certificate must be signed by a well-known CA.

Nested Schema for Nested Schema for settings.kafka_source.parser

Optional:

• audit_trails_v1_parser (Block List, Max: 1) Parse Audit Trails data. Empty struct. (see below for nested schema)
• cloud_logging_parser (Block List, Max: 1) Parse Cloud Logging data. Empty struct. (see below for nested schema)
• json_parser (Block List, Max: 1) Parse data in JSON format. (see below for nested schema)
• tskv_parser (Block List, Max: 1) Parse data if TSKV format. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.parser.audit_trails_v1_parser

Nested Schema for Nested Schema for settings.kafka_source.parser.cloud_logging_parser

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser

Optional:

• add_rest_column (Boolean) Add fields, that are not in the schema, into the _rest column.
• data_schema (Block List, Max: 1) Data parsing scheme. (see below for nested schema)
• null_keys_allowed (Boolean) Allow null keys. If false - null keys will be putted to unparsed data.
• unescape_string_values (Boolean) Allow unescape string values.

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser.data_schema

Optional:

• fields (Block List, Max: 1) (see below for nested schema)
• json_fields (String) Description of the data schema as JSON specification.

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser.data_schema.fields

Optional:

• fields (Block List) Description of the data schema in the array of fields structure. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser.data_schema.fields.fields

Optional:

• key (Boolean) Mark field as Primary Key.
• name (String) Field name.
• path (String) Path to the field.
• required (Boolean) Mark field as required.
• type (String) Field type, one of: INT64, INT32, INT16, INT8, UINT64, UINT32, UINT16, UINT8, DOUBLE, BOOLEAN, STRING, UTF8, ANY, DATETIME.

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser

Optional:

• add_rest_column (Boolean) Add fields, that are not in the schema, into the _rest column.
• data_schema (Block List, Max: 1) Data parsing scheme. (see below for nested schema)
• null_keys_allowed (Boolean) Allow null keys. If false - null keys will be putted to unparsed data.
• unescape_string_values (Boolean) Allow unescape string values.

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser.data_schema

Optional:

• fields (Block List, Max: 1) Description of the data schema in the array of fields structure. (see below for nested schema)
• json_fields (String) Description of the data schema as JSON specification.

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser.data_schema.fields

Optional:

• fields (Block List) (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser.data_schema.fields.fields

Optional:

• key (Boolean) Mark field as Primary Key.
• name (String) Field name.
• path (String) Path to the field.
• required (Boolean) Mark field as required.
• type (String) Field type, one of: INT64, INT32, INT16, INT8, UINT64, UINT32, UINT16, UINT8, DOUBLE, BOOLEAN, STRING, UTF8, ANY, DATETIME.

Nested Schema for Nested Schema for settings.kafka_source.transformer

Optional:

• buffer_flush_interval (String)
• buffer_size (String)
• cloud_function (String)
• invocation_timeout (String)
• number_of_retries (Number)
• service_account_id (String)

Nested Schema for Nested Schema for settings.kafka_target

Optional:

• auth (Block List, Max: 1) Authentication data. (see below for nested schema)
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• serializer (Block List, Max: 1) Data serialization settings. (see below for nested schema)
• topic_settings (Block List, Max: 1) Target topic settings. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.auth

Optional:

• no_auth (Block List, Max: 1) Connection without authentication data. (see below for nested schema)
• sasl (Block List, Max: 1) Authentication using sasl. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.auth.no_auth

Nested Schema for Nested Schema for settings.kafka_target.auth.sasl

Optional:

• mechanism (String)
• password (Block List, Max: 1) (see below for nested schema)
• user (String)

Nested Schema for Nested Schema for settings.kafka_target.auth.sasl.password

Optional:

• raw (String, Sensitive)

Nested Schema for Nested Schema for settings.kafka_target.connection

Optional:

• cluster_id (String) Identifier of the Managed Kafka cluster.
• on_premise (Block List, Max: 1) Connection settings of the on-premise Kafka server. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise

Optional:

• broker_urls (List of String) List of Kafka broker URLs.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.
• tls_mode (Block List, Max: 1) TLS settings for the server connection. Empty implies plaintext connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.kafka_target.serializer

Optional:

• serializer_auto (Block List, Max: 1) Empty block. Select data serialization format automatically. (see below for nested schema)
• serializer_debezium (Block List, Max: 1) Serialize data in json format. (see below for nested schema)
• serializer_json (Block List, Max: 1) Empty block. Serialize data in json format. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_auto

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_debezium

Optional:

• serializer_parameters (Block List) A list of Debezium parameters set by the structure of the key and value string fields. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_debezium.serializer_parameters

Optional:

• key (String)
• value (String)

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_json

Nested Schema for Nested Schema for settings.kafka_target.topic_settings

Optional:

• topic (Block List, Max: 1) All messages will be sent to one topic. (see below for nested schema)
• topic_prefix (String) Topic name prefix. Messages will be sent to topic with name <topic_prefix>..<table_name>.

Nested Schema for Nested Schema for settings.kafka_target.topic_settings.topic

Optional:

• save_tx_order (Boolean) Not to split events queue into separate per-table queues.
• topic_name (String) Full topic name.

Nested Schema for Nested Schema for settings.metrika_source

Optional:

• counter_ids (List of Number)
• streams (Block List) (see below for nested schema)
• token (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.metrika_source.streams

Optional:

• columns (List of String)
• type (String)

Nested Schema for Nested Schema for settings.metrika_source.token

Optional:

• raw (String, Sensitive)

Nested Schema for Nested Schema for settings.mongo_source

Optional:

• collections (Block List) The list of the MongoDB collections that should be transferred. If omitted, all available collections will be transferred. (see below for nested schema)
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• excluded_collections (Block List) The list of the MongoDB collections that should not be transferred. (see below for nested schema)
• secondary_preferred_mode (Boolean) Whether the secondary server should be preferred to the primary when copying data.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.

Nested Schema for Nested Schema for settings.mongo_source.collections

Optional:

• collection_name (String)
• database_name (String)

Nested Schema for Nested Schema for settings.mongo_source.connection

Optional:

• connection_options (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options

Optional:

• auth_source (String) Name of the database associated with the credentials.
• mdb_cluster_id (String) Identifier of the Managed MongoDB cluster.
• on_premise (Block List, Max: 1) Connection settings of the on-premise MongoDB server. (see below for nested schema)
• password (Block List, Max: 1) (see below for nested schema)
• user (String)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise

Optional:

• hosts (List of String) Host names of the replica set.
• port (Number) TCP Port number.
• replica_set (String) Replica set name.
• tls_mode (Block List, Max: 1) TLS settings for the server connection. Empty implies plaintext connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.password

Optional:

• raw (String, Sensitive)

Nested Schema for Nested Schema for settings.mongo_source.excluded_collections

Optional:

• collection_name (String)
• database_name (String)

Nested Schema for Nested Schema for settings.mongo_target

Optional:

• cleanup_policy (String) How to clean collections when activating the transfer. One of DISABLED, DROP or TRUNCATE.
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• database (String) If not empty, then all the data will be written to the database with the specified name; otherwise the database name is the same as in the source endpoint.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.

Nested Schema for Nested Schema for settings.mongo_target.connection

Optional:

• connection_options (Block List, Max: 1) Connection options. (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options

Optional:

• auth_source (String) Name of the database associated with the credentials.
• mdb_cluster_id (String) Identifier of the Managed MongoDB cluster.
• on_premise (Block List, Max: 1) Connection settings of the on-premise MongoDB server. (see below for nested schema)
• password (Block List, Max: 1) Password for the database access. (see below for nested schema)
• user (String) User for database access.

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise

Optional:

• hosts (List of String) Host names of the replica set.
• port (Number) TCP Port number.
• replica_set (String) Replica set name.
• tls_mode (Block List, Max: 1) TLS settings for the server connection. Empty implies plaintext connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.password

Optional:

• raw (String, Sensitive) Password for the database access.

Nested Schema for Nested Schema for settings.mysql_source

Optional:

• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• database (String) Name of the database to transfer.
• exclude_tables_regex (List of String) Opposite of include_table_regex. The tables matching the specified regular expressions will not be transferred.
• include_tables_regex (List of String) List of regular expressions of table names which should be transferred. A table name is formatted as schemaname.tablename. For example, a single regular expression may look like ^mydb.employees$.
• object_transfer_settings (Block List, Max: 1) Defines which database schema objects should be transferred, e.g. views, routines, etc. All of the attrubutes in the block are optional and should be either BEFORE_DATA, AFTER_DATA or NEVER. (see below for nested schema)
• password (Block List, Max: 1) Password for the database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• service_database (String)
• timezone (String) Timezone to use for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.
• user (String) User for the database access.

Nested Schema for Nested Schema for settings.mysql_source.connection

Optional:

• mdb_cluster_id (String) Identifier of the Managed MySQL cluster.
• on_premise (Block List, Max: 1) Connection settings of the on-premise MySQL server. (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise

Optional:

• hosts (List of String) List of host names of the MySQL server. Exactly one host is expected currently.
• port (Number) Port for the database connection.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.
• tls_mode (Block List, Max: 1) TLS settings for the server connection. Empty implies plaintext connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.mysql_source.object_transfer_settings

Optional:

• routine (String)
• tables (String)
• trigger (String)
• view (String)

Nested Schema for Nested Schema for settings.mysql_source.password

Optional:

• raw (String, Sensitive) Password for the database access.

Nested Schema for Nested Schema for settings.mysql_target

Optional:

• cleanup_policy (String) How to clean tables when activating the transfer. One of DISABLED, DROP or TRUNCATE.
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• database (String) Name of the database to transfer.
• password (Block List, Max: 1) Password for the database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• service_database (String) The name of the database where technical tables (__tm_keeper, __tm_gtid_keeper) will be created. Default is the value of the attribute database.
• skip_constraint_checks (Boolean) When true, disables foreign key checks. See foreign_key_checks. False by default.
• sql_mode (String) sql_mode to use when interacting with the server. Defaults to NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION.
• timezone (String) Timezone to use for parsing timestamps for saving source timezones. Accepts values from IANA timezone database. Default: local timezone.
• user (String) User for the database access.

Nested Schema for Nested Schema for settings.mysql_target.connection

Optional:

• mdb_cluster_id (String) Identifier of the Managed MySQL cluster.
• on_premise (Block List, Max: 1) Connection settings of the on-premise MySQL server. (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise

Optional:

• hosts (List of String) List of host names of the MySQL server. Exactly one host is expected currently.
• port (Number) Port for the database connection.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.
• tls_mode (Block List, Max: 1) TLS settings for the server connection. Empty implies plaintext connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.mysql_target.password

Optional:

• raw (String, Sensitive) Password for the database access.

Nested Schema for Nested Schema for settings.postgres_source

Optional:

• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• database (String) Name of the database to transfer.
• exclude_tables (List of String) List of tables which will not be transfered, formatted as schemaname.tablename.
• include_tables (List of String) List of tables to transfer, formatted as schemaname.tablename. If omitted or an empty list is specified, all tables will be transferred.
• object_transfer_settings (Block List, Max: 1) Defines which database schema objects should be transferred, e.g. views, functions, etc. All of the attributes in this block are optional and should be either BEFORE_DATA, AFTER_DATA or NEVER. (see below for nested schema)
• password (Block List, Max: 1) Password for the database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• service_schema (String) Name of the database schema in which auxiliary tables needed for the transfer will be created. Empty service_schema implies schema public.
• slot_gigabyte_lag_limit (Number) Maximum WAL size held by the replication slot, in gigabytes. Exceeding this limit will result in a replication failure and deletion of the replication slot. Unlimited by default.
• user (String) User for the database access.

Nested Schema for Nested Schema for settings.postgres_source.connection

Optional:

• mdb_cluster_id (String)
• on_premise (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise

Optional:

• hosts (List of String)
• port (Number)
• subnet_id (String)
• tls_mode (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.postgres_source.object_transfer_settings

Optional:

• cast (String)
• collation (String)
• constraint (String)
• default_values (String)
• fk_constraint (String)
• function (String)
• index (String)
• materialized_view (String)
• policy (String)
• primary_key (String)
• rule (String)
• sequence (String)
• sequence_owned_by (String)
• sequence_set (String)
• table (String)
• trigger (String)
• type (String)
• view (String)

Nested Schema for Nested Schema for settings.postgres_source.password

Optional:

• raw (String, Sensitive) Password for the database access.

Nested Schema for Nested Schema for settings.postgres_target

Optional:

• cleanup_policy (String)
• connection (Block List, Max: 1) Connection settings. (see below for nested schema)
• database (String) Name of the database to transfer.
• password (Block List, Max: 1) Password for the database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• user (String) User for the database access.

Nested Schema for Nested Schema for settings.postgres_target.connection

Optional:

• mdb_cluster_id (String) Identifier of the Managed PostgreSQL cluster.
• on_premise (Block List, Max: 1) Connection settings of the on-premise PostgreSQL server. (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise

Optional:

• hosts (List of String) List of host names of the PostgreSQL server. Exactly one host is expected currently.
• port (Number) Port for the database connection.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.
• tls_mode (Block List, Max: 1) TLS settings for the server connection. Empty implies plaintext connection. (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise.tls_mode

Optional:

• disabled (Block List, Max: 1) (see below for nested schema)
• enabled (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String)

Nested Schema for Nested Schema for settings.postgres_target.password

Optional:

• raw (String, Sensitive) Password for the database access.

Nested Schema for Nested Schema for settings.ydb_source

Optional:

• changefeed_custom_name (String) Custom name for changefeed.
• database (String) Database path in YDB where tables are stored. Example: /ru/transfer_manager/prod/data-transfer-yt.
• instance (String) Instance of YDB. Example: my-cute-ydb.yandex.cloud:2135.
• paths (List of String) A list of paths which should be uploaded. When not specified, all available tables are uploaded.
• sa_key_content (String, Sensitive) Authentication key.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• service_account_id (String) Service account ID for interaction with database.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.

Nested Schema for Nested Schema for settings.ydb_target

Optional:

• cleanup_policy (String) How to clean collections when activating the transfer. One of YDB_CLEANUP_POLICY_DISABLED or YDB_CLEANUP_POLICY_DROP.
• database (String) Database path in YDB where tables are stored. Example: /ru/transfer_manager/prod/data-transfer-yt.
• default_compression (String) Compression that will be used for default columns family on YDB table creation One of YDB_DEFAULT_COMPRESSION_UNSPECIFIED, YDB_DEFAULT_COMPRESSION_DISABLED, YDB_DEFAULT_COMPRESSION_LZ4.
• instance (String) Instance of YDB. Example: my-cute-ydb.yandex.cloud:2135.
• is_table_column_oriented (Boolean) Whether a column-oriented (i.e. OLAP) tables should be created. Default is false (create row-oriented OLTP tables).
• path (String) A path where resulting tables are stored.
• sa_key_content (String, Sensitive) Authentication key.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• service_account_id (String) Service account ID for interaction with database.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.

Nested Schema for Nested Schema for settings.yds_source

Optional:

• allow_ttl_rewind (Boolean) Should continue working, if consumer read lag exceed TTL of topic.
• consumer (String) Consumer.
• database (String) Database name.
• endpoint (String) YDS Endpoint.
• parser (Block List, Max: 1) Data parsing rules. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• service_account_id (String) Service account ID for interaction with database.
• stream (String) Stream.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.
• supported_codecs (List of String) List of supported compression codec.

Nested Schema for Nested Schema for settings.yds_source.parser

Optional:

• audit_trails_v1_parser (Block List, Max: 1) Parse Audit Trails data. Empty struct. (see below for nested schema)
• cloud_logging_parser (Block List, Max: 1) Parse Cloud Logging data. Empty struct. (see below for nested schema)
• json_parser (Block List, Max: 1) Parse data in json format. (see below for nested schema)
• tskv_parser (Block List, Max: 1) (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_source.parser.audit_trails_v1_parser

Nested Schema for Nested Schema for settings.yds_source.parser.cloud_logging_parser

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser

Optional:

• add_rest_column (Boolean)
• data_schema (Block List, Max: 1) Data parsing scheme. (see below for nested schema)
• null_keys_allowed (Boolean)
• unescape_string_values (Boolean)

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser.data_schema

Optional:

• fields (Block List, Max: 1) Description of the data schema in the array of fields structure. (see below for nested schema)
• json_fields (String) Description of the data schema as JSON specification.

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser.data_schema.fields

Optional:

• fields (Block List) Description of the data schema in the array of fields structure. (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser.data_schema.fields.fields

Optional:

• key (Boolean) Mark field as Primary Key.
• name (String) Field name.
• path (String) Path to the field.
• required (Boolean) Mark field as required.
• type (String) Field type, one of: INT64, INT32, INT16, INT8, UINT64, UINT32, UINT16, UINT8, DOUBLE, BOOLEAN, STRING, UTF8, ANY, DATETIME.

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser

Optional:

• add_rest_column (Boolean)
• data_schema (Block List, Max: 1) (see below for nested schema)
• null_keys_allowed (Boolean)
• unescape_string_values (Boolean)

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser.data_schema

Optional:

• fields (Block List, Max: 1) (see below for nested schema)
• json_fields (String)

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser.data_schema.fields

Optional:

• fields (Block List) (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser.data_schema.fields.fields

Optional:

• key (Boolean)
• name (String)
• path (String)
• required (Boolean)
• type (String)

Nested Schema for Nested Schema for settings.yds_target

Optional:

• database (String) Database.
• endpoint (String) YDS Endpoint.
• save_tx_order (Boolean) Save transaction order.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should use.
• serializer (Block List, Max: 1) Data serialization format. (see below for nested schema)
• service_account_id (String) Service account ID for interaction with database.
• stream (String) Stream.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the database. If omitted, the server has to be accessible via Internet.

Nested Schema for Nested Schema for settings.yds_target.serializer

Optional:

• serializer_auto (Block List, Max: 1) Empty block. Select data serialization format automatically. (see below for nested schema)
• serializer_debezium (Block List, Max: 1) Serialize data in json format. (see below for nested schema)
• serializer_json (Block List, Max: 1) Empty block. Serialize data in json format. (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_auto

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_debezium

Optional:

• serializer_parameters (Block List) A list of Debezium parameters set by the structure of the key and value string fields. (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_debezium.serializer_parameters

Optional:

• key (String)
• value (String)

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_json

ImportImport

The resource can be imported by using their resource ID. For getting the resource ID you can use Yandex Cloud Web Console or YC CLI.

# terraform import yandex_datatransfer_endpoint.<resource Name> <resource Id>
terraform import yandex_datatransfer_endpoint.my_dt_endpoint dtec2**********a38dp