yandex_datatransfer_endpoint (Resource)

Data Transfer endpoint. For more information, see the official
documentation

Example usageExample usage

//
// Create a new pair of Data Transfer Endpoints - Source & Target.
//
resource "yandex_datatransfer_endpoint" "pg_source" {
  name = "pg-test-source"
  settings {
    postgres_source {
      connection {
        on_premise {
          hosts = [
            "example.org"
          ]
          port = 5432
        }
      }
      slot_gigabyte_lag_limit = 100
      database                = "db1"
      user                    = "user1"
      password {
        raw = "123"
      }
    }
  }
}

resource "yandex_datatransfer_endpoint" "pg_target" {
  folder_id = "some_folder_id"
  name      = "pg-test-target2"
  settings {
    postgres_target {
      connection {
        mdb_cluster_id = "some_cluster_id"
      }
      security_groups = [/* list of security group ids */]
      database        = "db2"
      user            = "user2"
      password {
        raw = "321"
      }
    }
  }
}

SchemaSchema

OptionalOptional

• description (String) Description of the endpoint.
• endpoint_id (String) Identifier of the endpoint to return.

To get the endpoint ID, make an [EndpointService.List] request.

• folder_id (String) ID of the folder to create the endpoint in.

To get the folder ID, make a
[yandex.cloud.resourcemanager.v1.FolderService.List] request.

• id (String) Identifier of the endpoint to return.

To get the endpoint ID, make an [EndpointService.List] request.

• labels (Map of String) Endpoint labels as key:value pairs.

For details about the concept, see [documentation]( api-url-prefix
/resource-manager/concepts/labels).

• name (String) Name of the endpoint.

The name must be unique within the folder.

• settings (Block List) DataTransfer Endpoint Settings block (see below for nested schema)
• timeouts (Attributes) (see below for nested schema)

Nested Schema for Nested Schema for settings

Optional:

• clickhouse_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• clickhouse_target (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• kafka_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• kafka_target (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• metrika_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• mongo_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• mongo_target (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• mysql_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• mysql_target (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• postgres_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• postgres_target (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• ydb_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• ydb_target (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• yds_source (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)
• yds_target (Block List) package: yandex.cloud.datatransfer.v1
  filename: yandex/cloud/datatransfer/v1/endpoint.proto (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source

Optional:

• clickhouse_cluster_name (String) Name of the ClickHouse cluster. For Managed ClickHouse that is name of
  ShardGroup or managed cluster ID by default
• connection (Block List) Connection settings (see below for nested schema)
• exclude_tables (List of String) Exclude list of tables for replication. If none or empty list is presented -
  will replicate all tables. Can contain * patterns.
• include_tables (List of String) White list of tables for replication. If none or empty list is presented - will
  replicate all tables. Can contain * patterns.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.clickhouse_source.connection

Optional:

• connection_options (Block List) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options

Optional:

• connection_manager_connection (Block List) Get ClickHouse installation params and credentials from Connection Manager (see below for nested schema)
• database (String) Database name
• mdb_cluster_id (String) Identifier of the Managed ClickHouse cluster
• on_premise (Block List) Connection settings of the on-premise ClickHouse server (see below for nested schema)
• password (Block List) Password for the database access (see below for nested schema)
• user (String) User for database access. Required unless connection_manager_connection is used

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise

Optional:

• http_port (Number) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto
• native_port (Number) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto
• shards (Block List) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto (see below for nested schema)
• tls_mode (Block List) TLS settings for server connection. Disabled by default (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.shards

Optional:

• hosts (List of String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto
• name (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.clickhouse_source.connection.connection_options.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.clickhouse_target

Optional:

• alt_names (Block List) Table renaming rules in target (see below for nested schema)
• cleanup_policy (String) How to clean collections when activating the transfer. One of
  CLICKHOUSE_CLEANUP_POLICY_DISABLED or CLICKHOUSE_CLEANUP_POLICY_DROP
• clickhouse_cluster_name (String) Name of the ClickHouse cluster. For Managed ClickHouse that is name of
  ShardGroup or managed cluster ID by default.
• connection (Block List) Connection settings (see below for nested schema)
• is_schema_migration_disabled (Boolean) Whether can change table schema if schema changed on source
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• sharding (Block List) Shard selection rules for the data being transferred (see below for nested schema)
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.clickhouse_target.alt_names

Optional:

• from_name (String) Source table name
• to_name (String) Target table name

Nested Schema for Nested Schema for settings.clickhouse_target.connection

Optional:

• connection_options (Block List) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options

Optional:

• connection_manager_connection (Block List) Get ClickHouse installation params and credentials from Connection Manager (see below for nested schema)
• database (String) Database name
• mdb_cluster_id (String) Identifier of the Managed ClickHouse cluster
• on_premise (Block List) Connection settings of the on-premise ClickHouse server (see below for nested schema)
• password (Block List) Password for the database access (see below for nested schema)
• user (String) User for database access. Required unless connection_manager_connection is used

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise

Optional:

• http_port (Number) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto
• native_port (Number) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto
• shards (Block List) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto (see below for nested schema)
• tls_mode (Block List) TLS settings for server connection. Disabled by default (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.shards

Optional:

• hosts (List of String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto
• name (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.clickhouse_target.connection.connection_options.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.clickhouse_target.sharding

Optional:

• column_value_hash (Block List) Shard data by the hash value of the specified column (see below for nested schema)
• custom_mapping (Block List) A custom shard mapping by the value of the specified column (see below for nested schema)
• round_robin (Block List) Distribute incoming rows between ClickHouse shards in a round-robin manner.
  Specify as an empty block to enable (see below for nested schema)
• transfer_id (Block List) Shard data by ID of the transfer (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.column_value_hash

Optional:

• column_name (String) The name of the column to calculate hash from

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.custom_mapping

Optional:

• column_name (String) The name of the column to inspect when deciding the shard to chose for an
  incoming row
• mapping (Block List) The mapping of the specified column values to the shard names (see below for nested schema)

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.custom_mapping.mapping

Optional:

• column_value (Block List) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto (see below for nested schema)
• shard_name (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/clickhouse.proto

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.custom_mapping.mapping.column_value

Optional:

• string_value (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/common.proto

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.round_robin

Nested Schema for Nested Schema for settings.clickhouse_target.sharding.transfer_id

Nested Schema for Nested Schema for settings.kafka_source

Optional:

• auth (Block List) Authentication settings (see below for nested schema)
• connection (Block List) Connection settings (see below for nested schema)
• parser (Block List) Data parsing parameters. If not set, the source messages are read in raw (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• topic_name (String, Deprecated) Deprecated. Please use topic_names instead
  Full source topic name
• topic_names (List of String) List of full source topic names to read
• transformer (Block List) Transform data with a custom Cloud Function (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.auth

Optional:

• no_auth (Block List) No authentication (see below for nested schema)
• sasl (Block List) Authentication with SASL (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.auth.no_auth

Nested Schema for Nested Schema for settings.kafka_source.auth.sasl

Optional:

• mechanism (String) SASL mechanism for authentication, use one of: KAFKA_MECHANISM_SHA256,
  KAFKA_MECHANISM_SHA512
• password (Block List) Password for user (see below for nested schema)
• user (String) User name

Nested Schema for Nested Schema for settings.kafka_source.auth.sasl.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.kafka_source.connection

Optional:

• cluster_id (String) Managed Service for Kafka cluster ID.
  Set only one of: cluster_id/on_premise/connection_manager_connection
• connection_manager_connection (Block List) Get Kafka installation params and credentials from Connection Manager
  Set only one of: cluster_id/on_premise/connection_manager_connection (see below for nested schema)
• on_premise (Block List) Connection options for on-premise Kafka
  Set only one of: cluster_id/on_premise/connection_manager_connection (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.connection.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise

Optional:

• broker_urls (List of String) Kafka broker URLs
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet
• tls_mode (Block List) TLS settings for broker connection. Disabled by default. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.kafka_source.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.kafka_source.parser

Optional:

• audit_trails_v1_parser (Block List) Parse Audit Trails data. Empty struct (see below for nested schema)
• cloud_logging_parser (Block List) Parse Cloud Logging data. Empty struct (see below for nested schema)
• json_parser (Block List) Parse data in json format (see below for nested schema)
• tskv_parser (Block List) Parse data in tskv format (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.parser.audit_trails_v1_parser

Nested Schema for Nested Schema for settings.kafka_source.parser.cloud_logging_parser

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser

Optional:

• add_rest_column (Boolean) Will add _rest column for all unknown fields
• data_schema (Block List) Data parsing scheme (see below for nested schema)
• null_keys_allowed (Boolean) Allow null keys, if no - null keys will be putted to unparsed data
• unescape_string_values (Boolean) Unescape string values

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser.data_schema

Optional:

• fields (Block List) Description of the data schema in the array of fields structure (see below for nested schema)
• json_fields (String, Deprecated) Description of the data schema as JSON specification

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser.data_schema.fields

Optional:

• fields (Block List) Description of the column schema in the array of fields structure (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.parser.json_parser.data_schema.fields.fields

Optional:

• key (Boolean) Mark field as Primary Key
• name (String) Field name
• path (String) Path to the field
• required (Boolean) Mark field as required
• type (String) Field type, one of: INT64, INT32, INT16, INT8, UINT64, UINT32,
  UINT16, UINT8, DOUBLE, BOOLEAN, STRING, UTF8, ANY, DATETIME.

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser

Optional:

• add_rest_column (Boolean) Will add _rest column for all unknown fields
• data_schema (Block List) Data parsing scheme (see below for nested schema)
• null_keys_allowed (Boolean) Allow null keys, if no - null keys will be putted to unparsed data
• unescape_string_values (Boolean) Unescape string values

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser.data_schema

Optional:

• fields (Block List) Description of the data schema in the array of fields structure (see below for nested schema)
• json_fields (String, Deprecated) Description of the data schema as JSON specification

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser.data_schema.fields

Optional:

• fields (Block List) Description of the column schema in the array of fields structure (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_source.parser.tskv_parser.data_schema.fields.fields

Optional:

• key (Boolean) Mark field as Primary Key
• name (String) Field name
• path (String) Path to the field
• required (Boolean) Mark field as required
• type (String) Field type, one of: INT64, INT32, INT16, INT8, UINT64, UINT32,
  UINT16, UINT8, DOUBLE, BOOLEAN, STRING, UTF8, ANY, DATETIME.

Nested Schema for Nested Schema for settings.kafka_source.transformer

Optional:

• buffer_flush_interval (String) Flush interval
• buffer_size (String) Buffer size for function. Maximum 4 GB. Use value with units, i.e. 10 B, 20 kB,
  2.0 MB, 30 MB, 1.0 GB
• cloud_function (String) Cloud function
• invocation_timeout (String) Invocation timeout
• number_of_retries (Number) Number of retries
• service_account_id (String) Service account

Nested Schema for Nested Schema for settings.kafka_target

Optional:

• auth (Block List) Authentication settings (see below for nested schema)
• connection (Block List) Connection settings (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• serializer (Block List) Data serialization format settings (see below for nested schema)
• topic_settings (Block List) Target topic settings (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.auth

Optional:

• no_auth (Block List) No authentication (see below for nested schema)
• sasl (Block List) Authentication with SASL (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.auth.no_auth

Nested Schema for Nested Schema for settings.kafka_target.auth.sasl

Optional:

• mechanism (String) SASL mechanism for authentication, use one of: KAFKA_MECHANISM_SHA256,
  KAFKA_MECHANISM_SHA512
• password (Block List) Password for user (see below for nested schema)
• user (String) User name

Nested Schema for Nested Schema for settings.kafka_target.auth.sasl.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.kafka_target.connection

Optional:

• cluster_id (String) Managed Service for Kafka cluster ID.
  Set only one of: cluster_id/on_premise/connection_manager_connection
• connection_manager_connection (Block List) Get Kafka installation params and credentials from Connection Manager
  Set only one of: cluster_id/on_premise/connection_manager_connection (see below for nested schema)
• on_premise (Block List) Connection options for on-premise Kafka
  Set only one of: cluster_id/on_premise/connection_manager_connection (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.connection.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise

Optional:

• broker_urls (List of String) Kafka broker URLs
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet
• tls_mode (Block List) TLS settings for broker connection. Disabled by default. (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.kafka_target.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.kafka_target.serializer

Optional:

• serializer_auto (Block List) Empty block. Select the serialization format automatically (see below for nested schema)
• serializer_debezium (Block List) Serialize data in debezium json format (see below for nested schema)
• serializer_json (Block List) Empty block. Serialize data in json format (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_auto

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_debezium

Optional:

• serializer_parameters (Block List) A list of Debezium parameters set by the structure of the key and value
  string fields (see below for nested schema)

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_debezium.serializer_parameters

Optional:

• key (String) Name of the serializer parameter
• value (String) Value of the serializer parameter

Nested Schema for Nested Schema for settings.kafka_target.serializer.serializer_json

Nested Schema for Nested Schema for settings.kafka_target.topic_settings

Optional:

• topic (Block List) All messages will be sent to one topic (see below for nested schema)
• topic_prefix (String) Topic prefix
  Messages will be sent to topic with name <topic_prefix>..<table_name>.
  Analogue of the Debezium setting database.server.name.

Nested Schema for Nested Schema for settings.kafka_target.topic_settings.topic

Optional:

• save_tx_order (Boolean) Save transactions order
  Not to split events queue into separate per-table queues.
• topic_name (String) Full topic name

Nested Schema for Nested Schema for settings.metrika_source

Optional:

• counter_ids (List of Number) Counter IDs
• streams (Block List) Streams (see below for nested schema)
• token (Block List) Authentication token (see below for nested schema)

Nested Schema for Nested Schema for settings.metrika_source.streams

Optional:

• columns (List of String) Column names
• type (String) Stream type, one of: METRIKA_STREAM_TYPE_HITS, METRIKA_STREAM_TYPE_VISITS,
  METRIKA_STREAM_TYPE_HITS_V2

Nested Schema for Nested Schema for settings.metrika_source.token

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.mongo_source

Optional:

• collections (Block List) List of collections for replication. Empty list implies replication of all
  tables on the deployment. Allowed to use * as collection name. (see below for nested schema)
• connection (Block List) Connection settings (see below for nested schema)
• excluded_collections (Block List) List of forbidden collections for replication. Allowed to use * as collection
  name for forbid all collections of concrete schema. (see below for nested schema)
• secondary_preferred_mode (Boolean) Read mode for mongo client: whether the secondary server should be preferred to
  the primary when copying data
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.mongo_source.collections

Optional:

• collection_name (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/mongo.proto
• database_name (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/mongo.proto

Nested Schema for Nested Schema for settings.mongo_source.connection

Optional:

• connection_options (Block List) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/mongo.proto (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options

Optional:

• auth_source (String) Database name associated with the credentials
• connection_manager_connection (Block List) Get StoreDoc/MongoDB installation params and credentials from Connection Manager
  Use one of: mdb_cluster_id/on_premise/connection_manager_connection (see below for nested schema)
• mdb_cluster_id (String) Identifier of the Yandex StoreDoc cluster
  Use one of: mdb_cluster_id/on_premise/connection_manager_connection
• on_premise (Block List) Connection settings of the on-premise MongoDB server
  Use one of: mdb_cluster_id/on_premise/connection_manager_connection (see below for nested schema)
• password (Block List) Password for user (see below for nested schema)
• user (String) User name, required unless connection_manager_connection is used

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.connection_manager_connection

Optional:

• connection_id (String) ID of connectionmanager connection with mongodb/Yandex Storedoc installation
  parameters and credentials
• replica_set (String) Replica set name, used only for on-premise mongodb installations

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise

Optional:

• hosts (List of String) Host names of the replica set
• port (Number) TCP Port number
• replica_set (String) Replica set name
• tls_mode (Block List) TLS settings for the server connection. Empty implies plaintext connection (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.mongo_source.connection.connection_options.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.mongo_source.excluded_collections

Optional:

• collection_name (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/mongo.proto
• database_name (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/mongo.proto

Nested Schema for Nested Schema for settings.mongo_target

Optional:

• cleanup_policy (String) How to clean collections when activating the transfer. One of DISABLED, DROP
  or TRUNCATE
• connection (Block List) Connection settings (see below for nested schema)
• database (String) Database name. If not empty, then all the data will be written to the database
  with the specified name; otherwise the database name is the same as in the
  source endpoint
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.mongo_target.connection

Optional:

• connection_options (Block List) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/mongo.proto (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options

Optional:

• auth_source (String) Database name associated with the credentials
• connection_manager_connection (Block List) Get StoreDoc/MongoDB installation params and credentials from Connection Manager
  Use one of: mdb_cluster_id/on_premise/connection_manager_connection (see below for nested schema)
• mdb_cluster_id (String) Identifier of the Yandex StoreDoc cluster
  Use one of: mdb_cluster_id/on_premise/connection_manager_connection
• on_premise (Block List) Connection settings of the on-premise MongoDB server
  Use one of: mdb_cluster_id/on_premise/connection_manager_connection (see below for nested schema)
• password (Block List) Password for user (see below for nested schema)
• user (String) User name, required unless connection_manager_connection is used

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.connection_manager_connection

Optional:

• connection_id (String) ID of connectionmanager connection with mongodb/Yandex Storedoc installation
  parameters and credentials
• replica_set (String) Replica set name, used only for on-premise mongodb installations

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise

Optional:

• hosts (List of String) Host names of the replica set
• port (Number) TCP Port number
• replica_set (String) Replica set name
• tls_mode (Block List) TLS settings for the server connection. Empty implies plaintext connection (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.mongo_target.connection.connection_options.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.mysql_source

Optional:

• connection (Block List) Database connection settings (see below for nested schema)
• database (String) Name of the database to transfer

You can leave it empty, then it will be possible to transfer tables from several
databases at the same time from this source.

• exclude_tables_regex (List of String) Opposite of include_table_regex. The tables matching the specified regular
  expressions will not be transferred
• include_tables_regex (List of String) List of regular expressions of table names which should be transferred. A table
  name is formatted as schemaname.tablename. For example, a single regular
  expression may look like ^mydb.employees$
• object_transfer_settings (Block List) Schema migration
  Defines which database schema objects should be transferred, e.g. views,
  routines, etc.
  All of the attrubutes in the block are optional and should be either
  BEFORE_DATA, AFTER_DATA or NEVER." (see below for nested schema)
• password (Block List) Password for database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• service_database (String) Database for service tables
  Default: data source database. Here created technical tables (__tm_keeper,
  __tm_gtid_keeper).
• timezone (String) Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from
IANA timezone database. Default: local timezone.

• user (String) User for database access. Required unless connection manager connection is used

Nested Schema for Nested Schema for settings.mysql_source.connection

Optional:

• connection_manager_connection (Block List) Get Mysql installation params and credentials from Connection Manager (see below for nested schema)
• mdb_cluster_id (String) Managed Service for MySQL cluster ID
• on_premise (Block List) Connection options for on-premise MySQL (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_source.connection.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise

Optional:

• hosts (List of String) List of host names of the MySQL server. Exactly one host is expected
• port (Number) Port for the database connection
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet
• tls_mode (Block List) TLS settings for server connection. Disabled by default. (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mysql_source.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.mysql_source.object_transfer_settings

Optional:

• routine (String) Routines

CREATE PROCEDURE ... ; CREATE FUNCTION ... ;

• tables (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/mysql.proto
• trigger (String) Triggers

CREATE TRIGGER ...

• view (String) Views

CREATE VIEW ...

Nested Schema for Nested Schema for settings.mysql_source.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.mysql_target

Optional:

• cleanup_policy (String) Cleanup policy for activate, reactivate and reupload processes.
  One of DISABLED, DROP or TRUNCATE Default is DISABLED.
• connection (Block List) Database connection settings (see below for nested schema)
• database (String) Database name

Allowed to leave it empty, then the tables will be created in databases with the
same names as on the source. If this field is empty, then you must fill below db
schema for service table.

• is_schema_migration_disabled (Boolean) Whether can change table schema if schema changed on source
• password (Block List) Password for database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• service_database (String) Database schema for service table
  Default: db name. Here created technical tables (__tm_keeper, __tm_gtid_keeper).
• skip_constraint_checks (Boolean) Disable constraints checks
  When true, disables foreign key checks and unique checks. False by default.
  See
  foreign_key_checks.
  Recommend to disable for increase replication speed unless schema contains
  cascading operations
• sql_mode (String) sql_mode to use when
  interacting with the server.
  Defaults to NO_AUTO_VALUE_ON_ZERO,NO_DIR_IN_CREATE,NO_ENGINE_SUBSTITUTION
• timezone (String) Database timezone

Is used for parsing timestamps for saving source timezones. Accepts values from
IANA timezone database. Default: local timezone.

• user (String) User for database access. Required unless connection manager connection is used

Nested Schema for Nested Schema for settings.mysql_target.connection

Optional:

• connection_manager_connection (Block List) Get Mysql installation params and credentials from Connection Manager (see below for nested schema)
• mdb_cluster_id (String) Managed Service for MySQL cluster ID
• on_premise (Block List) Connection options for on-premise MySQL (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_target.connection.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise

Optional:

• hosts (List of String) List of host names of the MySQL server. Exactly one host is expected
• port (Number) Port for the database connection
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet
• tls_mode (Block List) TLS settings for server connection. Disabled by default. (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.mysql_target.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.mysql_target.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.postgres_source

Optional:

• connection (Block List) Database connection settings (see below for nested schema)
• database (String) Name of the database to transfer
• exclude_tables (List of String) List of tables which will not be transfered, formatted as schemaname.tablename
  If omitted or empty list is specified, all tables are replicated.
  Can contain schema_name.* patterns.
• include_tables (List of String) List of tables to transfer, formatted as schemaname.tablename.
  If omitted or an empty list is specified, all tables will be transferred.
  Can contain schema_name.* patterns.
• object_transfer_settings (Block List) Defines which database schema objects should be transferred, e.g. views,
  functions, etc.
  All of the attributes in this block are optional and should be either
  BEFORE_DATA, AFTER_DATA or NEVER (see below for nested schema)
• password (Block List) Password for database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• service_schema (String) Name of the database schema in which auxiliary tables needed for the transfer
  will be created (__consumer_keeper, __data_transfer_mole_finder).
  Empty service_schema implies schema public
• slot_gigabyte_lag_limit (Number) Maximum WAL size held by the replication slot (API - in bytes, terraform - in
  gigabytes);
  Exceeding this limit will result in a replication failure and deletion of the
  replication slot.
  Default is 50 gigabytes
• user (String) User for database access. Required unless Connection Manager connection is used.

Nested Schema for Nested Schema for settings.postgres_source.connection

Optional:

• connection_manager_connection (Block List) Get Postgres installation params and credentials from Connection Manager (see below for nested schema)
• mdb_cluster_id (String) Managed Service for PostgreSQL cluster ID
• on_premise (Block List) Connection options for on-premise PostgreSQL (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_source.connection.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise

Optional:

• hosts (List of String) PG installation hosts
• port (Number) PG port. Will be used if the cluster ID is not specified.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet
• tls_mode (Block List) TLS settings for server connection. Disabled by default. (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.postgres_source.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.postgres_source.object_transfer_settings

Optional:

• cast (String) Casts

CREATE CAST ...

• collation (String) Collations

CREATE COLLATION ...

• constraint (String) Constraints

ALTER TABLE ... ADD CONSTRAINT ...

• default_values (String) Default values

ALTER TABLE ... ALTER COLUMN ... SET DEFAULT ...

• fk_constraint (String) Foreign keys

ALTER TABLE ... ADD FOREIGN KEY ...

• function (String) Functions

CREATE FUNCTION ...

• index (String) Indexes

CREATE INDEX ...

• materialized_view (String) Materialized views

CREATE MATERIALIZED VIEW ...

• policy (String) Policies

CREATE POLICY ...

• primary_key (String) Primary keys

ALTER TABLE ... ADD PRIMARY KEY ...

• rule (String) Rules

CREATE RULE ...

• sequence (String) Sequences

CREATE SEQUENCE ...

• sequence_owned_by (String) Owned sequences

CREATE SEQUENCE ... OWNED BY ...

• sequence_set (String) package: yandex.cloud.datatransfer.v1.endpoint
  filename: yandex/cloud/datatransfer/v1/endpoint/postgres.proto
• table (String) Tables

CREATE TABLE ...

• trigger (String) Triggers

CREATE TRIGGER ...

• type (String) Types

CREATE TYPE ...

• view (String) Views

CREATE VIEW ...

Nested Schema for Nested Schema for settings.postgres_source.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.postgres_target

Optional:

• cleanup_policy (String) Cleanup policy for activate, reactivate and reupload processes.
  One of: DISABLED, DROP, TRUNCATE. Default is TRUNCATE
• connection (Block List) Database connection settings (see below for nested schema)
• database (String) Target database name
• is_schema_migration_disabled (Boolean) Whether can change table schema if schema changed on source
• password (Block List) Password for database access. (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• user (String) User for database access. Required unless Connection Manager connection is used

Nested Schema for Nested Schema for settings.postgres_target.connection

Optional:

• connection_manager_connection (Block List) Get Postgres installation params and credentials from Connection Manager (see below for nested schema)
• mdb_cluster_id (String) Managed Service for PostgreSQL cluster ID
• on_premise (Block List) Connection options for on-premise PostgreSQL (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_target.connection.connection_manager_connection

Optional:

• connection_id (String) ID of connection in Connection Manager with installation params and credetials
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise

Optional:

• hosts (List of String) PG installation hosts
• port (Number) PG port. Will be used if the cluster ID is not specified.
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet
• tls_mode (Block List) TLS settings for server connection. Disabled by default. (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise.tls_mode

Optional:

• disabled (Block List) Empty block designating that the connection is not secured, i.e. plaintext
  connection (see below for nested schema)
• enabled (Block List) TLS is used for the server connection (see below for nested schema)

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise.tls_mode.disabled

Nested Schema for Nested Schema for settings.postgres_target.connection.on_premise.tls_mode.enabled

Optional:

• ca_certificate (String) CA certificate

X.509 certificate of the certificate authority which issued the server's
certificate, in PEM format. When CA certificate is specified, TLS is used to
connect to the server. If CA certificate is empty, the server's certificate must
be signed by a well-known CA

Nested Schema for Nested Schema for settings.postgres_target.password

Optional:

• raw (String, Sensitive) Raw secret value

Nested Schema for Nested Schema for settings.ydb_source

Optional:

• changefeed_custom_consumer_name (String) Consumer for pre-created change feed if any
• changefeed_custom_name (String) Pre-created change feed if any
• database (String) Database path in YDB where tables are stored.
  Example: /ru/transfer_manager/prod/data-transfer-yt
• instance (String) Instance of YDB. example: ydb-ru-prestable.yandex.net:2135.
  If not specified, will be determined by database
• paths (List of String) A list of paths which should be uploaded. When not specified, all available
  tables are uploaded
• sa_key_content (String, Sensitive) Authorization Key
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• service_account_id (String) Service account ID for interaction with database
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database. If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.ydb_target

Optional:

• cleanup_policy (String) Cleanup policy determine how to clean collections when activating the transfer.
  One of YDB_CLEANUP_POLICY_DISABLED or YDB_CLEANUP_POLICY_DROP
• database (String) Database path in YDB where tables are stored.
  Example: /ru/transfer_manager/prod/data-transfer
• default_compression (String) Compression that will be used for default columns family on YDB table creation.
  One of YDB_DEFAULT_COMPRESSION_UNSPECIFIED,
  YDB_DEFAULT_COMPRESSION_DISABLED, YDB_DEFAULT_COMPRESSION_LZ4
• instance (String) Instance of YDB. example: ydb-ru-prestable.yandex.net:2135.
  If not specified, will be determined by database
• is_schema_migration_disabled (Boolean) Whether can change table schema if schema changed on source
• is_table_column_oriented (Boolean) Whether a column-oriented (i.e. OLAP) tables should be created.
  Default is false (create row-oriented OLTP tables)
• path (String) Path extension for database, each table will be layouted into this path
• sa_key_content (String, Sensitive) Authentication key
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• service_account_id (String) Service account ID for interaction with database
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database.
  If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.yds_source

Optional:

• allow_ttl_rewind (Boolean) Should continue working, if consumer read lag exceed TTL of topic
  False: stop the transfer in error state, if detected lost data. True: continue
  working with losing part of data
• consumer (String) Custom consumer - for important streams
• database (String) Database path in YDB for streams
  Example: /ru/transfer_manager/prod/data-transfer
• endpoint (String) YDS Endpoint for dedicated db
• parser (Block List) Data parsing rules (see below for nested schema)
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• service_account_id (String) Service account ID which has read access to the stream.
• stream (String) Stream to read
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database. If omitted, the server has to be accessible via Internet
• supported_codecs (List of String) List of supported compression codecs
  Options: YDS_COMPRESSION_CODEC_RAW, YDS_COMPRESSION_CODEC_ZSTD,
  YDS_COMPRESSION_CODEC_GZIP

Nested Schema for Nested Schema for settings.yds_source.parser

Optional:

• audit_trails_v1_parser (Block List) Parse Audit Trails data. Empty struct (see below for nested schema)
• cloud_logging_parser (Block List) Parse Cloud Logging data. Empty struct (see below for nested schema)
• json_parser (Block List) Parse data in json format (see below for nested schema)
• tskv_parser (Block List) Parse data in tskv format (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_source.parser.audit_trails_v1_parser

Nested Schema for Nested Schema for settings.yds_source.parser.cloud_logging_parser

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser

Optional:

• add_rest_column (Boolean) Will add _rest column for all unknown fields
• data_schema (Block List) Data parsing scheme (see below for nested schema)
• null_keys_allowed (Boolean) Allow null keys, if no - null keys will be putted to unparsed data
• unescape_string_values (Boolean) Unescape string values

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser.data_schema

Optional:

• fields (Block List) Description of the data schema in the array of fields structure (see below for nested schema)
• json_fields (String, Deprecated) Description of the data schema as JSON specification

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser.data_schema.fields

Optional:

• fields (Block List) Description of the column schema in the array of fields structure (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_source.parser.json_parser.data_schema.fields.fields

Optional:

• key (Boolean) Mark field as Primary Key
• name (String) Field name
• path (String) Path to the field
• required (Boolean) Mark field as required
• type (String) Field type, one of: INT64, INT32, INT16, INT8, UINT64, UINT32,
  UINT16, UINT8, DOUBLE, BOOLEAN, STRING, UTF8, ANY, DATETIME.

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser

Optional:

• add_rest_column (Boolean) Will add _rest column for all unknown fields
• data_schema (Block List) Data parsing scheme (see below for nested schema)
• null_keys_allowed (Boolean) Allow null keys, if no - null keys will be putted to unparsed data
• unescape_string_values (Boolean) Unescape string values

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser.data_schema

Optional:

• fields (Block List) Description of the data schema in the array of fields structure (see below for nested schema)
• json_fields (String, Deprecated) Description of the data schema as JSON specification

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser.data_schema.fields

Optional:

• fields (Block List) Description of the column schema in the array of fields structure (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_source.parser.tskv_parser.data_schema.fields.fields

Optional:

• key (Boolean) Mark field as Primary Key
• name (String) Field name
• path (String) Path to the field
• required (Boolean) Mark field as required
• type (String) Field type, one of: INT64, INT32, INT16, INT8, UINT64, UINT32,
  UINT16, UINT8, DOUBLE, BOOLEAN, STRING, UTF8, ANY, DATETIME.

Nested Schema for Nested Schema for settings.yds_target

Optional:

• compression_codec (String) Codec to use for output data compression. If not specified, no compression will
  be done
  Options: YDS_COMPRESSION_CODEC_RAW, YDS_COMPRESSION_CODEC_ZSTD,
  YDS_COMPRESSION_CODEC_GZIP
• database (String) Database path in YDB for streams
  Example: /ru/transfer_manager/prod/data-transfer
• endpoint (String) YDS Endpoint for dedicated db
• save_tx_order (Boolean) Save transaction order
  Not to split events queue into separate per-table queues.
  Incompatible with setting Topic prefix, only with Topic full name.
• security_groups (List of String) List of security groups that the transfer associated with this endpoint should
  use
• serializer (Block List) Data serialization format (see below for nested schema)
• service_account_id (String) Service account ID which has read access to the stream
• stream (String) Stream to write to
• subnet_id (String) Identifier of the Yandex Cloud VPC subnetwork to user for accessing the
  database. If omitted, the server has to be accessible via Internet

Nested Schema for Nested Schema for settings.yds_target.serializer

Optional:

• serializer_auto (Block List) Empty block. Select the serialization format automatically (see below for nested schema)
• serializer_debezium (Block List) Serialize data in debezium json format (see below for nested schema)
• serializer_json (Block List) Empty block. Serialize data in json format (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_auto

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_debezium

Optional:

• serializer_parameters (Block List) A list of Debezium parameters set by the structure of the key and value
  string fields (see below for nested schema)

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_debezium.serializer_parameters

Optional:

• key (String) Name of the serializer parameter
• value (String) Value of the serializer parameter

Nested Schema for Nested Schema for settings.yds_target.serializer.serializer_json

Nested Schema for Nested Schema for timeouts

Optional:

• create (String) A string that can be parsed as a duration consisting of numbers and unit suffixes, such as "30s" or "2h45m". Valid time units are "s" (seconds), "m" (minutes), "h" (hours).
• delete (String) A string that can be parsed as a duration consisting of numbers and unit suffixes, such as "30s" or "2h45m". Valid time units are "s" (seconds), "m" (minutes), "h" (hours). Setting a timeout for a Delete operation is only applicable if changes are saved into state before the destroy operation occurs.
• read (String) A string that can be parsed as a duration consisting of numbers and unit suffixes, such as "30s" or "2h45m". Valid time units are "s" (seconds), "m" (minutes), "h" (hours). Read operations occur during any refresh or planning operation when refresh is enabled.
• update (String) A string that can be parsed as a duration consisting of numbers and unit suffixes, such as "30s" or "2h45m". Valid time units are "s" (seconds), "m" (minutes), "h" (hours).

ImportImport

The resource can be imported by using their resource ID. For getting the resource ID you can use Yandex Cloud Web Console or YC CLI.

# terraform import yandex_datatransfer_endpoint.<resource Name> <resource Id>
terraform import yandex_datatransfer_endpoint.my_dt_endpoint dtec2**********a38dp