yandex_datatransfer_transfer (Resource)

Manages a Data Transfer transfer. For more information, see the official documentation.

Example usageExample usage

//
// Create a new pair of Data Transfer Endpoints Source & Target and Data Transfer.
//

// Create Data Transfer Endpoint "Source"
resource "yandex_datatransfer_endpoint" "pg_source" {
  name = "pg-test-source"
  settings {
    postgres_source {
      connection {
        on_premise {
          hosts = [
            "example.org"
          ]
          port = 5432
        }
      }
      slot_gigabyte_lag_limit = 100
      database                = "db1"
      user                    = "user1"
      password {
        raw = "123"
      }
    }
  }
}

// Create Data Transfer Endpoint "Target"
resource "yandex_datatransfer_endpoint" "pg_target" {
  folder_id = "some_folder_id"
  name      = "pg-test-target2"
  settings {
    postgres_target {
      connection {
        mdb_cluster_id = "some_cluster_id"
      }
      database = "db2"
      user     = "user2"
      password {
        raw = "321"
      }
    }
  }
}

// Create Data Transfer from "Source" to "Target"
resource "yandex_datatransfer_transfer" "pgpg_transfer" {
  folder_id = "some_folder_id"
  name      = "pgpg"
  source_id = yandex_datatransfer_endpoint.pg_source.id
  target_id = yandex_datatransfer_endpoint.pg_target.id
  type      = "SNAPSHOT_AND_INCREMENT"
  runtime {
    yc_runtime {
      job_count = 1
      upload_shard_params {
        job_count     = 4
        process_count = 1
      }
    }
  }
  transformation {
    transformers {
      # one of transformer
    }
    transformers {
      # one of transformer
    }
    # ...
  }
}

SchemaSchema

OptionalOptional

• description (String) The resource description.
• folder_id (String) The folder identifier that resource belongs to. If it is not provided, the default provider folder-id is used.
• labels (Map of String) A set of key/value label pairs which assigned to resource.
• name (String) The resource name.
• on_create_activate_mode (String) Activation action on create a new incremental transfer. It is not part of the transfer parameter and is used only on create. One of sync_activate, async_activate, dont_activate. The default is sync_activate.
• runtime (Block List, Max: 1) Runtime parameters for the transfer. (see below for nested schema)
• source_id (String) ID of the source endpoint for the transfer.
• target_id (String) ID of the target endpoint for the transfer.
• transformation (Block List, Max: 1) Transformation for the transfer. (see below for nested schema)
• type (String) Type of the transfer. One of SNAPSHOT_ONLY, INCREMENT_ONLY, SNAPSHOT_AND_INCREMENT

Read-OnlyRead-Only

• id (String) The ID of this resource.
• warning (String) Error description if transfer has any errors.

Nested Schema for Nested Schema for runtime

Optional:

• yc_runtime (Block List, Max: 1) YC Runtime parameters for the transfer. (see below for nested schema)

Nested Schema for Nested Schema for runtime.yc_runtime

Optional:

• job_count (Number) Number of workers in parallel replication.
• upload_shard_params (Block List, Max: 1) Parallel snapshot parameters. (see below for nested schema)

Nested Schema for Nested Schema for runtime.yc_runtime.upload_shard_params

Optional:

• job_count (Number) Number of workers.
• process_count (Number) Number of threads.

Nested Schema for Nested Schema for transformation

Optional:

• transformers (Block List) A list of transformers. You can specify exactly 1 transformer in each element of list. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers

Optional:

• convert_to_string (Block List, Max: 1) Convert column values to strings. (see below for nested schema)
• filter_columns (Block List, Max: 1) Set up a list of table columns to transfer. (see below for nested schema)
• filter_rows (Block List, Max: 1) This filter only applies to transfers with queues (Apache Kafka®) as a data source. When running a transfer, only the strings meeting the specified criteria remain in a changefeed. (see below for nested schema)
• mask_field (Block List, Max: 1) Mask field transformer allows you to hash data. (see below for nested schema)
• rename_tables (Block List, Max: 1) Set rules for renaming tables by specifying the current names of the tables in the source and new names for these tables in the target. (see below for nested schema)
• replace_primary_key (Block List, Max: 1) Override primary keys. (see below for nested schema)
• sharder_transformer (Block List, Max: 1) Set the number of shards for particular tables and a list of columns whose values will be used for calculating a hash to determine a shard. (see below for nested schema)
• table_splitter_transformer (Block List, Max: 1) Splits the X table into multiple tables (X_1, X_2, ..., X_n) based on data. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.convert_to_string

Optional:

• columns (Block List, Max: 1) List of the columns to transfer to the target tables using lists of included and excluded columns. (see below for nested schema)
• tables (Block List, Max: 1) Table filter. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.convert_to_string.columns

Optional:

• exclude_columns (List of String) List of columns that will be excluded to transfer.
• include_columns (List of String) List of columns that will be included to transfer.

Nested Schema for Nested Schema for transformation.transformers.convert_to_string.tables

Optional:

• exclude_tables (List of String) List of tables that will be excluded to transfer.
• include_tables (List of String) List of tables that will be included to transfer.

Nested Schema for Nested Schema for transformation.transformers.filter_columns

Optional:

• columns (Block List, Max: 1) List of the columns to transfer to the target tables using lists of included and excluded columns. (see below for nested schema)
• tables (Block List, Max: 1) Table filter. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.filter_columns.columns

Optional:

• exclude_columns (List of String)
• include_columns (List of String)

Nested Schema for Nested Schema for transformation.transformers.filter_columns.tables

Optional:

• exclude_tables (List of String)
• include_tables (List of String)

Nested Schema for Nested Schema for transformation.transformers.filter_rows

Optional:

• filter (String) Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. See details here.
• tables (Block List, Max: 1) Table filter. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.filter_rows.tables

Optional:

• exclude_tables (List of String)
• include_tables (List of String)

Nested Schema for Nested Schema for transformation.transformers.mask_field

Optional:

• columns (List of String) List of strings that specify the name of the column for data masking (a regular expression).
• function (Block List, Max: 1) Mask function. (see below for nested schema)
• tables (Block List, Max: 1) Table filter. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.mask_field.function

Optional:

• mask_function_hash (Block List, Max: 1) Hash mask function. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.mask_field.function.mask_function_hash

Optional:

• user_defined_salt (String) This string will be used in the HMAC(sha256, salt) function applied to the column data.

Nested Schema for Nested Schema for transformation.transformers.mask_field.tables

Optional:

• exclude_tables (List of String)
• include_tables (List of String)

Nested Schema for Nested Schema for transformation.transformers.rename_tables

Optional:

• rename_tables (Block List) List of renaming rules. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.rename_tables.rename_tables

Optional:

• new_name (Block List, Max: 1) Specify the new names for this table in the target. (see below for nested schema)
• original_name (Block List, Max: 1) Specify the current names of the table in the source. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.rename_tables.rename_tables.new_name

Optional:

• name (String)
• name_space (String)

Nested Schema for Nested Schema for transformation.transformers.rename_tables.rename_tables.original_name

Optional:

• name (String)
• name_space (String)

Nested Schema for Nested Schema for transformation.transformers.replace_primary_key

Optional:

• keys (List of String) List of columns to be used as primary keys.
• tables (Block List, Max: 1) Table filter. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.replace_primary_key.tables

Optional:

• exclude_tables (List of String)
• include_tables (List of String)

Nested Schema for Nested Schema for transformation.transformers.sharder_transformer

Optional:

• columns (Block List, Max: 1) List of the columns to transfer to the target tables using lists of included and excluded columns. (see below for nested schema)
• shards_count (Number) Number of shards.
• tables (Block List, Max: 1) Table filter. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.sharder_transformer.columns

Optional:

• exclude_columns (List of String)
• include_columns (List of String)

Nested Schema for Nested Schema for transformation.transformers.sharder_transformer.tables

Optional:

• exclude_tables (List of String)
• include_tables (List of String)

Nested Schema for Nested Schema for transformation.transformers.table_splitter_transformer

Optional:

• columns (List of String) List of strings that specify the columns in the tables to be partitioned.
• splitter (String) Specify the split string to be used for merging components in a new table name.
• tables (Block List, Max: 1) Table filter. (see below for nested schema)

Nested Schema for Nested Schema for transformation.transformers.table_splitter_transformer.tables

Optional:

• exclude_tables (List of String)
• include_tables (List of String)

ImportImport

The resource can be imported by using their resource ID. For getting the resource ID you can use Yandex Cloud Web Console or YC CLI.

# terraform import yandex_datatransfer_transfer.<resource Name> <resource Id>
terraform import yandex_datatransfer_endpoint.my_dt_transfer dttnc**********r3bkg