Data Transfer API, gRPC: TransferService.Create

Written by

Updated at October 17, 2024

gRPC request
CreateTransferRequest
Runtime
YcRuntime
ShardingUploadParams
Transformation
Transformer
MaskFieldTransformer
TablesFilter
MaskFunction
MaskFunctionHash
FilterColumnsTransformer
ColumnsFilter
RenameTablesTransformer
RenameTable
Table
ReplacePrimaryKeyTransformer
ToStringTransformer
SharderTransformer
TableSplitterTransformer
FilterRowsTransformer
operation.Operation

gRPC request

rpc Create (CreateTransferRequest) returns (operation.Operation)

CreateTransferRequest

{
  "sourceId": "string",
  "targetId": "string",
  "description": "string",
  "folderId": "string",
  "runtime": {
    // Includes only one of the fields `ycRuntime`
    "ycRuntime": {
      "jobCount": "int64",
      "uploadShardParams": {
        "jobCount": "int64",
        "processCount": "int64"
      }
    }
    // end of the list of possible fields
  },
  "type": "TransferType",
  "name": "string",
  "labels": "string",
  "transformation": {
    "transformers": [
      {
        // Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`
        "maskField": {
          "tables": {
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ]
          },
          "columns": [
            "string"
          ],
          "function": {
            // Includes only one of the fields `maskFunctionHash`
            "maskFunctionHash": {
              "userDefinedSalt": "string"
            }
            // end of the list of possible fields
          }
        },
        "filterColumns": {
          "tables": {
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ]
          },
          "columns": {
            "includeColumns": [
              "string"
            ],
            "excludeColumns": [
              "string"
            ]
          }
        },
        "renameTables": {
          "renameTables": [
            {
              "originalName": {
                "nameSpace": "string",
                "name": "string"
              },
              "newName": {
                "nameSpace": "string",
                "name": "string"
              }
            }
          ]
        },
        "replacePrimaryKey": {
          "tables": {
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ]
          },
          "keys": [
            "string"
          ]
        },
        "convertToString": {
          "tables": {
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ]
          },
          "columns": {
            "includeColumns": [
              "string"
            ],
            "excludeColumns": [
              "string"
            ]
          }
        },
        "sharderTransformer": {
          "tables": {
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ]
          },
          "columns": {
            "includeColumns": [
              "string"
            ],
            "excludeColumns": [
              "string"
            ]
          },
          "shardsCount": "int64"
        },
        "tableSplitterTransformer": {
          "tables": {
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ]
          },
          "columns": [
            "string"
          ],
          "splitter": "string"
        },
        "filterRows": {
          "tables": {
            "includeTables": [
              "string"
            ],
            "excludeTables": [
              "string"
            ]
          },
          "filter": "string",
          "filters": [
            "string"
          ]
        }
        // end of the list of possible fields
      }
    ]
  }
}

Field	Description
sourceId	string
targetId	string
description	string
folderId	string
runtime	Runtime
type	enum TransferType `TRANSFER_TYPE_UNSPECIFIED` `SNAPSHOT_AND_INCREMENT`: Snapshot and increment `SNAPSHOT_ONLY`: Snapshot `INCREMENT_ONLY`: Increment
name	string
labels	string
transformation	Transformation

Runtime

Field

Description

ycRuntime

YcRuntime

Includes only one of the fields ycRuntime.

YcRuntime

Field	Description
jobCount	int64
uploadShardParams	ShardingUploadParams

ShardingUploadParams

Field	Description
jobCount	int64
processCount	int64

Transformation is converting data using special transformer functions.
These functions are executed on a data stream, applied to each data change item,
and transform them.
A transformer can be run at both the metadata and data levels.
Data can only be transformed if the source and target are of different types.

Field

Description

transformers[]

Transformer

Transformers are set as a list.
When activating a transfer, a transformation plan is made for the tables that
match the specified criteria.
Transformers are applied to the tables in the sequence specified in the list.

Transformer

Some transformers may have limitations and only apply to some source-target
pairs.

Field	Description
maskField	MaskFieldTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.
filterColumns	FilterColumnsTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.
renameTables	RenameTablesTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.
replacePrimaryKey	ReplacePrimaryKeyTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.
convertToString	ToStringTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.
sharderTransformer	SharderTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.
tableSplitterTransformer	TableSplitterTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.
filterRows	FilterRowsTransformer Includes only one of the fields `maskField`, `filterColumns`, `renameTables`, `replacePrimaryKey`, `convertToString`, `sharderTransformer`, `tableSplitterTransformer`, `filterRows`.

MaskFieldTransformer

Mask field transformer allows you to hash data

Field	Description
tables	TablesFilter List of included and excluded tables
columns[]	string Specify the name of the column for data masking (a regular expression).
function	MaskFunction Mask function

TablesFilter

Filter tables using lists of included and excluded tables.

Field

Description

includeTables[]

string

List of tables that will be included to transfer

excludeTables[]

string

List of tables that will be excluded to transfer

MaskFunction

Mask function

Field

Description

maskFunctionHash

MaskFunctionHash

Hash mask function

Includes only one of the fields maskFunctionHash.

MaskFunctionHash

Hash data using HMAC

Field

Description

userDefinedSalt

string

This string will be used in the HMAC(sha256, salt) function applied to the
column data.

FilterColumnsTransformer

Set up a list of table columns to transfer

Field

Description

tables

TablesFilter

List of the tables to filter using lists of included and excluded tables.

columns

ColumnsFilter

List of the columns to transfer to the target tables using lists of included and
excluded columns.

ColumnsFilter

Filter columns using lists of included and excluded columns.

Field

Description

includeColumns[]

string

List of columns that will be included to transfer

excludeColumns[]

string

List of columns that will be excluded to transfer

RenameTablesTransformer

Set rules for renaming tables by specifying the current names of the tables in
the source and new names for these tables in the target.

Field

Description

renameTables[]

RenameTable

List of renaming rules

RenameTable

Specify rule for renaming table

Field

Description

originalName

Table

Specify the current names of the table in the source

newName

Table

Specify the new names for this table in the target

Table

Field	Description
nameSpace	string
name	string

ReplacePrimaryKeyTransformer

Override primary keys

Field

Description

tables

TablesFilter

List of included and excluded tables

keys[]

string

List of columns to be used as primary keys

ToStringTransformer

Convert column values to strings
The values will be converted depending on the source type
Conversion rules are described here:
https://cloud.yandex.com/en/docs/data-transfer/concepts/data-transformation#convert-to-string

Field

Description

tables

TablesFilter

List of included and excluded tables

columns

ColumnsFilter

List of included and excluded columns

SharderTransformer

Set the number of shards for particular tables and a list of columns whose
values will be used for calculating a hash to determine a shard.

Field	Description
tables	TablesFilter List of included and excluded tables
columns	ColumnsFilter List of included and excluded columns
shardsCount	int64 Number of shards

TableSplitterTransformer

A transfer splits the X table into multiple tables (X_1, X_2, ..., X_n) based on
data.
If a row was located in the X table before it was split, it is now in the X_i
table,
where i is determined by the column list and split string parameters.
Example:
If the column list has two columns, month of birth and gender, specified and the
split string states @,
information about an employee whose name is John and who was born on February
11, 1984,
from the Employees table will get to a new table named Employees@February@male.

Field	Description
tables	TablesFilter List of included and excluded tables
columns[]	string Specify the columns in the tables to be partitioned.
splitter	string Specify the split string to be used for merging components in a new table name.

FilterRowsTransformer

This filter only applies to transfers with queues (Logbroker or Apache Kafka®)
as a data source.
When running a transfer, only the strings meeting the specified criteria remain
in a changefeed.

Field	Description
tables	TablesFilter List of included and excluded tables.
filter	string Filtering criterion. This can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details here: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources. Deprecated: Use filters instead.
filters[]	string Data is transported if it satisfies at least one of filters. Consider that there is OR statement between filters. Each filter can be comparison operators for numeric, string, and Boolean values, comparison to NULL, and checking whether a substring is part of a string. Details in docs: https://yandex.cloud/en-ru/docs/data-transfer/concepts/data-transformation#append-only-sources.

operation.Operation

{
  "id": "string",
  "description": "string",
  "createdAt": "google.protobuf.Timestamp",
  "createdBy": "string",
  "modifiedAt": "google.protobuf.Timestamp",
  "done": "bool",
  "metadata": "google.protobuf.Any",
  // Includes only one of the fields `error`, `response`
  "error": "google.rpc.Status",
  "response": "google.protobuf.Any"
  // end of the list of possible fields
}

An Operation resource. For more information, see Operation.

Field	Description
id	string ID of the operation.
description	string Description of the operation. 0-256 characters long.
createdAt	google.protobuf.Timestamp Creation timestamp.
createdBy	string ID of the user or service account who initiated the operation.
modifiedAt	google.protobuf.Timestamp The time when the Operation resource was last modified.
done	bool If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
metadata	google.protobuf.Any Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
error	google.rpc.Status The error result of the operation in case of failure or cancellation. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.
response	google.protobuf.Any The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty. If the original method is the standard Create/Update, the response should be the target resource of the operation. Any method that returns a long-running operation should document the response type, if any. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.

Data Transfer API, gRPC: TransferService.Create

gRPC requestgRPC request

CreateTransferRequestCreateTransferRequest

RuntimeRuntime

YcRuntimeYcRuntime

ShardingUploadParamsShardingUploadParams

TransformationTransformation

TransformerTransformer

MaskFieldTransformerMaskFieldTransformer

TablesFilterTablesFilter

MaskFunctionMaskFunction

MaskFunctionHashMaskFunctionHash

FilterColumnsTransformerFilterColumnsTransformer

ColumnsFilterColumnsFilter

RenameTablesTransformerRenameTablesTransformer

RenameTableRenameTable

TableTable

ReplacePrimaryKeyTransformerReplacePrimaryKeyTransformer

ToStringTransformerToStringTransformer

SharderTransformerSharderTransformer

TableSplitterTransformerTableSplitterTransformer

FilterRowsTransformerFilterRowsTransformer

operation.Operationoperation.Operation

Was the article helpful?

gRPC request

CreateTransferRequest

Runtime

YcRuntime

ShardingUploadParams

Transformation

Transformer

MaskFieldTransformer

TablesFilter

MaskFunction

MaskFunctionHash

FilterColumnsTransformer

ColumnsFilter

RenameTablesTransformer

RenameTable

Table

ReplacePrimaryKeyTransformer

ToStringTransformer

SharderTransformer

TableSplitterTransformer

FilterRowsTransformer

operation.Operation