Связаться с экспертомПопробовать бесплатно
Статья создана
Открыть в Markdown

🚧 [Experimental] Create data transfer

Creates a transfer and returns an asynchronous operation. Use getLakehouseOperation to track completion.

Request

POST

https://api.datalens.tech/rpc/createDataTransfer

Headers

Name

Description

x-dl-api-version

Type: string

API version header.

Const: 3

Example: 3

Body

application/json
{
  "collectionId": "example",
  "name": "example",
  "description": "example",
  "labels": {},
  "config": {
    "source": {
      "postgresConnection": {
        "connectionId": "example",
        "database": "example"
      }
    },
    "destination": {
      "icebergNamespace": {
        "namespaceId": "example"
      }
    },
    "type": {
      "regular": {
        "cronExpression": "example",
        "tables": [
          null
        ]
      }
    },
    "tables": [
      "example"
    ]
  }
}

Name

Description

collectionId

Type: string

ID of the DataLens collection in which to create the transfer.

Min length: 1

Example: example

config
Type: object

destination
Type: object

icebergNamespace
Type: object

namespaceId

Type: string

ID of the destination Iceberg namespace.

Min length: 1

Example: example

Iceberg destination namespace settings.

Example
{
  "namespaceId": "example"
}

Destination settings.

Example
{
  "icebergNamespace": {
    "namespaceId": "example"
  }
}

source

Type: DataTransferSourceArgs

Exactly one source connection.

Example
{
  "postgresConnection": {
    "connectionId": "example",
    "database": "example"
  }
}

type

Type: DataTransferTypeArgs

Exactly one transfer mode: regular or one-time.

Example
{
  "regular": {
    "cronExpression": "example",
    "tables": [
      {
        "tableNamespace": "example",
        "tableName": "example",
        "cursorColumn": "example",
        "initialState": "example"
      }
    ]
  }
}

tables

Type: string[]

Source tables to transfer.

Example
[
  "example"
]

Transfer configuration.

Example
{
  "source": {
    "postgresConnection": {
      "connectionId": "example",
      "database": "example"
    }
  },
  "destination": {
    "icebergNamespace": {
      "namespaceId": "example"
    }
  },
  "type": {
    "regular": {
      "cronExpression": "example",
      "tables": [
        {}
      ]
    }
  },
  "tables": [
    "example"
  ]
}

name

Type: string

Name of the transfer.

Min length: 1

Max length: 63

Pattern: ^[a-zA-Z0-9_-]+$

Example: example

description

Type: string

Description of the transfer.

Max length: 256

Example: example

labels
Type: object

[additional]

Type: string

Max length: 63

Pattern: ^[-_0-9a-z]*$

Example: example

Transfer labels.

Example
{}

DataTransferSourceArgs

Exactly one source connection.

Any of 3 types
  • Type: object

    postgresConnection
    Type: object

    connectionId

    Type: string

    ID of the source DataLens connection.

    Min length: 1

    Example: example

    database

    Type: string

    Source database name.

    Min length: 1

    Max length: 50

    Example: example

    PostgreSQL source connection settings.

    Example
    {
  "connectionId": "example",
  "database": "example"
}
    Example
    {
  "postgresConnection": {
    "connectionId": "example",
    "database": "example"
  }
}
  • Type: object

    clickhouseConnection
    Type: object

    connectionId

    Type: string

    ID of the source DataLens connection.

    Min length: 1

    Example: example

    database

    Type: string

    Source database name.

    Min length: 1

    Max length: 50

    Example: example

    ClickHouse source connection settings.

    Example
    {
  "connectionId": "example",
  "database": "example"
}
    Example
    {
  "clickhouseConnection": {
    "connectionId": "example",
    "database": "example"
  }
}
  • Type: object

    mysqlConnection
    Type: object

    connectionId

    Type: string

    ID of the source DataLens connection.

    Min length: 1

    Example: example

    database

    Type: string

    Source database name.

    Min length: 1

    Max length: 50

    Example: example

    MySQL source connection settings.

    Example
    {
  "connectionId": "example",
  "database": "example"
}
    Example
    {
  "mysqlConnection": {
    "connectionId": "example",
    "database": "example"
  }
}
Example
{
  "postgresConnection": {
    "connectionId": "example",
    "database": "example"
  }
}

DataTransferIncrementalTableArgs

Name

Description

tableName

Type: string

Name of the source table.

Min length: 1

Example: example

tableNamespace

Type: string

Namespace or schema of the source table.

Min length: 1

Example: example

cursorColumn

Type: string

Column used to detect new and changed rows.

Example: example

initialState

Type: string

Initial cursor value for the first transfer run.

Example: example
Example
{
  "tableNamespace": "example",
  "tableName": "example",
  "cursorColumn": "example",
  "initialState": "example"
}

RegularDataTransferTypeArgs

Name

Description

regular
Type: object

cronExpression

Type: string

Five-field cron expression in UTC.

Example: example

tables

Type: DataTransferIncrementalTableArgs[]

Incremental snapshot settings.

Example
[
  {
    "tableNamespace": "example",
    "tableName": "example",
    "cursorColumn": "example",
    "initialState": "example"
  }
]

Regular transfer settings.

Example
{
  "cronExpression": "example",
  "tables": [
    {
      "tableNamespace": "example",
      "tableName": "example",
      "cursorColumn": "example",
      "initialState": "example"
    }
  ]
}
Example
{
  "regular": {
    "cronExpression": "example",
    "tables": [
      {
        "tableNamespace": "example",
        "tableName": "example",
        "cursorColumn": "example",
        "initialState": "example"
      }
    ]
  }
}

OneTimeDataTransferTypeArgs

Name

Description

oneTime
Type: object

One-time transfer settings.

Example
{}
Example
{
  "oneTime": {}
}

DataTransferTypeArgs

Exactly one transfer mode: regular or one-time.

Any of 2 types

  • Type: RegularDataTransferTypeArgs

    Example
    {
  "regular": {
    "cronExpression": "example",
    "tables": [
      {
        "tableNamespace": "example",
        "tableName": "example",
        "cursorColumn": "example",
        "initialState": "example"
      }
    ]
  }
}

  • Type: OneTimeDataTransferTypeArgs

    Example
    {
  "oneTime": {}
}
Example
{
  "regular": {
    "cronExpression": "example",
    "tables": [
      {
        "tableNamespace": "example",
        "tableName": "example",
        "cursorColumn": "example",
        "initialState": "example"
      }
    ]
  }
}

Responses

200 OK

Response

Body

application/json
{
  "id": "example",
  "description": "example",
  "createdAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "createdBy": "example",
  "modifiedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "done": true,
  "metadata": {},
  "error": {
    "code": 0.5,
    "message": "example",
    "details": [
      null
    ]
  },
  "response": {}
}

Name

Description

done

Type: boolean

Whether the operation has completed.

id

Type: string

Unique ID of the operation.

Example: example

metadata
Type: object

[additional]

Type: unknown

Example: null

Service-specific operation metadata.

Example
{}

createdAt
Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the operation was created.

Example
{
  "seconds": "example",
  "nanos": 0.5
}

createdBy

Type: string

ID of the operation creator.

Example: example

description

Type: string

Description of the operation.

Example: example

error
Type: object

code

Type: number

Operation error code.

message

Type: string

Operation error message.

Example: example

details

Type: unknown[]

Additional operation error details.

Example
[
  null
]

Operation error, if the operation failed.

Example
{
  "code": 0.5,
  "message": "example",
  "details": [
    null
  ]
}

modifiedAt
Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the operation was last modified.

Example
{
  "seconds": "example",
  "nanos": 0.5
}

response
Type: object

[additional]

Type: unknown

Example: null

Service-specific operation response.

Example
{}
Предыдущая
🚧 [Experimental] Get data transfer
Следующая
🚧 [Experimental] Update data transfer