Written by

🚧 [Experimental] Create Spark cluster

Creates a Spark cluster and returns an asynchronous operation.

Request

POST

https://api.datalens.tech/rpc/createSparkCluster

Headers

Name

Description

x-dl-api-version

Type: string

API version header.

Const: 3

Example: 3

Body

application/json

{
  "collectionId": "example",
  "cloudEnvironmentId": "example",
  "name": "example",
  "description": "example",
  "labels": {},
  "config": {
    "sparkVersion": "example",
    "resourcePools": {
      "driver": {
        "resourcePresetId": "example",
        "scalePolicy": {}
      },
      "executor": {
        "resourcePresetId": "example",
        "scalePolicy": {}
      }
    },
    "dependencies": {
      "pipPackages": [
        "example"
      ],
      "debPackages": [
        "example"
      ]
    },
    "logging": {
      "enabled": true
    }
  }
}

Name

Description

cloudEnvironmentId

Type: string

ID of the cloud environment in which to create the cluster.

Min length: 1

Max length: 50

Example: example

collectionId

Type: string

ID of the DataLens collection in which to create the cluster.

Min length: 1

Max length: 50

Example: example

config

Type: object

resourcePools

Type: object

driver

Type: object

resourcePresetId

Type: string

ID of the Spark resource preset.

Min length: 1

Max length: 50

Example: example

scalePolicy

Any of 2 types

Type: object

fixedScale

Type: object

size

Type: string

Number of instances in the resource pool.

Pattern: ^(?:[1-9]|[1-9]\d|100)$

Example: example

Example

{
  "size": "example"
}

Example

{
  "fixedScale": {
    "size": "example"
  }
}

Type: object

autoScale

Type: object

initialSize

Type: string

Initial number of instances in the resource pool.

Pattern: ^(?:0|[1-9]\d?|100)$

Example: example

maxSize

Type: string

Maximum number of instances in the resource pool.

Pattern: ^(?:[1-9]|[1-9]\d|100)$

Example: example

minSize

Type: string

Minimum number of instances in the resource pool.

Pattern: ^(?:0|[1-9]\d?|100)$

Example: example

Example

{
  "minSize": "example",
  "maxSize": "example",
  "initialSize": "example"
}

Example

{
  "autoScale": {
    "minSize": "example",
    "maxSize": "example",
    "initialSize": "example"
  }
}

Resource pool scaling policy.

Example

{
  "fixedScale": {
    "size": "example"
  }
}

Driver resource pool.

Example

{
  "resourcePresetId": "example",
  "scalePolicy": {
    "fixedScale": {
      "size": "example"
    }
  }
}

executor

Type: object

resourcePresetId

Type: string

ID of the Spark resource preset.

Min length: 1

Max length: 50

Example: example

scalePolicy

Any of 2 types

Type: object

fixedScale

Type: object

size

Type: string

Number of instances in the resource pool.

Pattern: ^(?:[1-9]|[1-9]\d|100)$

Example: example

Example

{
  "size": "example"
}

Example

{
  "fixedScale": {
    "size": "example"
  }
}

Type: object

autoScale

Type: object

initialSize

Type: string

Initial number of instances in the resource pool.

Pattern: ^(?:0|[1-9]\d?|100)$

Example: example

maxSize

Type: string

Maximum number of instances in the resource pool.

Pattern: ^(?:[1-9]|[1-9]\d|100)$

Example: example

minSize

Type: string

Minimum number of instances in the resource pool.

Pattern: ^(?:0|[1-9]\d?|100)$

Example: example

Example

{
  "minSize": "example",
  "maxSize": "example",
  "initialSize": "example"
}

Example

{
  "autoScale": {
    "minSize": "example",
    "maxSize": "example",
    "initialSize": "example"
  }
}

Resource pool scaling policy.

Example

{
  "fixedScale": {
    "size": "example"
  }
}

Executor resource pool.

Example

{
  "resourcePresetId": "example",
  "scalePolicy": {
    "fixedScale": {
      "size": "example"
    }
  }
}

Example

{
  "driver": {
    "resourcePresetId": "example",
    "scalePolicy": {
      "fixedScale": {
        "size": "example"
      }
    }
  },
  "executor": {
    "resourcePresetId": "example",
    "scalePolicy": {
      "fixedScale": {
        "size": "example"
      }
    }
  }
}

dependencies

Type: object

debPackages

Type: string[]

Debian packages to install in the cluster.

Example

[
  "example"
]

pipPackages

Type: string[]

Python packages to install in the cluster.

Example

[
  "example"
]

Packages to install in the cluster.

Example

{
  "pipPackages": [
    "example"
  ],
  "debPackages": [
    "example"
  ]
}

logging

Type: object

enabled

Type: boolean

Whether cluster logging is enabled.

Cluster logging configuration.

Example

{
  "enabled": true
}

sparkVersion

Type: string

Spark version. The service default is used when omitted.

Example: example

Example

{
  "sparkVersion": "example",
  "resourcePools": {
    "driver": {
      "resourcePresetId": "example",
      "scalePolicy": {
        "fixedScale": {}
      }
    },
    "executor": {
      "resourcePresetId": "example",
      "scalePolicy": {
        "fixedScale": {}
      }
    }
  },
  "dependencies": {
    "pipPackages": [
      "example"
    ],
    "debPackages": [
      "example"
    ]
  },
  "logging": {
    "enabled": true
  }
}

name

Type: string

Name of the Spark cluster.

Max length: 63

Pattern: ^[a-zA-Z0-9ЁёА-я]\S{1,61}[a-zA-Z0-9ЁёА-я]$

Example: example

description

Type: string

Description of the Spark cluster.

Max length: 200

Example: example

labels

Type: object

[additional]

Type: string

Max length: 63

Pattern: ^[-_0-9a-z]*$

Example: example

Spark cluster labels.

Example

{}

Responses

200 OK

Response

Body

application/json

{
  "id": "example",
  "description": "example",
  "createdAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "createdBy": "example",
  "modifiedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "done": true,
  "metadata": {},
  "error": {
    "code": 0.5,
    "message": "example",
    "details": [
      null
    ]
  },
  "response": {}
}

Name

Description

done

Type: boolean

Whether the operation has completed.

Type: string

Unique ID of the operation.

Example: example

metadata

Type: object

[additional]

Type: unknown

Example: null

Service-specific operation metadata.

Example

{}

createdAt

Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the operation was created.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

createdBy

Type: string

ID of the operation creator.

Example: example

description

Type: string

Description of the operation.

Example: example

error

Type: object

code

Type: number

Operation error code.

message

Type: string

Operation error message.

Example: example

details

Type: unknown[]

Additional operation error details.

Example

[
  null
]

Operation error, if the operation failed.

Example

{
  "code": 0.5,
  "message": "example",
  "details": [
    null
  ]
}

modifiedAt

Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the operation was last modified.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

response

Type: object

[additional]

Type: unknown

Example: null

Service-specific operation response.

Example

{}

🚧 [Experimental] Create Spark cluster

RequestRequest

HeadersHeaders

BodyBody

ResponsesResponses

200 OK200 OK

BodyBody

Was the article helpful?

Request

Headers

Body

Responses

200 OK

Body