Связаться с экспертомПопробовать бесплатно
Статья создана
Открыть в Markdown

🚧 [Experimental] List Spark clusters

Returns a list of Spark clusters available to the current DataLens tenant.

Request

POST

https://api.datalens.tech/rpc/listSparkClusters

Headers

Name

Description

x-dl-api-version

Type: string

API version header.

Const: 3

Example: 3

Body

application/json
{
  "pageSize": 0,
  "pageToken": "example",
  "filter": [
    "example"
  ],
  "collectionId": "example"
}

Name

Description

collectionId

Type: string

ID of the DataLens collection that contains the Spark clusters.

Min length: 1

Max length: 50

Example: example

filter

Type: string[]

Filter expressions applied to the Spark cluster list.

Example
[
  "example"
]

pageSize

Type: integer

Maximum number of Spark clusters to return. The default is 100.

Min value: 0

pageToken

Type: string

Token for the next page of Spark clusters.

Example: example

Responses

200 OK

Response

Body

application/json
{
  "sparkClusters": [
    {
      "id": "example",
      "clusterId": "example",
      "collectionId": "example",
      "cloudEnvironmentId": "example",
      "name": "example",
      "description": "example",
      "labels": {},
      "config": {
        "sparkVersion": "example",
        "resourcePools": {
          "driver": {},
          "executor": {}
        },
        "dependencies": {
          "pipPackages": [
            null
          ],
          "debPackages": [
            null
          ]
        },
        "logging": {
          "enabled": true
        }
      },
      "health": "HEALTH_UNKNOWN",
      "status": "STATUS_UNSPECIFIED",
      "entryId": "example"
    }
  ],
  "nextPageToken": "example"
}

Name

Description

nextPageToken

Type: string

Token for the next page of Spark clusters.

Example: example

sparkClusters

Type: SparkCluster[]

Spark clusters matching the request.

Example
[
  {
    "id": "example",
    "clusterId": "example",
    "collectionId": "example",
    "cloudEnvironmentId": "example",
    "name": "example",
    "description": "example",
    "labels": {},
    "config": {
      "sparkVersion": "example",
      "resourcePools": {
        "driver": {
          "resourcePresetId": "example",
          "scalePolicy": null
        },
        "executor": {
          "resourcePresetId": "example",
          "scalePolicy": null
        }
      },
      "dependencies": {
        "pipPackages": [
          "example"
        ],
        "debPackages": [
          "example"
        ]
      },
      "logging": {
        "enabled": true
      }
    },
    "health": "HEALTH_UNKNOWN",
    "status": "STATUS_UNSPECIFIED",
    "entryId": "example"
  }
]

SparkCluster

Name

Description

cloudEnvironmentId

Type: string

ID of the associated cloud environment.

Min length: 1

Example: example

clusterId

Type: string

ID of the managed Spark cluster.

Example: example

collectionId

Type: string

ID of the DataLens collection that contains the cluster.

Min length: 1

Example: example

config
Type: object

dependencies

Type: object | null

Packages installed in the cluster.

Example
{
  "pipPackages": [
    "example"
  ],
  "debPackages": [
    "example"
  ]
}

logging

Type: object | null

Cluster logging configuration.

Example
{
  "enabled": true
}

resourcePools
Type: object

driver
Type: object

resourcePresetId

Type: string

ID of the Spark resource preset.

Min length: 1

Max length: 50

Example: example

scalePolicy
Any of 2 types

  • Type: object

    fixedScale
    Type: object

    size

    Type: string

    Number of instances in the resource pool.

    Pattern: ^(?:[1-9]|[1-9]\d|100)$

    Example: example
    Example
    {
  "size": "example"
}

    scaleType

    Type: string

    Const: fixedScale

    Example: example
    Example
    {
  "fixedScale": {
    "size": "example"
  },
  "scaleType": "fixedScale"
}

  • Type: object

    autoScale
    Type: object

    initialSize

    Type: string

    Initial number of instances in the resource pool.

    Pattern: ^(?:0|[1-9]\d?|100)$

    Example: example

    maxSize

    Type: string

    Maximum number of instances in the resource pool.

    Pattern: ^(?:[1-9]|[1-9]\d|100)$

    Example: example

    minSize

    Type: string

    Minimum number of instances in the resource pool.

    Pattern: ^(?:0|[1-9]\d?|100)$

    Example: example
    Example
    {
  "minSize": "example",
  "maxSize": "example",
  "initialSize": "example"
}

    scaleType

    Type: string

    Const: autoScale

    Example: example
    Example
    {
  "autoScale": {
    "minSize": "example",
    "maxSize": "example",
    "initialSize": "example"
  },
  "scaleType": "autoScale"
}

Resource pool scaling policy.

Example
{
  "fixedScale": {
    "size": "example"
  },
  "scaleType": "fixedScale"
}

Driver resource pool.

Example
{
  "resourcePresetId": "example",
  "scalePolicy": {
    "fixedScale": {
      "size": "example"
    },
    "scaleType": "fixedScale"
  }
}

executor
Type: object

resourcePresetId

Type: string

ID of the Spark resource preset.

Min length: 1

Max length: 50

Example: example

scalePolicy
Any of 2 types

  • Type: object

    fixedScale
    Type: object

    size

    Type: string

    Number of instances in the resource pool.

    Pattern: ^(?:[1-9]|[1-9]\d|100)$

    Example: example
    Example
    {
  "size": "example"
}

    scaleType

    Type: string

    Const: fixedScale

    Example: example
    Example
    {
  "fixedScale": {
    "size": "example"
  },
  "scaleType": "fixedScale"
}

  • Type: object

    autoScale
    Type: object

    initialSize

    Type: string

    Initial number of instances in the resource pool.

    Pattern: ^(?:0|[1-9]\d?|100)$

    Example: example

    maxSize

    Type: string

    Maximum number of instances in the resource pool.

    Pattern: ^(?:[1-9]|[1-9]\d|100)$

    Example: example

    minSize

    Type: string

    Minimum number of instances in the resource pool.

    Pattern: ^(?:0|[1-9]\d?|100)$

    Example: example
    Example
    {
  "minSize": "example",
  "maxSize": "example",
  "initialSize": "example"
}

    scaleType

    Type: string

    Const: autoScale

    Example: example
    Example
    {
  "autoScale": {
    "minSize": "example",
    "maxSize": "example",
    "initialSize": "example"
  },
  "scaleType": "autoScale"
}

Resource pool scaling policy.

Example
{
  "fixedScale": {
    "size": "example"
  },
  "scaleType": "fixedScale"
}

Executor resource pool.

Example
{
  "resourcePresetId": "example",
  "scalePolicy": {
    "fixedScale": {
      "size": "example"
    },
    "scaleType": "fixedScale"
  }
}
Example
{
  "driver": {
    "resourcePresetId": "example",
    "scalePolicy": {
      "fixedScale": {
        "size": "example"
      },
      "scaleType": "fixedScale"
    }
  },
  "executor": {
    "resourcePresetId": "example",
    "scalePolicy": {
      "fixedScale": {
        "size": "example"
      },
      "scaleType": "fixedScale"
    }
  }
}

sparkVersion

Type: string

Spark version used by the cluster.

Example: example
Example
{
  "sparkVersion": "example",
  "resourcePools": {
    "driver": {
      "resourcePresetId": "example",
      "scalePolicy": {
        "fixedScale": {},
        "scaleType": "fixedScale"
      }
    },
    "executor": {
      "resourcePresetId": "example",
      "scalePolicy": {
        "fixedScale": {},
        "scaleType": "fixedScale"
      }
    }
  },
  "dependencies": {
    "pipPackages": [
      "example"
    ],
    "debPackages": [
      "example"
    ]
  },
  "logging": {
    "enabled": true
  }
}

description

Type: string

Description of the Spark cluster.

Example: example

entryId

Type: string

ID of the DataLens entry of the Spark cluster. Empty when the cluster has no entry.

Example: example

health

Type: string

Aggregated health of the Spark cluster.

Enum: HEALTH_UNKNOWN, ALIVE, DEAD, DEGRADED

id

Type: string

ID of the Spark cluster.

Min length: 1

Example: example

labels
Type: object

[additional]

Type: string

Example: example

Spark cluster labels.

Example
{}

name

Type: string

Name of the Spark cluster.

Min length: 1

Example: example

status

Type: string

Current status of the Spark cluster.

Enum: STATUS_UNSPECIFIED, CREATING, RUNNING, UPDATING, ERROR, STOPPING, STOPPED, STARTING
Example
{
  "id": "example",
  "clusterId": "example",
  "collectionId": "example",
  "cloudEnvironmentId": "example",
  "name": "example",
  "description": "example",
  "labels": {},
  "config": {
    "sparkVersion": "example",
    "resourcePools": {
      "driver": {
        "resourcePresetId": "example",
        "scalePolicy": {}
      },
      "executor": {
        "resourcePresetId": "example",
        "scalePolicy": {}
      }
    },
    "dependencies": {
      "pipPackages": [
        "example"
      ],
      "debPackages": [
        "example"
      ]
    },
    "logging": {
      "enabled": true
    }
  },
  "health": "HEALTH_UNKNOWN",
  "status": "STATUS_UNSPECIFIED",
  "entryId": "example"
}
Предыдущая
Overview
Следующая
🚧 [Experimental] Get Spark cluster