Статья создана

Request
- Headers
- Body
Responses
200 OK
- Body
- SparkApplication

🚧 [Experimental] List Spark applications

Returns a list of applications for the specified Spark cluster.

Request

POST

https://api.datalens.tech/rpc/listSparkApplications

Headers

Name

Description

x-dl-api-version

Type: string

API version header.

Const: 3

Example: 3

Body

application/json

{
  "clusterId": "example",
  "pageSize": 0,
  "pageToken": "example",
  "filter": [
    "example"
  ]
}

Name	Description
clusterId	Type: string ID of the Spark cluster. Min length: `1` Max length: `50` Example: `example`
filter	Type: string[] Filter conditions applied to the Spark application list, combined with AND. Each condition has the form `field="value"`, where field is one of `name`, `created_by`, `application_type` or `catalog_id`. Max items: `100` Example `[ "example" ]`
pageSize	Type: integer Maximum number of Spark applications to return. The default is 100. Min value: `0` Max value: `1000`
pageToken	Type: string Token for the next page of Spark applications. Max length: `200` Example: `example`

Responses

200 OK

Response

Body

application/json

{
  "applications": [
    {
      "id": "example",
      "clusterId": "example",
      "createdAt": {
        "seconds": "example",
        "nanos": 0.5
      },
      "startedAt": {
        "seconds": "example",
        "nanos": 0.5
      },
      "finishedAt": {
        "seconds": "example",
        "nanos": 0.5
      },
      "name": "example",
      "createdBy": "example",
      "status": "STATUS_UNSPECIFIED",
      "connectUrl": "example",
      "catalogs": [
        {}
      ],
      "sparkApplication": {
        "args": [
          null
        ],
        "jarFileUris": [
          null
        ],
        "fileUris": [
          null
        ],
        "archiveUris": [
          null
        ],
        "properties": {},
        "mainJarFileUri": "example",
        "mainClass": "example",
        "packages": [
          null
        ],
        "repositories": [
          null
        ],
        "excludePackages": [
          null
        ]
      },
      "applicationSpec": "sparkApplication"
    }
  ],
  "nextPageToken": "example"
}

Name

Description

applications

Type: SparkApplication[]

Spark applications matching the request.

Example

[
  {
    "id": "example",
    "clusterId": "example",
    "createdAt": {
      "seconds": "example",
      "nanos": 0.5
    },
    "startedAt": {
      "seconds": "example",
      "nanos": 0.5
    },
    "finishedAt": {
      "seconds": "example",
      "nanos": 0.5
    },
    "name": "example",
    "createdBy": "example",
    "status": "STATUS_UNSPECIFIED",
    "connectUrl": "example",
    "catalogs": [
      {
        "catalogId": "example"
      }
    ],
    "sparkApplication": {
      "args": [
        "example"
      ],
      "jarFileUris": [
        "example"
      ],
      "fileUris": [
        "example"
      ],
      "archiveUris": [
        "example"
      ],
      "properties": {},
      "mainJarFileUri": "example",
      "mainClass": "example",
      "packages": [
        "example"
      ],
      "repositories": [
        "example"
      ],
      "excludePackages": [
        "example"
      ]
    },
    "applicationSpec": "sparkApplication"
  }
]

nextPageToken

Type: string

Token for the next page of Spark applications.

Max length: 200

Example: example

SparkApplication

Any of 4 types

Type: object

applicationSpec

Type: string

Const: sparkApplication

Example: example

catalogs

Type: object[]

catalogId

Type: string

ID of the REST catalog.

Min length: 1

Max length: 50

Example: example

REST catalogs attached to the Spark application.

Example

[
  {
    "catalogId": "example"
  }
]

clusterId

Type: string

ID of the Spark cluster.

Min length: 1

Max length: 50

Example: example

connectUrl

Type: string

Spark Connect URL.

Example: example

createdAt

Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the Spark application was created.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

createdBy

Type: string

ID of the user who created the Spark application.

Min length: 1

Example: example

finishedAt

Type: object | null

Time when the Spark application was finished.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

Type: string

ID of the Spark application.

Min length: 1

Max length: 50

Example: example

name

Type: string

Name of the Spark application.

Example: example

sparkApplication

Type: object

archiveUris

Type: string[]

Archives extracted in the Spark working directory.

Max items: 100

Example

[
  "example"
]

args

Type: string[]

Arguments passed to the Spark driver.

Max items: 100

Example

[
  "example"
]

excludePackages

Type: string[]

Maven packages excluded during dependency resolution.

Max items: 100

Example

[
  "example"
]

fileUris

Type: string[]

Files copied to the Spark working directory.

Max items: 100

Example

[
  "example"
]

jarFileUris

Type: string[]

JAR files added to the Spark classpaths.

Max items: 100

Example

[
  "example"
]

mainClass

Type: string

Name of the driver main class.

Max length: 255

Example: example

mainJarFileUri

Type: string

URI of the JAR file that contains the main class.

Min length: 1

Max length: 2047

Example: example

packages

Type: string[]

Maven packages added to the Spark classpaths.

Max items: 100

Example

[
  "example"
]

properties

Type: object

[additional]

Type: string

Max length: 256

Example: example

Spark configuration properties.

Example

{}

repositories

Type: string[]

Additional Maven repositories.

Max items: 10

Example

[
  "example"
]

Example

{
  "args": [
    "example"
  ],
  "jarFileUris": [
    "example"
  ],
  "fileUris": [
    "example"
  ],
  "archiveUris": [
    "example"
  ],
  "properties": {},
  "mainJarFileUri": "example",
  "mainClass": "example",
  "packages": [
    "example"
  ],
  "repositories": [
    "example"
  ],
  "excludePackages": [
    "example"
  ]
}

startedAt

Type: object | null

Time when the Spark application was started.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

status

Type: string

Current status of the Spark application.

Enum: STATUS_UNSPECIFIED, PROVISIONING, PENDING, RUNNING, ERROR, DONE, CANCELLED, CANCELLING

Example

{
  "id": "example",
  "clusterId": "example",
  "createdAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "startedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "finishedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "name": "example",
  "createdBy": "example",
  "status": "STATUS_UNSPECIFIED",
  "connectUrl": "example",
  "catalogs": [
    {
      "catalogId": "example"
    }
  ],
  "sparkApplication": {
    "args": [
      "example"
    ],
    "jarFileUris": [
      "example"
    ],
    "fileUris": [
      "example"
    ],
    "archiveUris": [
      "example"
    ],
    "properties": {},
    "mainJarFileUri": "example",
    "mainClass": "example",
    "packages": [
      "example"
    ],
    "repositories": [
      "example"
    ],
    "excludePackages": [
      "example"
    ]
  },
  "applicationSpec": "sparkApplication"
}

Type: object

applicationSpec

Type: string

Const: pysparkApplication

Example: example

catalogs

Type: object[]

catalogId

Type: string

ID of the REST catalog.

Min length: 1

Max length: 50

Example: example

REST catalogs attached to the Spark application.

Example

[
  {
    "catalogId": "example"
  }
]

clusterId

Type: string

ID of the Spark cluster.

Min length: 1

Max length: 50

Example: example

connectUrl

Type: string

Spark Connect URL.

Example: example

createdAt

Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the Spark application was created.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

createdBy

Type: string

ID of the user who created the Spark application.

Min length: 1

Example: example

finishedAt

Type: object | null

Time when the Spark application was finished.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

Type: string

ID of the Spark application.

Min length: 1

Max length: 50

Example: example

name

Type: string

Name of the Spark application.

Example: example

pysparkApplication

Type: object

archiveUris

Type: string[]

Archives extracted in the Spark working directory.

Max items: 100

Example

[
  "example"
]

args

Type: string[]

Arguments passed to the PySpark driver.

Max items: 100

Example

[
  "example"
]

excludePackages

Type: string[]

Maven packages excluded during dependency resolution.

Max items: 100

Example

[
  "example"
]

fileUris

Type: string[]

Files copied to the Spark working directory.

Max items: 100

Example

[
  "example"
]

jarFileUris

Type: string[]

JAR files added to the Spark classpaths.

Max items: 100

Example

[
  "example"
]

mainPythonFileUri

Type: string

URI of the main Python file.

Min length: 1

Max length: 2047

Example: example

packages

Type: string[]

Maven packages added to the Spark classpaths.

Max items: 100

Example

[
  "example"
]

properties

Type: object

[additional]

Type: string

Max length: 256

Example: example

Spark configuration properties.

Example

{}

pythonFileUris

Type: string[]

Additional Python files passed to PySpark.

Max items: 100

Example

[
  "example"
]

repositories

Type: string[]

Additional Maven repositories.

Max items: 10

Example

[
  "example"
]

Example

{
  "args": [
    "example"
  ],
  "jarFileUris": [
    "example"
  ],
  "fileUris": [
    "example"
  ],
  "archiveUris": [
    "example"
  ],
  "properties": {},
  "mainPythonFileUri": "example",
  "pythonFileUris": [
    "example"
  ],
  "packages": [
    "example"
  ],
  "repositories": [
    "example"
  ],
  "excludePackages": [
    "example"
  ]
}

startedAt

Type: object | null

Time when the Spark application was started.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

status

Type: string

Current status of the Spark application.

Enum: STATUS_UNSPECIFIED, PROVISIONING, PENDING, RUNNING, ERROR, DONE, CANCELLED, CANCELLING

Example

{
  "id": "example",
  "clusterId": "example",
  "createdAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "startedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "finishedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "name": "example",
  "createdBy": "example",
  "status": "STATUS_UNSPECIFIED",
  "connectUrl": "example",
  "catalogs": [
    {
      "catalogId": "example"
    }
  ],
  "pysparkApplication": {
    "args": [
      "example"
    ],
    "jarFileUris": [
      "example"
    ],
    "fileUris": [
      "example"
    ],
    "archiveUris": [
      "example"
    ],
    "properties": {},
    "mainPythonFileUri": "example",
    "pythonFileUris": [
      "example"
    ],
    "packages": [
      "example"
    ],
    "repositories": [
      "example"
    ],
    "excludePackages": [
      "example"
    ]
  },
  "applicationSpec": "pysparkApplication"
}

Type: object

applicationSpec

Type: string

Const: sparkConnectApplication

Example: example

catalogs

Type: object[]

catalogId

Type: string

ID of the REST catalog.

Min length: 1

Max length: 50

Example: example

REST catalogs attached to the Spark application.

Example

[
  {
    "catalogId": "example"
  }
]

clusterId

Type: string

ID of the Spark cluster.

Min length: 1

Max length: 50

Example: example

connectUrl

Type: string

Spark Connect URL.

Example: example

createdAt

Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the Spark application was created.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

createdBy

Type: string

ID of the user who created the Spark application.

Min length: 1

Example: example

finishedAt

Type: object | null

Time when the Spark application was finished.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

Type: string

ID of the Spark application.

Min length: 1

Max length: 50

Example: example

name

Type: string

Name of the Spark application.

Example: example

sparkConnectApplication

Type: object

archiveUris

Type: string[]

Archives extracted in the Spark working directory.

Max items: 100

Example

[
  "example"
]

excludePackages

Type: string[]

Maven packages excluded during dependency resolution.

Max items: 100

Example

[
  "example"
]

fileUris

Type: string[]

Files copied to the Spark working directory.

Max items: 100

Example

[
  "example"
]

jarFileUris

Type: string[]

JAR files added to the Spark classpaths.

Max items: 100

Example

[
  "example"
]

packages

Type: string[]

Maven packages added to the Spark classpaths.

Max items: 100

Example

[
  "example"
]

properties

Type: object

[additional]

Type: string

Max length: 256

Example: example

Spark configuration properties.

Example

{}

repositories

Type: string[]

Additional Maven repositories.

Max items: 10

Example

[
  "example"
]

Example

{
  "jarFileUris": [
    "example"
  ],
  "fileUris": [
    "example"
  ],
  "archiveUris": [
    "example"
  ],
  "properties": {},
  "packages": [
    "example"
  ],
  "repositories": [
    "example"
  ],
  "excludePackages": [
    "example"
  ]
}

startedAt

Type: object | null

Time when the Spark application was started.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

status

Type: string

Current status of the Spark application.

Enum: STATUS_UNSPECIFIED, PROVISIONING, PENDING, RUNNING, ERROR, DONE, CANCELLED, CANCELLING

Example

{
  "id": "example",
  "clusterId": "example",
  "createdAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "startedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "finishedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "name": "example",
  "createdBy": "example",
  "status": "STATUS_UNSPECIFIED",
  "connectUrl": "example",
  "catalogs": [
    {
      "catalogId": "example"
    }
  ],
  "sparkConnectApplication": {
    "jarFileUris": [
      "example"
    ],
    "fileUris": [
      "example"
    ],
    "archiveUris": [
      "example"
    ],
    "properties": {},
    "packages": [
      "example"
    ],
    "repositories": [
      "example"
    ],
    "excludePackages": [
      "example"
    ]
  },
  "applicationSpec": "sparkConnectApplication"
}

Type: object

catalogs

Type: object[]

catalogId

Type: string

ID of the REST catalog.

Min length: 1

Max length: 50

Example: example

REST catalogs attached to the Spark application.

Example

[
  {
    "catalogId": "example"
  }
]

clusterId

Type: string

ID of the Spark cluster.

Min length: 1

Max length: 50

Example: example

connectUrl

Type: string

Spark Connect URL.

Example: example

createdAt

Type: object

seconds

Type: string

Number of seconds since the Unix epoch.

Example: example

nanos

Type: number

Fractional seconds in nanoseconds.

Time when the Spark application was created.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

createdBy

Type: string

ID of the user who created the Spark application.

Min length: 1

Example: example

finishedAt

Type: object | null

Time when the Spark application was finished.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

Type: string

ID of the Spark application.

Min length: 1

Max length: 50

Example: example

name

Type: string

Name of the Spark application.

Example: example

startedAt

Type: object | null

Time when the Spark application was started.

Example

{
  "seconds": "example",
  "nanos": 0.5
}

status

Type: string

Current status of the Spark application.

Enum: STATUS_UNSPECIFIED, PROVISIONING, PENDING, RUNNING, ERROR, DONE, CANCELLED, CANCELLING

Example

{
  "id": "example",
  "clusterId": "example",
  "createdAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "startedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "finishedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "name": "example",
  "createdBy": "example",
  "status": "STATUS_UNSPECIFIED",
  "connectUrl": "example",
  "catalogs": [
    {
      "catalogId": "example"
    }
  ]
}

Example

{
  "id": "example",
  "clusterId": "example",
  "createdAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "startedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "finishedAt": {
    "seconds": "example",
    "nanos": 0.5
  },
  "name": "example",
  "createdBy": "example",
  "status": "STATUS_UNSPECIFIED",
  "connectUrl": "example",
  "catalogs": [
    {
      "catalogId": "example"
    }
  ],
  "sparkApplication": {
    "args": [
      "example"
    ],
    "jarFileUris": [
      "example"
    ],
    "fileUris": [
      "example"
    ],
    "archiveUris": [
      "example"
    ],
    "properties": {},
    "mainJarFileUri": "example",
    "mainClass": "example",
    "packages": [
      "example"
    ],
    "repositories": [
      "example"
    ],
    "excludePackages": [
      "example"
    ]
  },
  "applicationSpec": "sparkApplication"
}

🚧 [Experimental] List Spark applications

RequestRequest

HeadersHeaders

BodyBody

ResponsesResponses

200 OK200 OK

BodyBody

SparkApplicationSparkApplication

Была ли статья полезна?

Request

Headers

Body

Responses

200 OK

Body

SparkApplication