🚧 [Experimental] Create Spark application
Creates a Spark application and returns an asynchronous operation.
Request
POST
https://api.datalens.tech/rpc/createSparkApplication
Headers
|
Name |
Description |
|
x-dl-api-version |
Type: string API version header. Const: Example: |
Body
application/json
{
"clusterId": "example",
"name": "example",
"catalogs": [
{
"catalogId": "example"
}
],
"sparkApplication": {
"args": [
"example"
],
"jarFileUris": [
"example"
],
"fileUris": [
"example"
],
"archiveUris": [
"example"
],
"properties": {},
"mainJarFileUri": "example",
"mainClass": "example",
"packages": [
"example"
],
"repositories": [
"example"
],
"excludePackages": [
"example"
]
}
}
Any of 3 types
-
Type: object
clusterId
Type: string
ID of the Spark cluster.
Min length:
1Max length:
50Example:
examplesparkApplication
Type: object
mainJarFileUri
Type: string
URI of the JAR file that contains the main class.
Min length:
1Max length:
2047Example:
examplearchiveUris
Type: string[]
Archives extracted in the Spark working directory.
Max items:
100Example
[ "example" ]args
Type: string[]
Arguments passed to the Spark driver.
Max items:
100Example
[ "example" ]excludePackages
Type: string[]
Maven packages excluded during dependency resolution.
Max items:
100Example
[ "example" ]fileUris
Type: string[]
Files copied to the Spark working directory.
Max items:
100Example
[ "example" ]jarFileUris
Type: string[]
JAR files added to the Spark classpaths.
Max items:
100Example
[ "example" ]mainClass
Type: string
Name of the driver main class.
Max length:
255Example:
examplepackages
Type: string[]
Maven packages added to the Spark classpaths.
Max items:
100Example
[ "example" ]properties
Type: object
[additional]
Type: string
Max length:
256Example:
exampleSpark configuration properties.
Example
{}repositories
Type: string[]
Additional Maven repositories.
Max items:
10Example
[ "example" ]Example
{ "args": [ "example" ], "jarFileUris": [ "example" ], "fileUris": [ "example" ], "archiveUris": [ "example" ], "properties": {}, "mainJarFileUri": "example", "mainClass": "example", "packages": [ "example" ], "repositories": [ "example" ], "excludePackages": [ "example" ] }catalogs
Type: object[]
catalogId
Type: string
ID of the REST catalog.
Min length:
1Max length:
50Example:
exampleREST catalogs to attach to the Spark application.
Example
[ { "catalogId": "example" } ]name
Type: string
Name of the Spark application.
Max length:
255Pattern:
^(?:|[a-z][-a-z0-9]{1,61}[a-z0-9])$Example:
exampleExample
{ "clusterId": "example", "name": "example", "catalogs": [ { "catalogId": "example" } ], "sparkApplication": { "args": [ "example" ], "jarFileUris": [ "example" ], "fileUris": [ "example" ], "archiveUris": [ "example" ], "properties": {}, "mainJarFileUri": "example", "mainClass": "example", "packages": [ "example" ], "repositories": [ "example" ], "excludePackages": [ "example" ] } } -
Type: object
clusterId
Type: string
ID of the Spark cluster.
Min length:
1Max length:
50Example:
examplepysparkApplication
Type: object
mainPythonFileUri
Type: string
URI of the main Python file.
Min length:
1Max length:
2047Example:
examplearchiveUris
Type: string[]
Archives extracted in the Spark working directory.
Max items:
100Example
[ "example" ]args
Type: string[]
Arguments passed to the PySpark driver.
Max items:
100Example
[ "example" ]excludePackages
Type: string[]
Maven packages excluded during dependency resolution.
Max items:
100Example
[ "example" ]fileUris
Type: string[]
Files copied to the Spark working directory.
Max items:
100Example
[ "example" ]jarFileUris
Type: string[]
JAR files added to the Spark classpaths.
Max items:
100Example
[ "example" ]packages
Type: string[]
Maven packages added to the Spark classpaths.
Max items:
100Example
[ "example" ]properties
Type: object
[additional]
Type: string
Max length:
256Example:
exampleSpark configuration properties.
Example
{}pythonFileUris
Type: string[]
Additional Python files passed to PySpark.
Max items:
100Example
[ "example" ]repositories
Type: string[]
Additional Maven repositories.
Max items:
10Example
[ "example" ]Example
{ "args": [ "example" ], "jarFileUris": [ "example" ], "fileUris": [ "example" ], "archiveUris": [ "example" ], "properties": {}, "mainPythonFileUri": "example", "pythonFileUris": [ "example" ], "packages": [ "example" ], "repositories": [ "example" ], "excludePackages": [ "example" ] }catalogs
Type: object[]
catalogId
Type: string
ID of the REST catalog.
Min length:
1Max length:
50Example:
exampleREST catalogs to attach to the Spark application.
Example
[ { "catalogId": "example" } ]name
Type: string
Name of the Spark application.
Max length:
255Pattern:
^(?:|[a-z][-a-z0-9]{1,61}[a-z0-9])$Example:
exampleExample
{ "clusterId": "example", "name": "example", "catalogs": [ { "catalogId": "example" } ], "pysparkApplication": { "args": [ "example" ], "jarFileUris": [ "example" ], "fileUris": [ "example" ], "archiveUris": [ "example" ], "properties": {}, "mainPythonFileUri": "example", "pythonFileUris": [ "example" ], "packages": [ "example" ], "repositories": [ "example" ], "excludePackages": [ "example" ] } } -
Type: object
clusterId
Type: string
ID of the Spark cluster.
Min length:
1Max length:
50Example:
examplesparkConnectApplication
Type: object
archiveUris
Type: string[]
Archives extracted in the Spark working directory.
Max items:
100Example
[ "example" ]excludePackages
Type: string[]
Maven packages excluded during dependency resolution.
Max items:
100Example
[ "example" ]fileUris
Type: string[]
Files copied to the Spark working directory.
Max items:
100Example
[ "example" ]jarFileUris
Type: string[]
JAR files added to the Spark classpaths.
Max items:
100Example
[ "example" ]packages
Type: string[]
Maven packages added to the Spark classpaths.
Max items:
100Example
[ "example" ]properties
Type: object
[additional]
Type: string
Max length:
256Example:
exampleSpark configuration properties.
Example
{}repositories
Type: string[]
Additional Maven repositories.
Max items:
10Example
[ "example" ]Example
{ "jarFileUris": [ "example" ], "fileUris": [ "example" ], "archiveUris": [ "example" ], "properties": {}, "packages": [ "example" ], "repositories": [ "example" ], "excludePackages": [ "example" ] }catalogs
Type: object[]
catalogId
Type: string
ID of the REST catalog.
Min length:
1Max length:
50Example:
exampleREST catalogs to attach to the Spark application.
Example
[ { "catalogId": "example" } ]name
Type: string
Name of the Spark application.
Max length:
255Pattern:
^(?:|[a-z][-a-z0-9]{1,61}[a-z0-9])$Example:
exampleExample
{ "clusterId": "example", "name": "example", "catalogs": [ { "catalogId": "example" } ], "sparkConnectApplication": { "jarFileUris": [ "example" ], "fileUris": [ "example" ], "archiveUris": [ "example" ], "properties": {}, "packages": [ "example" ], "repositories": [ "example" ], "excludePackages": [ "example" ] } }
Responses
200 OK
Response
Body
application/json
{
"id": "example",
"description": "example",
"createdAt": {
"seconds": "example",
"nanos": 0.5
},
"createdBy": "example",
"modifiedAt": {
"seconds": "example",
"nanos": 0.5
},
"done": true,
"metadata": {},
"error": {
"code": 0.5,
"message": "example",
"details": [
null
]
},
"response": {}
}
|
Name |
Description |
||||||
|
done |
Type: boolean Whether the operation has completed. |
||||||
|
id |
Type: string Unique ID of the operation. Example: |
||||||
|
metadata |
Type: object
Service-specific operation metadata. Example |
||||||
|
createdAt |
Type: object
Time when the operation was created. Example |
||||||
|
createdBy |
Type: string ID of the operation creator. Example: |
||||||
|
description |
Type: string Description of the operation. Example: |
||||||
|
error |
Type: object
Operation error, if the operation failed. Example |
||||||
|
modifiedAt |
Type: object
Time when the operation was last modified. Example |
||||||
|
response |
Type: object
Service-specific operation response. Example |