DataSphere Jobs API v2, gRPC: ProjectJobService.Create

Written by

Yandex Cloud

Updated at October 17, 2024

gRPC request
CreateProjectJobRequest
JobParameters
File
FileDesc
Environment
DockerImageSpec
PythonEnv
PipOptions
CloudInstanceType
ExtendedWorkingStorage
Argument
OutputDatasetDesc
GracefulShutdownParameters
operation.Operation
CreateProjectJobMetadata
CreateProjectJobResponse
StorageFile
File
FileDesc

Creates job.

gRPC request

rpc Create (CreateProjectJobRequest) returns (operation.Operation)

CreateProjectJobRequest

{
  "projectId": "string",
  "jobParameters": {
    "inputFiles": [
      {
        "desc": {
          "path": "string",
          "var": "string"
        },
        "sha256": "string",
        "sizeBytes": "int64",
        "compressionType": "FileCompressionType"
      }
    ],
    "outputFiles": [
      {
        "path": "string",
        "var": "string"
      }
    ],
    "s3MountIds": [
      "string"
    ],
    "datasetIds": [
      "string"
    ],
    "cmd": "string",
    "env": {
      "vars": "string",
      // Includes only one of the fields `dockerImageResourceId`, `dockerImageSpec`
      "dockerImageResourceId": "string",
      "dockerImageSpec": {
        "imageUrl": "string",
        "username": "string",
        // Includes only one of the fields `passwordPlainText`, `passwordDsSecretName`
        "passwordPlainText": "string",
        "passwordDsSecretName": "string"
        // end of the list of possible fields
      },
      // end of the list of possible fields
      "pythonEnv": {
        "condaYaml": "string",
        "localModules": [
          {
            "desc": {
              "path": "string",
              "var": "string"
            },
            "sha256": "string",
            "sizeBytes": "int64",
            "compressionType": "FileCompressionType"
          }
        ],
        "pythonVersion": "string",
        "requirements": [
          "string"
        ],
        "pipOptions": {
          "indexUrl": "string",
          "extraIndexUrls": [
            "string"
          ],
          "trustedHosts": [
            "string"
          ],
          "noDeps": "bool"
        }
      }
    },
    "attachProjectDisk": "bool",
    "cloudInstanceTypes": [
      {
        "name": "string"
      }
    ],
    "extendedWorkingStorage": {
      "type": "StorageType",
      "sizeGb": "int64"
    },
    "arguments": [
      {
        "name": "string",
        "value": "string"
      }
    ],
    "outputDatasets": [
      {
        "name": "string",
        "description": "string",
        "labels": "string",
        "sizeGb": "int64",
        "var": "string"
      }
    ],
    "gracefulShutdownParameters": {
      "timeout": "google.protobuf.Duration",
      "signal": "int64"
    }
  },
  "config": "string",
  "name": "string",
  "desc": "string",
  "dataTtl": "google.protobuf.Duration"
}

Field	Description
projectId	string ID of the project.
jobParameters	JobParameters Parameters of the job.
config	string Config of the job.
name	string Name of the job.
desc	string Description of the job.
dataTtl	google.protobuf.Duration Job data TTL.

JobParameters

Job parameters.

Field	Description
inputFiles[]	File List of input files.
outputFiles[]	FileDesc List of output files descriptions.
s3MountIds[]	string List of DataSphere S3 mount ids.
datasetIds[]	string List of DataSphere dataset ids.
cmd	string Job run command.
env	Environment Job environment description.
attachProjectDisk	bool Should project disk be attached to VM.
cloudInstanceTypes[]	CloudInstanceType VM specification.
extendedWorkingStorage	ExtendedWorkingStorage Extended working storage configuration.
arguments[]	Argument List of literal arguments.
outputDatasets[]	OutputDatasetDesc List of DataSets descriptions to create.
gracefulShutdownParameters	GracefulShutdownParameters Graceful shutdown settings.

File

Field	Description
desc	FileDesc
sha256	string SHA256 of the file.
sizeBytes	int64 File size in bytes.
compressionType	enum FileCompressionType File compression info `FILE_COMPRESSION_TYPE_UNSPECIFIED` `NONE` `ZIP`

FileDesc

Field

Description

path

string

Path of the file on filesystem.

var

string

Variable to use in cmd substitution.

Environment

Field	Description
vars	string Environment variables.
dockerImageResourceId	string DS docker image id. Includes only one of the fields `dockerImageResourceId`, `dockerImageSpec`.
dockerImageSpec	DockerImageSpec Includes only one of the fields `dockerImageResourceId`, `dockerImageSpec`.
pythonEnv	PythonEnv

DockerImageSpec

Field	Description
imageUrl	string Docker image URL.
username	string Username for container registry.
passwordPlainText	string Plaintext password. Includes only one of the fields `passwordPlainText`, `passwordDsSecretName`. Password for container registry.
passwordDsSecretName	string ID of DataSphere secret containing password. Includes only one of the fields `passwordPlainText`, `passwordDsSecretName`. Password for container registry.

PythonEnv

Field	Description
condaYaml	string Conda YAML.
localModules[]	File List of local modules descriptions.
pythonVersion	string Python version reduced to major.minor
requirements[]	string List of pip requirements
pipOptions	PipOptions Pip install options

PipOptions

Field	Description
indexUrl	string --index-url option
extraIndexUrls[]	string --extra-index-urls option
trustedHosts[]	string --trusted-hosts option
noDeps	bool --no-deps option

CloudInstanceType

Field

Description

name

string

Name of DataSphere VM configuration.

ExtendedWorkingStorage

Extended working storage configuration.

Field

Description

type

enum StorageType

STORAGE_TYPE_UNSPECIFIED
SSD

sizeGb

int64

Argument

Field	Description
name	string
value	string

OutputDatasetDesc

Field	Description
name	string Name to create dataset with
description	string Description to show in UI
labels	string
sizeGb	int64 Size of dataset to create
var	string Var name to replace in cmd, like in FileDesc

GracefulShutdownParameters

Field

Description

timeout

google.protobuf.Duration

signal

int64

default 15 (SIGTERM)

operation.Operation

{
  "id": "string",
  "description": "string",
  "createdAt": "google.protobuf.Timestamp",
  "createdBy": "string",
  "modifiedAt": "google.protobuf.Timestamp",
  "done": "bool",
  "metadata": {
    "projectId": "string",
    "jobId": "string"
  },
  // Includes only one of the fields `error`, `response`
  "error": "google.rpc.Status",
  "response": {
    "jobId": "string",
    "uploadFiles": [
      {
        "file": {
          "desc": {
            "path": "string",
            "var": "string"
          },
          "sha256": "string",
          "sizeBytes": "int64",
          "compressionType": "FileCompressionType"
        },
        "url": "string"
      }
    ]
  }
  // end of the list of possible fields
}

An Operation resource. For more information, see Operation.

Field	Description
id	string ID of the operation.
description	string Description of the operation. 0-256 characters long.
createdAt	google.protobuf.Timestamp Creation timestamp.
createdBy	string ID of the user or service account who initiated the operation.
modifiedAt	google.protobuf.Timestamp The time when the Operation resource was last modified.
done	bool If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
metadata	CreateProjectJobMetadata Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
error	google.rpc.Status The error result of the operation in case of failure or cancellation. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.
response	CreateProjectJobResponse The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty. If the original method is the standard Create/Update, the response should be the target resource of the operation. Any method that returns a long-running operation should document the response type, if any. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.

CreateProjectJobMetadata

Field

Description

projectId

string

ID of the project.

jobId

string

Job ID.

CreateProjectJobResponse

Field

Description

jobId

string

ID of the job.

uploadFiles[]

StorageFile

Files to upload with their presigned URLs for upload.

StorageFile

Field

Description

file

File

url

string

File URL.

File

Field	Description
desc	FileDesc
sha256	string SHA256 of the file.
sizeBytes	int64 File size in bytes.
compressionType	enum FileCompressionType File compression info `FILE_COMPRESSION_TYPE_UNSPECIFIED` `NONE` `ZIP`

FileDesc

Field

Description

path

string

Path of the file on filesystem.

var

string

Variable to use in cmd substitution.

DataSphere Jobs API v2, gRPC: ProjectJobService.Create

gRPC requestgRPC request

CreateProjectJobRequestCreateProjectJobRequest

JobParametersJobParameters

FileFile

FileDescFileDesc

EnvironmentEnvironment

DockerImageSpecDockerImageSpec

PythonEnvPythonEnv

PipOptionsPipOptions

CloudInstanceTypeCloudInstanceType

ExtendedWorkingStorageExtendedWorkingStorage

ArgumentArgument

OutputDatasetDescOutputDatasetDesc

GracefulShutdownParametersGracefulShutdownParameters

operation.Operationoperation.Operation

CreateProjectJobMetadataCreateProjectJobMetadata

CreateProjectJobResponseCreateProjectJobResponse

StorageFileStorageFile

FileFile

FileDescFileDesc

Was the article helpful?

gRPC request

CreateProjectJobRequest

JobParameters

File

FileDesc

Environment

DockerImageSpec

PythonEnv

PipOptions

CloudInstanceType

ExtendedWorkingStorage

Argument

OutputDatasetDesc

GracefulShutdownParameters

operation.Operation

CreateProjectJobMetadata

CreateProjectJobResponse

StorageFile

File

FileDesc