DataSphere Jobs API v2, gRPC: ProjectJobService.Execute

Статья создана

Yandex Cloud

Обновлена 17 октября 2024 г.

gRPC request
ExecuteProjectJobRequest
operation.Operation
ExecuteProjectJobMetadata
Job
JobParameters
File
FileDesc
Environment
DockerImageSpec
PythonEnv
PipOptions
CloudInstanceType
ExtendedWorkingStorage
Argument
OutputDatasetDesc
GracefulShutdownParameters
FileUploadError
OutputDataset
ExecuteProjectJobResponse
StorageFile
JobResult

Runs job execution.

gRPC request

rpc Execute (ExecuteProjectJobRequest) returns (operation.Operation)

ExecuteProjectJobRequest

{
  "jobId": "string"
}

Field

Description

jobId

string

ID of the job.

operation.Operation

{
  "id": "string",
  "description": "string",
  "createdAt": "google.protobuf.Timestamp",
  "createdBy": "string",
  "modifiedAt": "google.protobuf.Timestamp",
  "done": "bool",
  "metadata": {
    "job": {
      "id": "string",
      "name": "string",
      "desc": "string",
      "createdAt": "google.protobuf.Timestamp",
      "finishedAt": "google.protobuf.Timestamp",
      "status": "JobStatus",
      "config": "string",
      "createdById": "string",
      "projectId": "string",
      "jobParameters": {
        "inputFiles": [
          {
            "desc": {
              "path": "string",
              "var": "string"
            },
            "sha256": "string",
            "sizeBytes": "int64",
            "compressionType": "FileCompressionType"
          }
        ],
        "outputFiles": [
          {
            "path": "string",
            "var": "string"
          }
        ],
        "s3MountIds": [
          "string"
        ],
        "datasetIds": [
          "string"
        ],
        "cmd": "string",
        "env": {
          "vars": "string",
          // Includes only one of the fields `dockerImageResourceId`, `dockerImageSpec`
          "dockerImageResourceId": "string",
          "dockerImageSpec": {
            "imageUrl": "string",
            "username": "string",
            // Includes only one of the fields `passwordPlainText`, `passwordDsSecretName`
            "passwordPlainText": "string",
            "passwordDsSecretName": "string"
            // end of the list of possible fields
          },
          // end of the list of possible fields
          "pythonEnv": {
            "condaYaml": "string",
            "localModules": [
              {
                "desc": {
                  "path": "string",
                  "var": "string"
                },
                "sha256": "string",
                "sizeBytes": "int64",
                "compressionType": "FileCompressionType"
              }
            ],
            "pythonVersion": "string",
            "requirements": [
              "string"
            ],
            "pipOptions": {
              "indexUrl": "string",
              "extraIndexUrls": [
                "string"
              ],
              "trustedHosts": [
                "string"
              ],
              "noDeps": "bool"
            }
          }
        },
        "attachProjectDisk": "bool",
        "cloudInstanceTypes": [
          {
            "name": "string"
          }
        ],
        "extendedWorkingStorage": {
          "type": "StorageType",
          "sizeGb": "int64"
        },
        "arguments": [
          {
            "name": "string",
            "value": "string"
          }
        ],
        "outputDatasets": [
          {
            "name": "string",
            "description": "string",
            "labels": "string",
            "sizeGb": "int64",
            "var": "string"
          }
        ],
        "gracefulShutdownParameters": {
          "timeout": "google.protobuf.Duration",
          "signal": "int64"
        }
      },
      "dataExpiresAt": "google.protobuf.Timestamp",
      "dataCleared": "bool",
      "outputFiles": [
        {
          "desc": {
            "path": "string",
            "var": "string"
          },
          "sha256": "string",
          "sizeBytes": "int64",
          "compressionType": "FileCompressionType"
        }
      ],
      "logFiles": [
        {
          "desc": {
            "path": "string",
            "var": "string"
          },
          "sha256": "string",
          "sizeBytes": "int64",
          "compressionType": "FileCompressionType"
        }
      ],
      "diagnosticFiles": [
        {
          "desc": {
            "path": "string",
            "var": "string"
          },
          "sha256": "string",
          "sizeBytes": "int64",
          "compressionType": "FileCompressionType"
        }
      ],
      "dataSizeBytes": "int64",
      "startedAt": "google.protobuf.Timestamp",
      "statusDetails": "string",
      "actualCloudInstanceType": {
        "name": "string"
      },
      "parentJobId": "string",
      "fileErrors": [
        {
          // Includes only one of the fields `outputFileDesc`, `logFileName`
          "outputFileDesc": {
            "path": "string",
            "var": "string"
          },
          "logFileName": "string",
          // end of the list of possible fields
          "description": "string"
        }
      ],
      "outputDatasets": [
        {
          "desc": {
            "name": "string",
            "description": "string",
            "labels": "string",
            "sizeGb": "int64",
            "var": "string"
          },
          "id": "string"
        }
      ]
    }
  },
  // Includes only one of the fields `error`, `response`
  "error": "google.rpc.Status",
  "response": {
    "outputFiles": [
      {
        "file": {
          "desc": {
            "path": "string",
            "var": "string"
          },
          "sha256": "string",
          "sizeBytes": "int64",
          "compressionType": "FileCompressionType"
        },
        "url": "string"
      }
    ],
    "outputDatasets": [
      {
        "desc": {
          "name": "string",
          "description": "string",
          "labels": "string",
          "sizeGb": "int64",
          "var": "string"
        },
        "id": "string"
      }
    ],
    "result": {
      "returnCode": "int64"
    }
  }
  // end of the list of possible fields
}

An Operation resource. For more information, see Operation.

Field	Description
id	string ID of the operation.
description	string Description of the operation. 0-256 characters long.
createdAt	google.protobuf.Timestamp Creation timestamp.
createdBy	string ID of the user or service account who initiated the operation.
modifiedAt	google.protobuf.Timestamp The time when the Operation resource was last modified.
done	bool If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
metadata	ExecuteProjectJobMetadata Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
error	google.rpc.Status The error result of the operation in case of failure or cancellation. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.
response	ExecuteProjectJobResponse The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty. If the original method is the standard Create/Update, the response should be the target resource of the operation. Any method that returns a long-running operation should document the response type, if any. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.

ExecuteProjectJobMetadata

Field

Description

job

Job

Instance of the job.

Job

Instance of the job.

Field	Description
id	string ID of the job.
name	string Name of the job.
desc	string Description of the job.
createdAt	google.protobuf.Timestamp Create job timestamp.
finishedAt	google.protobuf.Timestamp Finish job timestamp.
status	enum JobStatus Status of the job. `JOB_STATUS_UNSPECIFIED` `CREATING` `EXECUTING` `UPLOADING_OUTPUT` `SUCCESS` `ERROR` `CANCELLED` `CANCELLING`
config	string Config of the job, copied from configuration file.
createdById	string ID of the user who created the job.
projectId	string ID of the project.
jobParameters	JobParameters
dataExpiresAt	google.protobuf.Timestamp Job data expiration timestamp.
dataCleared	bool Marks if the job data has been cleared.
outputFiles[]	File Output files of the job.
logFiles[]	File Job log files.
diagnosticFiles[]	File Job diagnostics files.
dataSizeBytes	int64 Job total data size.
startedAt	google.protobuf.Timestamp Start job timestamp.
statusDetails	string Details.
actualCloudInstanceType	CloudInstanceType Actual VM instance type, job is running on.
parentJobId	string Reference to the parent job.
fileErrors[]	FileUploadError Failed uploads.
outputDatasets[]	OutputDataset Created datasets.

JobParameters

Job parameters.

Field	Description
inputFiles[]	File List of input files.
outputFiles[]	FileDesc List of output files descriptions.
s3MountIds[]	string List of DataSphere S3 mount ids.
datasetIds[]	string List of DataSphere dataset ids.
cmd	string Job run command.
env	Environment Job environment description.
attachProjectDisk	bool Should project disk be attached to VM.
cloudInstanceTypes[]	CloudInstanceType VM specification.
extendedWorkingStorage	ExtendedWorkingStorage Extended working storage configuration.
arguments[]	Argument List of literal arguments.
outputDatasets[]	OutputDatasetDesc List of DataSets descriptions to create.
gracefulShutdownParameters	GracefulShutdownParameters Graceful shutdown settings.

File

Field	Description
desc	FileDesc
sha256	string SHA256 of the file.
sizeBytes	int64 File size in bytes.
compressionType	enum FileCompressionType File compression info `FILE_COMPRESSION_TYPE_UNSPECIFIED` `NONE` `ZIP`

FileDesc

Field

Description

path

string

Path of the file on filesystem.

var

string

Variable to use in cmd substitution.

Environment

Field	Description
vars	string Environment variables.
dockerImageResourceId	string DS docker image id. Includes only one of the fields `dockerImageResourceId`, `dockerImageSpec`.
dockerImageSpec	DockerImageSpec Includes only one of the fields `dockerImageResourceId`, `dockerImageSpec`.
pythonEnv	PythonEnv

DockerImageSpec

Field	Description
imageUrl	string Docker image URL.
username	string Username for container registry.
passwordPlainText	string Plaintext password. Includes only one of the fields `passwordPlainText`, `passwordDsSecretName`. Password for container registry.
passwordDsSecretName	string ID of DataSphere secret containing password. Includes only one of the fields `passwordPlainText`, `passwordDsSecretName`. Password for container registry.

PythonEnv

Field	Description
condaYaml	string Conda YAML.
localModules[]	File List of local modules descriptions.
pythonVersion	string Python version reduced to major.minor
requirements[]	string List of pip requirements
pipOptions	PipOptions Pip install options

PipOptions

Field	Description
indexUrl	string --index-url option
extraIndexUrls[]	string --extra-index-urls option
trustedHosts[]	string --trusted-hosts option
noDeps	bool --no-deps option

CloudInstanceType

Field

Description

name

string

Name of DataSphere VM configuration.

ExtendedWorkingStorage

Extended working storage configuration.

Field

Description

type

enum StorageType

STORAGE_TYPE_UNSPECIFIED
SSD

sizeGb

int64

Argument

Field	Description
name	string
value	string

OutputDatasetDesc

Field	Description
name	string Name to create dataset with
description	string Description to show in UI
labels	string
sizeGb	int64 Size of dataset to create
var	string Var name to replace in cmd, like in FileDesc

GracefulShutdownParameters

Field

Description

timeout

google.protobuf.Duration

signal

int64

default 15 (SIGTERM)

FileUploadError

Field	Description
outputFileDesc	FileDesc Includes only one of the fields `outputFileDesc`, `logFileName`.
logFileName	string Includes only one of the fields `outputFileDesc`, `logFileName`.
description	string

OutputDataset

Field

Description

desc

OutputDatasetDesc

Dataset description

string

Id of created dataset

ExecuteProjectJobResponse

Field	Description
outputFiles[]	StorageFile Uploaded output files with URLs.
outputDatasets[]	OutputDataset Created datasets
result	JobResult Result of the job.

StorageFile

Field

Description

file

File

url

string

File URL.

JobResult

Field

Description

returnCode

int64

Execution return code.

DataSphere Jobs API v2, gRPC: ProjectJobService.Execute

gRPC requestgRPC request

ExecuteProjectJobRequestExecuteProjectJobRequest

operation.Operationoperation.Operation

ExecuteProjectJobMetadataExecuteProjectJobMetadata

JobJob

JobParametersJobParameters

FileFile

FileDescFileDesc

EnvironmentEnvironment

DockerImageSpecDockerImageSpec

PythonEnvPythonEnv

PipOptionsPipOptions

CloudInstanceTypeCloudInstanceType

ExtendedWorkingStorageExtendedWorkingStorage

ArgumentArgument

OutputDatasetDescOutputDatasetDesc

GracefulShutdownParametersGracefulShutdownParameters

FileUploadErrorFileUploadError

OutputDatasetOutputDataset

ExecuteProjectJobResponseExecuteProjectJobResponse

StorageFileStorageFile

JobResultJobResult

Была ли статья полезна?

gRPC request

ExecuteProjectJobRequest

operation.Operation

ExecuteProjectJobMetadata

Job

JobParameters

File

FileDesc

Environment

DockerImageSpec

PythonEnv

PipOptions

CloudInstanceType

ExtendedWorkingStorage

Argument

OutputDatasetDesc

GracefulShutdownParameters

FileUploadError

OutputDataset

ExecuteProjectJobResponse

StorageFile

JobResult