DataSphere Jobs API v2, gRPC: ProjectJobService.Create

Written by

Yandex Cloud

Updated at December 17, 2024

gRPC request
CreateProjectJobRequest
JobParameters
File
FileDesc
Environment
DockerImageSpec
PythonEnv
PipOptions
CloudInstanceType
ExtendedWorkingStorage
Argument
OutputDatasetDesc
GracefulShutdownParameters
SparkParameters
operation.Operation
CreateProjectJobMetadata
CreateProjectJobResponse
StorageFile
File
FileDesc

Creates job.

gRPC request

rpc Create (CreateProjectJobRequest) returns (operation.Operation)

CreateProjectJobRequest

{
  "project_id": "string",
  "job_parameters": {
    "input_files": [
      {
        "desc": {
          "path": "string",
          "var": "string"
        },
        "sha256": "string",
        "size_bytes": "int64",
        "compression_type": "FileCompressionType"
      }
    ],
    "output_files": [
      {
        "path": "string",
        "var": "string"
      }
    ],
    "s3_mount_ids": [
      "string"
    ],
    "dataset_ids": [
      "string"
    ],
    "cmd": "string",
    "env": {
      "vars": "map<string, string>",
      // Includes only one of the fields `docker_image_resource_id`, `docker_image_spec`
      "docker_image_resource_id": "string",
      "docker_image_spec": {
        "image_url": "string",
        "username": "string",
        // Includes only one of the fields `password_plain_text`, `password_ds_secret_name`
        "password_plain_text": "string",
        "password_ds_secret_name": "string"
        // end of the list of possible fields
      },
      // end of the list of possible fields
      "python_env": {
        "conda_yaml": "string",
        "local_modules": [
          {
            "desc": {
              "path": "string",
              "var": "string"
            },
            "sha256": "string",
            "size_bytes": "int64",
            "compression_type": "FileCompressionType"
          }
        ],
        "python_version": "string",
        "requirements": [
          "string"
        ],
        "pip_options": {
          "index_url": "string",
          "extra_index_urls": [
            "string"
          ],
          "trusted_hosts": [
            "string"
          ],
          "no_deps": "bool"
        }
      }
    },
    "attach_project_disk": "bool",
    "cloud_instance_types": [
      {
        "name": "string"
      }
    ],
    "extended_working_storage": {
      "type": "StorageType",
      "size_gb": "int64"
    },
    "arguments": [
      {
        "name": "string",
        "value": "string"
      }
    ],
    "output_datasets": [
      {
        "name": "string",
        "description": "string",
        "labels": "map<string, string>",
        "size_gb": "int64",
        "var": "string"
      }
    ],
    "graceful_shutdown_parameters": {
      "timeout": "google.protobuf.Duration",
      "signal": "int64"
    },
    "spark_parameters": {
      "connector_id": "string"
    }
  },
  "config": "string",
  "name": "string",
  "desc": "string",
  "data_ttl": "google.protobuf.Duration"
}

Field	Description
project_id	string ID of the project.
job_parameters	JobParameters Parameters of the job.
config	string Config of the job.
name	string Name of the job.
desc	string Description of the job.
data_ttl	google.protobuf.Duration Job data TTL.

JobParameters

Job parameters.

Field	Description
input_files[]	File List of input files.
output_files[]	FileDesc List of output files descriptions.
s3_mount_ids[]	string List of DataSphere S3 mount ids.
dataset_ids[]	string List of DataSphere dataset ids.
cmd	string Job run command.
env	Environment Job environment description.
attach_project_disk	bool Should project disk be attached to VM.
cloud_instance_types[]	CloudInstanceType VM specification.
extended_working_storage	ExtendedWorkingStorage Extended working storage configuration.
arguments[]	Argument List of literal arguments.
output_datasets[]	OutputDatasetDesc List of DataSets descriptions to create.
graceful_shutdown_parameters	GracefulShutdownParameters Graceful shutdown settings.
spark_parameters	SparkParameters Spark connector settings.

File

Field	Description
desc	FileDesc
sha256	string SHA256 of the file.
size_bytes	int64 File size in bytes.
compression_type	enum FileCompressionType File compression info `FILE_COMPRESSION_TYPE_UNSPECIFIED` `NONE` `ZIP`

FileDesc

Field

Description

path

string

Path of the file on filesystem.

var

string

Variable to use in cmd substitution.

Environment

Field	Description
vars	object (map<string, string>) Environment variables.
docker_image_resource_id	string DS docker image id. Includes only one of the fields `docker_image_resource_id`, `docker_image_spec`.
docker_image_spec	DockerImageSpec Includes only one of the fields `docker_image_resource_id`, `docker_image_spec`.
python_env	PythonEnv

DockerImageSpec

Field	Description
image_url	string Docker image URL.
username	string Username for container registry.
password_plain_text	string Plaintext password. Includes only one of the fields `password_plain_text`, `password_ds_secret_name`. Password for container registry.
password_ds_secret_name	string ID of DataSphere secret containing password. Includes only one of the fields `password_plain_text`, `password_ds_secret_name`. Password for container registry.

PythonEnv

Field	Description
conda_yaml	string Conda YAML.
local_modules[]	File List of local modules descriptions.
python_version	string Python version reduced to major.minor
requirements[]	string List of pip requirements
pip_options	PipOptions Pip install options

PipOptions

Field	Description
index_url	string --index-url option
extra_index_urls[]	string --extra-index-urls option
trusted_hosts[]	string --trusted-hosts option
no_deps	bool --no-deps option

CloudInstanceType

Field

Description

name

string

Name of DataSphere VM configuration.

ExtendedWorkingStorage

Extended working storage configuration.

Field

Description

type

enum StorageType

STORAGE_TYPE_UNSPECIFIED
SSD

size_gb

int64

Argument

Field	Description
name	string
value	string

OutputDatasetDesc

Field	Description
name	string Name to create dataset with
description	string Description to show in UI
labels	object (map<string, string>)
size_gb	int64 Size of dataset to create
var	string Var name to replace in cmd, like in FileDesc

GracefulShutdownParameters

Field

Description

timeout

google.protobuf.Duration

signal

int64

default 15 (SIGTERM)

SparkParameters

Field

Description

connector_id

string

ID of the Spark connector.

operation.Operation

{
  "id": "string",
  "description": "string",
  "created_at": "google.protobuf.Timestamp",
  "created_by": "string",
  "modified_at": "google.protobuf.Timestamp",
  "done": "bool",
  "metadata": {
    "project_id": "string",
    "job_id": "string"
  },
  // Includes only one of the fields `error`, `response`
  "error": "google.rpc.Status",
  "response": {
    "job_id": "string",
    "upload_files": [
      {
        "file": {
          "desc": {
            "path": "string",
            "var": "string"
          },
          "sha256": "string",
          "size_bytes": "int64",
          "compression_type": "FileCompressionType"
        },
        "url": "string"
      }
    ]
  }
  // end of the list of possible fields
}

An Operation resource. For more information, see Operation.

Field	Description
id	string ID of the operation.
description	string Description of the operation. 0-256 characters long.
created_at	google.protobuf.Timestamp Creation timestamp.
created_by	string ID of the user or service account who initiated the operation.
modified_at	google.protobuf.Timestamp The time when the Operation resource was last modified.
done	bool If the value is `false`, it means the operation is still in progress. If `true`, the operation is completed, and either `error` or `response` is available.
metadata	CreateProjectJobMetadata Service-specific metadata associated with the operation. It typically contains the ID of the target resource that the operation is performed on. Any method that returns a long-running operation should document the metadata type, if any.
error	google.rpc.Status The error result of the operation in case of failure or cancellation. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.
response	CreateProjectJobResponse The normal response of the operation in case of success. If the original method returns no data on success, such as Delete, the response is google.protobuf.Empty. If the original method is the standard Create/Update, the response should be the target resource of the operation. Any method that returns a long-running operation should document the response type, if any. Includes only one of the fields `error`, `response`. The operation result. If `done == false` and there was no failure detected, neither `error` nor `response` is set. If `done == false` and there was a failure detected, `error` is set. If `done == true`, exactly one of `error` or `response` is set.

CreateProjectJobMetadata

Field

Description

project_id

string

ID of the project.

job_id

string

Job ID.

CreateProjectJobResponse

Field

Description

job_id

string

ID of the job.

upload_files[]

StorageFile

Files to upload with their presigned URLs for upload.

StorageFile

Field

Description

file

File

url

string

File URL.

File

Field	Description
desc	FileDesc
sha256	string SHA256 of the file.
size_bytes	int64 File size in bytes.
compression_type	enum FileCompressionType File compression info `FILE_COMPRESSION_TYPE_UNSPECIFIED` `NONE` `ZIP`

FileDesc

Field

Description

path

string

Path of the file on filesystem.

var

string

Variable to use in cmd substitution.

DataSphere Jobs API v2, gRPC: ProjectJobService.Create

gRPC requestgRPC request

CreateProjectJobRequestCreateProjectJobRequest

JobParametersJobParameters

FileFile

FileDescFileDesc

EnvironmentEnvironment

DockerImageSpecDockerImageSpec

PythonEnvPythonEnv

PipOptionsPipOptions

CloudInstanceTypeCloudInstanceType

ExtendedWorkingStorageExtendedWorkingStorage

ArgumentArgument

OutputDatasetDescOutputDatasetDesc

GracefulShutdownParametersGracefulShutdownParameters

SparkParametersSparkParameters

operation.Operationoperation.Operation

CreateProjectJobMetadataCreateProjectJobMetadata

CreateProjectJobResponseCreateProjectJobResponse

StorageFileStorageFile

FileFile

FileDescFileDesc

Was the article helpful?

gRPC request

CreateProjectJobRequest

JobParameters

File

FileDesc

Environment

DockerImageSpec

PythonEnv

PipOptions

CloudInstanceType

ExtendedWorkingStorage

Argument

OutputDatasetDesc

GracefulShutdownParameters

SparkParameters

operation.Operation

CreateProjectJobMetadata

CreateProjectJobResponse

StorageFile

File

FileDesc