DataSphere CLI

You run DataSphere Jobs jobs using the DataSphere CLI utility.

To install DataSphere CLI, in a Python virtual environment, run the following command:

pip install datasphere

As soon as the installation is complete, you can view help by running this command with the -h flag:

datasphere -h

Command result:

usage: datasphere [-h] [-t TOKEN] [-l {ERROR,WARNING,INFO,DEBUG}] [--log-config LOG_CONFIG] {version,project} ...

positional arguments:
  {version,project}
    version             Show version

options:
  -h, --help            show this help message and exit
  -t TOKEN, --token TOKEN
                        YC OAuth token, see https://yandex.cloud/en/docs/iam/concepts/authorization/oauth-token
  -l {ERROR,WARNING,INFO,DEBUG}, --log-level {ERROR,WARNING,INFO,DEBUG}
                        Logging level
  --log-config LOG_CONFIG
                        Custom logging config

DataSphere CLI commandsDataSphere CLI commands

You can manage jobs using the datasphere project job section commands:

• Running and restoring a job session.
• Viewing job information.
• Canceling a job.

Running jobsRunning jobs

To run the job, run the following command:

datasphere project job execute -p <project_ID> -c <configuration_file>

Where:

• <project_ID>: ID of the DataSphere project you will run the job in.
• <configuration_file>: Path to the job configuration file.

Running a job locks the shell session until the job completes. The job code operation logs will be output to the standard stdout output and stderr error streams. The job execution system logs will be written to a separate file in the user's working directory.

If the shell session is interrupted during job execution, the job will continue to run in DataSphere, but the execution logs will not be saved. To resume logging, recover the session by running the following command:

datasphere project job attach --id <job_ID>

You can find out the job ID in the jobs widget of the DataSphere UI on the project page.

Tracking and logging will resume after the job session is restored.

Viewing job informationViewing job information

You can view all past and current project jobs by running the following command:

datasphere project job list -p <project_ID>

The response will return a table with the following fields:

• Job ID.
• Name.
• Description.
• Status.
• Job start and end date (if already completed).
• Name of the user who ran the job.

To view information about a specific job, run the following command:

datasphere project job get --id <job_ID>

Canceling a jobCanceling a job

You can stop and cancel a job in two ways:

1. If you have a shell session running with a job in progress, click Ctrl + C.

2. If you want to stop a job that is not related with an active shell session, run the following command:

   datasphere project job cancel --id <job_ID>
   

The running job will be stopped.

Job logsJob logs

When running a job through DataSphere CLI, the shell first notifies the user to save the logs in the user's working directory. For example:

2024-05-16 12:42:35,447 - [INFO] - logs file path: C:\Temp\datasphere\job_2024-05-16T12-42-35.427056

After running the job, you can find the following files in the user's working directory:

• stdout.txt: Standard output stream of the user program.
• stderr.txt: Standard error message stream.
• system.log: System log of the VM configuration and environment package installation.
• log.txt: General DataSphere CLI log which records the progress of the job.
• docker_stats.tsv: Log of the resources consumed by the Docker image, such as utilized CPU power, read and write speeds, used RAM, and boot speed. You can also get this information by running the docker stats command.
• gpu_stats.tsv: Log of GPU utilization, which includes the number of cores, utilized power, and video memory.

To change the directory for storing logs, use the following command:

datasphere --log-dir <new_directory>