Working with Docker images

You can configure the environment for running your code using Docker images.

Yandex DataSphere enables you to create repositories of Docker images and select an image to use in a project. The selected image will be used when running code in all project notebooks.

Creating a Docker imageCreating a Docker image

1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

2. In the top-right corner, click Create resource. In the pop-up window, select Docker.

3. Select the Docker image location:

   Container Registry

   DataSphere

   The image will be built in the Yandex Container Registry registry. Running a project with an image from the registry depends on the image size and takes more time. You cannot publish an image from the registry in the community.

   To build an image in Container Registry, the community administrator must specify the registry in community settings and enable a service agent.

   Fill out the fields as follows:

   • Tag: Image name and tag in <name>:<tag> format, e.g., tensorflow:1.0.0.

   • Project storage mounting point: Path to the directory whose contents you must have in the Docker image build context. The entire contents of this directory will take up space on the disk used to build your image. . is the root directory.

   • Description: Docker image description.

   • Docker template: Template with the minimum set of instructions you need to build a functional Docker image with the selected Python version.

   • Docker file: Instructions for creating a Docker image.

     Edit the contents of the field. For example, the following code will create a Docker image with python_3_8 based on the original TensorFlow image:

     FROM tensorflow/tensorflow:2.7.0-gpu
     
     RUN set -e \
       && useradd -ms /bin/bash --uid 1000 jupyter \
       && pip install --no-cache-dir --upgrade pip \
       && pip install --no-cache-dir nptyping==1.4.4 pandas==1.4.1 opencv-python-headless==4.5.5.62 scikit-learn==1.0.2 \
       && ln -s /usr/bin/python3 /usr/local/bin/python3
     

   The Docker image will be built and optimized for DataSphere projects. You can publish such a Docker image in the community. The resource will be placed in a dedicated storage you are going to pay for separately.

   Fill out the fields as follows:

   • Disk size, GB: Desired storage size in GB.

   • Docker name: Image name, e.g., tensorflow.

   • Tag: Image tag, e.g., 1.0.0.

   • Project storage mounting point: Path to the project directory where the created Docker image will reside; . indicates the root directory.

   • Description: Docker image description.

   • Docker template: Template of the script to install Python.

   • Docker file: Instructions for creating a Docker image.

     Edit the contents of the field. For example, the following code will create a Docker image with python_3_8 based on the original TensorFlow image:

     FROM tensorflow/tensorflow:2.7.0-gpu
     
     RUN set -e \
       && useradd -ms /bin/bash --uid 1000 jupyter \
       && pip install --no-cache-dir --upgrade pip \
       && pip install --no-cache-dir nptyping==1.4.4 pandas==1.4.1 opencv-python-headless==4.5.5.62 scikit-learn==1.0.2 \
       && ln -s /usr/bin/python3 /usr/local/bin/python3
     

4. Optionally, activate Authentication data and specify your Docker Hub account username and password.

5. Click Build.

   This will create a Docker image with TensorFlow packages to enable GPU-based computations.

6. Select the project in your community or on the DataSphere home page in the Recent projects tab.

7. Under Project resources, select Docker.

Applying a Docker image to a projectApplying a Docker image to a project

1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

2. Under Project resources, select Docker.

3. Next to the image in question, click and select Activate.

4. Open the project in JupyterLab and wait for it to load.

5. Open the notebook tab and make sure the custom image environment is available in your project. For example, for the TensorFlow image, create and run a cell with the following code:

   import tensorflow as tf
   tf.config.list_physical_devices('GPU')
   

6. Select the VM configuration. For example, to run a TensorFlow image, select a g1.1 GPU configuration. In the VM widget, the Environment field should give the full name of the image.

   Result:

   ...
   [PhysicalDevice(name='/physical_device:GPU:0', device_type='GPU')]
   

To return to the default environment, follow these steps:

1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

2. Under Project resources, select Docker.
3. Next to the default image, click and select Activate.

Sharing a Docker imageSharing a Docker image

A Docker image can be shared in the community only by the community administrator and only if the image is hosted in DataSphere.

To learn more about roles in DataSphere, see Access management in DataSphere.

1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

2. Under Project resources, click Docker.
3. Select the image from the list.
4. Go to the Access tab.
5. Enable the visibility option next to the name of the community you want to share the Docker image in.

To make a Docker image available for use in a different project, the project admin needs to add that image on the Shared tab.

Deleting a Docker imageDeleting a Docker image

You cannot delete a Docker image if it is available for the entire community.

1. Select the project in your community or on the DataSphere home page in the Recent projects tab.

2. Under Project resources, click Docker.
3. In the list of Docker images, select the one you want to delete.
4. Click and select Delete.
5. Click Confirm.

You will see a message saying that the resource has been deleted.