Yandex DataSphere release notes

This section shows what changed in Yandex DataSphere.

Release as of 11/11/24Release as of 11/11/24

Now you can use a service agent to work with Yandex Cloud services from DataSphere notebooks, e.g., issue authorization tokens. To enable this feature in a community, follow this guide. For more information on how service agents work, see the Identity and Access Management documentation.

Release as of 02/09/24Release as of 02/09/24

• Added examples of operations with YandexART and open-source foundational models to initial notebooks.
• When working with Yandex Data Processing using a Spark connector, you can now synchronize the environment
• Fixed some bugs and added minor performance improvements.

Release as of 30/07/24Release as of 30/07/24

• Added a feature to create communities in different availability zones: ru-central1-a and ru-central1-b.
• You can now connect to your nodes an additional disk of 10 to 4,096 GB.
• Fixed some bugs and added minor performance improvements.

Release as of 04/07/24Release as of 04/07/24

• DataSphere projects now have a new type of resources: Spark connectors for integration with Yandex Data Processing.
• Improved creating nodes.
• Improved linking a billing account to a community.
• Fixed some bugs and added minor performance improvements.

Release as of 18/06/24Release as of 18/06/24

• Now you can deploy node instances in different availability zones: ru-central1-a and ru-central1-b.
• Now you can rerun jobs.
• Python 3.7 is no longer supported.
• Fixed some bugs and added minor performance improvements.

Release as of 03/04/24Release as of 03/04/24

1. Updated configurations of Yandex Data Processing temporary clusters.
2. Now you can use XGBoost and LightGBM models to deploy nodes from models.
3. Added delivering input variables in fulfillment APIs.
4. Improved creating nodes from Docker images.
5. Fixed some bugs and added minor performance improvements.

Release as of 27/03/24Release as of 27/03/24

Model tuning in DataSphere now works based on the new YandexGPT Pro model.

Release as of 01/03/24Release as of 01/03/24

The Serverless mode is no longer supported.

Release as of 27/02/24Release as of 27/02/24

1. Added the option to run a notebook in Dedicated mode to the API.
2. Improved logs and metrics for nodes.
3. Fixed bugs and added minor improvements in platform performance.

Release as of 29/01/24Release as of 29/01/24

1. Updated the NVIDIA driver to version 535.
2. Added support for multi-login to multiple organizations in various federations.
3. Added the option to pause and resume a running node.
4. Fixed bugs and added minor improvements in platform performance.

Release as of 15/01/24Release as of 15/01/24

1. Added self-service problem-solving tools to the project page.
2. In DataSphere Jobs, now you can use your project resources: secrets, S3 connectors, environment dockers, datasets, and project disk.
3. Fixed bugs and added minor improvements in platform performance.

Release as of 20/12/23Release as of 20/12/23

1. Added new configuration, gt4.1 (1 GPU NVIDIA T4).
2. The g2.mig configuration (1 GPU MIG NVIDIA Ampere A100) is obsolete.
3. A new node type from the model resource is available.
4. Selecting a configuration in Dedicated mode will display its current availability.
5. Fixed bugs and added minor improvements in platform performance.

Release as of 10/10/23Release as of 10/10/23

1. You can test fine-tuned YandexGPT models right in DataSphere. YandexGPT Playground in DataSphere is available after fine-tuning to users with access to YandexGPT API.
2. You can now connect your DataSphere project to JupyterLab from a local IDE.
3. Fixed bugs and added minor improvements in platform performance.

Release as of 25/09/23Release as of 25/09/23

1. With DataSphere Jobs, cloud computing resources in DataSphere can now be used from a user's local environment.
2. DataSphere projects now have a new type of resources: Models.
3. Optimized JupyterLab 3 (available in dedicated mode) by adding new extensions.
4. YandexGPT model tuning is now available at the Preview stage.
5. Fixed bugs and added minor improvements in platform performance.

Release as of 18/09/23Release as of 18/09/23

1. A new DS Default (Python 3.10) system image is used by default.
2. Community administrators can now manage permissions to use the functionality.
3. Improved working with community and project lists.
4. Fixed bugs and added minor improvements in platform performance.

Release as of 21/07/23Release as of 21/07/23

1. Updated the JupyterLab extension for working with GIT.
2. Community administrators can now manage permissions to use computing resources.
3. Community and project members can now be added before they accept an invitation to join an organization.
4. Improved the Docker image build editor.
5. Added an example of operations with YandexGPT API to initial notebooks.
6. The process of starting a project is now more obvious and transparent.
7. Fixed bugs and added minor improvements in platform performance.

Release as of 20/06/23Release as of 20/06/23

1. Added a page with a list of all user projects.
2. Updated initial notebooks.
3. Fixed bugs and added minor improvements in platform performance.

Release as of 23/05/23Release as of 23/05/23

1. DataSphere now supports a new Dedicated operation mode.
2. In the Dedicated mode, the IDE version is updated to JupyterLab 3.5.3.
3. You can now select an organization in an optimized way.
4. Operations with community and project members are now easier to perform.
5. Fixed bugs and added minor improvements in platform performance.

Release as of 29/03/23Release as of 29/03/23

1. You can now work with labels to label resources.
2. Fixed bugs and added minor improvements.

Release as of 24/03/23Release as of 24/03/23

1. Added a tool to migrate projects to the new DataSphere version.
2. Fixed bugs and added minor improvements.

Release as of 02/03/23Release as of 02/03/23

1. You can now use the new DataSphere version via the API.
2. Fixed bugs and added minor improvements.

Release as of 19/01/23Release as of 19/01/23

1. The service now displays inherited roles of community and project members.
2. Optimized the advanced settings for projects.
3. Updated the snippets for working with S3, Yandex Disk, and Google Drive.
4. Fixed bugs and added minor improvements.

Release as of 20/10/22Release as of 20/10/22

Greatly improved the Apache Spark™ cluster functionality:

1. DataSphere now has a new type of resources: Yandex Data Processing templates.
2. You can now configure a livy session when using Yandex Data Processing clusters.
3. DataSphere now supports the Spark SQL library.

Release as of 23/09/22Release as of 23/09/22

Meet our large DataSphere update: new interface, communities, resources, and many other features for ML development.

Release as of 11/01/2022Release as of 11/01/2022

1. Added new computing resource configurations:
   • g2.mig (1 MIG NVIDIA Ampere A100)
   • g2.2 (2 GPU NVIDIA Ampere A100)
   • g2.4 (4 GPU NVIDIA Ampere A100)
2. Updated the introductory Welcome notebooks in Russian and English.
3. Fixed bugs, added minor improvements.

Release as of 18/11/2021Release as of 18/11/2021

1. You can now resize project storage.
2. You can now set usage limits on individual folders and projects to manage costs.
3. Projects take less time to open.
4. Fixed bugs, added minor improvements.

Release as of 24/09/2021Release as of 24/09/2021

1. You can now connect to S3 object storage from the interface.
2. Added rapid model deployment from Python code cells.

Release as of 16/02/2021Release as of 16/02/2021

Added a new state saving mode: Autosave Commit Mode.

Release as of 10/02/2021Release as of 10/02/2021

1. The approach to state serialization has been revised. The old mode has changed and become a bit more intuitive, also now you can enable autosave mode.
2. Added GPU usage indicator.
3. Added the option to contact support from the service.

Release as of 24/12/2020Release as of 24/12/2020

1. Introduced Early Access Version: a new mode of running DataSphere.

   Early Access Version is a pre-release version of the system where all the key new features will be announced.

   Note

   How do I use it?

   You can select the Early Access Version operating mode for your project in its entirety. To activate this mode, from the project menu, select File and then Enable Early Access Version.
   You can revert to the regular mode anytime: click File in your project menu and select Disable Early Access Version.

   Where to view it:

   To find out what's new in the release and how you can use it, see our new notebook: What's new in Early Access?.

2. Memory and CPU usage indicators added in DataSphere: CPU core and memory usage is now shown directly in the notebook interface.

3. TensorBoard support added.

4. Implemented asynchronous background execution of operations in specially designated cells.

Release as of 08/12/2020Release as of 08/12/2020

An introductory Welcome notebook has been released (in Russian). It explains how the service works and how you can quickly get started.

Release as of 23/11/2020Release as of 23/11/2020

1. Code completion fixed.
2. Added support for widgets.

Release as of 11/11/2020Release as of 11/11/2020

1. Implemented support for TensorFlow version 2.x.
2. Implemented support for updating pre-installed libraries, such as TensorFlow.
3. Improved the algorithm for detecting changed variables: now only the actually changed variables are included in commits, which decreases the time needed to save the state.
4. Added a snippet to work with the SPARK cluster.

Release as of 01/10/2020Release as of 01/10/2020

1. Yandex DataSphere has become a paid generally available (GA) feature.

   Per-second billing is used, so you pay only for the computation time (you are not charged for using notebooks).
   Pricing is based on a billing unit. One billing unit represents the cost of using one CPU core for one second.

   The number of units and the cost depend on the computing resource configuration.

2. Added the option to use the bash command.

   The %%bash command is still available directly, but you can use its functionality as follows:

   • Specify #!S:bash in the cell headers (S indicates the type of VM instance to run bash on).

   Limitations:

   • Background jobs are not supported, for example, sshd.
   • Launching pip is not supported. Continue to use magic for pip.

3. Full-featured integration with Apache Spark™ is now available. You can compute on the existing Yandex Data Processing clusters or even create temporary Yandex Data Processing clusters from DataSphere directly.

4. Added versioning and support of checkpoints.

5. Added new configuration types: M (8 cores, 0 gpu) and XL (32 cores, 4 gpu Nvidia v100).