Creating a workspace
You can create an environment from scratch or create a new environment based on an existing one and edit the parameters.
Creating a workspace
-
In the management console
, select the folder. -
Navigate
to Managed Service for Apache Spark™. -
In the left-hand panel, select
Environments. -
Click Create environment.
-
Under General information:
-
Specify a name for the environment.
-
Optionally, enter a description for the environment.
-
Optionally, create labels:
- Click Add label.
- Add a label in
key: valueformat. - Press Enter.
-
-
Under Base environment, select a base environment from the list.
-
Under Packages:
- Pip packages: pip package names separated by spaces to install additional libraries and applications into the environment.
- Deb packages: deb package names separated by spaces to install additional libraries and applications into the environment.
You can set version restrictions for the installed packages, e.g.:
py4j>=0.10.9.7 pandas>=1.05 grpcio>=1.48,<1.57 grpcio-status>=1.48,<1.57 googleapis-common-protos==1.56.4The package name format and version are defined by the install command:
pip installfor pip packages andapt installfor deb packages.Warning
You cannot change the package list after the environment has been created. If you want to change the list of packages, create a new environment based on an existing one and make the changes.
-
Click Create.
-
Get an IAM token for API authentication and put it into an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
Get a list of base environments by calling EnvironmentService/ListBase:
grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/spark/v1/environment/environment_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "folder_id": "<folder_ID>" }' \ spark.api.cloud.yandex.net:443 \ yandex.cloud.spark.v1.environment.EnvironmentService.ListBaseYou can get the folder ID with the list of folders in the cloud.
-
Use the EnvironmentService/Create call to create your environment:
grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/spark/v1/environment/environment_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "base_id": "<base_environment_ID>", "folder_id": "<folder_ID>", "name": "<environment_name>", "description": "<environment_description>", "labels": { "<label_key>": "<label_value>" }, "config": { "pip_packages": [ "<pip_package_1>", "<pip_package_2>" ], "deb_packages": [ "<deb_package_1>", "<deb_package_2>" ] } }' \ spark.api.cloud.yandex.net:443 \ yandex.cloud.spark.v1.environment.EnvironmentService.CreateWhere:
-
base_id: Base environment ID you got in the previous step. -
folder_id: Folder ID. -
name: Environment name. -
description: Environment description. -
labels: Environment labels. -
config: Parameters of packages you need to install into the environment:pip_packages: List of pip packages.deb_packages: List of deb packages.
You can set version restrictions for the installed packages, e.g.:
"pip_packages": [ "pandas==2.1.1", "scikit-learn>=1.0.0", "clickhouse-driver~=0.2.0" ]The package name format and version are defined by the install commands:
pip installfor pip packages andapt installfor deb packages.
-
-
Check the server response to make sure your request was successful.
Creating a new environment based on an existing one
- In the management console
, select the folder. - Navigate
to Managed Service for Apache Spark™. - In the left-hand panel, select
Environments. - Select the required environment from the list, navigate to it and click
Create a copy of the environment on the top panel. - Add or change the environment parameters.
- Click Create.