How to automate project setup using APIs
With DataSphere communities, you can arrange team work and shared access to different materials and resources. For example, you can publish Docker images with pre-installed libraries, connections to Yandex Object Storage buckets, and ready-made datasets created in a single project to grant all community projects access to them. This may be useful when preparing training tasks or when multiple developers are working on the same research.
You can use the DataSphere
To create and set up a community and multiple projects:
If you no longer need the resources you created, delete them.
Getting started
Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.
- On the DataSphere home page
, click Try for free and select an account to log in with: Yandex ID or your working account in the identity federation (SSO). - Select the Yandex Cloud Organization organization you are going to use in Yandex Cloud.
- Create a community.
- Link your billing account to the DataSphere community you are going to work in. Make sure that you have a billing account linked and its status is
ACTIVE
orTRIAL_ACTIVE
. If you do not have a billing account yet, create one in the DataSphere interface.
Required paid resources
The automation cost includes a fee for using DataSphere computing resources.
Get an IAM token
To access your organization from DataSphere, you need an IAM token.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
Get an IAM token:
yc iam create-token
Alert
If you are the owner of the cloud and you use your own account to access the API, remember that the owner of the cloud can perform any operations with cloud resources.
We recommend using a service account to work with the API. This way, you can assign only the roles that are necessary.
-
Log in
to your Yandex account. -
Get an OAuth token from Yandex.OAuth. To do this, follow this link
, click Allow, and copy the OAuth token you got. -
Exchange the OAuth token for an IAM token:
-
Using curl
in Bash:curl -X POST \ -d '{"yandexPassportOauthToken":"<OAuth_token>"}' \ https://iam.api.cloud.yandex.net/iam/v1/tokens
-
Using the built-in PowerShell function:
$yandexPassportOauthToken = "<OAuth_token>" $Body = @{ yandexPassportOauthToken = "$yandexPassportOauthToken" } | ConvertTo-Json -Compress Invoke-RestMethod -Method 'POST' -Uri 'https://iam.api.cloud.yandex.net/iam/v1/tokens' -Body $Body -ContentType 'Application/json' | Select-Object -ExpandProperty iamToken
-
Create projects
To create projects, copy and paste the code into notebook cells and run them.
-
Open the DataSphere project:
-
Select the relevant project in your community or on the DataSphere homepage
in the Recent projects tab. - Click Open project in JupyterLab and wait for the loading to complete.
- Open the notebook tab.
-
-
Specify the IAM token you obtained:
iam_token = "<IAM_token>"
-
Import the required library:
import requests
-
Output the list of all available organizations and their IDs:
res = requests.get('https://resource-manager.api.cloud.yandex.net/resource-manager/v1/clouds', headers={"Authorization" : "Bearer {}".format(iam_token)}) res.json()
-
Specify the ID of the organization to create a community in:
ORGANIZATION_ID = "<organization_id>"
-
Get a list of available billing accounts:
res = requests.get('https://billing.api.cloud.yandex.net/billing/v1/billingAccounts', headers={"Authorization" : "Bearer {}".format(iam_token)}) res.json()
-
Create a community by substituting its name and description and the ID of the active billing account:
data={} data['name'] = "<community_name>" data['description'] = "<community_description>" data['organizationId'] = ORGANIZATION_ID data['billingAccountId'] = "<billing_account_ID>" res = requests.post('https://datasphere.api.cloud.yandex.net/datasphere/v2/communities', json=data, headers={"Authorization" : "Bearer {}".format(iam_token)}) community_res = res.json() community_res
-
Get a list of role IDs in DataSphere:
res = requests.get('https://iam.api.cloud.yandex.net/iam/v1/roles', headers={"Authorization" : "Bearer {}".format(iam_token)}) roles = res.json()['roles'] datasphere_roles = [role for role in roles if 'datasphere' in role['id']] datasphere_roles
-
Get a list of organization members:
res = requests.get("https://organization-manager.api.cloud.yandex.net/organization-manager/v1/organizations/{}/users".format(ORGANIZATION_ID), headers={"Authorization" : "Bearer {}".format(iam_token)}) res.json()
You will need the IDs from the
sub
field at the next step. -
Create projects, set limits, and set up roles for the members:
# List the IDs of the members from the 'sub' field to create projects for user_organization_ids = ['<member_1_ID>','<member_2_ID>'] projects = {} for user_id in user_organization_ids: # Create a project for the user data={} data['name'] = "Student {}".format(user_id) data['communityId'] = community_res['metadata']['communityId'] data['description'] = "This is a workplace and create code and store resources" data['limits'] = { # Set a limit on the maximum number of units per hour for the project "maxUnitsPerHour": 10000, # Set a limit on the maximum number of units per cell run for the project "maxUnitsPerExecution": 5000 } res = requests.post('https://datasphere.api.cloud.yandex.net/datasphere/v2/projects', json=data, headers={"Authorization" : "Bearer {}".format(iam_token)}) print("Project for {} is created with response: {}".format(user_id, res)) data={} data['communityId'] = community_res['metadata']['communityId'] data['projectNamePattern'] = "Student {}".format(user_id) res = requests.get('https://datasphere.api.cloud.yandex.net/datasphere/v2/projects', json=data, headers={"Authorization" : "Bearer {}".format(iam_token)}) projects[user_id] = res.json()['projects'][0] project_id = res.json()['projects'][0]['id'] # Add yourself with the datasphere.community-projects.admin role # and the user with the datasphere.community-projects.developer role to the project data={} data['accessBindings'] = [{ "roleId": 'datasphere.community-projects.admin', "subject": { "id": "<project_admin_ID>", # Specify the project administrator ID "type": "userAccount" }}, { "roleId": 'datasphere.community-projects.developer', "subject": { "id": "<project_developer_ID>", # Specify the project developer ID "type": "userAccount" }} ] res = requests.post('https://datasphere.api.cloud.yandex.net/datasphere/v2/projects/{}:setAccessBindings'.\ format(project_id), json=data, headers={"Authorization" : "Bearer {}".format(iam_token)}) print("Admin was added to project {} with response: {}".format(project_id, res))
-
Get a list of the projects you created:
projects
How to delete the resources you created
To stop paying for the resources you created, delete the project.