How to automate project setup using APIs

With DataSphere communities, you can arrange team work and shared access to different materials and resources. For example, you can publish Docker images with pre-installed libraries, connections to Yandex Object Storage buckets, and ready-made datasets created in a single project to grant all community projects access to them. This may be useful when preparing training tasks or when multiple developers are working on the same research.

You can use the DataSphere interface to create and set up projects in DataSphere. However, if you need to set up a number of identical projects, you can automate operations using the DataSphere API and Cloud Organization API. To do this, you will need a preset custom community with a project to invoke the DataSphere API methods from.

To create and set up a community and multiple projects:

1. Get an IAM token.
2. Create projects.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Before getting started, register in Yandex Cloud, set up a community, and link your billing account to it.

1. On the DataSphere home page, click Try for free and select an account to log in with: Yandex ID or your working account in the identity federation (SSO).
2. Select the Yandex Cloud Organization organization you are going to use in Yandex Cloud.
3. Create a community.
4. Link your billing account to the DataSphere community you are going to work in. Make sure that you have a billing account linked and its status is ACTIVE or TRIAL_ACTIVE. If you do not have a billing account yet, create one in the DataSphere interface.

Required paid resourcesRequired paid resources

The automation cost includes a fee for using DataSphere computing resources.

Get an IAM tokenGet an IAM token

To access your organization from DataSphere, you need an IAM token.

yc iam create-token

Create projectsCreate projects

To create projects, copy and paste the code into notebook cells and run them.

1. Open the DataSphere project:

   1. Select the relevant project in your community or on the DataSphere homepage in the Recent projects tab.

   2. Click Open project in JupyterLab and wait for the loading to complete.
   3. Open the notebook tab.

2. Specify the IAM token you obtained:

   iam_token = "<IAM_token>"
   

3. Import the required library:

   import requests
   

4. Output the list of all available organizations and their IDs:

   res = requests.get('https://resource-manager.api.cloud.yandex.net/resource-manager/v1/clouds',
                       headers={"Authorization" : "Bearer {}".format(iam_token)})
   
   res.json()
   

5. Specify the ID of the organization to create a community in:

   ORGANIZATION_ID = "<organization_id>"
   

6. Get a list of available billing accounts:

   res = requests.get('https://billing.api.cloud.yandex.net/billing/v1/billingAccounts',
                       headers={"Authorization" : "Bearer {}".format(iam_token)})
   res.json()
   

7. Create a community by substituting its name and description and the ID of the active billing account:

   data={}
   data['name'] = "<community_name>"
   data['description'] = "<community_description>"
   data['organizationId'] = ORGANIZATION_ID
   data['billingAccountId'] = "<billing_account_ID>"
   
   res = requests.post('https://datasphere.api.cloud.yandex.net/datasphere/v2/communities',
                       json=data,
                       headers={"Authorization" : "Bearer {}".format(iam_token)})
   community_res = res.json()
   community_res
   

8. Get a list of role IDs in DataSphere:

   res = requests.get('https://iam.api.cloud.yandex.net/iam/v1/roles',
                       headers={"Authorization" : "Bearer {}".format(iam_token)})
   roles = res.json()['roles']
   datasphere_roles = [role for role in roles if 'datasphere' in role['id']]
   datasphere_roles
   

9. Get a list of organization members:

   res = requests.get("https://organization-manager.api.cloud.yandex.net/organization-manager/v1/organizations/{}/users".format(ORGANIZATION_ID),
                       headers={"Authorization" : "Bearer {}".format(iam_token)})
   res.json()
   

   You will need the IDs from the sub field at the next step.

10. Create projects, set limits, and set up roles for the members:

    # List the IDs of the members from the 'sub' field to create projects for
    user_organization_ids = ['<member_1_ID>','<member_2_ID>']
    
    projects = {}
    for user_id in user_organization_ids:
      # Create a project for the user
       data={}
       data['name'] = "Student {}".format(user_id)
       data['communityId'] = community_res['metadata']['communityId']
       data['description'] = "This is a workplace and create code and store resources"
       data['limits'] = {
         # Set a limit on the maximum number of units per hour for the project
         "maxUnitsPerHour": 10000,
         # Set a limit on the maximum number of units per cell run for the project
         "maxUnitsPerExecution": 5000
       }
       res = requests.post('https://datasphere.api.cloud.yandex.net/datasphere/v2/projects',
                           json=data,
                           headers={"Authorization" : "Bearer {}".format(iam_token)})
       print("Project for {} is created with response: {}".format(user_id, res))
    
       data={}
       data['communityId'] = community_res['metadata']['communityId']
       data['projectNamePattern'] = "Student {}".format(user_id)
       res = requests.get('https://datasphere.api.cloud.yandex.net/datasphere/v2/projects',
                           json=data,
                           headers={"Authorization" : "Bearer {}".format(iam_token)})
       projects[user_id] = res.json()['projects'][0]
       project_id = res.json()['projects'][0]['id']
    
       # Add yourself with the datasphere.community-projects.admin role
       # and the user with the datasphere.community-projects.developer role to the project
       data={}
       data['accessBindings'] = [{
         "roleId": 'datasphere.community-projects.admin',
         "subject": {
               "id": "<project_admin_ID>", # Specify the project administrator ID
               "type": "userAccount"
         }},
         {
         "roleId": 'datasphere.community-projects.developer',
         "subject": {
               "id": "<project_developer_ID>", # Specify the project developer ID
               "type": "userAccount"
         }}
       ]
       res = requests.post('https://datasphere.api.cloud.yandex.net/datasphere/v2/projects/{}:setAccessBindings'.\
                           format(project_id),
                           json=data,
                           headers={"Authorization" : "Bearer {}".format(iam_token)})
       print("Admin was added to project {} with response: {}".format(project_id, res))
    

11. Get a list of the projects you created:

    projects
    

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you created, delete the project.