Connecting to an S3 storage
You can manage your connection to an S3 object storage by using the S3 Connector resource on your project page in the DataSphere interface.
To connect to an object storage from the notebook code, follow the steps given in Connecting to S3 using the library.
Note
Avoid using S3 storage in FUSE
Getting started
Get an access key from your S3 storage provider. To do this in Yandex Object Storage, follow these steps:
- Create a service account.
- Assign the new account a role with either read or read and write permissions.
- Create an access key for the service account.
Creating an S3 connector
-
Select the relevant project in your community or on the DataSphere homepage
in the Recent projects tab. - Optionally, in the top-right corner, click Create resource. In the pop-up window, select Secret and create a secret with the secret part of the static access key for your service account. You can also create a secret when creating an S3 connector.
- In the top-right corner, click Create resource. In the pop-up window, select S3 Connector.
- Fill in the fields as follows:
-
Name: Name of the new connector. The naming requirements are as follows:
- The name must be from 3 to 63 characters long.
- It may contain uppercase and lowercase Latin and Russian letters, numbers, hyphens, underscores, and spaces.
- The first character must be a letter. The last character cannot be a hyphen, underscore, or space.
-
Description: Description of the new connector. This is an optional parameter.
-
Endpoint: Storage host. For Object Storage, set it to
https://storage.yandexcloud.net/
. -
Bucket: Name of the storage bucket.
Warning
Do not use buckets with periods in their names for connection. You can learn more about buckets here.
-
Mount name: Name of the volume for mounting the bucket into the project file system. The naming requirements are as follows:
- It must be 2 to 63 characters long.
- It may contain lowercase Latin letters, numbers, and hyphens.
- It must start with a letter and cannot end with a hyphen.
-
Static access key ID: ID of the static access key used to connect to the storage.
-
Static access key: In the list, select a secret that contains the secret part of the static access key or create a new secret.
-
Mode: Object storage access mode, Read only or Read and write.
-
- Click Create.
Note
You need to set up a NAT gateway for any subnet linked to the project.
Attaching an S3 storage to a project
Go to the S3 connector page and click Activate. Once activated, the bucket will be available in the /s3/
folder of the JupyterLab file browser, and you will be able to view it as a file system.
Using an S3 storage in a project
You can access files in the connected bucket from your project code. Select the file you need in the attached S3 storage on the S3 Mounts
Detaching an S3 storage
- On the project page, under Project resources, click S3 Connector.
- Select the connector and go to the resource page.
- Click Deactivate in the top-right corner of the page.
You can re-attach the S3 storage to your project whenever you need it.
Sharing an S3 connector
Note
You can only share resources within a single organization between communities created in the same availability zone.
-
Select the relevant project in your community or on the DataSphere homepage
in the Recent projects tab. - Under Project resources, click
S3 Connector. - Select the S3 connector from the list.
- Go to the Access tab.
- Enable the visibility option next to the name of the community you want to share the S3 connector with.
To make an S3 connector available for use in a different project, the project admin needs to add it on the Shared tab.
Deleting an S3 connector
You can only delete a deactivated connector that is not available to any community.
-
Select the relevant project in your community or on the DataSphere homepage
in the Recent projects tab. - Under Project resources, click
S3 Connector. - In the list of S3 connectors, select the one you want to delete. Click
and select Delete. - Click Confirm.
You will see a message saying that the secret has been deleted.
Warning
The actual deletion of resources can take up to 72 hours.