Configuring access to Object Storage from a ClickHouse® cluster
Managed Service for ClickHouse® supports using Yandex Object Storage to:
- Add ML models, data format schemas, and custom geobases.
- Process data in an object storage if this data is in any format ClickHouse® supports
.
To access Object Storage bucket data from a cluster, set up password-free access to the bucket using a service account:
Note
The service account will send SQL queries to Object Storage. In this case, keys and secrets are ignored.
If no service account is specified, SQL queries will be sent:
- Anonymously, if the key and secret are not provided.
- Using the key, if the key and secret are provided.
See also Examples of working with objects.
Before you begin, assign the iam.serviceAccounts.user role or higher to your Yandex Cloud account. You will need this role in the following cases:
- To create or update a cluster and attach it to a service account.
- To restore a cluster attached to a service account from its backup.
Attach the service account to the cluster
-
When creating or updating a cluster, either select an existing service account or create a new one.
-
Assign the relevant roles to the service account from the
storage.*role group, e.g.,storage.viewerandstorage.uploader.
Tip
To connect Managed Service for ClickHouse® clusters to Object Storage, we recommend using dedicated service accounts. This allows you to work with any buckets, including those to which public access cannot or should not be granted.
Configure access permissions
-
In the management console
, select the folder containing your bucket. If you have no bucket, create one and upload the data you need to it. -
Select Object Storage.
-
Set up the bucket ACL or object ACL:
- In the list of buckets or objects, select the item in question and click
. - Click Configure ACL or Object ACL.
- In the Select a user drop-down list, specify the service account attached to the cluster.
- Set the required permissions for the service account from the drop-down list.
- Click Add and Save.
Note
Revoke access permissions from one or more users as needed by clicking Cancel in the appropriate row.
- In the list of buckets or objects, select the item in question and click
-
If encryption is enabled for the bucket, assign to the service account the kms.keys.encrypterDecrypter role for the encryption key linked to the bucket.
Examples of working with objects
You can get a bucket object URL in https://storage.yandexcloud.net/<bucket_name>/<object_name> format to work with geotags and schemas or to use the s3 table function and the S3 table engine.
The S3 table engine is similar to FileSELECT and INSERT.
The s3 table function offers the same features as the S3 table engine, without the need to create a table first.
For example, if your Object Storage bucket contains a file named table.tsv that stores table data in TSV format, you can create either a table or a function to access this file. You need to set up password-free access and get a link to table.tsv first.
-
Assign the
managed-clickhouse.editorandstorage.uploaderroles to the service account attached to the Managed Service for ClickHouse® cluster. -
Create a table:
CREATE TABLE test (n Int32) ENGINE = S3('https://storage.yandexcloud.net/<bucket_name>/table.tsv', 'TSV'); -
Run test queries to the table:
INSERT INTO test VALUES (1); SELECT * FROM test; ┌─n─┐ │ 1 │ └───┘
-
Assign the
managed-clickhouse.editorandstorage.uploaderroles to the service account attached to the Managed Service for ClickHouse® cluster. -
Insert data:
INSERT INTO FUNCTION s3('https://storage.yandexcloud.net/<bucket_name>/table.tsv', 'TSV', 'n Int32') VALUES (1); -
Run this test query:
SELECT * FROM s3('https://storage.yandexcloud.net/<bucket_name>/table.tsv', 'TSV', 'n Int32'); ┌─n─┐ │ 1 │ └───┘
ClickHouse® is a registered trademark of ClickHouse, Inc