Working with hybrid storage
Hybrid storage enables you to store frequently used data in cluster storage (on the Managed Service for ClickHouse® cluster's network drives) while keeping rarely used data in Yandex Object Storage.
Enabling hybrid storage for a cluster
Warning
Once hybrid storage is enabled, you cannot disable it.
Enabling hybrid storage when creating a cluster
-
In the management console
, select the folder where you want to create a ClickHouse® cluster. -
Navigate
to Managed Service for ClickHouse. -
Click Create cluster and set the cluster parameters. To learn more about parameters, see the guide on creating a cluster.
-
Enable Hybrid storage under DBMS settings and, optionally, set the hybrid storage settings:
-
Data caching: Enables caching data accessed from the object storage in the cluster storage. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing.
By default, data caching is enabled.
-
Cache size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage.
-
Min. share of free space on cluster disks: Sets the minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is
0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
-
Click Create cluster.
If you do not have the Yandex Cloud CLI yet, install and initialize it.
The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.
If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
To create a cluster with hybrid storage enabled and, optionally, configure its settings:
-
View the description of the CLI command for creating a cluster:
yc managed-clickhouse cluster create --help -
Run this command, specifying hybrid storage settings:
yc managed-clickhouse cluster create \ ... --cloud-storage=true \ --cloud-storage-data-cache=<cache_data> \ --cloud-storage-data-cache-max-size=<maximum_memory_in_bytes> \ --cloud-storage-move-factor=<share_of_free_space> \ --cloud-storage-prefer-not-to-merge=<disable_merging_data_parts> ...Where:
--cloud-storage: Enables hybrid storage.
-
--cloud-storage-data-cache: Enables caching data accessed from the object storage in the cluster storage. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
--cloud-storage-data-cache-max-size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
--cloud-storage-move-factor: Sets the minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to Yandex Object Storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
--cloud-storage-prefer-not-to-merge: Disables merging of data parts in cluster and object storages. To disable merges, set the parameter totrueor provide it with no value. To keep merges enabled, set the parameter tofalseor do not provide it in the CLI command when creating a cluster.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
To learn more about the
yc managed-clickhouse cluster createcommand's other parameters, see the guide on creating a cluster.
-
Create a Terraform configuration file describing your infrastructure.
-
Add the
cloud_storagesection to the configuration file:resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" { ... cloud_storage = { enabled = true move_factor = <share_of_free_space> data_cache_enabled = <cache_data> data_cache_max_size = <maximum_memory_in_bytes> prefer_not_to_merge = <disable_merging_data_parts> } }Where:
enabled: Enables hybrid storage.
-
move_factor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
data_cache_enabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
data_cache_max_size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
prefer_not_to_merge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
-
Make sure the Terraform configuration file is correct:
-
In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
-
Run this command:
terraform validateTerraform will show any errors found in your configuration files.
-
-
Create a cluster:
-
Run this command to view the intended changes:
terraform planIf you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
-
If everything looks correct, apply the changes:
-
Run this command:
terraform apply -
Confirm updating the resources.
-
Wait for the operation to complete.
-
-
Timeouts
The Terraform provider sets the following timeouts for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring from a backup: 60 minutes.
- Updating a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the timeout are aborted.
How to change these limits
Add the timeouts section to your cluster description, such as the following:
resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" {
...
timeouts = {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
-
Get an IAM token for API authentication and put it into an environment variable:
export IAM_TOKEN="<IAM_token>" -
Call the Cluster.Create method, e.g., via the following cURL
request:-
Create a file named
body.jsonand paste the following code into it:{ <cluster_parameters> ... "configSpec": { "cloudStorage": { "enabled": true, "moveFactor": "<share_of_free_space>", "dataCacheEnabled": <cache_data>, "dataCacheMaxSize": "<maximum_memory_in_bytes>", "preferNotToMerge": <disable_merging_data_parts> }, ... } }Where:
enabled: Enables hybrid storage.
-
moveFactor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
dataCacheEnabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
dataCacheMaxSize: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
preferNotToMerge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
To learn more about other parameters, see the guide on creating a cluster.
-
Run this request:
curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters' \ --data '@body.json'
-
-
View the server response to make sure your request was successful.
-
Get an IAM token for API authentication and put it into an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
Call the ClusterService.Create method, e.g., via the following gRPCurl
request:-
Create a file named
body.jsonand paste the following code into it:{ <cluster_parameters> ... "config_spec": { "cloud_storage": { "enabled": true, "move_factor": "<share_of_free_space>", "data_cache_enabled": <cache_data>, "data_cache_max_size": "<maximum_memory_in_bytes>", "prefer_not_to_merge": <disable_merging_data_parts> }, ... } }Where:
enabled: Enables hybrid storage.
-
move_factor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
data_cache_enabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
data_cache_max_size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
prefer_not_to_merge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
To learn more about other parameters, see the guide on creating a cluster.
-
Run this request:
grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d @ \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.Create \ < body.json
-
-
Check the server response to make sure your request was successful.
Enabling hybrid storage for an existing cluster
-
In the management console
, select the folder containing the cluster. -
Navigate
to Managed Service for ClickHouse. -
Select your cluster and click Edit in the top panel.
-
Enable Hybrid storage under DBMS settings and, optionally, set the hybrid storage settings:
-
Data caching: Enables caching data accessed from the object storage in the cluster storage. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing.
By default, data caching is enabled.
-
Cache size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage.
-
Min. share of free space on cluster disks: Sets the minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is
0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
-
Click Save changes.
If you do not have the Yandex Cloud CLI yet, install and initialize it.
The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.
If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
To enable hybrid storage in a cluster and, optionally, configure its settings, run this command:
yc managed-clickhouse cluster update <cluster_name_or_ID> \
--cloud-storage=true \
--cloud-storage-data-cache=<cache_data> \
--cloud-storage-data-cache-max-size=<maximum_memory_in_bytes> \
--cloud-storage-move-factor=<share_of_free_space> \
--cloud-storage-prefer-not-to-merge=<disable_merging_data_parts>
Where:
<cluster_name_or_ID>: Cluster name or ID which you can get with the list of clusters in the folder.--cloud-storage: Enables hybrid storage.
-
--cloud-storage-data-cache: Enables caching data accessed from the object storage in the cluster storage. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
--cloud-storage-data-cache-max-size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
--cloud-storage-move-factor: Sets the minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to Yandex Object Storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
--cloud-storage-prefer-not-to-merge: Disables merging of data parts in cluster and object storages. To disable merges, set the parameter totrueor provide it with no value. To keep merges enabled, set the parameter tofalseor do not provide it in the CLI command when creating a cluster.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
To enable hybrid storage in a cluster and, optionally, configure its settings:
-
Open the current Terraform configuration file with the infrastructure plan.
To learn how to create this file, see Creating a cluster.
For the complete list of configurable Managed Service for ClickHouse® cluster fields, see this Terraform provider guide.
-
Add the
cloud_storagesection to the Managed Service for ClickHouse® cluster description:resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" { ... cloud_storage = { enabled = true move_factor = <share_of_free_space> data_cache_enabled = <cache_data> data_cache_max_size = <maximum_memory_in_bytes> prefer_not_to_merge = <disable_merging_data_parts> } }Where:
enabled: Enables hybrid storage.
-
move_factor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
data_cache_enabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
data_cache_max_size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
prefer_not_to_merge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
-
Make sure the Terraform configuration file is correct:
-
In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
-
Run this command:
terraform validateTerraform will show any errors found in your configuration files.
-
-
Confirm resource changes:
-
Run this command to view the intended changes:
terraform planIf you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
-
If everything looks correct, apply the changes:
-
Run this command:
terraform apply -
Confirm updating the resources.
-
Wait for the operation to complete.
-
-
Timeouts
The Terraform provider sets the following timeouts for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring from a backup: 60 minutes.
- Updating a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the timeout are aborted.
How to change these limits
Add the timeouts section to your cluster description, such as the following:
resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" {
...
timeouts = {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
-
Get an IAM token for API authentication and put it into an environment variable:
export IAM_TOKEN="<IAM_token>" -
Call the Cluster.Update method, e.g., via the following cURL
request:Warning
The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the
updateMaskparameter as a single comma-separated string.curl \ --request PATCH \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>' \ --data '{ "updateMask": "<list_of_settings_to_update>", "configSpec": { "cloudStorage": { "enabled": true, "moveFactor": "<share_of_free_space>", "dataCacheEnabled": <cache_data>, "dataCacheMaxSize": "<maximum_memory_in_bytes>", "preferNotToMerge": <disable_merging_data_parts> } } }'Where:
-
<cluster_ID>: Cluster ID which you can get with the list of clusters in the folder. -
updateMask: Comma-separated string of settings to update. -
configSpec.cloudStorage: Hybrid storage settings:enabled: Enables hybrid storage.
-
moveFactor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
dataCacheEnabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
dataCacheMaxSize: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
preferNotToMerge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
-
-
Check the server response to make sure your request was successful.
-
Get an IAM token for API authentication and put it into an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
Call the ClusterService.Update method, e.g., via the following gRPCurl
request:Warning
The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the
update_maskparameter as an array ofpaths[]strings.Format for listing settings
"update_mask": { "paths": [ "<setting_1>", "<setting_2>", ... "<setting_N>" ] }grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "update_mask": { "paths": [ <list_of_settings_to_update> ] }, "config_spec": { "cloud_storage": { "enabled": true, "move_factor": "<share_of_free_space>", "data_cache_enabled": <cache_data>, "data_cache_max_size": "<maximum_memory_in_bytes>", "prefer_not_to_merge": <disable_merging_data_parts> } } }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.UpdateWhere:
-
<cluster_ID>: Cluster ID which you can get with the list of clusters in the folder. -
update_mask: List of settings to update as an array of strings (paths[]). -
config_spec.cloud_storage: Hybrid storage settings:enabled: Enables hybrid storage.
-
move_factor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
data_cache_enabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
data_cache_max_size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
prefer_not_to_merge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
You can get the cluster ID with the list of clusters in the folder.
-
-
Check the server response to make sure your request was successful.
Changing the hybrid storage settings
If a Managed Service for ClickHouse® cluster has hybrid storage enabled, you can change its settings.
-
In the management console
, select the folder containing the cluster. -
Navigate
to Managed Service for ClickHouse. -
Select your cluster and click Edit in the top panel.
-
Under DBMS settings, edit the hybrid storage settings:
-
Data caching: Enables caching data accessed from the object storage in the cluster storage. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing.
By default, data caching is enabled.
-
Cache size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage.
-
Min. share of free space on cluster disks: Sets the minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is
0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
-
Click Save changes.
If you do not have the Yandex Cloud CLI yet, install and initialize it.
The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id.
If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.
To edit hybrid storage settings, run this command:
yc managed-clickhouse cluster update <cluster_name_or_ID> \
--cloud-storage-data-cache=<cache_data> \
--cloud-storage-data-cache-max-size=<maximum_memory_in_bytes> \
--cloud-storage-move-factor=<share_of_free_space> \
--cloud-storage-prefer-not-to-merge=<disable_merging_data_parts>
Where:
<cluster_name_or_ID>: Cluster name or ID which you can get with the list of clusters in the folder.
-
--cloud-storage-data-cache: Enables caching data accessed from the object storage in the cluster storage. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
--cloud-storage-data-cache-max-size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
--cloud-storage-move-factor: Sets the minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to Yandex Object Storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
--cloud-storage-prefer-not-to-merge: Disables merging of data parts in cluster and object storages. To disable merges, set the parameter totrueor provide it with no value. To keep merges enabled, set the parameter tofalseor do not provide it in the CLI command when creating a cluster.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
-
Open the current Terraform configuration file with the infrastructure plan.
To learn how to create this file, see Creating a cluster.
For the complete list of configurable Managed Service for ClickHouse® cluster fields, see this Terraform provider guide.
-
In the Managed Service for ClickHouse® cluster description, edit the
cloud_storagesection attributes:resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" { ... cloud_storage = { ... move_factor = <share_of_free_space> data_cache_enabled = <cache_data> data_cache_max_size = <maximum_memory_in_bytes> prefer_not_to_merge = <disable_merging_data_parts> } }Where:
-
move_factor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
data_cache_enabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
data_cache_max_size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
prefer_not_to_merge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
-
-
Make sure the Terraform configuration file is correct:
-
In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
-
Run this command:
terraform validateTerraform will show any errors found in your configuration files.
-
-
Confirm resource changes:
-
Run this command to view the intended changes:
terraform planIf you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
-
If everything looks correct, apply the changes:
-
Run this command:
terraform apply -
Confirm updating the resources.
-
Wait for the operation to complete.
-
-
Timeouts
The Terraform provider sets the following timeouts for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring from a backup: 60 minutes.
- Updating a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the timeout are aborted.
How to change these limits
Add the timeouts section to your cluster description, such as the following:
resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" {
...
timeouts = {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
-
Get an IAM token for API authentication and put it into an environment variable:
export IAM_TOKEN="<IAM_token>" -
Call the Cluster.Update method, e.g., via the following cURL
request:Warning
The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the
updateMaskparameter as a single comma-separated string.curl \ --request PATCH \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>' \ --data '{ "updateMask": "<list_of_settings_to_update>", "configSpec": { "cloudStorage": { "moveFactor": "<share_of_free_space>", "dataCacheEnabled": <cache_data>, "dataCacheMaxSize": "<maximum_memory_in_bytes>", "preferNotToMerge": <disable_merging_data_parts> } } }'Where:
-
<cluster_ID>: Cluster ID which you can get with the list of clusters in the folder. -
updateMask: Comma-separated string of settings to update. -
configSpec.cloudStorage: Hybrid storage settings:-
moveFactor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
dataCacheEnabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
dataCacheMaxSize: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
preferNotToMerge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
-
-
-
Check the server response to make sure your request was successful.
-
Get an IAM token for API authentication and put it into an environment variable:
export IAM_TOKEN="<IAM_token>" -
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapiBelow, we assume that the repository contents reside in the
~/cloudapi/directory. -
Call the ClusterService.Update method, e.g., via the following gRPCurl
request:Warning
The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the
update_maskparameter as an array ofpaths[]strings.Format for listing settings
"update_mask": { "paths": [ "<setting_1>", "<setting_2>", ... "<setting_N>" ] }grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "update_mask": { "paths": [ <list_of_settings_to_update> ] }, "config_spec": { "cloud_storage": { "move_factor": "<share_of_free_space>", "data_cache_enabled": <cache_data>, "data_cache_max_size": "<maximum_memory_in_bytes>", "prefer_not_to_merge": <disable_merging_data_parts> } } }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.UpdateWhere:
-
<cluster_ID>: Cluster ID which you can get with the list of clusters in the folder. -
update_mask: List of settings to update as an array of strings (paths[]). -
config_spec.cloud_storage: Hybrid storage settings:-
move_factor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
data_cache_enabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
data_cache_max_size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
prefer_not_to_merge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
-
-
-
Check the server response to make sure your request was successful.
Setting data storage policies
In a Managed Service for ClickHouse® cluster with hybrid storage enabled, three storage policies are available: default, local, and object_storage. For DB tables, you can specify one of them. You cannot create new storage policies or update the existing ones.
Getting information about policies
To view policy settings, run this query:
SELECT *
FROM system.storage_policies;
Setting a policy when creating a table
To set a policy when creating a table, run this query:
CREATE TABLE <table_name> (
<table_schema>
) ENGINE = MergeTree
...
SETTINGS storage_policy = '<storage_policy>';
Setting a policy for an existing a table
To set or edit a policy for an existing a table, run this query:
ALTER TABLE <table_name>
MODIFY SETTING storage_policy = '<storage_policy>';
Monitoring space taken up by data in Object Storage
To monitor the amount of space MergeTreech_s3_disk_parts_size metric in Yandex Monitoring.
-
Open the management console
. -
Navigate
to Monitoring. -
Open the Metric Explorer section.
-
Run this request:
"ch_s3_disk_parts_size"{service="managed-clickhouse", resource_type="cluster", node="by_host", resource_id="<cluster_ID>", subcluster_name="clickhouse_subcluster"}