Managing ClickHouse® cluster hosts
You can perform the following actions on ClickHouse® hosts:
- Get a list of cluster hosts.
- Create a host.
- Update ClickHouse® host settings.
- Restart a host.
- Remove a host.
For information about moving ClickHouse® hosts to a different availability zone, see this guide.
Warning
If you have created a cluster without ClickHouse® Keeper support, then before adding new hosts to any of the shards, enable fault tolerance using hosts ZooKeeper.
Getting a list of cluster hosts
- Go to the folder page
and select Managed Service for ClickHouse. - Click the cluster name and select the Hosts tab.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To get a list of cluster hosts, run the command:
yc managed-clickhouse host list \
--cluster-name=<cluster_name>
+----------------------------+--------------+---------+--------+---------------+
| NAME | CLUSTER ID | ROLE | HEALTH | ZONE ID |
+----------------------------+--------------+---------+--------+---------------+
| rc1b...mdb.yandexcloud.net | c9qp71dk1... | MASTER | ALIVE | ru-central1-b |
| rc1a...mdb.yandexcloud.net | c9qp71dk1... | REPLICA | ALIVE | ru-central1-a |
+----------------------------+--------------+---------+--------+---------------+
You can request the cluster name with a list of clusters in the folder.
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.listHosts method and make a request, e.g., via cURL
:curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts'
You can request the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService/ListHosts call and make a request, e.g., via gRPCurl
:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.ListHosts
You can request the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
Creating a host
The number of hosts in Managed Service for ClickHouse® clusters is limited by the CPU and RAM quotas available to DB clusters in your cloud. To check the resources in use, open the Quotas
Using the CLI, Terraform, and API, you can create multiple hosts in a cluster in one go.
-
Go to the folder page
and select Managed Service for ClickHouse. -
Click the cluster name and go to the Hosts tab.
-
Click Create host.
-
Specify the host parameters:
- Availability zone.
- Subnet (if the required subnet is not on the list, create it).
- Select Public access if the host must be accessible from outside Yandex Cloud.
- Shard name.
- Select the Copy data schema option to copy the schema from a random replica to the new host.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To create one or more hosts:
-
Request a list of cluster subnets to select one for the new host:
yc vpc subnet list
Result:
+-----------+-----------+------------+---------------+------------------+ | ID | NAME | NETWORK ID | ZONE | RANGE | +-----------+-----------+------------+---------------+------------------+ | b0cl69... | default-d | enp6rq7... | ru-central1-d | [172.16.0.0/20] | | e2lkj9... | default-b | enp6rq7... | ru-central1-b | [10.10.0.0/16] | | e9b0ph... | a-2 | enp6rq7... | ru-central1-a | [172.16.32.0/20] | | e9b9v2... | default-a | enp6rq7... | ru-central1-a | [172.16.16.0/20] | +-----------+-----------+------------+---------------+------------------+
If the required subnet is not in the list, create it.
-
View the description of the CLI command for creating hosts:
yc managed-clickhouse hosts add --help
-
Run the command for creating hosts.
Specify one or more
--host
parameters in the command, one for each host to be created.The command for creating a single host looks like this:
yc managed-clickhouse hosts add \ --cluster-name=<cluster_name> \ --host zone-id=<availability_zone>,` `subnet-id=<subnet_ID>,` `assign-public-ip=<public_access_to_host>,` `shard-name=<shard_name>,` `type=clickhouse
Where
assign-public-ip
indicates whether the host is reachable from the internet over a public IP,true
orfalse
.To copy the data schema from a random replica to the new host, set the
--copy-schema
optional parameter.Managed Service for ClickHouse® will run the operation for creating hosts.
The subnet ID should be specified if the availability zone contains multiple subnets; otherwise, Managed Service for ClickHouse® will automatically select a single subnet. You can request the cluster name with a list of clusters in the folder.
-
Open the current Terraform configuration file with an infrastructure plan.
For more information about how to create this file, see Creating clusters.
-
Add one or more
host
sections to the Managed Service for ClickHouse® cluster description, one for each host to be created.A single
host
section looks like this:resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" { ... host { type = "CLICKHOUSE" zone = "<availability_zone>" subnet_id = "<subnet_ID>" assign_public_ip = <public_access_to_host> } }
Where
assign_public_ip
indicates whether the host is reachable from the internet over a public IP,true
orfalse
. -
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
For more information, see the Terraform provider documentation
Time limits
A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring one from a backup: 60 minutes.
- Editing a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.addHosts method and make a request, e.g., via cURL
:curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:batchCreate' \ --data '{ "hostSpecs": [ { "type": "CLICKHOUSE", "zoneId": "<availability_zone>", "subnetId": "<subnet_ID>", "shardName": "<shard_name>", "assignPublicIp": <public_access_to_host> }, { <similar_settings_for_new_host_2> }, { ... }, { <similar_settings_for_new_host_N> } ], "copySchema": <copying_data_schema> }'
Where:
-
hostSpecs
: Array with settings for the new hosts. One array element contains settings for a single host and has the following structure:type
: Host type, which is alwaysCLICKHOUSE
for ClickHouse® hosts.zoneId
: Availability zone.subnetId
: Subnet ID.assignPublicIp
: Internet access to the host via a public IP address,true
orfalse
.
-
copySchema
: Enables or disables copying the data schema from a random replica to the new hosts,true
orfalse
.
You can request the cluster ID with a list of clusters in the folder.
-
-
View the server response to make sure the request was successful.
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService/AddHosts call and make a request, e.g., via gRPCurl
:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "host_specs": [ { "type": "CLICKHOUSE", "zone_id": "<availability_zone>", "subnet_id": "<subnet_ID>", "shard_name": "<shard_name>", "assign_public_ip": <public_access_to_host> }, { <similar_settings_for_new_host_2> }, { ... }, { <similar_settings_for_new_host_N> } ], "copy_schema": <whether_to_copy_the_data_schema> }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.AddHosts
Where:
-
host_specs
: Array with settings for the new hosts. One array element contains settings for a single host and has the following structure:type
: Host type, which is alwaysCLICKHOUSE
for ClickHouse® hosts.zone_id
: Availability zone.subnet_id
: Subnet ID.assign_public_ip
: Internet access to the host via a public IP address,true
orfalse
.
-
copy_schema
: Enables or disables copying the data schema from a random replica to the new hosts,true
orfalse
.
You can request the cluster ID with a list of clusters in the folder.
-
-
View the server response to make sure the request was successful.
Warning
If you cannot connect to the host after you created it, check that the cluster security group is configured correctly for the host's subnet.
Use the copy data schema option only if the schema is the same on all replica hosts of the cluster.
Changing a host
You can modify public access settings for every host in a Managed Service for ClickHouse® cluster.
To change the parameters of the cluster host:
- Go to the folder page
and select Managed Service for ClickHouse. - Click the cluster name and open the Hosts tab.
- Click
in the required host row and select Edit. - Enable Public access if the host must be accessible from outside Yandex Cloud.
- Click Save.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To change the parameters of the host, run the command:
yc managed-clickhouse host update <host_name> \
--cluster-name=<cluster_name> \
--assign-public-ip=<public_access_to_host>
Where assign-public-ip
indicates whether the host is reachable from the internet over a public IP, true
or false
.
You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.
-
Open the current Terraform configuration file with an infrastructure plan.
For more information about how to create this file, see Creating clusters.
-
In the host, use the
host
section to add or edit theassign_public_ip
parameter.resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" { ... host { ... assign_public_ip = <public_access_to_host> } ... }
Where
assign_public_ip
indicates whether the host is reachable from the internet over a public IP,true
orfalse
. -
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
For more information, see the Terraform provider documentation
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.updateHosts method and make a request, e.g., via cURL
:Warning
This API method overrides all parameters of the object being modified that were not explicitly passed in the request to the default values. To avoid this, list the settings you want to change in the
updateMask
parameter (one line separated by commas).curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:batchUpdate' \ --data '{ "updateHostSpecs": [ { "hostName": "<host_name>", "updateMask": "assignPublicIp", "assignPublicIp": <public_access_to_host> } ] }'
Where
updateHostSpecs[]
is a list of hosts to change and their parameters. Its individual elements have the following structure:-
hostName
: Host name that you can get with a list of hosts in the cluster. -
updateMask
: List of parameters to update as a single string, separated by commas.Here we specified just a single parameter,
assignPublicIp
. -
assignPublicIp
: Internet access to the host via a public IP address,true
orfalse
.
You can request the cluster ID with a list of clusters in the folder.
-
-
View the server response to make sure the request was successful.
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService/UpdateHosts call and make a request, e.g., via gRPCurl
:Warning
This API method will assign default values to all object parameters not explicitly set in the request. To avoid this, list the settings you want to change in the
update_mask
parameter as an array ofpaths[]
strings.Format for listing settings
"update_mask": { "paths": [ "<setting_1>", "<setting_2>", ... "<setting_N>" ] }
grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "update_host_specs": [ { "host_name": "<host_name>", "update_mask": { "paths": [ "assign_public_ip" ] }, "assign_public_ip": <public_access_to_host> }] }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.UpdateHosts
Where
update_host_specs[]
is a list of hosts to change and their parameters. Its individual elements have the following structure:-
host_name
: Host name that you can request with a list of hosts in the cluster. -
update_mask
: List of parameters to update as an array ofpaths[]
strings.Here we specified just a single parameter,
assign_public_ip
. -
assign_public_ip
: Internet access to the host via a public IP address,true
orfalse
.
You can request the cluster ID with a list of clusters in the folder.
-
-
View the server response to make sure the request was successful.
Warning
If you cannot connect to the host after you changed it, check that the cluster security group is configured correctly for the host's subnet.
Restarting a host
You may need to restart hosts to promptly address such issues as the following:
- Resource overrun
- Memory leak
- Deadlock between requests
- Unresponsive ClickHouse® operations and internal processes
To restart a host:
- Go to the folder page
and select Managed Service for ClickHouse. - Click the cluster name and go to the Hosts tab.
- In the required host row, click
and select Restart. - Confirm the host restart.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To restart a host, run this command:
yc managed-clickhouse host restart <host_name> \
--cluster-name=<cluster_name>
You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.restartHosts method and make a request, e.g., via cURL
:curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:restartHosts' \ --data '{ "hostNames": [ <list_of_host_names> ] }'
Where
hostNames
is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.You can request the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService/RestartHosts call and make a request, e.g., via gRPCurl
:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "host_names": [ <list_of_host_names> ] }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.RestartHosts
Where
host_names
is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.You can request the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
Removing a host
You can use the CLI, Terraform, and API to delete multiple hosts from a cluster in one go.
Warning
You cannot delete a host from a cluster or shard if the relevant limit for the minimum number of hosts was reached.
You cannot delete hosts used for ClickHouse® Keeper placement if you enabled support of this replication mechanism when creating the cluster.
- Go to the folder page
and select Managed Service for ClickHouse. - Click the cluster name and open the Hosts tab.
- Click
in the required host row and select Delete.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To delete one or more hosts from the cluster, run the following command, providing the names of the hosts you want to delete. Use the space character as the separator.
The command for deleting a single host looks like this:
yc managed-clickhouse hosts delete --cluster-name=<cluster_name> \
<host_name>
You can request the host names with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.
-
Open the current Terraform configuration file with an infrastructure plan.
For more information about how to create this file, see Creating clusters.
-
In the Managed Service for ClickHouse® cluster description, delete one or more
host
sections of theCLICKHOUSE
type. -
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Type
yes
and press Enter.-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
For more information, see the Terraform provider documentation
Time limits
A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring one from a backup: 60 minutes.
- Editing a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.deleteHosts method and make a request, e.g., via cURL
:curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:batchDelete' \ --data '{ "hostNames": [ <list_of_host_names> ] }'
Where
hostNames
is an array of strings. Each string is the name of a host to delete. You can request host names with a list of hosts in the cluster.You can request the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
-
Get an IAM token for API authentication and place it in the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService/DeleteHosts call and make a request, e.g., via gRPCurl
:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "host_names": [ <list_of_host_names> ] }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.DeleteHosts
Where
host_names
is an array of strings. Each string is the name of a host to delete. You can request host names with a list of hosts in the cluster.You can request the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
ClickHouse® is a registered trademark of ClickHouse, Inc