Managing ZooKeeper hosts
Single-host shards are not fault-tolerant and do not offer data replication. To make such shards fault-tolerant, add more hosts to them. If a cluster already contains a multi-host shard, you can immediately add ClickHouse® hosts to the appropriate shard. Otherwise, you must first enable fault tolerance and only then will you be able to add ClickHouse® hosts.
Warning
If fault tolerance is already enabled for the cluster and ZooKeeper hosts are created, you cannot delete all these hosts because there are always at least three ZooKeeper hosts in the cluster.
You can perform the following actions on ZooKeeper hosts in fault-tolerant clusters:
- Get a list of cluster hosts.
- Use ZooKeeper hosts to enable fault tolerance.
- Add a host.
- Restart a host.
- Remove a host.
A fault-tolerant cluster can contain a total of three to five ZooKeeper hosts.
For information about moving ZooKeeper hosts to a different availability zone, see this guide.
Getting a list of cluster hosts
- Go to the folder page
and select Managed Service for ClickHouse. - Click the name of the cluster you need and select the Hosts tab.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To get a list of cluster hosts, run the command:
yc managed-clickhouse host list \
--cluster-name=<cluster_name>
+----------------------------+--------------+---------+--------+---------------+
| NAME | CLUSTER ID | ROLE | HEALTH | ZONE ID |
+----------------------------+--------------+---------+--------+---------------+
| rc1b...mdb.yandexcloud.net | c9qp71dk1... | MASTER | ALIVE | ru-central1-b |
| rc1a...mdb.yandexcloud.net | c9qp71dk1... | REPLICA | ALIVE | ru-central1-a |
+----------------------------+--------------+---------+--------+---------------+
You can request the cluster name with the list of clusters in the folder.
-
Get an IAM token for API authentication and put it into the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.listHosts method and make a request, e.g., via cURL
:curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts'
You can get the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
-
Get an IAM token for API authentication and put it into the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService/ListHosts call and make a request, e.g., via gRPCurl
:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.ListHosts
You can get the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
Enabling fault tolerance for the cluster
- In the management console
, go to the folder page and select Managed Service for ClickHouse. - Click the cluster name and open the Hosts tab.
- At the top right, click Set up ZooKeeper hosts.
- Specify the host class.
- Set up the storage settings.
- Change the ZooKeeper host settings, if required. To do this, hover over the host's row and click
. - Click Save changes.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To enable fault tolerance for a cluster:
-
View a description of the CLI command for adding ZooKeeper hosts:
yc managed-clickhouse cluster add-zookeeper --help
-
Run the operation with the default host characteristics:
yc managed-clickhouse cluster add-zookeeper <cluster_name> \ --host zone-id=ru-central1-d,subnet-name=default-d \ --host zone-id=ru-central1-a,subnet-name=default-a \ --host zone-id=ru-central1-b,subnet-name=default-b
If the network hosting the cluster contains exactly 3 subnets, each per availability zone, you do not have to explicitly specify subnets for the hosts: Managed Service for ClickHouse® automatically distributes hosts over the subnets.
You can request the cluster name with the list of clusters in the folder.
-
Open the current Terraform configuration file with an infrastructure plan.
For more information about creating this file, see Creating clusters.
-
Make sure the configuration file describes three subnets, one for each availability zone. Add the missing ones, if required:
resource "yandex_vpc_network" "<network_name>" { name = "<network_name>" } resource "yandex_vpc_subnet" "<subnet_name_in_ru-central1-a_zone>" { name = "<subnet_name_in_ru-central1-a_zone>" zone = "ru-central1-a" network_id = yandex_vpc_network.<network_name>.id v4_cidr_blocks = [ "<ru-central1-a_zone_subnet_IP_address_range>" ] } resource "yandex_vpc_subnet" "<subnet_name_in_ru-central1-b_zone>" { name = "<subnet_name_in_ru-central1-b_zone>" zone = "ru-central1-b" network_id = yandex_vpc_network.<network_name>.id v4_cidr_blocks = [ "<ru-central1-b_zone_subnet_IP_address_range>" ] } resource "yandex_vpc_subnet" "<subnet_name_in_ru-central1-d_zone>" { name = "<subnet_name_in_ru-central1-d_zone>" zone = "ru-central1-d" network_id = yandex_vpc_network.<network_name>.id v4_cidr_blocks = [ "<ru-central1-d_zone_subnet_IP_address_range>" ] }
-
Add the required number of
CLICKHOUSE
typehost
blocks to the ClickHouse® cluster description.ClickHouse® host requirements:
- Minimum host class:
b1.medium
. - If there is more than one host, they must reside in different availability zones.
If necessary, change the class of existing ClickHouse® hosts and availability zone and add the required number of new hosts.
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" { name = "<cluster_name>" ... clickhouse { resources { resource_preset_id = "<host_class>" disk_type_id = "<disk_type>" disk_size = <storage_size_in_GB> } } ... host { type = "CLICKHOUSE" zone = "ru-central1-a" subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-a_zone>.id } ... }
Where
resource_preset_id
isb1.medium
host class or higher. - Minimum host class:
-
Add at least 3
ZOOKEEPER
typehost
blocks to the ClickHouse® cluster description.ZooKeeper host requirements:
- Each availability zone must have at least one host.
- Minimum host class:
b1.medium
. - Disk type:
network-ssd
. - The minimum storage size is 10 GB.
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" { ... zookeeper { resources { resource_preset_id = "<host_class>" disk_type_id = "network-ssd" disk_size = <storage_size_in_GB> } } ... host { type = "ZOOKEEPER" zone = "ru-central1-a" subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-a_zone>.id } host { type = "ZOOKEEPER" zone = "ru-central1-b" subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-b_zone>.id } host { type = "ZOOKEEPER" zone = "ru-central1-d" subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-d_zone>.id } }
Where
resource_preset_id
isb1.medium
host class or higher. -
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
For more information, see the Terraform provider documentation
Time limits
A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring one from a backup: 60 minutes.
- Editing a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
To enable fault tolerance for a cluster, use the addZookeeper method for the Cluster resource or the ClusterService/AddZookeeper gRPC API call. When adding, specify the settings for three ZooKeeper hosts by listing them in the hostSpecs
parameter.
Note
ZooKeeper hosts get the following specs by default:
- Host class:
b2.medium
. - Network SSD storage (
network-ssd
): 10 GB.
Adding a ZooKeeper host
- In the management console
, go to the folder page and select Managed Service for ClickHouse. - Click the name of the cluster you need and select the Hosts tab.
- Click Add ZooKeeper hosts.
- If required, change the host settings.
- Click Save.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To add a host to a cluster:
-
Collect the necessary information:
-
Request the subnet ID by running the command:
yc vpc subnet list
If the required subnet is not in the list, create it.
-
You can get the cluster name with a list of clusters in the folder.
-
-
View a description of the CLI command for adding a host:
yc managed-clickhouse host add --help
-
Run the add ZooKeeper host command:
yc managed-clickhouse hosts add \ --cluster-name <cluster_name> \ --host zone-id=<availability_zone>,subnet-id=<subnet_ID>,type=zookeeper
-
Open the current Terraform configuration file with an infrastructure plan.
For more information about creating this file, see Creating clusters.
-
Add the
ZOOKEEPER
typehost
blocks to the Managed Service for ClickHouse® cluster description:
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
...
host {
type = "ZOOKEEPER"
zone = "<availability_zone>"
subnet_id = yandex_vpc_subnet.<subnet_name_in_the_selected_availability_zone>.id
}
...
}
-
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Confirm updating the resources.
-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
For more information, see the Terraform provider documentation
Time limits
A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring one from a backup: 60 minutes.
- Editing a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
To add a ZooKeeper host, use the addHosts REST API method for the Cluster resource or the ClusterService/AddHosts gRPC API call and provide the following in the request:
- ID of the cluster you want the host to reside in, in the
clusterId
parameter. To find out the cluster ID, get a list of clusters in the folder. - Host settings in the
hostSpecs
parameter (in addition, specify theZOOKEEPER
type in thehostSpecs.type
parameter). Do not specify settings for multiple hosts in this parameter because ZooKeeper hosts are added to the cluster one by one unlike ClickHouse® hosts, which can be added several at a time.
Restarting a ZooKeeper host
You may need to restart hosts to promptly address such issues as the following:
- Resource overrun
- Memory leak
- Deadlock between requests
- Unresponsive ClickHouse® operations and internal processes
To restart a host:
- Go to the folder page
and select Managed Service for ClickHouse. - Click the cluster name and go to the Hosts tab.
- In the host's row, click
and select Restart. - Confirm the host restart.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To restart a host, run this command:
yc managed-clickhouse host restart <host_name> \
--cluster-name=<cluster_name>
You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.
-
Get an IAM token for API authentication and put it into the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.restartHosts method and make a request, e.g., via cURL
:curl \ --request POST \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:restartHosts' \ --data '{ "hostNames": [ <list_of_host_names> ] }'
Where
hostNames
is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.You can get the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
-
Get an IAM token for API authentication and put it into the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService/RestartHosts call and make a request, e.g., via gRPCurl
:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "host_names": [ <list_of_host_names> ] }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.clickhouse.v1.ClusterService.RestartHosts
Where
host_names
is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.You can get the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
Deleting a ZooKeeper host
- In the management console
, go to the folder page and select Managed Service for ClickHouse. - Click the name of the cluster you need and select the Hosts tab.
- Hover over the host's row and click
. - In the window that opens, click Delete.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To remove a host from the cluster, run:
yc managed-clickhouse hosts delete <host_name> \
--cluster-name=<cluster_name>
You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.
-
Open the current Terraform configuration file with an infrastructure plan.
For more information about creating this file, see Creating clusters.
-
Delete the
ZOOKEEPER
typehost
block from the Managed Service for ClickHouse® cluster description. -
Make sure the settings are correct.
-
Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.
-
Run the command:
terraform validate
If there are errors in the configuration files, Terraform will point to them.
-
-
Type
yes
and press Enter.-
Run the command to view planned changes:
terraform plan
If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.
-
If you are happy with the planned changes, apply them:
-
Run the command:
terraform apply
-
Confirm the update of resources.
-
Wait for the operation to complete.
-
-
For more information, see the Terraform provider documentation
Time limits
A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:
- Creating a cluster, including by restoring one from a backup: 60 minutes.
- Editing a cluster: 90 minutes.
- Deleting a cluster: 30 minutes.
Operations exceeding the set timeout are interrupted.
How do I change these limits?
Add the timeouts
block to the cluster description, for example:
resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
...
timeouts {
create = "1h30m" # 1 hour 30 minutes
update = "2h" # 2 hours
delete = "30m" # 30 minutes
}
}
To delete a ZooKeeper host, use the deleteHosts REST API method for the Cluster resource or the ClusterService/DeleteHosts gRPC API call and provide the following in the request:
- ID of the cluster the host resides in, in the
clusterId
parameter. To find out the cluster ID, get a list of clusters in the folder. - Host name in the
hostNames
parameter. To find out the name, get a list of hosts in the cluster.
ClickHouse® is a registered trademark of ClickHouse, Inc