Managing ZooKeeper hosts

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of cluster hosts, run the command:

yc managed-clickhouse host list \
   --cluster-name=<cluster_name>

+----------------------------+--------------+---------+--------+---------------+
|            NAME            |  CLUSTER ID  |  ROLE   | HEALTH |    ZONE ID    |
+----------------------------+--------------+---------+--------+---------------+
| rc1b...mdb.yandexcloud.net | c9qp71dk1... | MASTER  | ALIVE  | ru-central1-b |
| rc1a...mdb.yandexcloud.net | c9qp71dk1... | REPLICA | ALIVE  | ru-central1-a |
+----------------------------+--------------+---------+--------+---------------+

You can request the cluster name with a list of clusters in the folder.

1. Get an IAM token for API authentication and place it in the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.listHosts method and make a request, e.g., via cURL:

   curl \
       --request GET \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts'
   

   You can request the cluster ID with a list of clusters in the folder.

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and place it in the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/ListHosts call and make a request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d '{
               "cluster_id": "<cluster_ID>"
           }' \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.clickhouse.v1.ClusterService.ListHosts
   

   You can request the cluster ID with a list of clusters in the folder.

4. View the server response to make sure the request was successful.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To enable fault tolerance for a cluster:

1. View a description of the CLI command for adding ZooKeeper hosts:

   yc managed-clickhouse cluster add-zookeeper --help
   

2. Run the operation with the default host characteristics:

   yc managed-clickhouse cluster add-zookeeper <cluster_name> \
      --host zone-id=ru-central1-d,subnet-name=default-d \
      --host zone-id=ru-central1-a,subnet-name=default-a \
      --host zone-id=ru-central1-b,subnet-name=default-b
   

   If the network hosting the cluster contains exactly 3 subnets, each per availability zone, you do not have to explicitly specify subnets for the hosts: Managed Service for ClickHouse® automatically distributes hosts over the subnets.

   You can request the cluster name with a list of clusters in the folder.

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about how to create this file, see Creating clusters.

2. Make sure the configuration file describes three subnets, one for each availability zone. Add the missing ones, if required:

   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name_in_the_ru-central1-a_zone>" {
     name           = "<subnet_name_in_the_ru-central1-a_zone>"
     zone           = "ru-central1-a"
     network_id     = yandex_vpc_network.<network_name>.id
     v4_cidr_blocks = [ "<subnet_IP_address_range_in_the_ru-central1-a_zone>" ]
   }
   
   resource "yandex_vpc_subnet" "<subnet_name_in_the_ru-central1-b_zone>" {
     name           = "<subnet_name_in_the_ru-central1-b_zone>"
     zone           = "ru-central1-b"
     network_id     = yandex_vpc_network.<network_name>.id
     v4_cidr_blocks = [ "<subnet_IP_address_range_in_the_ru-central1-b_zone>" ]
   }
   
   resource "yandex_vpc_subnet" "<subnet_name_in_the_ru-central1-d_zone>" {
     name           = "<subnet_name_in_the_ru-central1-d_zone>"
     zone           = "ru-central1-d"
     network_id     = yandex_vpc_network.<network_name>.id
     v4_cidr_blocks = [ "<subnet_IP_address_range_in_the_ru-central1-d_zone>" ]
   }
   

3. Add the required number of CLICKHOUSE type host blocks to the ClickHouse® cluster description.

   ClickHouse® host requirements:

   • Minimum host class: b1.medium.
   • If there are several hosts, they must be located in different availability zones.

   If necessary, change the class of existing ClickHouse® hosts and availability zone and add the required number of new hosts.

   resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
     name = "<cluster_name>"
     ...
     clickhouse {
       resources {
         resource_preset_id = "<host_class>"
         disk_type_id       = "<disk_type>"
         disk_size          = <storage_size_GB>
       }
     }
     ...
     host {
       type      = "CLICKHOUSE"
       zone      = "ru-central1-a"
       subnet_id = yandex_vpc_subnet.<name_of_subnet_in_availability_zone_ru-central1-a>.id
     }
     ...
   }
   

   Where resource_preset_id is host class: b1.medium or higher.

4. Add at least 3 ZOOKEEPER type host blocks to the ClickHouse® cluster description.

   ZooKeeper host requirements:

   • Each availability zone must have at least one host.
   • Minimum host class: b1.medium.
   • Disk type: network-ssd.
   • The minimum storage size is 10 GB.

   resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
     ...
     zookeeper {
       resources {
         resource_preset_id = "<host_class>"
         disk_type_id       = "network-ssd"
         disk_size          = <storage_size_GB>
       }
     }
     ...
     host {
       type      = "ZOOKEEPER"
       zone      = "ru-central1-a"
       subnet_id = yandex_vpc_subnet.<subnet_name_in_the_ru-central1-a_zone>.id
     }
     host {
       type      = "ZOOKEEPER"
       zone      = "ru-central1-b"
       subnet_id = yandex_vpc_subnet.<subnet_name_in_the_ru-central1-b_zone>.id
     }
     host {
       type      = "ZOOKEEPER"
       zone      = "ru-central1-d"
       subnet_id = yandex_vpc_subnet.<subnet_name_in_the_ru-central1-d_zone>.id
     }
   }
   

   Where resource_preset_id is host class: b1.medium or higher.

5. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

6. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To enable fault tolerance for a cluster, use the addZookeeper method for the Cluster resource or the ClusterService/AddZookeeper gRPC API call. For the added hosts, specify the settings for three ZooKeeper hosts by listing them in the hostSpecs parameter.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To add a host to a cluster:

1. Collect the necessary information:

   • Request the subnet ID by running the command:

     yc vpc subnet list
     

     If the required subnet is not in the list, create it.

   • You can get the cluster name with a list of clusters in the folder.

2. View a description of the CLI command for adding a host:

   yc managed-clickhouse host add --help
   

3. Run the add ZooKeeper host command:

   yc managed-clickhouse hosts add \
     --cluster-name <cluster_name> \
     --host zone-id=<availability_zone>,subnet-id=<subnet_ID>,type=zookeeper
   

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about how to create this file, see Creating clusters.

2. Add a host block of the ZOOKEEPER type to the Managed Service for ClickHouse® cluster description:

   resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
     ...
     host {
       type      = "ZOOKEEPER"
       zone      = "<availability_zone>"
       subnet_id = yandex_vpc_subnet.<name_of_subnet_in_selected_availability_zone>.id
     }
     ...
   }
   

3. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

4. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To add a ZooKeeper host, use the addHosts REST API method for the Cluster resource or the ClusterService/AddHosts gRPC API call and provide the following in the request:

• In the clusterId parameter, the ID of the cluster where you want to locate the host. To find out the cluster ID, get a list of clusters in the folder.
• Settings for the host, in the hostSpecs parameter (in addition, specify the ZOOKEEPER type in the hostSpecs.type parameter). Do not specify settings for multiple hosts in this parameter because ZooKeeper hosts are added to the cluster one by one unlike ClickHouse® hosts, which can be added several at a time.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To restart a host, run this command:

yc managed-clickhouse host restart <host_name> \
   --cluster-name=<cluster_name>

You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.

1. Get an IAM token for API authentication and place it in the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.restartHosts method and make a request, e.g., via cURL:

   curl \
       --request POST \
       --header "Authorization: Bearer $IAM_TOKEN" \
       --header "Content-Type: application/json" \
       --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:restartHosts' \
       --data '{
                 "hostNames": [
                   <list_of_host_names>
                 ]
               }'
   

   Where hostNames is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.

   You can request the cluster ID with a list of clusters in the folder.

3. View the server response to make sure the request was successful.

1. Get an IAM token for API authentication and place it in the environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/RestartHosts call and make a request, e.g., via gRPCurl:

   grpcurl \
       -format json \
       -import-path ~/cloudapi/ \
       -import-path ~/cloudapi/third_party/googleapis/ \
       -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
       -rpc-header "Authorization: Bearer $IAM_TOKEN" \
       -d '{
               "cluster_id": "<cluster_ID>",
               "host_names": [
                 <list_of_host_names>
               ]
           }' \
       mdb.api.cloud.yandex.net:443 \
       yandex.cloud.mdb.clickhouse.v1.ClusterService.RestartHosts
   

   Where host_names is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.

   You can request the cluster ID with a list of clusters in the folder.

4. View the server response to make sure the request was successful.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To remove a host from the cluster, run:

yc managed-clickhouse hosts delete <host_name> \
  --cluster-name=<cluster_name>

You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about how to create this file, see Creating clusters.

2. In the Managed Service for ClickHouse® cluster description, delete the ZOOKEEPER type host block.

3. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

4. Type yes and press Enter.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To delete a ZooKeeper host, use the deleteHosts REST API method for the Cluster resource or the ClusterService/DeleteHosts gRPC API call and provide the following in the request:

• ID of the cluster in which the host is located, in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• Host name, in the hostNames parameter. To find out the name, get a list of hosts in the cluster.