Contact UsGet started
© 2024 Direct Cursus Technology L.L.C.

Managing ZooKeeper hosts

Written by
Updated at December 4, 2024

Single-host shards are not fault-tolerant and do not offer data replication. To make such shards fault-tolerant, add more hosts to them. If a cluster already contains a multi-host shard, you can immediately add ClickHouse® hosts to the appropriate shard. Otherwise, you must first enable fault tolerance and only then will you be able to add ClickHouse® hosts.

Warning

If fault tolerance is already enabled for the cluster and ZooKeeper hosts are created, you cannot delete all these hosts because there are always at least three ZooKeeper hosts in the cluster.

You can perform the following actions on ZooKeeper hosts in fault-tolerant clusters:

A fault-tolerant cluster can contain a total of three to five ZooKeeper hosts.

For information about moving ZooKeeper hosts to a different availability zone, see this guide.

Getting a list of cluster hosts

  1. Go to the folder page and select Managed Service for ClickHouse.
  2. Click the name of the cluster you need and select the Hosts tab.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of cluster hosts, run the command:

yc managed-clickhouse host list \
   --cluster-name=<cluster_name>

+----------------------------+--------------+---------+--------+---------------+
|            NAME            |  CLUSTER ID  |  ROLE   | HEALTH |    ZONE ID    |
+----------------------------+--------------+---------+--------+---------------+
| rc1b...mdb.yandexcloud.net | c9qp71dk1... | MASTER  | ALIVE  | ru-central1-b |
| rc1a...mdb.yandexcloud.net | c9qp71dk1... | REPLICA | ALIVE  | ru-central1-a |
+----------------------------+--------------+---------+--------+---------------+

You can request the cluster name with the list of clusters in the folder.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Use the Cluster.listHosts method and make a request, e.g., via cURL:

    curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts'

    You can get the cluster ID with a list of clusters in the folder.

  3. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Use the ClusterService/ListHosts call and make a request, e.g., via gRPCurl:

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
            "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.clickhouse.v1.ClusterService.ListHosts

    You can get the cluster ID with a list of clusters in the folder.

  4. View the server response to make sure the request was successful.

Enabling fault tolerance for the cluster

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the cluster name and open the Hosts tab.
  3. At the top right, click Set up ZooKeeper hosts.
  4. Specify the host class.
  5. Set up the storage settings.
  6. Change the ZooKeeper host settings, if required. To do this, hover over the host's row and click .
  7. To convert non-replicated tables to replicated, enable Convert non-replicated tables.
  8. Click Save changes.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To enable fault tolerance for a cluster:

  1. View a description of the CLI command for adding ZooKeeper hosts:

    yc managed-clickhouse cluster add-zookeeper --help

  2. Run the operation with the default host characteristics:

    yc managed-clickhouse cluster add-zookeeper <cluster_name> \
   --host zone-id=ru-central1-d,subnet-name=default-d \
   --host zone-id=ru-central1-a,subnet-name=default-a \
   --host zone-id=ru-central1-b,subnet-name=default-b

    If the network hosting the cluster contains exactly 3 subnets, each per availability zone, you do not have to explicitly specify subnets for the hosts: Managed Service for ClickHouse® automatically distributes hosts over the subnets.

    To convert non-replicated tables to replicated, add the --convert-tables-to-replicated parameter to the command.

    You can request the cluster name with a list of clusters in the folder.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Make sure the configuration file describes three subnets, one for each availability zone. Add the missing ones, if required:

    resource "yandex_vpc_network" "<network_name>" {
  name = "<network_name>"
}

resource "yandex_vpc_subnet" "<subnet_name_in_ru-central1-a_zone>" {
  name           = "<subnet_name_in_ru-central1-a_zone>"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.<network_name>.id
  v4_cidr_blocks = [ "<ru-central1-a_zone_subnet_IP_address_range>" ]
}

resource "yandex_vpc_subnet" "<subnet_name_in_ru-central1-b_zone>" {
  name           = "<subnet_name_in_ru-central1-b_zone>"
  zone           = "ru-central1-b"
  network_id     = yandex_vpc_network.<network_name>.id
  v4_cidr_blocks = [ "<ru-central1-b_zone_subnet_IP_address_range>" ]
}

resource "yandex_vpc_subnet" "<subnet_name_in_ru-central1-d_zone>" {
  name           = "<subnet_name_in_ru-central1-d_zone>"
  zone           = "ru-central1-d"
  network_id     = yandex_vpc_network.<network_name>.id
  v4_cidr_blocks = [ "<ru-central1-d_zone_subnet_IP_address_range>" ]
}

  3. Add the required number of CLICKHOUSE type host blocks to the ClickHouse® cluster description.

    ClickHouse® host requirements:

    • Minimum host class: b1.medium.
    • If there is more than one host, they must reside in different availability zones.

    If necessary, change the class of existing ClickHouse® hosts and availability zone and add the required number of new hosts.

    resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  name = "<cluster_name>"
  ...
  clickhouse {
    resources {
      resource_preset_id = "<host_class>"
      disk_type_id       = "<disk_type>"
      disk_size          = <storage_size_in_GB>
    }
  }
  ...
  host {
    type      = "CLICKHOUSE"
    zone      = "ru-central1-a"
    subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-a_zone>.id
  }
  ...
}

    Where resource_preset_id is the host class: b1.medium or higher.

  4. Add at least three ZOOKEEPER type host blocks to the ClickHouse® cluster description.

    ZooKeeper host requirements:

    • Each availability zone must have at least one host.
    • Minimum host class: b1.medium.
    • Disk type: network-ssd.
    • The minimum storage size is 10 GB.
    resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  zookeeper {
    resources {
      resource_preset_id = "<host_class>"
      disk_type_id       = "network-ssd"
      disk_size          = <storage_size_in_GB>
    }
  }
  ...
  host {
    type      = "ZOOKEEPER"
    zone      = "ru-central1-a"
    subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-a_zone>.id
  }
  host {
    type      = "ZOOKEEPER"
    zone      = "ru-central1-b"
    subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-b_zone>.id
  }
  host {
    type      = "ZOOKEEPER"
    zone      = "ru-central1-d"
    subnet_id = yandex_vpc_subnet.<subnet_name_in_ru-central1-d_zone>.id
  }
}

    Where resource_preset_id is the host class: b1.medium or higher.

  5. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  6. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:

  • Creating a cluster, including by restoring one from a backup: 60 minutes.
  • Editing a cluster: 90 minutes.
  • Deleting a cluster: 30 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To enable fault tolerance for a cluster, use the addZookeeper method for the Cluster resource or the ClusterService/AddZookeeper gRPC API call and provide the following in the request:

  • Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
  • Settings for three ZooKeeper hosts in the hostSpecs parameter.
  • Whether or not to convert non-replicated tables to replicated in the convertTablesToReplicated parameter.

Note

ZooKeeper hosts get the following specs by default:

  • Host class: b2.medium.
  • Network SSD storage (network-ssd): 10 GB.

Adding a ZooKeeper host

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the name of the cluster you need and select the Hosts tab.
  3. Click Add ZooKeeper hosts.
  4. If required, change the host settings.
  5. Click Save.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To add a host to a cluster:

  1. Collect the necessary information:

  2. View a description of the CLI command for adding a host:

    yc managed-clickhouse host add --help

  3. Run the add ZooKeeper host command:

    yc managed-clickhouse hosts add \
  --cluster-name <cluster_name> \
  --host zone-id=<availability_zone>,subnet-id=<subnet_ID>,type=zookeeper

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Add a ZOOKEEPER type host block to the Managed Service for ClickHouse® cluster description.

 resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
   ...
   host {
     type      = "ZOOKEEPER"
     zone      = "<availability_zone>"
     subnet_id = yandex_vpc_subnet.<subnet_name_in_selected_availability_zone>.id
   }
   ...
 }

  1. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  2. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:

  • Creating a cluster, including by restoring one from a backup: 60 minutes.
  • Editing a cluster: 90 minutes.
  • Deleting a cluster: 30 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To add a ZooKeeper host, use the addHosts REST API method for the Cluster resource or the ClusterService/AddHosts gRPC API call and provide the following in the request:

  • ID of the cluster you want the host to reside in, in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
  • Host settings in the hostSpecs parameter (also specify the ZOOKEEPER type in the hostSpecs.type parameter). Do not specify settings for multiple hosts in this parameter because ZooKeeper hosts are added to the cluster one by one unlike ClickHouse® hosts, which can be added several at a time.

Restarting a ZooKeeper host

You may need to restart hosts to promptly address such issues as the following:

  • Resource overrun
  • Memory leak
  • Deadlock between requests
  • Unresponsive ClickHouse® operations and internal processes

To restart a host:

  1. Go to the folder page and select Managed Service for ClickHouse.
  2. Click the cluster name and go to the Hosts tab.
  3. In the required host row, click and select Restart.
  4. Confirm the host restart.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To restart a host, run this command:

yc managed-clickhouse host restart <host_name> \
   --cluster-name=<cluster_name>

You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Use the Cluster.restartHosts method and make a request, e.g., via cURL:

    curl \
    --request POST \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:restartHosts' \
    --data '{
              "hostNames": [
                <list_of_host_names>
              ]
            }'

    Where hostNames is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.

    You can get the cluster ID with a list of clusters in the folder.

  3. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Use the ClusterService/RestartHosts call and make a request, e.g., via gRPCurl:

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
            "cluster_id": "<cluster_ID>",
            "host_names": [
              <list_of_host_names>
            ]
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.clickhouse.v1.ClusterService.RestartHosts

    Where host_names is an array of strings. Each string is the name of a host to restart. You can request host names with a list of hosts in the cluster.

    You can get the cluster ID with a list of clusters in the folder.

  4. View the server response to make sure the request was successful.

Converting non-replicated tables to replicated

To automatically convert non-replicated MergeTree tables to replicated ReplicatedMergeTree tables, add ZooKeeper hosts with table conversion enabled.

For more information, see the Replication section and the ClickHouse® documentation.

To convert non-replicated tables to replicated:

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.

  2. Click the name of the cluster you need and select the  Hosts tab.

  3. At the top right, click Set up ZooKeeper hosts.

  4. Add ZooKeeper hosts](#add-zk-host).

    On the host adding page, the Convert non-replicated tables option is enabled by default.

  5. To keep non-replicated tables, disable Convert non-replicated tables. To enable support of replicated tables, leave this option enabled.

  6. Click Save changes.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To convert non-replicated tables to replicated ones, run this command:

yc managed-clickhouse cluster add-zookeeper \
   --name <cluster_name> \
   --resource-preset <host_class> \
   --disk-size <storage_size_in_GB> \
   --disk-type <disk_type> \
   --host zone-id=<availability_zone>,subnet-name=<subnet_name> \
   --host <similar_settings_for_new_host_2> \
   --host ... \
   --host <similar_settings_for_new_host_N> \
   --convert-tables-to-replicated

In the command, specify no less than three --host parameters, as the cluster must have at least three ZooKeeper hosts.

The --convert-tables-to-replicated parameter enables conversion of non-replicated MergeTree tables to replicated ReplicatedMergeTree tables.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Use the Cluster.addZookeeper method and make a request, e.g., via cURL:

    curl \
  --request POST \
  --header "Authorization: Bearer $IAM_TOKEN" \
  --header "Content-Type: application/json" \
  --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>:addZookeeper' \
  --data '{
           "resources": {
             "resourcePresetId": "<host_class>",
             "diskSize": "<storage_size_in_bytes>",
             "diskTypeId": "<disk_type>"
           },
           "hostSpecs": [
             {
               "zoneId": "<availability_zone>",
               "type": "ZOOKEEPER",
               "subnetId": "<subnet_ID>",
               "shardName": "<shard_name>",
               "assignPublicIp": <public_access_to_host>
             },
             { <similar_settings_for_new_host_2> },
             { ... },
             { <similar_settings_for_new_host_N> }
           ],
           "convertTablesToReplicated": true
         }'

    Where:

    • resources: Resources for ZooKeeper hosts.

    • hostSpecs: Array with settings for the new hosts. One array element contains settings for a single host and the cluster must have at least three ZooKeeper hosts. An array element has the following structure.

      • type: Host type, which is always ZOOKEEPER for ZooKeeper hosts.
      • zoneId: Availability zone.
      • subnetId: Subnet ID.
      • shardName: Name of the shard the host is added to.
      • assignPublicIp: Internet access to the host via a public IP address, true or false.

    • convertTablesToReplicated: Conversion of non-replicated MergeTree tables to replicated ReplicatedMergeTree tables, true or false.

    You can get the cluster ID with a list of clusters in the folder.

  3. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Use the ClusterService.AddZookeeper call and make a request, e.g., via gRPCurl:

    grpcurl \
  -format json \
  -import-path ~/cloudapi \
  -import-path ~/cloudapi/third_party/googleapis/ \
  -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
  -rpc-header "Authorization: Bearer $IAM_TOKEN" \
  -d '{
        "cluster_id": "<cluster_ID>",
        "resources": {
          "resource_preset_id": "<host_class>",
          "disk_size": "<storage_size_in_bytes>",
          "disk_type_id": "<disk_type>"
        },
        "host_specs": [
          {
            "type": "ZOOKEEPER",
            "zone_id": "<availability_zone>",
            "subnet_id": "<subnet_ID>",
            "shard_name": "<shard_name>",
            "assign_public_ip": <public_access_to_host>
          },
          { <similar_settings_for_new_host_2> },
          { ... },
          { <similar_settings_for_new_host_N> }
        ],
        "convert_tables_to_replicated": true
      }' \
  mdb.api.cloud.yandex.net:443 \
  yandex.cloud.mdb.clickhouse.v1.ClusterService.AddZookeeper

    Where:

    • resources: Resources for ZooKeeper hosts.

    • host_specs: Array with settings for the new hosts. One array element contains settings for a single host and the cluster must have at least three ZooKeeper hosts.

      An array element has the following structure.

      • type: Host type, which is always ZOOKEEPER for ZooKeeper hosts.
      • zone_id: Availability zone.
      • subnet_id: Subnet ID.
      • shard_name: Name of the shard the host is added to.
      • assign_public_ip: Internet access to the host via a public IP address, true or false.

    • convert_tables_to_replicated: Conversion of non-replicated MergeTree tables to replicated ReplicatedMergeTree tables, true or false.

    You can get the cluster ID with a list of clusters in the folder.

  4. View the server response to make sure the request was successful.

Removing a ZooKeeper host

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the name of the cluster you need and select the Hosts tab.
  3. Hover over the host's row and click .
  4. In the window that opens, click Delete.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To remove a host from the cluster, run:

yc managed-clickhouse hosts delete <host_name> \
  --cluster-name=<cluster_name>

You can request the host name with a list of cluster hosts, and the cluster name, with a list of clusters in the folder.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Delete the ZOOKEEPER type host block from the Managed Service for ClickHouse® cluster description.

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Type yes and press Enter.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:

  • Creating a cluster, including by restoring one from a backup: 60 minutes.
  • Editing a cluster: 90 minutes.
  • Deleting a cluster: 30 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To delete a ZooKeeper host, use the deleteHosts REST API method for the Cluster resource or the ClusterService/DeleteHosts gRPC API call and provide the following in the request:

ClickHouse® is a registered trademark of ClickHouse, Inc.

Previous
Stopping and starting a cluster
Next
Managing ClickHouse® hosts