Migrating ClickHouse® cluster hosts to a different availability zone

ClickHouse® and ZooKeeper hosts of a Managed Service for ClickHouse® cluster reside in Yandex Cloud availability zones. Follow this guide to migrate ClickHouse® and ZooKeeper hosts to a different availability zone. If you want to migrate hosts with built-in ClickHouse® Keeper, contact support.

Migrating ClickHouse® hostsMigrating ClickHouse® hosts

1. Make sure the migration will only move replicated tables on the ReplicatedMergeTree family engine.

   Non-replicated tables will be lost during migration.

2. If the cluster does not have a coordination service, enable one. Without a coordination service, you will not be able to add new hosts to shards and perform migration.

3. Create a subnet in your target availability zone.

4. Add a host to your cluster:

   Management console

   CLI

   Terraform

   REST API

   gRPC API

   1. In the management console, select the folder the cluster is in.

   2. Navigate to Managed Service for ClickHouse.

   3. Click the cluster name and navigate to the Hosts tab.

   4. Click Create host.

   5. Specify the following host settings:

      • Target availability zone for your hosts.
      • New subnet.
      • To make the host accessible from outside Yandex Cloud, select Public access.

   6. Click Save.

   If you do not have the Yandex Cloud CLI yet, install and initialize it.

   The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

   Run this command:

   yc managed-clickhouse host add \
      --cluster-name <cluster_name> \
      --host type=clickhouse,`
            `zone-id=<availability_zone>,`
            `subnet-id=<new_subnet_ID>,`
            `assign-public-ip=<allow_public_access_to_host>
   

   You can get the cluster name with the list of clusters in the folder. In the zone-id argument, specify the target availability zone for your hosts.

   1. In the Terraform infrastructure configuration file, add a new ClickHouse® host to the hosts section:

      resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" {
        ...
        hosts = {
          ...
          <host_name> = {
            type             = "CLICKHOUSE"
            zone             = "<availability_zone>"
            subnet_id        = "<new_subnet_ID>"
            assign_public_ip = <allow_public_access_to_host>
            shard_name       = "<shard_name>"
          }
        }
      }
      

      In the zone attribute, specify the target availability zone for your hosts.

   2. Validate your configuration.

      1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

      2. Run this command:

         terraform validate
         

         Terraform will show any errors found in your configuration files.

   3. Confirm updating the resources.

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

   1. Get an IAM token for API authentication and place it in an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Call the Cluster.AddHosts method, e.g., via the following cURL request:

      curl \
          --request POST \
          --header "Authorization: Bearer $IAM_TOKEN" \
          --header "Content-Type: application/json" \
          --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:batchCreate' \
          --data '{
                    "hostSpecs": [
                      {
                        "type": "CLICKHOUSE",
                        "zoneId": "<availability_zone>",
                        "subnetId": "<subnet_ID>",
                        "assignPublicIp": <allow_public_access_to_host>
                      }
                    ]
                  }'
      

      Where hostSpecs is the array of settings for the new hosts. Each array element contains the configuration for a single host and has the following structure:

      • type: Host type, which is always CLICKHOUSE for ClickHouse® hosts.
      • zoneId: Availability zone.
      • subnetId: Subnet ID.
      • assignPublicIp: Internet access to the host via a public IP address, true or false.

      You can request the cluster ID with the list of clusters in the folder.

   3. View the server response to make sure your request was successful.

   1. Get an IAM token for API authentication and put it into an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Clone the cloudapi repository:

      cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
      

      Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

   3. Call the ClusterService.AddHosts method, e.g., via the following gRPCurl request:

      grpcurl \
          -format json \
          -import-path ~/cloudapi/ \
          -import-path ~/cloudapi/third_party/googleapis/ \
          -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
          -rpc-header "Authorization: Bearer $IAM_TOKEN" \
          -d '{
                  "cluster_id": "<cluster_ID>",
                  "host_specs": [
                      {
                          "type": "CLICKHOUSE",
                          "zone_id": "<availability_zone>",
                          "subnet_id": "<subnet_ID>",
                          "assign_public_ip": <allow_public_access_to_host>
                      }
                  ]
              }' \
          mdb.api.cloud.yandex.net:443 \
          yandex.cloud.mdb.clickhouse.v1.ClusterService.AddHosts
      

      Where host_specs is the array of settings for the new hosts. Each array element contains the configuration for a single host and has the following structure:

      • type: Host type, which is always CLICKHOUSE for ClickHouse® hosts.
      • zone_id: Availability zone.
      • subnet_id: Subnet ID.
      • assign_public_ip: Internet access to the host via a public IP address, true or false.

      You can request the cluster ID with the list of clusters in the folder.

   4. View the server response to make sure your request was successful.

5. To connect to the database after migration, specify the new host’s FQDN in your backend or client, e.g., in your application code or graphical IDE. Delete the original host's FQDN in the source availability zone.

   You can get this FQDN from the list of hosts in your cluster:

   yc managed-clickhouse host list --cluster-name <cluster_name>
   

   You will see the FQDN under NAME in the command output. Alternatively, you can connect using a special FQDN.

6. Delete the hosts in the source availability zone:

   Management console

   CLI

   Terraform

   REST API

   gRPC API

   1. In the management console, select the folder the cluster is in.
   2. Navigate to Managed Service for ClickHouse.
   3. Click the name of your cluster and select the Hosts tab.
   4. Click in the host row, select Delete, and confirm the deletion.

   Run the following command for each host:

   yc managed-clickhouse host delete <host_FQDN> --cluster-name <cluster_name>
   

   1. In your Terraform infrastructure configuration file, delete the ClickHouse® host with the source availability zone from the hosts section.

   2. Validate your configuration.

      1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

      2. Run this command:

         terraform validate
         

         Terraform will show any errors found in your configuration files.

   3. Type yes and press Enter.

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

   1. Call the Cluster.DeleteHosts method, e.g., via the following cURL request:

      curl \
         --request POST \
         --header "Authorization: Bearer $IAM_TOKEN" \
         --header "Content-Type: application/json" \
         --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:batchDelete' \
         --data '{
                   "hostNames": [
                     "<host_FQDN>"
                   ]
                 }'
      

      Where hostNames is the array containing the host you want to delete.

      You can only specify one host FQDN per request. If you need to delete multiple hosts, make a separate request for each of them.

   2. View the server response to make sure your request was successful.

   1. Call the ClusterService.DeleteHosts method, e.g., via the following gRPCurl request:

      grpcurl \
         -format json \
         -import-path ~/cloudapi/ \
         -import-path ~/cloudapi/third_party/googleapis/ \
         -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
         -rpc-header "Authorization: Bearer $IAM_TOKEN" \
         -d '{
               "cluster_id": "<cluster_ID>",
               "host_names": [
                 "<host_FQDN>"
               ]
             }' \
         mdb.api.cloud.yandex.net:443 \
         yandex.cloud.mdb.clickhouse.v1.ClusterService.DeleteHosts
      

      Where host_names is the array containing the host you want to delete.

      You can only specify one host FQDN per request. If you need to delete multiple hosts, make a separate request for each of them.

   2. View the server response to make sure your request was successful.

7. Wait for the cluster state to change to Alive. In the management console, go to Managed Service for ClickHouse. You can check the cluster state in the Availability column.

Migrating ZooKeeper hostsMigrating ZooKeeper hosts

1. Create a subnet in your target availability zone.

2. Add a host to your cluster:

   Management console

   CLI

   Terraform

   REST API

   gRPC API

   1. In the management console, select the folder the cluster is in.
   2. Navigate to Managed Service for ClickHouse.
   3. Click the cluster name and navigate to the Hosts tab.
   4. Click Create ZooKeeper host.
   5. Specify the new subnet and the availability zone to move the hosts to.
   6. Click Save.

   If you do not have the Yandex Cloud CLI yet, install and initialize it.

   The folder used by default is the one specified when creating the CLI profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also specify a different folder for any command using --folder-name or --folder-id. If you access a resource by its name, the search will be limited to the default folder. If you access a resource by its ID, the search will be global, i.e., through all folders based on access permissions.

   Run this command:

   yc managed-clickhouse host add \
      --cluster-name <cluster_name> \
      --host type=zookeeper,`
            `zone-id=<availability_zone>,`
            `subnet-id=<new_subnet_ID>,`
            `assign-public-ip=<allow_public_access_to_host>
   

   You can get the cluster name with the list of clusters in the folder. In the zone-id argument, specify the target availability zone for your hosts.

   1. In the Terraform infrastructure configuration file, add a new ZooKeeper host to the hosts section:

      resource "yandex_mdb_clickhouse_cluster_v2" "<cluster_name>" {
        ...
        hosts = {
          ...
          <host_name> = {
            type      = "ZOOKEEPER"
            zone      = "<availability_zone>"
            subnet_id = "<new_subnet_ID>"
          }
        }
      }
      

      In the zone attribute, specify the target availability zone for your hosts.

   2. Validate your configuration.

      1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

      2. Run this command:

         terraform validate
         

         Terraform will show any errors found in your configuration files.

   3. Confirm updating the resources.

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

   1. Get an IAM token for API authentication and place it in an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Call the Cluster.AddHosts method, e.g., via the following cURL request:

      curl \
          --request POST \
          --header "Authorization: Bearer $IAM_TOKEN" \
          --header "Content-Type: application/json" \
          --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:batchCreate' \
          --data '{
                    "hostSpecs": [
                      {
                        "type": "ZOOKEEPER",
                        "zoneId": "<availability_zone>",
                        "subnetId": "<subnet_ID>",
                        "assignPublicIp": <allow_public_access_to_host>
                      }
                    ]
                  }'
      

      Where hostSpecs is the array of settings for the new host. Each array element contains the configuration for a single host and has the following structure:

      • type: Host type, ZOOKEEPER.
      • zoneId: Availability zone.
      • subnetId: Subnet ID.
      • assignPublicIp: Internet access to the host via a public IP address, true or false.

      You can request the cluster ID with the list of clusters in the folder.

   3. Check the server response to make sure your request was successful.

   1. Get an IAM token for API authentication and put it into an environment variable:

      export IAM_TOKEN="<IAM_token>"
      

   2. Clone the cloudapi repository:

      cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
      

      Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

   3. Call the ClusterService.AddHosts method, e.g., via the following gRPCurl request:

      grpcurl \
          -format json \
          -import-path ~/cloudapi/ \
          -import-path ~/cloudapi/third_party/googleapis/ \
          -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
          -rpc-header "Authorization: Bearer $IAM_TOKEN" \
          -d '{
                  "cluster_id": "<cluster_ID>",
                  "host_specs": [
                      {
                          "type": "ZOOKEEPER",
                          "zone_id": "<availability_zone>",
                          "subnet_id": "<subnet_ID>",
                          "assign_public_ip": <allow_public_access_to_host>
                      }
              }' \
          mdb.api.cloud.yandex.net:443 \
          yandex.cloud.mdb.clickhouse.v1.ClusterService.AddHosts
      

      Where host_specs is the array of settings for the new hosts. Each element of the host_specs array contains settings for a single host and has the following structure:

      • type: ZOOKEEPER host type.
      • zone_id: Availability zone.
      • subnet_id: Subnet ID.
      • assign_public_ip: Internet access to the host via a public IP address, true or false.

      You can request the cluster ID with the list of clusters in the folder.

   4. View the server response to make sure your request was successful.

3. Delete the hosts in the source availability zone:

   Management console

   CLI

   Terraform

   REST API

   gRPC API

   1. In the management console, select the folder the cluster is in.
   2. Navigate to Managed Service for ClickHouse.
   3. Click the name of your cluster and select the Hosts tab.
   4. Click in the host row, select Delete, and confirm the deletion.

   Run the following command for each host:

   yc managed-clickhouse host delete <host_FQDN> --cluster-name <cluster_name>
   

   1. In your Terraform infrastructure configuration file, delete the ZooKeeper host with the source availability zone from the hosts section.

   2. Validate your configuration.

      1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

      2. Run this command:

         terraform validate
         

         Terraform will show any errors found in your configuration files.

   3. Type yes and press Enter.

      1. Run this command to view the planned changes:

         terraform plan
         

         If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

         1. Run this command:

            terraform apply
            

         2. Confirm updating the resources.

         3. Wait for the operation to complete.

   1. Call the Cluster.DeleteHosts method, e.g., via the following cURL request:

      curl \
         --request POST \
         --header "Authorization: Bearer $IAM_TOKEN" \
         --header "Content-Type: application/json" \
         --url 'https://mdb.api.cloud.yandex.net/managed-clickhouse/v1/clusters/<cluster_ID>/hosts:batchDelete' \
         --data '{
                   "hostNames": [
                     "<host_FQDN>"
                   ]
                 }'
      

      Where hostNames is the array containing the host you want to delete.

      You can only specify one host FQDN per request. If you need to delete multiple hosts, make a separate request for each of them.

   2. View the server response to make sure your request was successful.

   1. Call the ClusterService.DeleteHosts method, e.g., via the following gRPCurl request:

      grpcurl \
         -format json \
         -import-path ~/cloudapi/ \
         -import-path ~/cloudapi/third_party/googleapis/ \
         -proto ~/cloudapi/yandex/cloud/mdb/clickhouse/v1/cluster_service.proto \
         -rpc-header "Authorization: Bearer $IAM_TOKEN" \
         -d '{
               "cluster_id": "<cluster_ID>",
               "host_names": [
                 "<host_FQDN>"
               ]
             }' \
         mdb.api.cloud.yandex.net:443 \
         yandex.cloud.mdb.clickhouse.v1.ClusterService.DeleteHosts
      

      Where host_names is the array containing the host you want to delete.

      You can only specify one host FQDN per request. If you need to delete multiple hosts, make a separate request for each of them.

   2. View the server response to make sure your request was successful.

4. Wait for the cluster state to change to Alive. In the management console, go to Managed Service for ClickHouse. You can check the cluster state in the Availability column.

Specifics of migration in Yandex Data TransferSpecifics of migration in Yandex Data Transfer

If your cluster is used as an endpoint when transferring data with Data Transfer, and the transfer type is Replication or Snapshot and increment, restart the transfer after migrating the cluster. This way, the transfer will get data about the cluster's new topology.

You do not need to restart Snapshot transfers, as information about the new topology is provided automatically while activating them.

To restart a transfer, choose one of the two methods:

• Deactivate the transfer and wait for its status to change to Stopped. Next, reactivate the transfer and wait for its status to change to Replicating.
• Update any setting for the transfer or endpoint.

For more information, see Migrating a Data Transfer transfer and endpoints to a different availability zone.

ClickHouse® is a registered trademark of ClickHouse, Inc.