Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Managing shards in a ClickHouse® cluster

Written by
Updated at November 18, 2024

You can group several shards of a ClickHouse® cluster in a shard group and save tables in this group.

Listing shard groups in a cluster

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the cluster name and open the Shard groups tab.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To get a list of shard groups in a cluster, run the command:

yc managed-clickhouse shard-groups list \
   --cluster-name=<cluster_name>

You can request the cluster name with the list of clusters in the folder.

To get a list of shard groups in a cluster, use the listShardGroups REST API method for the Cluster resource or the ClusterService/ListShardGroups gRPC API call and provide the cluster ID in the clusterId request parameter.

To find out the cluster ID, get a list of clusters in the folder.

Viewing detailed information about a shard group

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the cluster name and open the Shard groups tab.
  3. Select a shard group to view detailed information.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To view detailed information about a shard group in a cluster, run the command:

yc managed-clickhouse shard-groups get \
  --cluster-name=<cluster_name> \
  --name=<shard_group_name>

You can request the cluster name with the list of clusters in the folder.

To get detailed information about a shard group, use the getShardGroup REST API method for the Cluster resource or the ClusterService/GetShardGroup gRPC API call and provide the following in the request:

Creating a shard group

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the cluster name and open the Shard groups tab.
  3. Click Create shard group.
  4. Fill in the form fields and click Apply.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a shard group in a cluster, run the command:

yc managed-clickhouse shard-groups create \
  --cluster-name=<cluster_name> \
  --name=<shard_group_name> \
  --description=<shard_group_description> \
  --shards=<list_of_shard_names>

Where --shards is the list of list of shard names to include in the group.

You can request the cluster name with the list of clusters in the folder.

You can request a shard name with a list of shards in the cluster.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Add the shard_group description section to the Managed Service for ClickHouse® cluster description.

    resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  shard_group {
    name        = "<shard_group_name>"
    description = "<optional_shard_group_description>"
    shard_names = [
      # List of shards in the group
      "<shard_1_name>",
      ...
      "<shard_N_name>"
    ]
  }
}

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:

  • Creating a cluster, including by restoring one from a backup: 60 minutes.
  • Editing a cluster: 90 minutes.
  • Deleting a cluster: 30 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To create a shard group, use the createShardGroup REST API method for the Cluster resource or the ClusterService/CreateShardGroup gRPC API call and provide the following in the request:

  • ID of the cluster in which you want to create a group, in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
  • Name of the shard group, in the shardGroupName parameter.
  • List of shard names to include in the group, in the shardNames parameter. To find out the names, get a list of shards in the cluster.
  • Description of the shard group, in the description parameter, if required.

Changing a shard group

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the cluster name and open the Shard groups tab.
  3. Click for the shard group you need and select Edit.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change a shard group in a cluster, run the command:

yc managed-clickhouse shard-groups update \
  --cluster-name=<cluster_name> \
  --name=<shard_group_name> \
  --description=<new_description_for_shard_group> \
  --shards=<new_list_of_shard_names>

Where --shards is the new list of shard names to include in the group.

This command replaces the existing list of shards in the group with the new one provided in the --shards parameter. Before running the command, make sure that you added all the appropriate shards in the new list.

You can request the cluster name with the list of clusters in the folder.

You can request the name of the shard group with a list of shard groups in the cluster.

You can request a shard name with a list of shards in the cluster.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. In the Managed Service for ClickHouse® cluster description, change the shard_group section to include the required shard group:

    resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  shard_group {
    name        = "<new_name_for_shard_group>"
    description = "<new_description_for_shard_group>"
    shard_names = [
      # New list of shards in the group
      "<shard_1_name>",
      ...
      "<shard_N_name>"
    ]
  }
}

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:

  • Creating a cluster, including by restoring one from a backup: 60 minutes.
  • Editing a cluster: 90 minutes.
  • Deleting a cluster: 30 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To update a shard group, use the updateShardGroup REST API method for the Cluster resource or the ClusterService/UpdateShardGroup gRPC API call and provide the following in the request:

  • ID of the cluster in which you want to change a group, in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
  • Name of the shard group, in the shardGroupName parameter. To find out the name, get a list of shard groups in the cluster.
  • New description for the shard group, in the description parameter, if required.
  • New list of shard names to included in the group, in the shardNames parameter, if required. To find out the names, get a list of shards in the cluster. This list will replace the current one, so make sure that you added all the appropriate shards in the new list.
  • Names of parameters to change, in the updateMask parameter.

Warning

The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

Deleting a group of shards

Deleting a group of shards doesn't affect the shards in the group: they are kept in the cluster.

Tables created on the deleted group are kept, but they are disabled and attempts to query them result in errors. However, you can delete these tables before or after you delete the shard group.

  1. In the management console, go to the folder page and select Managed Service for ClickHouse.
  2. Click the cluster name and open the Shard groups tab.
  3. Click for the shard group you need and select Delete.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To delete a shard group in a cluster, run the command:

yc managed-clickhouse shard-groups delete \
   --cluster-name=<cluster_name> \
   --name=<shard_group_name>

You can request the cluster name with the list of clusters in the folder.

You can request the name of the shard group with a list of shard groups in the cluster.

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. Delete the shard_group description section for the appropriate group from the Managed Service for ClickHouse® cluster description.

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Type yes and press Enter.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

A Terraform provider sets the timeout for Managed Service for ClickHouse® cluster operations:

  • Creating a cluster, including by restoring one from a backup: 60 minutes.
  • Editing a cluster: 90 minutes.
  • Deleting a cluster: 30 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_clickhouse_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To delete a shard group, use the deleteShardGroup REST API method for the Cluster resource or the ClusterService/DeleteShardGroup gRPC API call and provide the following in the request:

ClickHouse® is a registered trademark of ClickHouse, Inc.

Previous
Managing shards
Next
Viewing cluster logs