Storage in Managed Service for ClickHouse®
Managed Service for ClickHouse® allows you to use network and local storage drives for database clusters. Network drives are based on network blocks, i.e., virtual drives in the Yandex Cloud infrastructure. Local drives are physically located on the database host servers.
When creating a cluster, you can select the following disk types for data storage:
-
Network HDDs (
network-hdd): Most cost-effective option for clusters with low requirements for read and write performance. -
Network SSDs (
network-ssd): Compromise solution: slower than local SSDs, network SSDs ensure data integrity in the event of Yandex Cloud hardware failure. -
Non-replicated SSDs (
network-ssd-nonreplicated): Network disks with higher performance achieved by eliminating redundancy.You can only expand this type of storage in 93 GB increments.
-
Ultra high-speed network SSDs with three replicas (
network-ssd-io-m3): Network disks that deliver performance equivalent to non-replicated SSDs while ensuring redundancy.You can only increase the size of these disks in 93 GB increments.
-
Local SSDs (
local-ssd): Highest-performing disks.You can expand such a storage as follows:
- For Intel Cascade Lake: Only in 100 GB increments.
- For Intel Ice Lake and AMD Zen 4: Only in 368 GB increments.
For clusters with hosts residing in the
ru-central1-davailability zone, local SSD storage is not available if using Intel Cascade Lake.
Block
Note
Up to 5% of disk space is reserved for system use, so the disks may have less available space than indicated when creating a cluster.
For more information about sizes and performance of different disk types, see this Yandex Compute Cloud guide.
Hybrid storage
Hybrid storage in Managed Service for ClickHouse® integrates two data storage tiers:
- Cluster storage: Frequently accessed (hot) data resides on fast storage media (cluster network drives).
- Object storage: Rarely used (cold) data resides in the less expensive and slower Yandex Object Storage.
Policies define whether to write new data to cluster or object storage. For example, under the default policy, new data is initially written to cluster storage. ClickHouse® handles this data using its standard workflows, including segmentation into partitions and data parts. Then ClickHouse® monitors data access frequency and automatically moves colder files to object storage. Automatic reverse migration from object to cluster storage is not supported, but you can move data manually using the MOVE PARTITION and MOVE PART commands.
The object storage uses a service bucket with unlimited storage capacity. Its storage class is standard, and you cannot change it. The Object Storage limits apply to the object storage.
Storing cold data and its backups in hybrid storage counts towards the total cluster usage cost.
You can enable hybrid storage when creating or updating a cluster.
Warning
Once hybrid storage is enabled, you cannot disable it.
Pros and cons of hybrid storage
Key benefits of using hybrid storage include:
- Fast network drives deliver high-speed processing for active data. This achieves low latency and high speeds for analytical queries.
- Rarely used (cold) data resides in the less expensive and slower object storage, reducing storage costs.
- Automated data management. You can set conditions, such as disk capacity thresholds or TTL, to trigger automatic data migration between storage tiers, or move data manually.
- Simplified scaling. When you add a new replica to a cluster with hybrid storage, only the hot data on network drives is copied. Object storage data is not duplicated. This reduces resource consumption and allows for faster scaling.
- Access to data via a single interface. ClickHouse® abstracts its storage infrastructure from data consumers, so users can run both rare queries against cold data and frequent queries to hot data via a single interface.
Hybrid storage constraints:
- Slower access to object storage.
- Hybrid storage is only available for MergeTree tables
. For other table engines, data is only placed in the cluster storage. - Once hybrid storage is enabled in a cluster, you cannot revert to standard storage.
Available storage policies
You can distribute data between cluster and object storage using storage policies set at the database table level.
Note
You cannot create new storage policies or update the existing ones.
In a Managed Service for ClickHouse® cluster with hybrid storage enabled, the following storage policies are available:
-
default: Cluster automatically manages data placement depending on the following:- Hybrid storage settings.
- Table TTL settings.
-
local: For a table with this policy, rows can only be placed in the cluster storage. No data is moved between storage tiers. -
object_storage: For a table with this policy, rows can only be placed in the object storage. No data is moved between storage tiers.
Storage policies do not affect merge operations for data parts.
For more information about storage policies and their settings, see this ClickHouse® guide
Merge operations
Inserting data into MergeTree tables creates immutable data parts sorted by primary key. Over time, the number of these parts can grow, creating query processing overhead. To address this, ClickHouse® runs regular background merges of small data parts into larger ones.
Merge operations
- prefer_not_to_merge: Enables or disables merging data parts within object storage.
max_data_part_size_bytes: Sets the maximum merged data part size in object storage. You can only configure this setting by submitting a request to support.
You can also adjust merge behavior using settings available in the cluster. Use the merge keyword to search for those settings. Examples include:
- Max bytes to merge at max space in pool
- Merge max block size
- Merge tree max bytes to use cache
Hybrid storage settings
A Managed Service for ClickHouse® cluster with enabled hybrid storage has the following settings:
-
move_factor: Minimum percentage of free space in the cluster storage. If your free space percentage is below this value, the data will be moved to object storage. The minimum value is0, the maximum value is1, and the default value is0.01.Data parts are queued up in descending order by size. The number of data parts that will be moved is determined by the condition regarding the specified minimum share of free space in cluster storage.
-
data_cache_enabled: Allows caching data accessed from the object storage in the cluster storage,trueorfalse. As a result, the cold data accessed from the object storage is moved to high-speed drives for faster processing. The default value istrue(allowed). -
data_cache_max_size: Sets the maximum cache size, in bytes, allocated in the cluster storage for data accessed from the object storage. If no value is set, the maximum cache size defaults to half the size of the cluster storage. -
prefer_not_to_merge: Disable merging of data parts in the cluster and object storage,trueorfalse. To disable merging, set totrue. To leave merging enabled, set tofalse.Once inserted into the table, the data is saved as a data part and sorted by its primary key. Then, ClickHouse® runs a background merge of data parts belonging to the same partition into a larger data part within 10 to 15 minutes after the insertion. You can use the system.parts
system table to view the merged data parts and partitions.
You can specify hybrid storage settings when creating or updating a cluster.
For more on hybrid storage settings, see this ClickHouse® guide
TTL
TTL (time-to-live) is a tool you can use to restrict the lifetime of your data. Once the specified time interval expires, you can automatically delete, move, or aggregate table rows and columns. You can configure moving rows with the expired TTL to the object storage and set the TTL value when creating a table or later on.
If there is enough free space in the cluster storage, only the rows with the expired TTL are moved to the object storage. Thus you can move the data to the object storage in parts before your cluster storage is filled up.
While specifying TTL for hybrid storage is optional, doing so gives you explicit control over what data is moved to object storage. Without a defined TTL, data is only moved to the object storage when you run out of space in your cluster storage.
Data is moved between cluster and object storage in parts
For more information on configuring TTL, see this ClickHouse® guide
Selecting the disk type when creating a cluster
The number of hosts you can create together with a ClickHouse® cluster depends on the selected disk type:
-
With local SSD storage (
local-ssd), you can create a cluster with two or more hosts.Storage on local SSDs increases your cluster costs: you pay for the cluster even if it is stopped. For more information, see the pricing policy.
-
With non-replicated network SSD storage (
network-ssd-nonreplicated), you can create a cluster with three or more hosts. -
You can add any number of hosts within the current quota when using the following disk types:
- Network HDDs (
network-hdd) - Network SSDs (
network-ssd) - Ultra high-speed network SSDs with three replicas (
network-ssd-io-m3)
- Network HDDs (
For information about limits on the number of hosts per cluster, see Quotas and limits.
Disk encryption
When creating or restoring a cluster from a backup, you can encrypt the storage disk with a custom KMS key. To encrypt a disk of an already created cluster, disable encryption, or encrypt a disk with a different key, create a backup of the cluster and restore it with the new settings.
Warning
Local disk encryption is not supported for dedicated hosts.
To create an encrypted disk, you need the kms.keys.user role or higher.
If you deactivate the key used to encrypt a disk, access to the data will be suspended until you reactivate the key.
Alert
If you delete the key used to encrypt a disk or its version, you will irrevocably lose access to your data. For more information, see this Key Management Service guide.
Managing disk space
In Managed Service for ClickHouse®, if your storage runs out of space, INSERT queries, background merges, and mutations are suspended. They will resume automatically after you expand the storage.
To monitor your storage utilization, set up alerts in Yandex Monitoring.
Automatic storage expansion
To prevent situations where the disk runs out of free space and insert queries, background merges, and mutations get suspended, set up automatic storage size increase for ClickHouse® and a coordination service, ClickHouse® Keeper or ZooKeeper. This will trigger storage increase when you reach a preset threshold, i.e., a percentage of the total storage size. There are two thresholds:
-
Scheduled expansion threshold: To schedule such an expansion, an algorithm analyzes data from the last few hours and estimates how quickly the storage is filling up. If the calculations show that the specified threshold will be exceeded by the start of the nearest maintenance window, the system schedules a storage expansion. When initiating maintenance, the system will check whether the threshold is indeed exceeded, and if so, expand the storage.
-
Immediate expansion threshold: When reached, the storage expands immediately.
You can use either one or both thresholds. If you set both, make sure the immediate expansion threshold is not lower than the scheduled one.
For a scheduled expansion, you need to set up a maintenance window schedule.
Upon reaching the threshold, the storage will be expanded depending on the disk type:
-
For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.
-
For non-replicated SSDs and ultra high-speed network SSDs with three replicas, by 93 GB.
-
For local SSDs:
- Intel Cascade Lake cluster, by 100 GB.
- In an Intel Ice Lake or AMD Zen 4 cluster, by 368 GB.
If the threshold is reached again, the storage will be automatically expanded until it reaches the specified maximum. After that, you can set a new maximum storage size.
The automatic storage size increase settings for ClickHouse® apply to all existing shards. New shards will use the settings of the oldest shard.
You can configure automatic storage expansion when creating or updating a cluster.
Warning
- You cannot decrease the storage size.
- When scaling your storage, the cluster hosts will be unavailable.