Using hybrid storage in Yandex Managed Service for ClickHouse®

Hybrid storage allows you to store frequently used data on the network disks of the Managed Service for ClickHouse® cluster and rarely used data in Yandex Object Storage. Automatically moving data between these storage tiers is only supported for MergeTree tables. For more information, see Disk types in Managed Service for ClickHouse®.

To use hybrid storage:

1. Create a table.
2. Populate the table with data.
3. Check the placement of data in a cluster.
4. Run a test query.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Prepare the infrastructurePrepare the infrastructure

Set up the command line toolsSet up the command line tools

1. Install the curl and unxz tools:

   apt-get update && apt-get install curl xz-utils
   

2. Set up clickhouse-client and use it to connect to the database.

Explore the test dataset (optional)Explore the test dataset (optional)

To demonstrate how hybrid storage works, we are going to use the Yandex Metrica anonymized hit data (hits_v1). This dataset contains information about almost 9 million hits for the week from March 17, 2014 to March 23, 2014.

When creating the tutorial.hits_v1 table, you are going to configure it so that all the more recent table data, starting from March 21, 2014, is in network storage, and older data, from March 17, 2014 to March 20, 2014, is in object storage.

Create a tableCreate a table

Create the tutorial.hits_v1 table that uses hybrid storage. To do this, run an SQL query by substituting <schema> with a table schema from the ClickHouse® documentation:

CREATE TABLE tutorial.hits_v1
(
   <schema>
)
ENGINE = MergeTree()
PARTITION BY EventDate
ORDER BY (CounterID, EventDate, intHash32(UserID))
SAMPLE BY intHash32(UserID)
TTL EventDate + toIntervalDay(dateDiff('day', toDate('2014-03-20'), now())) TO DISK 'object_storage'
SETTINGS index_granularity = 8192

CREATE TABLE tutorial.hits_v1
(
    `WatchID` UInt64,
    `JavaEnable` UInt8,
    `Title` String,
    `GoodEvent` Int16,
    `EventTime` DateTime,
    `EventDate` Date,
    `CounterID` UInt32,
    `ClientIP` UInt32,
    `ClientIP6` FixedString(16),
    `RegionID` UInt32,
    `UserID` UInt64,
    `CounterClass` Int8,
    `OS` UInt8,
    `UserAgent` UInt8,
    `URL` String,
    `Referer` String,
    `URLDomain` String,
    `RefererDomain` String,
    `Refresh` UInt8,
    `IsRobot` UInt8,
    `RefererCategories` Array(UInt16),
    `URLCategories` Array(UInt16),
    `URLRegions` Array(UInt32),
    `RefererRegions` Array(UInt32),
    `ResolutionWidth` UInt16,
    `ResolutionHeight` UInt16,
    `ResolutionDepth` UInt8,
    `FlashMajor` UInt8,
    `FlashMinor` UInt8,
    `FlashMinor2` String,
    `NetMajor` UInt8,
    `NetMinor` UInt8,
    `UserAgentMajor` UInt16,
    `UserAgentMinor` FixedString(2),
    `CookieEnable` UInt8,
    `JavascriptEnable` UInt8,
    `IsMobile` UInt8,
    `MobilePhone` UInt8,
    `MobilePhoneModel` String,
    `Params` String,
    `IPNetworkID` UInt32,
    `TraficSourceID` Int8,
    `SearchEngineID` UInt16,
    `SearchPhrase` String,
    `AdvEngineID` UInt8,
    `IsArtifical` UInt8,
    `WindowClientWidth` UInt16,
    `WindowClientHeight` UInt16,
    `ClientTimeZone` Int16,
    `ClientEventTime` DateTime,
    `SilverlightVersion1` UInt8,
    `SilverlightVersion2` UInt8,
    `SilverlightVersion3` UInt32,
    `SilverlightVersion4` UInt16,
    `PageCharset` String,
    `CodeVersion` UInt32,
    `IsLink` UInt8,
    `IsDownload` UInt8,
    `IsNotBounce` UInt8,
    `FUniqID` UInt64,
    `HID` UInt32,
    `IsOldCounter` UInt8,
    `IsEvent` UInt8,
    `IsParameter` UInt8,
    `DontCountHits` UInt8,
    `WithHash` UInt8,
    `HitColor` FixedString(1),
    `UTCEventTime` DateTime,
    `Age` UInt8,
    `Sex` UInt8,
    `Income` UInt8,
    `Interests` UInt16,
    `Robotness` UInt8,
    `GeneralInterests` Array(UInt16),
    `RemoteIP` UInt32,
    `RemoteIP6` FixedString(16),
    `WindowName` Int32,
    `OpenerName` Int32,
    `HistoryLength` Int16,
    `BrowserLanguage` FixedString(2),
    `BrowserCountry` FixedString(2),
    `SocialNetwork` String,
    `SocialAction` String,
    `HTTPError` UInt16,
    `SendTiming` Int32,
    `DNSTiming` Int32,
    `ConnectTiming` Int32,
    `ResponseStartTiming` Int32,
    `ResponseEndTiming` Int32,
    `FetchTiming` Int32,
    `RedirectTiming` Int32,
    `DOMInteractiveTiming` Int32,
    `DOMContentLoadedTiming` Int32,
    `DOMCompleteTiming` Int32,
    `LoadEventStartTiming` Int32,
    `LoadEventEndTiming` Int32,
    `NSToDOMContentLoadedTiming` Int32,
    `FirstPaintTiming` Int32,
    `RedirectCount` Int8,
    `SocialSourceNetworkID` UInt8,
    `SocialSourcePage` String,
    `ParamPrice` Int64,
    `ParamOrderID` String,
    `ParamCurrency` FixedString(3),
    `ParamCurrencyID` UInt16,
    `GoalsReached` Array(UInt32),
    `OpenstatServiceName` String,
    `OpenstatCampaignID` String,
    `OpenstatAdID` String,
    `OpenstatSourceID` String,
    `UTMSource` String,
    `UTMMedium` String,
    `UTMCampaign` String,
    `UTMContent` String,
    `UTMTerm` String,
    `FromTag` String,
    `HasGCLID` UInt8,
    `RefererHash` UInt64,
    `URLHash` UInt64,
    `CLID` UInt32,
    `YCLID` UInt64,
    `ShareService` String,
    `ShareURL` String,
    `ShareTitle` String,
    `ParsedParams` Nested(
        Key1 String,
        Key2 String,
        Key3 String,
        Key4 String,
        Key5 String,
        ValueDouble Float64),
    `IslandID` FixedString(16),
    `RequestNum` UInt32,
    `RequestTry` UInt8
)
ENGINE = MergeTree()
PARTITION BY EventDate
ORDER BY (CounterID, EventDate, intHash32(UserID))
SAMPLE BY intHash32(UserID)
TTL EventDate + toIntervalDay(dateDiff('day', toDate('2014-03-20'), now())) TO DISK 'object_storage'
SETTINGS index_granularity = 8192

The TTL ... expression sets a policy for handing expiring data:

1. TTL sets the lifetime of a table row. In our case, it is the number of days from the current date to March 20, 2014.
2. For the table data, the EventDate value is checked:
   • If the number of days from the current date to EventDate is less than the TTL value (i.e., the lifetime has not expired yet), this data is kept in network disk storage.
   • If the number of days from the current date to EventDate is greater than or equal to the TTL value (that is, the lifetime has already expired), this data will be placed in the object storage according to the TO DISK 'object_storage' policy.

You do not need to specify TTL for hybrid storage, but this allows you to explicitly control which data will be in Object Storage. If you do not specify TTL, data will be placed in object storage only when you run out of space in your network disk storage. For more information, see Disk types in Managed Service for ClickHouse®.

Data is moved between storage on network disks and object storage in parts, not line by line. Make sure to choose the TTL expression and partitioning key so that the TTL matches for all the rows in the data part. Otherwise, you may have problems moving data into object storage as TTL expires if one data part contains data intended for different storage levels. At the most basic level, the TTL expression should use the same columns as in the partitioning key, like in the example above, where the EventDate column is used.

For more information on setting up TTL, see the ClickHouse® documentation.

Populate the table with dataPopulate the table with data

1. Disconnect from the database.

2. Download the test dataset:

   curl https://storage.yandexcloud.net/doc-files/managed-clickhouse/hits_v1.tsv.xz | unxz --threads=`nproc` > hits_v1.tsv
   

   The size of the downloaded dataset is about 10 GB.

3. Insert data from this dataset into ClickHouse® using clickhouse-client:

   clickhouse-client \
       --host <ClickHouse®_host_FQDN> \
       --secure \
       --user <username> \
       --database tutorial \
       --port 9440 \
       --ask-password \
       --query "INSERT INTO tutorial.hits_v1 FORMAT TSV" \
       --max_insert_block_size=100000 < hits_v1.tsv
   

   You can obtain the host FQDN with a list of hosts in the cluster.

4. Wait for the operation to complete, as data insertion may take some time.

For more information, see the ClickHouse® documentation.

Check the placement of data in a clusterCheck the placement of data in a cluster

1. Connect to the database.

2. Check where the table rows are placed:

   SELECT
       table,
       partition,
       name,
       rows,
       disk_name
   FROM system.parts
   WHERE active AND (table = 'hits_v1') AND (database = 'tutorial')
   

   Table partitions for which the EventDate value is outside the given TTL should reside on the disk named object_storage, i.e., in object storage. All other partitions should reside on the default disk:

   ┌─table───┬─partition──┬─name───────────────┬───rows─┬─disk_name──────┐
   │ hits_v1 │ 2014-03-17 │ 20140317_6_80_2    │ 571657 │ object_storage │
   │ hits_v1 │ 2014-03-17 │ 20140317_86_125_1  │ 287545 │ object_storage │
   │ ...                                                                 │
   │ hits_v1 │ 2014-03-20 │ 20140320_109_145_1 │ 250484 │ object_storage │
   │ hits_v1 │ 2014-03-20 │ 20140320_149_200_1 │ 420081 │ object_storage │
   │ hits_v1 │ 2014-03-21 │ 20140321_3_57_1    │ 612616 │ default        │
   │ hits_v1 │ 2014-03-21 │ 20140321_65_65_0   │  53382 │ default        │
   │ ...                                                                 │
   │ hits_v1 │ 2014-03-23 │ 20140323_191_191_0 │  11145 │ default        │
   │ hits_v1 │ 2014-03-23 │ 20140323_197_197_0 │  98910 │ default        │
   └─────────┴────────────┴────────────────────┴────────┴────────────────┘
   

3. Check the number of rows at each of the storage levels:

   SELECT
    sum(rows),
    disk_name
   FROM system.parts
   WHERE active AND (database = 'tutorial') AND (table = 'hits_v1')
   GROUP BY disk_name
   

   As a result, you will see the distribution of table rows for the storage levels:

   ┌─sum(rows)─┬─disk_name──────┐
   │   2711246 │ default        │
   │   6162652 │ object_storage │
   └───────────┴────────────────┘
   

As you can see from the SQL command results, the data in the table was successfully distributed in hybrid storage between different storage levels.

Run a test queryRun a test query

Run a test query to the tutorial.hits_v1 table that engages with data on multiple storage levels at once:

SELECT
    URLDomain AS Domain,
    AVG(SendTiming) AS AvgSendTiming
FROM tutorial.hits_v1
WHERE (EventDate >= '2014-03-19') AND (EventDate <= '2014-03-22')
GROUP BY Domain
ORDER BY AvgSendTiming DESC
LIMIT 10

Result:

┌─Domain──────────────────────────────┬──────AvgSendTiming─┐
│ realty.ru.msn.com.travel            │ 101166.85714285714 │
│ podbor.ru.msn.com.uazbukatusprosima │  76429.16666666667 │
│ club.metalia-woman                  │ 64872.333333333336 │
│ avito.rusfootki                     │              51099 │
│ papas.drimmirkvart                  │ 50325.642857142855 │
│ apps.oyunuoyna.com.uazbukadelight   │ 32761.666666666668 │
│ voyeurhit                           │          31501.625 │
│ yandex.ru.com.travesti.net          │            31427.5 │
│ sozcu.com.ua.alm.slands             │              29439 │
│ hasters.ru                          │ 18365.666666666668 │
└─────────────────────────────────────┴────────────────────┘

As you can see from the SQL request result, from the user's point of view, the table is a single entity: ClickHouse® successfully queries this table regardless of where the data is actually located in it.

(Optional step) Monitor the amount of space used by data in Object Storage(Optional step) Monitor the amount of space used by data in Object Storage

To monitor the amount of space used by MergeTree table chunks in Object Storage, use the ch_s3_disk_parts_size metric in Yandex Monitoring:

1. In the management console, select Monitoring.

2. Go to Metric Explorer.

3. Run this request:

   "ch_s3_disk_parts_size"{service="managed-clickhouse", resource_type="cluster", node="by_host", resource_id="<cluster_ID>", subcluster_name="clickhouse_subcluster"}
   

Delete the resources you createdDelete the resources you created

Delete the resources you no longer need to avoid paying for them:

ClickHouse® is a registered trademark of ClickHouse, Inc.