Configuring Yandex Managed Service for ClickHouse® for Graphite
Yandex Managed Service for ClickHouse® can be used as data storage for Graphite
The GraphiteMergeTree
Note
If you do not need decimation and aggregation or averaging, you can use any ClickHouse® table engine
To configure a database to work with Graphite:
- Prepare the environment.
- Create a cluster.
- Register the rollup configuration in a cluster.
- Create and set up a virtual machine.
- Connect the virtual machine to a database.
- Create a GraphiteMergeTree table.
- Configure Graphite.
If you no longer need the resources you created, delete them.
Prepare the environment
Sign up for Yandex Cloud and create a billing account:
- Go to the management console
and log in to Yandex Cloud or create an account if you do not have one yet. - On the Yandex Cloud Billing
page, make sure you have a billing account linked and it has theACTIVE
orTRIAL_ACTIVE
status. If you do not have a billing account, create one.
If you have an active billing account, you can go to the cloud page
Learn more about clouds and folders.
Required paid resources
The cost of maintaining your ClickHouse® database for Graphite includes:
- Fee for the cluster computing resources, storage, and backups (see Yandex Managed Service for ClickHouse® pricing).
- Fee for a running VM to manage a database (see Yandex Compute Cloud pricing).
Create a cluster
- Create a Managed Service for ClickHouse® cluster in any suitable configuration with the
db1
database and public access to all of its hosts. Save the database name, username, and password. - In the management console
, select Managed Service for ClickHouse® and go to the new cluster. Save the cluster ID in the Overview tab. - In the top-right corner, click Connect. In the Shell tab, save the
--host
parameter from the Sample connection string field, e.g.,rc1a-2sqal8f0********.mdb.yandexcloud.net
. This is the cluster host FQDN you will need later on.
Register the rollup configuration in a cluster
Register the rollup
configuration in your cluster to decimate and aggregate or average out the database contents for Graphite:
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
-
Prepare a YAML file named
graphite-rollup.yaml
with a description ofrollup
parameters, e.g.:name: test_rollup patterns: - regexp: click_cost function: max retention: - age: 86400 precision: 60
-
Run this command:
yc managed-clickhouse cluster add-graphite-rollup <cluster_ID> --rollup-file-name <yaml_file_path>
Where:
<cluster_ID>
: Cluster ID.--rollup-file-name
: Path tographite-rollup.yaml
.
For more information about the
managed-clickhouse cluster add-graphite-rollup
command, see the CLI reference.
Use the REST API update method by providing the required rollup
parameters in the request body:
"graphiteRollup": [
{
"name": "test_rollup",
"patterns": [
{
"regexp": "click_cost",
"function": "max",
"retention": [
{
"age": "86400",
"precision": "60"
}
]
}
]
}
]
Create and set up a virtual machine
-
In the same cloud network where the cluster is located, create a VM on Linux.
-
Connect to the VM via SSH.
-
Connect the ClickHouse® DEB repository
:sudo apt update && sudo apt install -y apt-transport-https ca-certificates dirmngr && \ sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv E0C56BD4 && \ echo "deb https://packages./deb stable main" | sudo tee \ /etc/apt/sources.list.d/clickhouse.list
-
Install the dependencies and the
clickhouse-client
client application:sudo apt update && sudo apt install -y clickhouse-client
-
Download the configuration file for
clickhouse-client
:mkdir -p ~/.clickhouse-client && \ wget "https://storage.yandexcloud.net/doc-files/clickhouse-client.conf.example" \ --output-document ~/.clickhouse-client/config.xml
-
Get an SSL certificate:
sudo mkdir --parents && \ sudo wget "" \ --output-document && \ sudo chmod 655 && \ sudo update-ca-certificates
Connect the virtual machine to a database
-
If you are using security groups for a cloud network, configure them to allow all required traffic between the cluster and the VM.
-
Run the ClickHouse® CLI with the following parameters: instead of
<host_FQDN>
,<db_name>
,<db_username>
and<DB_user_password>
, specify the previously saved parameters.clickhouse-client --host <host_FQDN> \ --secure \ --user <db_username> \ --database <db_name> \ --password <db_user_password> \ --port 9440
Create a GraphiteMergeTree table
In the ClickHouse® CLI interface, run a query to create a GraphiteMergeTreerollup
section described earlier:
CREATE TABLE GraphiteTable
(
Path String,
Time DateTime,
Value Int64,
Version UInt64
)
ENGINE = GraphiteMergeTree('test_rollup')
PARTITION BY Time
ORDER BY cityHash64(Version, Path)
Configure Graphite
Configure Graphite to save metrics in the ClickHouse® cluster. In this case, data is decimated automatically by the ClickHouse® server according to the parameters you specified.
-
Install the
carbon-clickhouse
utility in the system running Graphite:wget https://github.com/go-graphite/carbon-clickhouse/releases/download/v0.11.2/carbon-clickhouse_0.11.2_amd64.deb && \ sudo apt-get install $(pwd)/carbon-clickhouse_0.11.2_amd64.deb
-
Configure
carbon-clickhouse
. -
Run
carbon-clickhouse
:sudo systemctl enable carbon-clickhouse && \ sudo systemctl start carbon-clickhouse
-
Install the
graphite-clickhouse
utility:wget https://github.com/go-graphite/graphite-clickhouse/releases/download/v0.13.2/graphite-clickhouse_0.13.2_amd64.deb && \ sudo apt-get install $(pwd)/graphite-clickhouse_0.13.2_amd64.deb
-
Configure
graphite-clickhouse
. -
Run
graphite-clickhouse
:sudo systemctl enable graphite-clickhouse && \ sudo systemctl start graphite-clickhouse
For more information about configuring Graphite, see the relevant documentation
.
How to delete the resources you created
To remove the rollup
configuration from a cluster:
- Delete all the tables that use this configuration.
- Run the
yc managed-clickhouse cluster remove-graphite-rollup
command.
For more information about the command, see the CLI reference.
Alert
Removal of the rollup
configuration without first deleting the tables that use it may result in cluster failure.
Delete the resources you no longer need to avoid paying for them:
ClickHouse® is a registered trademark of ClickHouse, Inc