Asynchronously replicating data from Yandex Metrica to ClickHouse® using Yandex Data Transfer

With Data Transfer, you can transfer data from a Yandex Metrica tag to a ClickHouse® cluster. Such a transfer ensures the completeness of the data you transfer from Yandex Metrica and allows:

• Processing data with ClickHouse® tools.
• Streaming data from ClickHouse® to other locations.
• Visualizing data using Yandex DataLens or other services.

To transfer data:

1. Prepare and activate your transfer.
2. Test the transfer.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost includes:

• Managed Service for ClickHouse® cluster fee: Using computing resources allocated to hosts (including ZooKeeper hosts) and disk space (see Managed Service for ClickHouse® pricing).
• Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• Metrica Pro fee.
• Per-transfer fee: Using computing resources and the number of transferred data rows (see Data Transfer pricing).

Getting startedGetting started

Set up your infrastructure:

1. Select a Yandex Metrica tag or create and install a new one.

2. Create a Managed Service for ClickHouse® target cluster in any suitable configuration.

Prepare and activate your transferPrepare and activate your transfer

1. Create an endpoint for the Metrica source.

   Hits and sessions are transferred as separate tables.

2. Create an endpoint for the target:

   • Database type: ClickHouse

   • Endpoint parameters → Connection type: Managed cluster

     Select a target cluster from the list and specify its connection settings.

3. Create a transfer of the Replication type that will use the created endpoints.

4. Activate your transfer.

A transfer only moves the current data and does not affect historical data. If you deactivate and activate the transfer again:

1. Data obtained by the Yandex Metrica tag before the transfer was disabled will not be transferred.

2. Depending on the cleanup policy selected in the target endpoint, the existing data tables will be:

   • Drop: Deleted along with the data and created again with the same names.
   • Truncate: Purged of existing data without removing the tables and schemas.
   • Don't cleanup: Used for further data writes.

Test the transferTest the transfer

1. Wait for the transfer status to change to Replicating.

2. Make sure the data from the Yandex Metrica tag has been moved to the Managed Service for ClickHouse® database:

   1. Connect to the cluster using clickhouse-client.

   2. Check whether the hit and session tables have appeared in the database:

      SELECT table FROM system.tables
      WHERE database = '<ClickHouse®_database_name>'
      

      Result:

      ┌─table───────────────────────┐
      │ hits_dt...                  │
      │ visits_dt...                │
      └─────────────────────────────┘
      

   3. Check whether the hit and session tables contain data from the tag:

      SELECT * FROM <name_of_hit_or_visit_table>
      

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the transfer.
2. Delete the endpoints for both the source and target.
3. Delete the Managed Service for ClickHouse® cluster.
4. Delete the Yandex Metrica tag from your Yandex Metrica Pro account.

ClickHouse® is a registered trademark of ClickHouse, Inc.