Asynchronously replicating data from Yandex Metrica to ClickHouse® using Yandex Data Transfer

With Data Transfer, you can transfer data from a Yandex Metrica tag to a ClickHouse® cluster. Such a transfer ensures the data transferred from Yandex Metrica are complete and allows:

• Processing data with ClickHouse® tools.
• Streaming data from ClickHouse® to other locations.
• Visualizing data using Yandex DataLens or other services.

To transfer data:

1. Prepare and activate the transfer.
2. Test the transfer.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Prepare the infrastructure:

1. Select a Yandex Metrica tag or create and install a new one.

2. Create a Managed Service for ClickHouse® target cluster with any suitable configuration.

Prepare and activate the transferPrepare and activate the transfer

1. Create an endpoint for the Metrica source.

   Hits and sessions are transferred as separate tables.

2. Create an endpoint for the target:

   • Database type: ClickHouse

   • Endpoint parameters → Connection type: Managed cluster

     Select a target cluster from the list and specify its connection settings.

3. Create a transfer of the Replication type that will use the created endpoints.

4. Activate your transfer.

A transfer only moves the current data and does not affect historical data. If you deactivate and activate the transfer again:

1. Data obtained by the Yandex Metrica tag before the transfer was disabled will not be transferred.

2. Depending on the cleanup policy selected in the target endpoint, the existing data tables will be:

   • Drop: Deleted along with the data and created again with the same names.
   • Truncate: Purged of existing data without removing the tables and schemas.
   • Don't cleanup: Used for further data writes.

Test the transferTest the transfer

1. Wait for the transfer status to change to Replicating.

2. Make sure the data from the Yandex Metrica tag has been moved to the Managed Service for ClickHouse® database:

   1. Connect to the cluster using clickhouse-client:

   2. Check whether the hit and session tables have appeared in the database:

      SELECT table FROM system.tables
      WHERE database = '<ClickHouse®_database_name>'
      

      Result:

      ┌─table───────────────────────┐
      │ hits_dt...                  │
      │ visits_dt...                │
      └─────────────────────────────┘
      

   3. Check whether the hit and session tables contain data from the tag:

      SELECT * FROM <name_of_hit_or_session_table>
      

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

1. Delete the transfer.
2. Delete endpoints for both the source and target.
3. Delete the Managed Service for ClickHouse® cluster.
4. Delete the Yandex Metrica tag from your Yandex Metrica Pro account.

ClickHouse® is a registered trademark of ClickHouse, Inc.