Yandex Cloud
Documentation
PresentationContact UsTry it for free
© 2026 Direct Cursus Technology L.L.C.
Tutorials
    • All tutorials
      • Adding data to a ClickHouse® DB
      • Fetching data from Managed Service for Apache Kafka® to Managed Service for ClickHouse®
      • Fetching data from Managed Service for Apache Kafka® to ksqlDB
      • Fetching data from RabbitMQ to Managed Service for ClickHouse®
      • Migrating data to Managed Service for ClickHouse® using ClickHouse®
      • Migrating data to Managed Service for ClickHouse® using Data Transfer
      • Migrating databases from Google BigQuery to Managed Service for ClickHouse®
      • Migrating data from Managed Service for OpenSearch to Managed Service for ClickHouse® with a storage change using Data Transfer
      • Delivering data from Managed Service for MySQL® to Managed Service for ClickHouse® using Data Transfer
      • Exchanging data between Managed Service for ClickHouse® and Yandex Data Processing
      • Configuring Managed Service for ClickHouse® for Graphite
      • Saving a data stream from Data Streams to Managed Service for ClickHouse®
      • Asynchronously replicating data from PostgreSQL to ClickHouse®
      • Asynchronous replication of data from Yandex Metrica to ClickHouse® using Data Transfer
      • Using hybrid storage in Managed Service for ClickHouse®
      • Sharding Managed Service for ClickHouse® tables
      • Executing analytical queries in Managed Service for PostgreSQL with processing in Managed Service for ClickHouse®
      • Loading data from Yandex Direct to a Managed Service for ClickHouse® data mart using Cloud Functions, Object Storage, and Data Transfer
      • Loading data from Object Storage to Managed Service for ClickHouse® using Data Transfer
      • Loading data from Managed Service for YDB to Managed Service for ClickHouse® using Data Transfer
      • Yandex Managed Service for ClickHouse® integration with Microsoft SQL Server via ClickHouse® JDBC Bridge
      • Yandex Managed Service for ClickHouse® integration with Oracle via ClickHouse® JDBC Bridge

In this article:

  • Required paid resources
  • Getting started
  • Set up your infrastructure
  • Configure additional settings
  • Set up the RabbitMQ integration for the Managed Service for ClickHouse® cluster
  • Create a RabbitMQ engine table in your Managed Service for ClickHouse® cluster
  • Send the test data to the RabbitMQ queue
  • Make sure the test data appears in the Managed Service for ClickHouse® cluster table
  • Delete the resources you created
  1. Building a data platform
  2. ClickHouse®
  3. Fetching data from RabbitMQ to Managed Service for ClickHouse®

Fetching data from RabbitMQ to Yandex Managed Service for ClickHouse®

Written by
Yandex Cloud
Updated at October 27, 2026
View in Markdown
  • Required paid resources
  • Getting started
    • Set up your infrastructure
    • Configure additional settings
  • Set up the RabbitMQ integration for the Managed Service for ClickHouse® cluster
  • Create a RabbitMQ engine table in your Managed Service for ClickHouse® cluster
  • Send the test data to the RabbitMQ queue
  • Make sure the test data appears in the Managed Service for ClickHouse® cluster table
  • Delete the resources you created

You can stream data from RabbitMQ to a Managed Service for ClickHouse® cluster in real time. Managed Service for ClickHouse® will automatically ingest data from exchanges of specified RabbitMQ queues using its RabbitMQ table engine as an intermediate listener, before inserting it into a permanent storage table.

To set up data delivery from RabbitMQ to Managed Service for ClickHouse®:

  1. Set up RabbitMQ integration for your Managed Service for ClickHouse® cluster.
  2. Create a RabbitMQ table in the Managed Service for ClickHouse® cluster.
  3. Send the test data to the RabbitMQ queue.
  4. Make sure the test data appears in the Managed Service for ClickHouse® cluster table.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

  • Managed Service for ClickHouse® cluster: use of computing resources allocated to hosts, as well as the storage and backup size (see Managed Service for ClickHouse® pricing).
  • Public IP addresses if public access is enabled for cluster hosts (see Yandex Virtual Private Cloud pricing).
  • VM instance: use of computing resources, storage, public IP address, and OS (see Yandex Compute Cloud pricing).

Getting startedGetting started

Set up your infrastructureSet up your infrastructure

Manually
Terraform
  1. Create a Managed Service for ClickHouse® cluster with your preferred configuration and add the database named db1. Enable public access to the cluster during creation so you can connect to it from your local machine. Connections from within the Yandex Cloud network are enabled by default.

    Note

    Public access to cluster hosts is required if you plan to connect to the cluster via the internet. This connection option is simpler and is recommended for the purposes of this guide. You can connect to non-public hosts as well but only from Yandex Cloud virtual machines located in the same cloud network as the cluster.

    Integration with RabbitMQ is available during cluster setup. In this example, however, we will configure the integration at a later stage.

  2. Create a virtual machine for RabbitMQ. Enable public access to the VM during creation so you can connect to it from your local machine. Connections from the Yandex Cloud network are enabled by default.

  1. If you do not have Terraform yet, install it.

  2. Get the authentication credentials. You can add them to environment variables or specify them later in the provider configuration file.

  3. Configure and initialize a provider. Instead of manually creating the provider configuration file, you can download it.

  4. Place the configuration file in a separate working directory and specify the parameter values. If you have not added the authentication credentials to environment variables, specify them in the configuration file.

  5. Download the clickhouse-cluster-and-vm-for-rabbitmq.tf configuration file to your current working directory.

    This file describes:

    • Network.
    • Subnet.
    • Default security group and inbound internet rules for your cluster and VM.
    • Managed Service for ClickHouse® cluster.
    • Virtual machine.
  6. In the clickhouse-cluster-and-vm-for-rabbitmq.tf file, specify the following:

    • Username and password that will be used to access the Managed Service for ClickHouse® cluster.
    • ID of the public, non-GPU Ubuntu image to use for the VM.
    • Username and path to the SSH public key for VM access. By default, the pre-configured image ignores the specified username and automatically creates a user named ubuntu. Use it to connect to the VM.
  7. Validate your Terraform configuration files using this command:

    terraform validate
    

    Terraform will display any configuration errors detected in your files.

  8. Create the required infrastructure:

    1. Run this command to view the intended changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply
        
      2. Confirm updating the resources.

      3. Wait for the operation to complete.

    All the required resources will be created in the specified folder. You can check resource availability and their settings in the management console.

Configure additional settingsConfigure additional settings

  1. Connect to a virtual machine over SSH.

    1. Install RabbitMQ:

      sudo apt update && sudo apt install rabbitmq-server --yes
      
    2. Create a user for RabbitMQ:

      sudo rabbitmqctl add_user <username> <password>
      
    3. Grant this user permissions to connect to the server:

      sudo rabbitmqctl set_permissions -p / <username> ".*" ".*" ".*" && \
      sudo rabbitmqctl set_topic_permissions -p / <username> amq.topic "cars" "cars"
      
  2. Install the amqp-publish and amqp-declare-queue tools for RabbitMQ and jq for processing JSON streams:

    sudo apt update && sudo apt install amqp-tools --yes && sudo apt-get install jq --yes
    
  3. Use amqp-declare-queue to create a queue named cars in RabbitMQ:

    amqp-declare-queue \
        --url=amqp://<username>:<password>@<IP_address_or_FQDN_of_the_RabbitMQ_server>:5672 \
        --queue=cars
    
  4. Install clickhouse-client to connect to the database in your Managed Service for ClickHouse® cluster.

    1. Add the ClickHouse® DEB repository:

      sudo apt update && sudo apt install --yes apt-transport-https ca-certificates dirmngr && \
      sudo apt-key adv --keyserver hkp://keyserver.ubuntu.com:80 --recv E0C56BD4 && \
      echo "deb https://packages.clickhouse.com/deb stable main" | sudo tee \
      /etc/apt/sources.list.d/clickhouse.list
      
    2. Install the dependencies:

      sudo apt update && sudo apt install clickhouse-client --yes
      
    3. Download the clickhouse-client configuration file:

      mkdir -p ~/.clickhouse-client && \
      wget "https://storage.yandexcloud.net/doc-files/clickhouse-client.conf.example" \
        --output-document ~/.clickhouse-client/config.xml
      

    Verify that you can establish an SSL connection to the Managed Service for ClickHouse® cluster via clickhouse-client.

Set up the RabbitMQ integration for the Managed Service for ClickHouse® clusterSet up the RabbitMQ integration for the Managed Service for ClickHouse® cluster

Manually
Terraform

In the Managed Service for ClickHouse® cluster settings, navigate to the DBMS settings → Rabbitmq section and specify the username and password for RabbitMQ authentication.

Add the clickhouse.config.rabbitmq block containing RabbitMQ username and password to the cluster configuration:

resource "yandex_mdb_clickhouse_cluster_v2" "clickhouse-cluster" {
  ...
  clickhouse = {
    ...
    config = {
      rabbitmq = {
        username = "<username>"
        password = "<password>"
      }
    }
    ...
  }
}

Create a RabbitMQ engine table in your Managed Service for ClickHouse® clusterCreate a RabbitMQ engine table in your Managed Service for ClickHouse® cluster

Suppose, you publish the following JSON car sensor data to the RabbitMQ exchange named exchange, which routes it to the cars queue:

  • device_id: Device string identifier.

  • datetime: Date and time of data generation in YYYY-MM-DD HH:MM:SS format.

  • Car coordinates:

    • latitude: Latitude.
    • longitude: Longitude.
    • altitude: Height above mean sea level.
  • speed: Current speed.

  • battery_voltage: Battery voltage for electric cars. null for ICE vehicles.

  • cabin_temperature: Temperature inside the car.

  • fuel_level: Fuel level for ICE cars. null for electric cars.

This data will be transmitted as RabbitMQ messages. Each message will contain a string containing a serialized JSON object with the following structure:

{"device_id":"iv9a94th6rzt********","datetime":"2020-06-05 17:27:00","latitude":"55.70329032","longitude":"37.65472196","altitude":"427.5","speed":"0","battery_voltage":"23.5","cabin_temperature":"17","fuel_level":null}

For table inserts, the Managed Service for ClickHouse® cluster will use the JSONEachRow format that parses rows from RabbitMQ messages into the required column values.

In the Managed Service for ClickHouse® cluster, create a table to store data incoming from RabbitMQ:

  1. Connect to the db1 database on your Managed Service for ClickHouse® cluster via clickhouse-client.

  2. Run this query:

    CREATE TABLE IF NOT EXISTS db1.cars (
        device_id String,
        datetime DateTime,
        latitude Float32,
        longitude Float32,
        altitude Float32,
        speed Float32,
        battery_voltage Nullable(Float32),
        cabin_temperature Float32,
        fuel_level Nullable(Float32)
    ) ENGINE = RabbitMQ
    SETTINGS
        rabbitmq_host_port = '<internal_IP_address_of_VM_with_RabbitMQ>:5672',
        rabbitmq_routing_key_list = 'cars',
        rabbitmq_exchange_name = 'exchange',
        rabbitmq_format = 'JSONEachRow';
    

This table will be automatically populated with messages consumed from the cars queue, which is bound to RabbitMQ exchange. When reading the data, Managed Service for ClickHouse® uses the authentication credentials provided earlier.

Send the test data to the RabbitMQ queueSend the test data to the RabbitMQ queue

  1. Create a file named sample.json with test data:

    {
        "device_id": "iv9a94th6rzt********",
        "datetime": "2020-06-05 17:27:00",
        "latitude": 55.70329032,
        "longitude": 37.65472196,
        "altitude": 427.5,
        "speed": 0,
        "battery_voltage": 23.5,
        "cabin_temperature": 17,
        "fuel_level": null
    }
    
    {
        "device_id": "rhibbh3y08qm********",
        "datetime": "2020-06-06 09:49:54",
        "latitude": 55.71294467,
        "longitude": 37.66542005,
        "altitude": 429.13,
        "speed": 55.5,
        "battery_voltage": null,
        "cabin_temperature": 18,
        "fuel_level": 32
    }
    
    {
        "device_id": "iv9a94th6rzt********",
        "datetime": "2020-06-07 15:00:10",
        "latitude": 55.70985913,
        "longitude": 37.62141918,
        "altitude": 417.0,
        "speed": 15.7,
        "battery_voltage": 10.3,
        "cabin_temperature": 17,
        "fuel_level": null
    }
    
  2. Use jq and amqp-publish to send data from sample.json to the previously created cars queue via exchange.

    jq \
    --raw-output \
    --compact-output . ./sample.json |\
    amqp-publish \
    --url=amqp://<RabbitMQ_username>:<password>@<IP_address_or_FQDN_of_the_RabbitMQ_server>:5672 \
    --routing-key=cars \
    --exchange=exchange
    

Make sure the test data appears in the Managed Service for ClickHouse® cluster tableMake sure the test data appears in the Managed Service for ClickHouse® cluster table

To access the data, use a materialized view. Once a materialized view is attached to a RabbitMQ table, it starts gathering data in the background automatically. This enables the system to continuously consume messages from RabbitMQ and convert them to the required format using SELECT.

Note

A message from the queue can be read by ClickHouse® only once. Therefore, instead of reading data directly from the table, use a materialized view.

To create a materialized view for the db1.cars table:

  1. Connect to the db1 database on your Managed Service for ClickHouse® cluster via clickhouse-client.

  2. Run the following queries:

    CREATE TABLE IF NOT EXISTS db1.cars_data_source (
        device_id String,
        datetime DateTime,
        latitude Float32,
        longitude Float32,
        altitude Float32,
        speed Float32,
        battery_voltage Nullable(Float32),
        cabin_temperature Float32,
        fuel_level Nullable(Float32)
    ) ENGINE MergeTree()
      ORDER BY device_id;
    
    CREATE MATERIALIZED VIEW db1.cars_view TO db1.cars_data_source
      AS SELECT * FROM db1.cars;
    

To retrieve all data from the db1.cars_view materialized view:

  1. Connect to the db1 database on your Managed Service for ClickHouse® cluster via clickhouse-client.

  2. Run this query:

    SELECT * FROM db1.cars_view;
    

The query results will show all data sent to RabbitMQ.

Delete the resources you createdDelete the resources you created

Delete the resources you no longer need to avoid paying for them:

Manually
Terraform
  • Delete the Yandex Managed Service for ClickHouse® cluster.
  • Delete the virtual machine.
  • If you reserved public static IP addresses, release and delete them.
  1. In the terminal window, go to the directory containing the infrastructure plan.

    Warning

    Make sure the directory has no Terraform manifests with the resources you want to keep. Terraform deletes all resources that were created using the manifests in the current directory.

  2. Delete resources:

    1. Run this command:

      terraform destroy
      
    2. Confirm deleting the resources and wait for the operation to complete.

    All the resources described in the Terraform manifests will be deleted.

ClickHouse® is a registered trademark of ClickHouse, Inc.

Was the article helpful?

Previous
Fetching data from Managed Service for Apache Kafka® to ksqlDB
Next
Migrating data to Managed Service for ClickHouse® using ClickHouse®
© 2026 Direct Cursus Technology L.L.C.