Ingesting data into storage systems

Data from mobile phones, various smart devices, or external services can come in massive numbers of small batches. The communication circuits used for transmission are often slow, and the communication time, limited. Yandex Data Streams receives data arriving at high frequency and speed and forms outbound data batches for the target systems, thus maintaining optimal operating modes for sources and targets. Using an API gateway to receive messages enables you to implement a custom data transmission protocol.

In this use case, the API gateway receives incoming data and sends it to the data stream. The stream buffers the data and sends it to the ClickHouse® database cluster through a transfer.

To set up data ingestion:

1. Get your cloud ready.
2. Set up your environment.
3. Create a ClickHouse® cluster.
4. Create a data stream.
5. Create an API gateway.
6. Create a transfer.
7. Test sending and receiving data.

If you no longer need to ingest data, delete the associated resources.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a linked billing account with an ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of supporting data ingestion into storage systems includes:

• Fee for API gateway requests (see Yandex API Gateway pricing).
• Data stream maintenance fee (see Yandex Data Streams pricing).
• Fee for transferring data between sources and targets (see Yandex Data Transfer pricing).
• Fee for a continuously running Managed Service for ClickHouse® cluster (see Managed Service for ClickHouse® pricing).

Set up your environmentSet up your environment

Create a service account and assign it the editor role for your folder.

Create a ClickHouse® clusterCreate a ClickHouse® cluster

1. In the management console, select the folder where you want to create a DB cluster.
2. Select Managed Service for ClickHouse®.
3. Click Create cluster.
4. Configure your ClickHouse® cluster:
   1. Under Basic parameters:
   • Enter a name for the cluster.
   • Select the service account you created earlier.
   1. Under Database, specify the DВ name, username, and password.
   2. Under Hosts, click . Enable Public access and click Save.
   3. Under Additional settings, enable the following options:
   • Access from Data Transfer.
   • Access from the management console.
   1. Specify the remaining cluster settings by following this guide.
5. Click Create cluster.

Wait for the cluster to start. When the cluster is ready for use, its status will change to Alive.

Create a data streamCreate a data stream

1. In the management console, select the folder where you want to create a data stream.
2. Select Data Streams.
3. Click Create stream.
4. Specify an existing serverless database in YDB or create a new one. If you chose to create a new database, click Refresh after creating it to refresh the list of databases.
5. Enter a name for the stream.
6. Click Create.

Wait for the stream to start. When the stream is ready for use, its status will change from CREATING to ACTIVE.

Create an API gatewayCreate an API gateway

1. On the page of the created stream, click Actions and select API Gateway.

2. Name your API gateway.

3. Under Specification, replace the service_account_id key value with the ID of the service account you created earlier.

   Save the values of the Name and Service domain fields, as you will need them later.

4. Click Create.

Wait for the API gateway to start. When the API gateway is ready for use, its status will change from CREATING to ACTIVE.

Create a transferCreate a transfer

1. In the management console, select the folder where you want to create a transfer.
2. Select Yandex Data Transfer.
3. Click Create data transfer.
4. Name the transfer.
5. Create a source endpoint:
   1. Next to Source, click Create new.
   2. Name the endpoint.
   3. In the Database type list, select Yandex Data Streams.
   4. Select a database for the source.
   5. Enter the name of the stream you created earlier.
   6. Select the service account you created earlier.
   7. Click Create.
6. Create a target endpoint:
   1. Next to Target, click Create new.
   2. Name the endpoint.
   3. In the Database type list, select ClickHouse.
   4. Select the MDB cluster you created earlier.
   5. Enter the DB name, username, and password of the cluster you created earlier.
   6. Click Create.
7. Click Create.
8. Click next to the name of the created transfer and select Activate.

Wait until the transfer gets activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Test sending and receiving dataTest sending and receiving data

1. Send data to the storage system:

   curl --request POST --data 'test massage' https://<url>/<paths>
   

   Where:

   • <url>: API gateway Service domain you saved earlier.
   • <paths>: API gateway Name you saved earlier.

2. In the management console, select the Managed Service for ClickHouse® cluster you created earlier.

3. On the left-hand panel, select SQL.

4. Enter the username and password and click Connect.

5. In the list, select select the previously created database.

6. Select a DB table.

If everything is set up properly, the table will show a new entry containing system data and the sent message.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you used in this scenario:

1. Delete the API gateway.
2. Delete the transfer.
3. Delete the endpoints.
4. Delete the data stream.
5. Delete the ClickHouse® cluster.

ClickHouse® is a registered trademark of ClickHouse, Inc.