Entering data into storage systems

Data from mobile phones, various smart devices, or external services can come in massive numbers of small batches. The communication circuits used for transmission are often slow, and the communication time limited. Yandex Data Streams receives data arriving at high frequency and speed and forms outbound data batches for the target systems, thus maintaining optimal operating modes for sources and targets. Using an API gateway to receive messages enables you to implement a custom data transmission protocol.

In this use case, the API gateway accepts incoming data and sends it to the data stream. Data is buffered in the stream and transmitted using a transfer to a cluster of ClickHouse® databases.

To set up data input:

1. Get your cloud ready.
2. Set up your environment.
3. Create a ClickHouse® cluster.
4. Create a data stream.
5. Create an API gateway.
6. Create a transfer.
7. Test sending and receiving data.

If you no longer need data input, delete the resources used.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of supporting data input into storage systems includes:

• Fee for requests to the API gateway (see Yandex API Gateway pricing).
• Data stream maintenance fees (see Yandex Data Streams pricing).
• Fees for transmitting data between sources and targets (see Yandex Data Transfer pricing).
• Fee for a continuously running Managed Service for ClickHouse® cluster (see Managed Service for ClickHouse® pricing).

Set up your environmentSet up your environment

Create a service account and assign it the editor role for your folder.

Create a ClickHouse® clusterCreate a ClickHouse® cluster

1. In the management console, select the folder where you want to create a DB cluster.
2. Select Managed Service for ClickHouse®.
3. Click Create cluster.
4. Specify the ClickHouse® cluster settings:
   1. Under Basic parameters:
   • Enter a name for the cluster.
   • Select the service account you created earlier.
   1. Under Database, specify the DB name, username, and password.
   2. Under Hosts, click . Enable Public access and click Save.
   3. Under Additional settings, enable the following options:
   • Access from Data Transfer
   • Access from the management console
   1. Specify the remaining cluster parameters according to the instructions.
5. Click Create cluster.

Wait for the cluster to start. When the cluster is ready for use, its status will change to Alive.

Create a data streamCreate a data stream

1. In the management console, select the folder to create a data stream in.
2. Select Data Streams.
3. Click Create stream.
4. Specify an existing serverless YDB database or create a new one. If you chose to create a new database, click Refresh after creating it to refresh the list of databases.
5. Enter a name for the stream.
6. Click Create.

Wait for the stream to start. When the stream is ready for use, its status will change from CREATING to ACTIVE.

Create an API gatewayCreate an API gateway

1. On the page of the created stream, click Actions and select API Gateway.

2. Enter the name of the API gateway.

3. Under Specification, replace the service_account_id key value with the ID of the service account you created earlier.

   Save the values of the Name and Service domain fields, you will need them later.

4. Click Create.

Wait for the API gateway to start. When the API gateway is ready for use, its status will change from CREATING to ACTIVE.

Create a transferCreate a transfer

1. In the management console, select the folder where you want to create a transfer.
2. Select Yandex Data Transfer.
3. Click Create data transfer.
4. Name the transfer.
5. Create a source endpoint:
   1. In the Source line, click Create new.
   2. Enter a name for the endpoint.
   3. In the Database type list, select Yandex Data Streams.
   4. Select a database for the source.
   5. Enter the name of the previously created stream.
   6. Select the service account you created earlier.
   7. Click Create.
6. Create a target endpoint:
   1. In the Target line, click Create new.
   2. Enter a name for the endpoint.
   3. In the Database type list, select ClickHouse.
   4. Select the MDB cluster you created earlier.
   5. Enter the DB name, username, and password of the cluster you created earlier.
   6. Click Create.
7. Click Create.
8. Click next to the name of the created transfer and select Activate.

Wait until the transfer is activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Test sending and receiving dataTest sending and receiving data

1. Send data to the storage system:

   curl --request POST --data 'test massage' https://<url>/<paths>
   

   Where:

   • <url>: The earlier saved Service domain value of the API gateway.
   • <paths>: The earlier saved Name value of the API gateway.

2. In the management console, select the Managed Service for ClickHouse® cluster you created earlier.

3. On the left-hand panel, select SQL.

4. Enter the username and password and click Connect.

5. In the list, select the database you created earlier.

6. Select the DB table.

If the configuration process is successful, a record containing service data and the sent message appears in the table.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you used:

1. Delete the API gateway.
2. Delete the transfer.
3. Delete the endpoints.
4. Delete the data stream.
5. Delete the ClickHouse® cluster.

ClickHouse® is a registered trademark of ClickHouse, Inc.