Contact UsGet started

Replicating logs to Object Storage using Data Streams

Written by
Updated at June 11, 2024

With Yandex Data Streams, you can set up automatic replication of service and user application logs to Yandex Object Storage.

The solution works as described below:

  1. Logs, for example, from a VM instance, are sent to a Cloud Logging log group.
  2. The log group settings specify the Data Streams stream where the logs are transmitted automatically.
  3. A Data Transfer transfer is set up to fetch data from the stream and save it to an Object Storage bucket.

To set up log replication:

  1. Prepare your cloud.
  2. Configure the environment.
  3. Create a bucket.
  4. Create a data stream.
  5. Connect the stream to the log group.
  6. Create a transfer.
  7. Test sending and receiving data.

If you no longer want to store logs, delete the resources allocated to them.

Getting started

Sign up for Yandex Cloud and create a billing account:

  1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

The cost of data storage support includes:

Configure the environment

  1. Create a service account, e.g., logs-sa, with the editor role assigned for the folder.
  2. Set up the transfer of logs to the log group. For example, you can transfer logs from a VM instance or add test records to the log group.

Create a bucket

  1. In the management console, select the folder where you want to create a bucket.
  2. Select Object Storage.
  3. Click Create bucket.
  4. Enter the name of the bucket.
  5. In the Storage class field, select Cold.
  6. Click Create bucket.

Create a data stream

  1. In the management console, select the folder to create a data stream in.
  2. Select Data Streams.
  3. Click Create stream.
  4. Specify an existing serverless YDB database or create a new one. If you have created a new database, click to update the database list.
  5. Enter the data stream name: logs-stream.
  6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from Creating to Active.

Connect the stream to the log group

  1. In the management console, select Cloud Logging.
  2. Next to the log group where the logs are sent, click and select Edit.
  3. In the Data stream field, select the logs-stream created before.
  4. Click Save.

Create a transfer

  1. In the management console, select the folder where you want to create a transfer.
  2. Select Data Transfer.
  3. Create a source endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Source.
    3. Enter the endpoint name, for example, logs-source.
    4. In the Database type list, select Yandex Data Streams.
    5. Select the database you specified in the settings of the stream you created earlier.
    6. Enter the stream name: logs-stream.
    7. Select the logs-sa service account you created earlier.
    8. Under Advanced settings, specify the conversion rules for the CloudLogging parser data.
    9. Click Create.
  4. Create a target endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Target.
    3. Enter the endpoint name, for example, logs-receiver.
    4. In the Database type list, select Object Storage.
    5. Enter the name of the previously created bucket.
    6. Select the previously created logs-sa service account.
    7. In the Serialization format field, select JSON.
    8. Click Create.
  5. Create a transfer:
    1. In the Transfers tab, click Create transfer.
    2. Enter the transfer name, e.g., logs-transfer.
    3. Select the previously created source endpoint, logs-source.
    4. Select the previously created target endpoint, logs-receiver.
    5. Click Create.
  6. Click next to the created transfer and select Activate.

Wait until the transfer is activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Test sending and receiving data

  1. In the management console, navigate to the folder with the new data stream, transfer, and bucket.
  2. Select Data Streams.
  3. Select the logs-stream data stream.
  4. Go to the Monitoring tab and check the stream activity charts.
  5. Select Data Transfer.
  6. Select logs-transfer.
  7. Go to the Monitoring tab and check the transfer activity charts.
  8. Select Object Storage.
  9. Select the previously created bucket.
  10. Make sure that you have objects in the bucket. Download and review the log files you got.

How to delete the resources you created

To stop paying for the resources you created:

  1. Delete the transfer.
  2. Delete the endpoints.
  3. Delete the data stream.
  4. Delete the objects from the bucket.
  5. Delete the bucket.
Previous
Replicating logs to Object Storage using Fluent Bit
Next
Migrating data to Yandex Object Storage using Yandex Data Transfer