Contact UsTry it for free
© 2026 Direct Cursus Technology L.L.C.

Replicating logs to Yandex Object Storage using Yandex Data Streams

Written by
Updated at March 24, 2026

With Yandex Data Streams, you can set up automatic replication of service and user application logs to Yandex Object Storage.

The solution works as follows:

  1. Your Cloud Logging log group receives logs, e.g., from a VM.
  2. The log group is set up to automatically forward logs to a specific data stream in Data Streams.
  3. A Data Transfer transfer is set up to fetch data from the stream and save it to an Object Storage bucket.

To set up log replication:

  1. Get your cloud ready.
  2. Set up your environment.
  3. Create a bucket.
  4. Create a data stream.
  5. Connect the stream to the log group.
  6. Create a transfer.
  7. Test sending and receiving data.

If you no longer want to store logs, delete the resources allocated to them.

Getting started

Sign up for Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or create a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

  • Data Streams (see Data Streams pricing). The cost depends on the pricing model:

    • Based on allocated resources: You pay a fixed hourly rate for the established throughput limit and message retention period, and additionally for the number of units of actually written data.
    • On-demand: You pay for the performed read/write operations, the amount of read or written data, and the actual storage used for messages that are still within their retention period.

  • Managed Service for YDB database, operating in serverless mode: data operations, amount of stored data and backups (see Managed Service for YDB pricing).

  • Object Storage bucket: use of storage, data operations (see Object Storage pricing).

Set up your environment

  1. Create a service account, e.g., logs-sa, with the editor role for the folder.
  2. Set up the transfer of logs to the log group. For example, you can transfer logs from a VM or add test records to the log group.

Create a bucket

  1. In the management console, select the folder where you want to create a bucket.
  2. Go to Object Storage.
  3. Click Create bucket.
  4. Enter a name for the bucket.
  5. In the Storage class field, select Cold.
  6. Click Create bucket.

Create a data stream

  1. In the management console, select the folder where you want to create a data stream.
  2. Go to Data Streams.
  3. Click Create stream.
  4. Specify an existing serverless database in YDB or create a new one. If you have created a new database, click to update the database list.
  5. Name the data stream, e.g., logs-stream.
  6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from Creating to Active.

Connect the stream to the log group

  1. In the management console, select the folder with the log group.
  2. Go to Cloud Logging.
  3. Next to the log group receiving the logs, click and select Edit.
  4. In the Data stream field, select logs-stream you created earlier.
  5. Click Save.

Create a transfer

  1. In the management console, select the folder where you want to create a transfer.
  2. Go to Data Transfer.
  3. Create a source endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Source.
    3. Enter the endpoint name, e.g., logs-source.
    4. From the Database type list, select Yandex Data Streams.
    5. Select the database you specified in the settings of the stream you created earlier.
    6. Name the data stream: logs-stream.
    7. Select the logs-sa service account you created earlier.
    8. Under Advanced settings, specify the conversion rules for the CloudLogging parser data.
    9. Click Create.
  4. Create a target endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Target.
    3. Enter the endpoint name, e.g., logs-receiver.
    4. From the Database type list, select Object Storage.
    5. Enter the name of the previously created bucket.
    6. Select the logs-sa service account you created earlier.
    7. In the Serialization format field, select JSON.
    8. Click Create.
  5. Create a transfer:
    1. In the Transfers tab, click Create transfer.
    2. Enter the transfer name, e.g., logs-transfer.
    3. Select the logs-source source endpoint you created earlier.
    4. Select the logs-receiver target endpoint you created earlier.
    5. Click Create.
  6. Click next to the new transfer and select Activate.

Wait until the transfer gets activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Test sending and receiving data

  1. In the management console, navigate to the folder with the new data stream, transfer, and bucket.
  2. Go to Data Streams.
  3. Select the data stream named logs-stream.
  4. Go to the Monitoring tab and check the stream activity charts.
  5. Go to Data Transfer.
  6. Select the logs-transfer transfer.
  7. Go to the Monitoring tab and check the transfer activity charts.
  8. Go to Object Storage.
  9. Select the previously created bucket.
  10. Make sure that you have objects in the bucket. Download and review the log files you got.

Delete the resources you created

To reduce the consumption of resources, delete those you do not need:

  1. Delete the transfer.
  2. Delete the endpoints.
  3. Delete the data stream.
  4. Delete the objects from the bucket.
  5. Delete the bucket.
Previous
Replicating logs to Object Storage using Fluent Bit
Next
Visualizing logs in Grafana using the Yandex Cloud Logging plugin