Contact UsGet started
© 2025 Direct Cursus Technology L.L.C.

Replicating logs to Yandex Object Storage using Fluent Bit

Written by
Updated at May 7, 2025

Data aggregators enable you to transmit data, e.g., logs, from VM instances to log monitoring and data storage services.

In this tutorial, you will learn how to replicate VM logs automatically to an Object Storage bucket using the Fluent Bit logging processor.

The solution described below works in the following way:

  1. Fluent Bit runs on an active VM as a systemd module.
  2. Fluent Bit collects logs as per the configuration settings and sends them to a Data Streams-enabled stream over the Amazon Kinesis Data Streams protocol.
  3. In your working folder, you set up a Data Transfer-enabled transfer that fetches data from the stream and saves it to an Object Storage bucket.

To set up log replication:

  1. Get your cloud ready.
  2. Set up your environment.
  3. Create an Object Storage bucket for storing your logs.
  4. Create a Data Streams-enabled data stream.
  5. Create a Data Transfer-enabled transfer.
  6. Install Fluent Bit.
  7. Connect Fluent Bit to your data stream.
  8. Test sending and receiving data.

If you no longer want to store logs, delete the resources allocated to them.

Get your cloud ready

Sign up in Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or register a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

The cost of data storage support includes:

Set up your environment

  1. Create a service account, e.g., logs-sa, with the editor role for the folder.

  2. Create a static access key for the service account. Save the ID and secret key. You will need them to log in to AWS.

  3. Create a VM from a public Ubuntu 20.04 image. Under Access, specify the service account you created at the previous step.

  4. Connect to the VM over SSH.

  5. Install the AWS CLI utility on your VM.

  6. Run this command:

    aws configure

  7. Enter the following, one by one:

    • AWS Access Key ID [None]:: Service account key ID.
    • AWS Secret Access Key [None]:: Service account secret key.
    • Default region name [None]:: ru-central1.

Create a bucket

  1. In the management console, select the folder where you want to create a bucket.
  2. Select Object Storage.
  3. Click Create bucket.
  4. Enter the name of the bucket.
  5. In the Storage class field, select Cold.
  6. Click Create bucket.

Create a data stream

  1. In the management console, select the folder to create a data stream in.
  2. Select Data Streams.
  3. Click Create stream.
  4. Specify an existing serverless YDB database or create a new one. If you have created a new database, click to update the database list.
  5. Enter the data stream name: logs-stream.
  6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from Creating to Active.

Create a transfer

  1. In the management console, select the folder where you want to create a transfer.
  2. Select Data Transfer.
  3. Create a source endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Source.
    3. Enter the endpoint name, for example, logs-source.
    4. In the Database type list, select Yandex Data Streams.
    5. Select the database you specified in the settings of the stream you created earlier.
    6. Enter the stream name: logs-stream.
    7. Select the logs-sa service account you created earlier.
    8. Under Advanced settings, specify the conversion rules for the CloudLogging parser data.
    9. Click Create.
  4. Create a target endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Target.
    3. Enter the endpoint name, for example, logs-receiver.
    4. In the Database type list, select Object Storage.
    5. Enter the name of the previously created bucket.
    6. Select the previously created logs-sa service account.
    7. In the Serialization format field, select JSON.
    8. Click Create.
  5. Create a transfer:
    1. In the Transfers tab, click Create transfer.
    2. Enter the transfer name, e.g., logs-transfer.
    3. Select the previously created source endpoint, logs-source.
    4. Select the previously created target endpoint, logs-receiver.
    5. Click Create.
  6. Click next to the created transfer and select Activate.

Wait until the transfer is activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Install Fluent Bit

Note

This tutorial uses the current Fluent Bit version, 1.9.

  1. To install Fluent Bit on your VM, run this command:

    curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

    For more information on how to install Fluent Bit, see this Fluent Bit guide.

  2. Start fluent-bit:

    sudo systemctl start fluent-bit

  3. Make sure the fluent-bit status is active:

    sudo systemctl status fluent-bit

    The result should include the active (running) status and logs for the embedded cpu plugin that Fluent Bit starts collecting by default as soon as installation is complete:

    ● fluent-bit.service - Fluent Bit
 Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
 Active: active (running) since Thu 2022-09-08 10:23:03 UTC; 10s ago
   Docs: https://docs.fluentbit.io/manual/
Main PID: 1328 (fluent-bit)
  Tasks: 4 (limit: 2310)
 Memory: 2.8M
 CGroup: /system.slice/fluent-bit.service
         └─1328 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

 Sep 08 10:23:03 ycl-20 fluent-bit[1328]: [2022/09/08 10:23:03] [ info] [output:stdout:stdout.0] worker #0 started
 Sep 08 10:23:05 ycl-20 fluent-bit[1328]: [0] cpu.local: [1662632584.114661597, {"cpu_p"=>1.000000, "user_p"=>0.000000, >
 Sep 08 10:23:06 ycl-20 fluent-bit[1328]: [0] cpu.local: [1662632585.114797726, {"cpu_p"=>0.000000, "user_p"=>0.000000, >
 ...

Connect Fluent Bit to your data stream

Note

If running Fluent Bit version below 1.9, which comes with the td-agent-bit package, edit the /etc/td-agent-bit/td-agent-bit.conf and /lib/systemd/system/td-agent-bit.service files and restart td-agent-bit.

  1. Open /etc/fluent-bit/fluent-bit.conf:

    sudo vim  /etc/fluent-bit/fluent-bit.conf

  2. Add the OUTPUT section with the kinesis_streams plugin settings:

    [OUTPUT]
    Name  kinesis_streams
    Match *
    region ru-central-1
    stream /<region>/<folder_ID>/<database_ID>/<stream_name>
    endpoint https://yds.serverless.yandexcloud.net

    Where:

    • stream: ID of the Data Streams-enabled data stream.

      For example, your stream ID will appear as /ru-central1/aoeu1kuk2dht********/cc8029jgtuab********/logs-stream if:

      • logs-stream: Stream name.
      • ru-central1: Region
      • aoeu1kuk2dht********: Folder ID.
      • cc8029jgtuab********: YDB database ID.

    For more information on how to install Fluent Bit, see this Fluent Bit guide.

  3. Open /lib/systemd/system/fluent-bit.service:

    sudo vim  /lib/systemd/system/fluent-bit.service

  4. Add the environment variables that include paths to the access key files to the SERVICE section:

    Environment=AWS_CONFIG_FILE=/home/<username>/.aws/config
Environment=AWS_SHARED_CREDENTIALS_FILE=/home/<username>/.aws/credentials

    Where <username> is the username you specified in the VM settings.

  5. Restart fluent-bit:

    sudo systemctl daemon-reload
sudo systemctl restart fluent-bit

  6. Check the fluent-bit status. It must not contain any error messages:

    sudo systemctl status fluent-bit

    Result:

    Sep 08 16:51:19 ycl-20 fluent-bit[3450]: Fluent Bit v1.9.8
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: * Copyright (C) 2015-2022 The Fluent Bit Authors
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: * https://fluentbit.io
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [fluent bit] version=1.9.8, commit=, pid=3450
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [cmetrics] version=0.3.6
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [sp] stream processor started
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [output:kinesis_streams:kinesis_streams.1] worker #0 started
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [output:stdout:stdout.0] worker #0 started

Test sending and receiving data

  1. In the management console, navigate to the folder with the new data stream, transfer, and bucket.
  2. Select Data Streams.
  3. Select the logs-stream data stream.
  4. Go to the Monitoring tab and check the stream activity charts.
  5. Select Data Transfer.
  6. Select logs-transfer.
  7. Go to the Monitoring tab and check the transfer activity charts.
  8. Select Object Storage.
  9. Select the previously created bucket.
  10. Make sure that you have objects in the bucket. Download and review the log files you got.

How to delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  1. Delete the transfer.
  2. Delete the endpoints.
  3. Delete the data stream.
  4. Delete the objects from the bucket.
  5. Delete the bucket.
Previous
Developing a Telegram bot
Next
Replicating logs to Object Storage using Data Streams