Contact UsGet started

Replicating logs to Object Storage using Fluent Bit

Written by
Updated at May 23, 2024

Data aggregators enable you to transmit data, e.g., logs, from VM instances to log monitoring and data storage services.

In this tutorial, you will learn how to replicate VM logs automatically to an Object Storage bucket using the Fluent Bit logging processor.

The solution described below works in the following way:

  1. Fluent Bit runs on an active VM as a systemd module.
  2. Fluent Bit collects logs according to the configuration settings and sends them to a Data Streams stream via the Amazon Kinesis Data Streams protocol.
  3. In the working folder, you set up a Data Transfer transfer that fetches data from the stream and saves it to an Object Storage bucket.

To set up log replication:

  1. Prepare your cloud.
  2. Configure the environment.
  3. Create an Object Storage bucket for storing your logs.
  4. Create a data stream Data Streams.
  5. Create a transfer Data Transfer.
  6. Install Fluent Bit.
  7. Connect Fluent Bit to a data stream.
  8. Test sending and receiving data.

If you no longer want to store logs, delete the resources allocated to them.

Prepare your cloud

Sign up for Yandex Cloud and create a billing account:

  1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

The cost of data storage support includes:

Configure the environment

  1. Create a service account, e.g., logs-sa, with the editor role assigned for the folder.

  2. Create a static access key for the service account. Save the ID and private key. You will need them to log in to AWS.

  3. Create a VM from a public Ubuntu 20.04 image. Under Access, specify the service account that you created in the previous step.

  4. Connect to the VM via SSH.

  5. Install the AWS CLI utility on the VM.

  6. Run this command:

    aws configure

  7. Enter the following one by one:

    • AWS Access Key ID [None]:: Service account key ID.
    • AWS Secret Access Key [None]:: Secret access key of the service account.
    • Default region name [None]: ru-central1 region.

Create a bucket

  1. In the management console, select the folder where you want to create a bucket.
  2. Select Object Storage.
  3. Click Create bucket.
  4. Enter the name of the bucket.
  5. In the Storage class field, select Cold.
  6. Click Create bucket.

Create a data stream

  1. In the management console, select the folder to create a data stream in.
  2. Select Data Streams.
  3. Click Create stream.
  4. Specify an existing serverless YDB database or create a new one. If you have created a new database, click to update the database list.
  5. Enter the data stream name: logs-stream.
  6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from Creating to Active.

Create a transfer

  1. In the management console, select the folder where you want to create a transfer.
  2. Select Data Transfer.
  3. Create a source endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Source.
    3. Enter the endpoint name, for example, logs-source.
    4. In the Database type list, select Yandex Data Streams.
    5. Select the database you specified in the settings of the stream you created earlier.
    6. Enter the stream name: logs-stream.
    7. Select the logs-sa service account you created earlier.
    8. Under Advanced settings, specify the conversion rules for the CloudLogging parser data.
    9. Click Create.
  4. Create a target endpoint:
    1. In the Endpoints tab, click Create endpoint.
    2. In the Direction field, select Target.
    3. Enter the endpoint name, for example, logs-receiver.
    4. In the Database type list, select Object Storage.
    5. Enter the name of the previously created bucket.
    6. Select the previously created logs-sa service account.
    7. In the Serialization format field, select JSON.
    8. Click Create.
  5. Create a transfer:
    1. In the Transfers tab, click Create transfer.
    2. Enter the transfer name, e.g., logs-transfer.
    3. Select the previously created source endpoint, logs-source.
    4. Select the previously created target endpoint, logs-receiver.
    5. Click Create.
  6. Click next to the created transfer and select Activate.

Wait until the transfer is activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Install Fluent Bit

Note

This tutorial uses the current Fluent Bit version 1.9.

  1. To install Fluent Bit to your VM, run the command:

    curl https://raw.githubusercontent.com/fluent/fluent-bit/master/install.sh | sh

    For more information on how to install Fluent Bit, see the official documentation.

  2. Run the fluent-bit service:

    sudo systemctl start fluent-bit

  3. Check the fluent-bit service status, it should be active:

    sudo systemctl status fluent-bit

    The result should include the active (running) status and logs for the embedded cpu plugin that Fluent Bit starts collecting by default once installation is complete:

    ● fluent-bit.service - Fluent Bit
 Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
 Active: active (running) since Thu 2022-09-08 10:23:03 UTC; 10s ago
   Docs: https://docs.fluentbit.io/manual/
Main PID: 1328 (fluent-bit)
  Tasks: 4 (limit: 2310)
 Memory: 2.8M
 CGroup: /system.slice/fluent-bit.service
         └─1328 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

 Sep 08 10:23:03 ycl-20 fluent-bit[1328]: [2022/09/08 10:23:03] [ info] [output:stdout:stdout.0] worker #0 started
 Sep 08 10:23:05 ycl-20 fluent-bit[1328]: [0] cpu.local: [1662632584.114661597, {"cpu_p"=>1.000000, "user_p"=>0.000000, >
 Sep 08 10:23:06 ycl-20 fluent-bit[1328]: [0] cpu.local: [1662632585.114797726, {"cpu_p"=>0.000000, "user_p"=>0.000000, >
 ...

Connect Fluent Bit to the data stream

Note

If you are running Fluent Bit version below 1.9 that comes with the td-agent-bit package, edit the /etc/td-agent-bit/td-agent-bit.conf, /lib/systemd/system/td-agent-bit.service files and restart the td-agent-bit service.

  1. Open the /etc/fluent-bit/fluent-bit.conf file:

    sudo vim  /etc/fluent-bit/fluent-bit.conf

  2. Add the OUTPUT section with the kinesis_streams plugin settings:

    [OUTPUT]
    Name  kinesis_streams
    Match *
    region ru-central-1
    stream /<region>/<folder_ID>/<database_ID>/<data_stream_ID>
    endpoint https://yds.serverless.yandexcloud.net

    Where:

    • data-stream: Data Streams data stream ID.

      For example, your stream ID will appear as /ru-central1/aoeu1kuk2dht********/cc8029jgtuab********/logs-stream if:

      • logs-stream: Stream name
      • ru-central1: Region
      • aoeu1kuk2dht********: Folder ID
      • cc8029jgtuab********: YDB database ID

    For more information on how to install Fluent Bit, see the official documentation.

  3. Open the /lib/systemd/system/fluent-bit.service file:

    sudo vim  /lib/systemd/system/fluent-bit.service

  4. To the SERVICE section, add the environment variables that include paths to files with access keys:

    Environment=AWS_CONFIG_FILE=/home/<username>/.aws/config
Environment=AWS_SHARED_CREDENTIALS_FILE=/home/<username>/.aws/credentials

    Where <username> is the username that you specified in the VM settings.

  5. Restart the fluent-bit service:

    sudo systemctl daemon-reload
sudo systemctl restart fluent-bit

  6. Check the status of the fluent-bit service. It should not include any error messages:

    sudo systemctl status fluent-bit

    Result:

    Sep 08 16:51:19 ycl-20 fluent-bit[3450]: Fluent Bit v1.9.8
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: * Copyright (C) 2015-2022 The Fluent Bit Authors
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: * Fluent Bit is a CNCF sub-project under the umbrella of Fluentd
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: * https://fluentbit.io
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [fluent bit] version=1.9.8, commit=, pid=3450
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [storage] version=1.2.0, type=memory-only, sync=normal, checksum=disabled, max_chunks_up=128
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [cmetrics] version=0.3.6
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [sp] stream processor started
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [output:kinesis_streams:kinesis_streams.1] worker #0 started
Sep 08 16:51:19 ycl-20 fluent-bit[3450]: [2022/09/08 16:51:19] [ info] [output:stdout:stdout.0] worker #0 started

Test sending and receiving data

  1. In the management console, navigate to the folder with the new data stream, transfer, and bucket.
  2. Select Data Streams.
  3. Select the logs-stream data stream.
  4. Go to the Monitoring tab and check the stream activity charts.
  5. Select Data Transfer.
  6. Select logs-transfer.
  7. Go to the Monitoring tab and check the transfer activity charts.
  8. Select Object Storage.
  9. Select the previously created bucket.
  10. Make sure that you have objects in the bucket. Download and review the log files you got.

How to delete the resources you created

Some resources are not free of charge. To avoid paying for them, delete the resources you no longer need:

  1. Delete the transfer.
  2. Delete the endpoints.
  3. Delete the data stream.
  4. Delete the objects from the bucket.
  5. Delete the bucket.
Previous
Transferring logs through Unified Agent HTTP input to Cloud Logging
Next
Replicating logs to Object Storage using Data Streams