Storing application runtime logs

To analyze information about application events for any period, application runtime logs need to be stored securely.

Applications do not usually send their logs to storage systems directly. Instead, they use aggregators such as fluentd, fluentbit, or logstash. Aggregators can write data directly to storage systems, but for greater reliability, data first goes into an intermediate buffer (data stream bus, message broker) and only from there to storage systems.

This approach lets developers focus on application functionality and delegate log delivery and storage to special systems.

Using this tutorial, you'll learn how to save application runtime logs to Yandex Object Storage.

To set up application runtime log storage:

1. Prepare your cloud.
2. Configure the environment.
3. Create a bucket for storing logs.
4. Create a data stream.
5. Create a transfer.
6. Install Fluentd.
7. Connect Fluentd to your data stream.
8. Test sending and receiving data.

If you no longer need to store any logs, delete the resources in use.

Prepare your cloudPrepare your cloud

Sign up for Yandex Cloud and create a billing account:

1. Go to the management console and log in to Yandex Cloud or create an account if you do not have one yet.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one.

If you have an active billing account, you can go to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of application log storage support includes:

• Data stream maintenance fees (see Yandex Data Streams pricing).
• Fees for transmitting data between sources and targets (see Yandex Data Transfer pricing).
• Data storage fees (see Yandex Object Storage pricing).

Configure the environmentConfigure the environment

1. Create a service account and assign it the editor role for your folder.
2. Create a static access key.

You'll need the ID and secret key in the next steps.

Create a bucket for storing logsCreate a bucket for storing logs

1. In the management console, select the folder you want to create a bucket in.
2. In the list of services, select Object Storage.
3. Click Create bucket.
4. Name the bucket.
5. In the Object read access, Object listing access, and Read access to settings fields, select Limited.
6. In the Storage class field, select Cold.
7. Click Create bucket.

Create a data streamCreate a data stream

1. In the management console, select the folder to create a data stream in.
2. Select Yandex Data Streams.
3. Click Create stream.
4. Specify an existing serverless YDB database or create a new one. If you chose to create a new database, click Update after creating it to update the list of databases.
5. Enter a name for the stream.
6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from CREATING to ACTIVE.

Create a transferCreate a transfer

1. On the page of the created stream, click Actions and select Create data transfer.
2. Create a source endpoint:
   1. In the Direction field, select Source.
   2. Enter a name for the endpoint.
   3. In the Database type list, select Yandex Data Streams.
   4. Select a database for the source.
   5. Enter the name of the previously created stream.
   6. Select the service account you created earlier.
   7. Click Create.
3. Create a target endpoint:
   1. Click Create endpoint.
   2. In the Direction field, select Target.
   3. Enter a name for the endpoint.
   4. In the Database type list, select Object Storage.
   5. Enter the name of the previously created bucket.
   6. Select the service account you created earlier.
   7. Click Create.
4. Create a transfer:
   1. In the left-hand panel, select Transfers.
   2. Click Create transfer.
   3. Name the transfer.
   4. Select the previously created source endpoint.
   5. Select the previously created target endpoint.
   6. Click Create.
   7. Click next to the name of the created transfer and select Activate.

Wait until the transfer is activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Install FluentdInstall Fluentd

1. Download and install Fluentd.
2. Install the Fluentd plugin to support the AWS Kinesis Data Streams protocol. This protocol will be used for streaming data.

sudo td-agent-gem install fluent-plugin-kinesis

Connect Fluentd to your data streamConnect Fluentd to your data stream

1. On the page of the created stream, click Connect and go to the Fluentd tab.
2. Copy the sample configuration file and paste it into the /etc/td-agent/td-agent.conf file. Replace <key_id> and <secret> with the previously obtained ID and secret key.

<system>
  log_level debug
</system>
<source>
  @type http
  @id input_http
  port 8888
</source>
<match kinesis>
  @type copy
  <store>
    @type stdout
  </store>
  <store>
    @type kinesis_streams

    aws_key_id <key_id>
    aws_sec_key <secret>

    # kinesis stream name
    stream_name /ru-central1/b1gia92mbaom********/etnhstu01nin********/my-stream

    # region
    region ru-central-1

    endpoint https://yds.serverless.yandexcloud.net

    <buffer>
      flush_interval 5s
    </buffer>
  </store>
</match>

Test sending and receiving dataTest sending and receiving data

To send data to the stream using Fluentd, run the command:

curl --request POST --data 'json={"user_id":"user1", "score": 100}' http://localhost:8888/kinesis

If the setup is successful, the Fluentd /var/log/td-agent/td-agent.log operation log will feature messages about receiving the data and writing it to Yandex Data Streams via the AWS Kinesis Data Streams protocol:

...
2022-04-20 19:36:37.770311035 +0000 kinesis: {"user_id":"user1","score":100}
2022-04-20 19:36:42 +0000 [debug]: #0 /ru-central1/b1gia92mbaom********/etnhstu01nin********/my-stream: Write chunk 5dd1b1ca1bd788e49185aa681e8132b9 /   1 records /    0 KB
2022-04-20 19:36:42 +0000 [debug]: #0 /ru-central1/b1gia92mbaom********/etnhstu01nin********/my-stream: Finish writing chunk
...

The created bucket will contain a file with the message sent.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you used:

1. Delete the transfer.
2. Delete the endpoints.
3. Delete the data stream.
4. Delete the bucket.