Storing application runtime logs

Analyzing information about application events over any period requires secure storage of application runtime logs.

Applications do not usually send their logs to storage systems directly. Instead, they use aggregators such as fluentd, fluentbit, or logstash. Aggregators can write data directly to storage systems; however, for greater reliability, data first goes into an intermediate buffer (data stream bus, message broker) and only from there, to storage systems.

This approach lets developers focus on application features and delegate log delivery and storage to special systems.

In this tutorial, you will learn how to save application runtime logs to Yandex Object Storage.

To set up application runtime log storage:

1. Get your cloud ready.
2. Set up your environment.
3. Create a bucket for storing logs.
4. Create a data stream.
5. Create a transfer.
6. Install Fluentd.
7. Connect Fluentd to your data stream.
8. Test sending and receiving data.

If you no longer need to store any logs, delete the resources in use.

Get your cloud readyGet your cloud ready

Sign up in Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or register a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can navigate to the cloud page to create or select a folder for your infrastructure to operate in.

Learn more about clouds and folders.

Required paid resourcesRequired paid resources

The cost of application log storage support includes:

• Fee for data stream maintenance (see Yandex Data Streams pricing).
• Fee for transmitting data between sources and targets (see Yandex Data Transfer pricing).
• Fee for data storage (see Yandex Object Storage pricing).

Set up your environmentSet up your environment

1. Create a service account and assign it the editor role for your folder.
2. Create a static access key.

You will need the ID and secret key at the next steps.

Create a bucket for storing logsCreate a bucket for storing logs

1. In the management console, select the folder where you want to create a bucket.
2. From the list of services, select Object Storage.
3. Click Create bucket.
4. Name the bucket.
5. In the Object read access, Object listing access, and Read access to settings fields, select Limited.
6. In the Storage class field, select Cold.
7. Click Create bucket.

Create a data streamCreate a data stream

1. In the management console, select the folder to create a data stream in.
2. Select Yandex Data Streams.
3. Click Create stream.
4. Specify an existing serverless YDB database or create a new one. If you chose to create a new database, click Refresh after creating it to refresh the list of databases.
5. Enter a name for the stream.
6. Click Create.

Wait for the stream to start. Once the stream is ready for use, its status will change from CREATING to ACTIVE.

Create a transferCreate a transfer

1. On the page of the stream you created, click Actions and select Create data transfer.
2. Create a source endpoint:
   1. In the Direction field, select Source.
   2. Enter a name for the endpoint.
   3. From the Database type list, select Yandex Data Streams.
   4. Select a database for the source.
   5. Enter the name of the stream you created earlier.
   6. Select the service account you created earlier.
   7. Click Create.
3. Create a target endpoint:
   1. Click Create endpoint.
   2. In the Direction field, select Target.
   3. Enter a name for the endpoint.
   4. From the Database type list, select Object Storage.
   5. Enter the name of the bucket you created earlier.
   6. Select the service account you created earlier.
   7. Click Create.
4. Create a transfer:
   1. In the left-hand panel, select Transfers.
   2. Click Create transfer.
   3. Name the transfer.
   4. Select the source endpoint you created earlier.
   5. Select the target endpoint you created earlier.
   6. Click Create.
   7. Click next to the name of the created transfer and select Activate.

Wait until the transfer is activated. Once the transfer is ready for use, its status will change from Creating to Replicating.

Install FluentdInstall Fluentd

1. Download and install Fluentd.
2. Install the Fluentd plugin to support the AWS Kinesis Data Streams protocol. This protocol will be responsible for streaming data.

sudo td-agent-gem install fluent-plugin-kinesis

Connect Fluentd to your data streamConnect Fluentd to your data stream

1. On the page of the stream you created, click Connect and navigate to the Fluentd tab.
2. Copy the configuration file example and paste it into the /etc/td-agent/td-agent.conf file. Replace <key_id> and <secret> with the ID and secret key you got earlier.

<system>
  log_level debug
</system>
<source>
  @type http
  @id input_http
  port 8888
</source>
<match kinesis>
  @type copy
  <store>
    @type stdout
  </store>
  <store>
    @type kinesis_streams

    aws_key_id <key_id>
    aws_sec_key <secret>

    # kinesis stream name
    stream_name /ru-central1/b1gia92mbaom********/etnhstu01nin********/my-stream

    # region
    region ru-central-1

    endpoint https://yds.serverless.yandexcloud.net

    <buffer>
      flush_interval 5s
    </buffer>
  </store>
</match>

Test sending and receiving dataTest sending and receiving data

To send data to your stream using Fluentd, run the command:

curl --request POST --data 'json={"user_id":"user1", "score": 100}' http://localhost:8888/kinesis

If the setup is successful, the Fluentd operation log named /var/log/td-agent/td-agent.log will feature messages about receiving the data and writing it to Yandex Data Streams over the AWS Kinesis Data Streams protocol:

...
2022-04-20 19:36:37.770311035 +0000 kinesis: {"user_id":"user1","score":100}
2022-04-20 19:36:42 +0000 [debug]: #0 /ru-central1/b1gia92mbaom********/etnhstu01nin********/my-stream: Write chunk 5dd1b1ca1bd788e49185aa681e8132b9 /   1 records /    0 KB
2022-04-20 19:36:42 +0000 [debug]: #0 /ru-central1/b1gia92mbaom********/etnhstu01nin********/my-stream: Finish writing chunk
...

The bucket you created will contain a file with the sent message.

How to delete the resources you createdHow to delete the resources you created

To stop paying for the resources you used:

1. Delete the transfer.
2. Delete the endpoints.
3. Delete the data stream.
4. Delete the bucket.