Fluentd

Written by

Updated at August 15, 2025

Note

You can create a trigger that will invoke a function in Cloud Functions or run a container in Serverless Containers when data is sent to a stream. Read more about triggers for Data Streams.

Download and install Fluentd.
Install the Fluentd plugin to add AWS Kinesis Data Streams protocol support. This protocol will be used for data delivery.
```
sudo fluent-gem install fluent-plugin-kinesis
```
In the management console, select the folder containing your data stream.
Select Data Streams.
Select the data stream.
Click Connect and navigate to the Fluentd tab.

Copy the example configuration and paste it into the /etc/td-agent/td-agent.conf file.

Example configuration:

<system>
  log_level debug
</system>
<source>
  @type http
  @id input_http
  port 8888
</source>
<match kinesis>
  @type copy
  <store>
    @type stdout
  </store>
  <store>
    @type kinesis_streams

    aws_key_id <access_key_ID>
    aws_sec_key <secret_key>

    # kinesis stream name
    stream_name /ru-central1/aoegtvhtp8ob********/cc8004q4lbo6********/test

    # region
    region ru-central-1

    endpoint https://yds.serverless.yandexcloud.net

    <buffer>
      flush_interval 5s
    </buffer>
  </store>
</match>

Where:

<access_key_ID>: Static access key ID.
<secret_key>: Secret part of the static access key.

Send test data to Fluentd:

curl \
 --request POST \
 --data 'json={"user_id":"user1", "score": 100}' \
 http://localhost:8888/kinesis

If the setup is successful, the Fluentd /var/log/td-agent/td-agent.log operational log will show a message confirming data receipt and its transmission to Yandex Data Streams over the AWS Kinesis Data Streams protocol:

kinesis: {"json":"message"}
DEBUG -- : [Aws::Kinesis::Client 200 0.628973 0 retries] put_records(stream_name:"/ru-central1/aoeu1kuk2dht********/cc8029jgtuab********/fluentd_stream",records:[{data:"{\"message\":\"Write chunk 5c0cf5c556654e99cac84*********** /   2 records /    0 KB\"}\n",partition_key:"6ec03a4e3ba832c85e802***********"},{data:"{\"message\":\"Finish writing chunk\"}\n",partition_key:"8ada32f7373e1ab4c48fb***********"},{data:"{\"json\":\"message\"}\n",partition_key:"70f21f2decfc90b6f1975***********"}])

Fluentd

Was the article helpful?