Transferring logs from a VM to Cloud Logging

Written by

Improved by

Updated at May 5, 2025

Getting started
Create a systemd service that generates logs
Install and configure Fluent Bit
Enable the plugin
View the logs
Delete the resources you created

The Fluent Bit log processor allows you to transfer logs from VM instances to Yandex Cloud Logging. To transfer logs, you will use the Fluent Bit plugin for Yandex Cloud Logging module.

To set up log transfer:

Create a `systemd` service that generates logs

Create a directory:

sudo mkdir /usr/local/bin/logtest
sudo chown $USER /usr/local/bin/logtest
cd /usr/local/bin/logtest

Create a file named logtest.py:

import logging
import random
import sys
import time

from systemdlogging.toolbox import check_for_systemd
from systemdlogging.toolbox import SystemdFormatter
from systemdlogging.toolbox import SystemdHandler

# You can replace these lines with the `systemdlogging.toolbox.init_systemd_logging` call.
# This code is a sample.

# Check whether `systemd` is available.
systemd_ok = check_for_systemd()

if systemd_ok:
    handler = SystemdHandler()
    # syslog_id: Value to use in the `SYSLOG_IDENTIFIER` field.
    handler.syslog_id = ''
    handler.setFormatter(SystemdFormatter())

    # Get an instance of the logger object.
    logger = logging.getLogger()
    logger.addHandler(handler)

else:
    logger = logging.getLogger(__name__)
    # Output logs to STDOUT.
    handler = logging.StreamHandler(stream=sys.stdout)
    handler.setFormatter(logging.Formatter(
        '[%(levelname)s] %(code)d %(message)s'
    ))
    logger.addHandler(handler)

# Optionally, configure the default logging level.
logger.setLevel(logging.DEBUG)

# Generate URL-like values.
PATHS = [
    '/',
    '/admin',
    '/hello',
    '/docs',
]

PARAMS = [
    'foo',
    'bar',
    'query',
    'search',
    None
]


def fake_url():
    path = random.choice(PATHS)
    param = random.choice(PARAMS)
    if param:
        val = random.randint(0, 100)
        param += '=%s' % val
    code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0]
    return '?'.join(filter(None, [path, param])), code


if __name__ == '__main__':
    while True:
        # Create a pair consisting of a code and a URL value.
        path, code = fake_url()
        # If the code is 200, write to the `Info` level log.
        # Cloud Logging uses the textual logging level value; therefore, in `context`, provide
        # the additional `SEVERITY` value.
        # This value will end up in `journald`, where you can read it in the `yc-logging` plugin.
        if code == 200:
            logger.info(
                'Path: %s',
                path,
                extra={"code": code, "context": {"SEVERITY": "info"}},
            )
        # Otherwise, write to the `Error` level log.
        else:
            logger.error(
                'Error: %s',
                path,
                extra={"code": code, "context": {"SEVERITY": "error"}},
            )

        # To avoid log cluttering, wait for 1 second.
        time.sleep(1)

Upgrade the versions of the installed packages:
```
sudo apt-get update
```

Create a virtual environment and install the necessary dependencies:

sudo apt install python3-pip python3.8-venv

python3 -m venv venv
source venv/bin/activate
pip3 install systemd-logging

Create a file named logtest.sh:

#!/bin/bash

SCRIPT_PATH=$(dirname "$(realpath "$0")")
. "$SCRIPT_PATH/venv/bin/activate"
python "$SCRIPT_PATH/logtest.py"

systemd will call logtest.sh to run the service; therefore, make the file executable:
```
chmod +x logtest.sh
```

Create a file named logtest.service:

[Unit]
Description=Sample to show logging from a Python application to systemd
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/logtest/logtest.sh
Restart=on-abort

[Install]
WantedBy=multi-user.target

Create a symbolic link for logtest.service:

sudo ln -s "$(pwd)/logtest.service" /etc/systemd/system/logtest.service

Restart systemd:
```
sudo systemctl daemon-reload
```

Check logtest.service status:

systemctl status logtest.service

Result:

● logtest.service - Sample to show logging from a Python application to systemd
 Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled)
 Active: inactive (dead)

Start logtest.service:
```
sudo systemctl start logtest.service
```

Recheck logtest.service status; the service must now be active:

systemctl status logtest.service

Result:

● logtest.service - Sample to show logging from a Python application to systemd
     Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled)
     Active: active (running) since Wed 2022-02-02 18:49:03 UTC; 13h ago
   Main PID: 6550 (logtest.sh)
      Tasks: 2 (limit: 2311)
     Memory: 8.5M
     CGroup: /system.slice/logtest.service
             ├─6550 /bin/bash /usr/local/bin/logtest/logtest.sh
             └─6568 python /usr/local/bin/logtest/logtest.py

Install and configure Fluent Bit

Add the GPG key you used to sign the packages in the Fluent Bit repository:

wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

Add the following line to the /etc/apt/sources.list file:

deb https://packages.fluentbit.io/ubuntu/focal focal main

Upgrade the versions of the installed packages:
```
sudo apt-get update
```
Install the fluent-bit package:
```
sudo apt-get install fluent-bit
```
Start fluent-bit:
```
sudo systemctl start fluent-bit
```

Check fluent-bit status; the service must be active:

systemctl status fluent-bit

Result:

● fluent-bit.service - Fluent Bit
     Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-04-30 09:00:58 UTC; 3h 35min ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 589764 (fluent-bit)
      Tasks: 9 (limit: 2219)
     Memory: 18.8M
        CPU: 2.543s
     CGroup: /system.slice/fluent-bit.service
             └─589764 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Enable the plugin

Clone the repository with the plugin code:

git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git

Write the package versions to the environment variables:
```
cd fluent-bit-plugin-yandex/
export fluent_bit_version=3.0.3
export golang_version=1.22.2
export plugin_version=dev
```
Where:
- fluent_bit_version: To check the version, use the /opt/fluent-bit/bin/fluent-bit --version command.
- golang_version: Go compiler version. To check the version, use the go version command.
Exit the Python virtual environment and provide permissions for the plugin directory:
```
 deactivate
```
```
 sudo chown -R $USER:$USER /fluent-bit-plugin-yandex
```

Compile the yc-logging.so library:

CGO_ENABLED=1 go build -buildmode=c-shared \
    -o ./yc-logging.so \
    -ldflags "-X main.PluginVersion=${plugin_version}" \
    -ldflags "-X main.FluentBitVersion=${fluent_bit_version}"

Copy the yc-logging.so library to the fluent-bit library directory:

sudo cp yc-logging.so /usr/lib/fluent-bit/plugins/yc-logging.so

Add the yc-logging.so library path to the /etc/fluent-bit/plugins.conf plugin settings file:
```
[PLUGINS]
    Path /usr/lib/fluent-bit/plugins/yc-logging.so
```

Add the fluent-bit settings to the /etc/fluent-bit/fluent-bit.conf file:

[INPUT]
    Name  systemd
    Tag   host.*
    Systemd_Filter  _SYSTEMD_UNIT=logtest.service

[OUTPUT]
    Name            yc-logging
    Match           *
    resource_type   logtest
    folder_id       <folder_ID>
    message_key     MESSAGE
    level_key       SEVERITY
    default_level   WARN
    authorization   instance-service-account

Where:

folder_id: ID of the folder whose default log group will get the logs.
authorization: Authorization settings. Specify instance-service-account to log in as a service account you specified under Access when creating the VM.

Restart fluent-bit:
```
sudo systemctl restart fluent-bit
```

View the logs

Management console

CLI

API

In the management console, go to the folder you specified in the fluent-bit settings.
Select Cloud Logging.
Click the row with the default log group.
Navigate to the Logs tab.
The page that opens will show the log group records.

If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

To view records in the log group, run this command:

yc logging read --folder-id=<folder_ID>

Where --folder-id is the ID specified in the fluent-bit settings.

To view log group records, use the LogReadingService/Read gRPC API call.

Delete the resources you created

If you no longer need the resources you created, delete them:

Transferring logs from a VM to Cloud Logging

Getting startedGetting started

Create a service that generates logsCreate a systemd service that generates logs

Install and configure Fluent BitInstall and configure Fluent Bit

Enable the pluginEnable the plugin

View the logsView the logs

Delete the resources you createdDelete the resources you created

Was the article helpful?

Getting started

Create a `systemd` service that generates logs

Install and configure Fluent Bit

Enable the plugin

View the logs

Delete the resources you created