Transferring logs from a VM instance to Cloud Logging

Written by

Improved by

Updated at November 20, 2024

Getting started
Create a systemd service that generates logs
Install and configure Fluent Bit
Enable the plugin
View the logs
Delete the resources you created

The Fluent Bit log processor allows you to transfer the cluster logs from VM instances to Yandex Cloud Logging. The Fluent Bit plugin for Yandex Cloud Logging module is used to transfer logs.

To set up log transfer:

Getting started

Create a service account with the logging.writer role for the folder.
Create a VM from a public Ubuntu 20.04 image. Under Access, specify the service account that you created in the previous step.
Connect to the VM over SSH.

On the VM instance, install:

Go version 1.17 or higher:

wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
tar -xzf go1.17.6.linux-amd64.tar.gz
export PATH=$PWD/go/bin:$PATH

Git:
```
sudo apt install git
```

Create a systemd service that generates logs

Create a directory:

sudo mkdir /usr/local/bin/logtest
sudo chown $USER /usr/local/bin/logtest
cd /usr/local/bin/logtest

Create a file named logtest.py:

import logging
import random
import sys
import time

from systemdlogging.toolbox import check_for_systemd
from systemdlogging.toolbox import SystemdFormatter
from systemdlogging.toolbox import SystemdHandler

# You can replace the following few lines with the `systemdlogging.toolbox.init_systemd_logging` call
# This is a demo code

# Check if `systemd` is available
systemd_ok = check_for_systemd()

if systemd_ok:
    handler = SystemdHandler()
    # syslog_id: Value to use in the `SYSLOG_IDENTIFIER` field
    handler.syslog_id = ''
    handler.setFormatter(SystemdFormatter())

    # Get an instance of the logger object
    logger = logging.getLogger()
    logger.addHandler(handler)

else:
    logger = logging.getLogger(__name__)
    # Output logs to STDOUT
    handler = logging.StreamHandler(stream=sys.stdout)
    handler.setFormatter(logging.Formatter(
        '[%(levelname)s] %(code)d %(message)s'
    ))
    logger.addHandler(handler)

# Configure the default logging level (optional)
logger.setLevel(logging.DEBUG)

# Generate URL-like values
PATHS = [
    '/',
    '/admin',
    '/hello',
    '/docs',
]

PARAMS = [
    'foo',
    'bar',
    'query',
    'search',
    None
]


def fake_url():
    path = random.choice(PATHS)
    param = random.choice(PARAMS)
    if param:
        val = random.randint(0, 100)
        param += '=%s' % val
    code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0]
    return '?'.join(filter(None, [path, param])), code


if __name__ == '__main__':
    while True:
        # Create a code and URL value pair
        path, code = fake_url()
        # If the code is 200, write to the log with the `Info` level
        # Cloud Logging uses the textual logging level value so, in `context`, provide
        # the additional `SEVERITY` value
        # It will end up in `journald` where you can read it in the `yc-logging` plugin
        if code == 200:
            logger.info(
                'Path: %s',
                path,
                extra={"code": code, "context": {"SEVERITY": "info"}},
            )
        # Otherwise, with the `Error` level
        else:
            logger.error(
                'Error: %s',
                path,
                extra={"code": code, "context": {"SEVERITY": "error"}},
            )

        # Wait for 1 second to avoid log cluttering
        time.sleep(1)

Upgrade the versions of installed packages:
```
sudo apt-get update
```

Create a virtual environment and install the necessary dependencies:

sudo apt install python3-pip python3.8-venv

python3 -m venv venv
source venv/bin/activate
pip3 install systemd-logging

Create a file named logtest.sh:

#!/bin/bash

SCRIPT_PATH=$(dirname "$(realpath "$0")")
. "$SCRIPT_PATH/venv/bin/activate"
python "$SCRIPT_PATH/logtest.py"

systemd will call logtest.sh to run the service, so make the file executable:
```
chmod +x logtest.sh
```

Create a file named logtest.service:

[Unit]
Description=Sample to show logging from a Python application to systemd
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/logtest/logtest.sh
Restart=on-abort

[Install]
WantedBy=multi-user.target

Create a symbolic link for logtest.service:

sudo ln -s "$(pwd)/logtest.service" /etc/systemd/system/logtest.service

Restart systemd:
```
sudo systemctl daemon-reload
```

Check logtest.service status:

systemctl status logtest.service

Result:

● logtest.service - Sample to show logging from a Python application to systemd
 Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled)
 Active: inactive (dead)

Start logtest.service:
```
sudo systemctl start logtest.service
```

Recheck logtest.service status, it must now be active:

systemctl status logtest.service

Result:

● logtest.service - Sample to show logging from a Python application to systemd
     Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled)
     Active: active (running) since Wed 2022-02-02 18:49:03 UTC; 13h ago
   Main PID: 6550 (logtest.sh)
      Tasks: 2 (limit: 2311)
     Memory: 8.5M
     CGroup: /system.slice/logtest.service
             ├─6550 /bin/bash /usr/local/bin/logtest/logtest.sh
             └─6568 python /usr/local/bin/logtest/logtest.py

Install and configure Fluent Bit

Add the GPG key that you used to sign packages in the Fluent Bit repository:

wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

Add the following line to the /etc/apt/sources.list file:

deb https://packages.fluentbit.io/ubuntu/focal focal main

Upgrade the versions of installed packages:
```
sudo apt-get update
```
Install the fluent-bit package:
```
sudo apt-get install fluent-bit
```
Start fluent-bit:
```
sudo systemctl start fluent-bit
```

Check fluent-bit status, it must be active:

systemctl status fluent-bit

Result:

● fluent-bit.service - Fluent Bit
     Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-04-30 09:00:58 UTC; 3h 35min ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 589764 (fluent-bit)
      Tasks: 9 (limit: 2219)
     Memory: 18.8M
        CPU: 2.543s
     CGroup: /system.slice/fluent-bit.service
             └─589764 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Enable the plugin

Clone the repository with the plugin code:

git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git

Write the package versions to the environment variables:
```
cd fluent-bit-plugin-yandex/
export fluent_bit_version=3.0.3
export golang_version=1.22.2
export plugin_version=dev
```
Where:
- fluent_bit_version: fluent-bit version. To check the version, use the /opt/fluent-bit/bin/fluent-bit --version command.
- golang_version: Go compiler version. To check the version, use the go version command.
Exit the Python virtual environment and provide permissions for the plugin directory:
```
 deactivate
```
```
 sudo chown -R $USER:$USER /fluent-bit-plugin-yandex
```

Compile the yc-logging.so library:

CGO_ENABLED=1 go build -buildmode=c-shared \
    -o ./yc-logging.so \
    -ldflags "-X main.PluginVersion=${plugin_version}" \
    -ldflags "-X main.FluentBitVersion=${fluent_bit_version}"

Copy the yc-logging.so library to the fluent-bit library directory:

sudo cp yc-logging.so /usr/lib/fluent-bit/plugins/yc-logging.so

Add the path to the yc-logging.so library to the /etc/fluent-bit/plugins.conf file with plugin settings:
```
[PLUGINS]
    Path /usr/lib/fluent-bit/plugins/yc-logging.so
```

Add the fluent-bit settings to the /etc/fluent-bit/fluent-bit.conf file:

[INPUT]
    Name  systemd
    Tag   host.*
    Systemd_Filter  _SYSTEMD_UNIT=logtest.service

[OUTPUT]
    Name            yc-logging
    Match           *
    resource_type   logtest
    folder_id       <folder_ID>
    message_key     MESSAGE
    level_key       SEVERITY
    default_level   WARN
    authorization   instance-service-account

Where:

folder_id: ID of the folder to whose default log group the logs will be sent.
authorization: Authorization settings. Specify instance-service-account to log in under the service account you specified under Access when creating the VM.

Restart fluent-bit:
```
sudo systemctl restart fluent-bit
```

View the logs

Management console

CLI

API

In the management console, go to the folder you specified in the fluent-bit settings.
Select Cloud Logging.
Click the row with the default log group.
Go to the Logs tab.
The page that opens will show the log group records.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To view records in the log group, run this command:

yc logging read --folder-id=<folder_ID>

Wjere --folder-id is the folder ID specified in the fluent-bit settings.

To view log group records, use the LogReadingService/Read gRPC API call.

Delete the resources you created

If you no longer need the resources you created, delete them:

Transferring logs from a VM instance to Cloud Logging

Getting startedGetting started

Create a systemd service that generates logsCreate a systemd service that generates logs

Install and configure Fluent BitInstall and configure Fluent Bit

Enable the pluginEnable the plugin

View the logsView the logs

Delete the resources you createdDelete the resources you created

Was the article helpful?

Getting started

Create a systemd service that generates logs

Install and configure Fluent Bit

Enable the plugin

View the logs

Delete the resources you created