Contact UsGet started

Transferring logs from a VM to Yandex Cloud Logging

Written by
Improved by
Updated at May 13, 2025

The Fluent Bit log processor allows you to transfer logs from VM instances to Yandex Cloud Logging. To transfer logs, you will use the Fluent Bit plugin for Yandex Cloud Logging module.

To set up log transfer:

  1. Create a systemd service that generates logs.
  2. Install and configure Fluent Bit.
  3. Enable the plugin.

Getting started

  1. Create a service account with the logging.writer role for the folder.
  2. Create a VM from a public Ubuntu 20.04 image. Under Access, specify the service account you created at the previous step.
  3. Connect to the VM over SSH.
  4. On the VM, install:

    • Go version 1.17 or higher:

      wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz
tar -xzf go1.17.6.linux-amd64.tar.gz
export PATH=$PWD/go/bin:$PATH

    • Git:

      sudo apt install git

Create a systemd service that generates logs

  1. Create a directory:

    sudo mkdir /usr/local/bin/logtest
sudo chown $USER /usr/local/bin/logtest
cd /usr/local/bin/logtest

  2. Create a file named logtest.py:

    import logging
import random
import sys
import time

from systemdlogging.toolbox import check_for_systemd
from systemdlogging.toolbox import SystemdFormatter
from systemdlogging.toolbox import SystemdHandler

# You can replace these lines with the `systemdlogging.toolbox.init_systemd_logging` call.
# This code is a sample.

# Check whether `systemd` is available.
systemd_ok = check_for_systemd()

if systemd_ok:
    handler = SystemdHandler()
    # syslog_id: Value to use in the `SYSLOG_IDENTIFIER` field.
    handler.syslog_id = ''
    handler.setFormatter(SystemdFormatter())

    # Get an instance of the logger object.
    logger = logging.getLogger()
    logger.addHandler(handler)

else:
    logger = logging.getLogger(__name__)
    # Output logs to STDOUT.
    handler = logging.StreamHandler(stream=sys.stdout)
    handler.setFormatter(logging.Formatter(
        '[%(levelname)s] %(code)d %(message)s'
    ))
    logger.addHandler(handler)

# Optionally, configure the default logging level.
logger.setLevel(logging.DEBUG)

# Generate URL-like values.
PATHS = [
    '/',
    '/admin',
    '/hello',
    '/docs',
]

PARAMS = [
    'foo',
    'bar',
    'query',
    'search',
    None
]


def fake_url():
    path = random.choice(PATHS)
    param = random.choice(PARAMS)
    if param:
        val = random.randint(0, 100)
        param += '=%s' % val
    code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0]
    return '?'.join(filter(None, [path, param])), code


if __name__ == '__main__':
    while True:
        # Create a pair consisting of a code and a URL value.
        path, code = fake_url()
        # If the code is 200, write to the `Info` level log.
        # Cloud Logging uses the textual logging level value; therefore, in `context`, provide
        # the additional `SEVERITY` value.
        # This value will end up in `journald`, where you can read it in the `yc-logging` plugin.
        if code == 200:
            logger.info(
                'Path: %s',
                path,
                extra={"code": code, "context": {"SEVERITY": "info"}},
            )
        # Otherwise, write to the `Error` level log.
        else:
            logger.error(
                'Error: %s',
                path,
                extra={"code": code, "context": {"SEVERITY": "error"}},
            )

        # To avoid log cluttering, wait for 1 second.
        time.sleep(1)

  3. Upgrade the versions of the installed packages:

    sudo apt-get update

  4. Create a virtual environment and install the necessary dependencies:

    sudo apt install python3-pip python3.8-venv

    python3 -m venv venv
source venv/bin/activate
pip3 install systemd-logging

  5. Create a file named logtest.sh:

    #!/bin/bash

SCRIPT_PATH=$(dirname "$(realpath "$0")")
. "$SCRIPT_PATH/venv/bin/activate"
python "$SCRIPT_PATH/logtest.py"

  6. systemd will call logtest.sh to run the service; therefore, make the file executable:

    chmod +x logtest.sh

  7. Create a file named logtest.service:

    [Unit]
Description=Sample to show logging from a Python application to systemd
After=network.target

[Service]
Type=simple
ExecStart=/usr/local/bin/logtest/logtest.sh
Restart=on-abort

[Install]
WantedBy=multi-user.target

  8. Create a symbolic link for logtest.service:

    sudo ln -s "$(pwd)/logtest.service" /etc/systemd/system/logtest.service

  9. Restart systemd:

    sudo systemctl daemon-reload

  10. Check logtest.service status:

    systemctl status logtest.service

    Result:

    ● logtest.service - Sample to show logging from a Python application to systemd
 Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled)
 Active: inactive (dead)

  11. Start logtest.service:

    sudo systemctl start logtest.service

  12. Recheck logtest.service status; the service must now be active:

    systemctl status logtest.service

    Result:

    ● logtest.service - Sample to show logging from a Python application to systemd
     Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled)
     Active: active (running) since Wed 2022-02-02 18:49:03 UTC; 13h ago
   Main PID: 6550 (logtest.sh)
      Tasks: 2 (limit: 2311)
     Memory: 8.5M
     CGroup: /system.slice/logtest.service
             ├─6550 /bin/bash /usr/local/bin/logtest/logtest.sh
             └─6568 python /usr/local/bin/logtest/logtest.py

Install and configure Fluent Bit

  1. Add the GPG key you used to sign the packages in the Fluent Bit repository:

    wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -

  2. Add the following line to the /etc/apt/sources.list file:

    deb https://packages.fluentbit.io/ubuntu/focal focal main

  3. Upgrade the versions of the installed packages:

    sudo apt-get update

  4. Install the fluent-bit package:

    sudo apt-get install fluent-bit

  5. Start fluent-bit:

    sudo systemctl start fluent-bit

  6. Check fluent-bit status; the service must be active:

    systemctl status fluent-bit

    Result:

    ● fluent-bit.service - Fluent Bit
     Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
     Active: active (running) since Tue 2024-04-30 09:00:58 UTC; 3h 35min ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 589764 (fluent-bit)
      Tasks: 9 (limit: 2219)
     Memory: 18.8M
        CPU: 2.543s
     CGroup: /system.slice/fluent-bit.service
             └─589764 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf

Enable the plugin

  1. Clone the repository with the plugin code:

    git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git

  2. Write the package versions to the environment variables:

    cd fluent-bit-plugin-yandex/
export fluent_bit_version=3.0.3
export golang_version=1.22.2
export plugin_version=dev

    Where:

    • fluent_bit_version: To check the version, use the /opt/fluent-bit/bin/fluent-bit --version command.
    • golang_version: Go compiler version. To check the version, use the go version command.

  3. Exit the Python virtual environment and provide permissions for the plugin directory:

     deactivate

     sudo chown -R $USER:$USER /fluent-bit-plugin-yandex

  4. Compile the yc-logging.so library:

    CGO_ENABLED=1 go build -buildmode=c-shared \
    -o ./yc-logging.so \
    -ldflags "-X main.PluginVersion=${plugin_version}" \
    -ldflags "-X main.FluentBitVersion=${fluent_bit_version}"

  5. Copy the yc-logging.so library to the fluent-bit library directory:

    sudo cp yc-logging.so /usr/lib/fluent-bit/plugins/yc-logging.so

  6. Add the yc-logging.so library path to the /etc/fluent-bit/plugins.conf plugin settings file:

    [PLUGINS]
    Path /usr/lib/fluent-bit/plugins/yc-logging.so

  7. Add the fluent-bit settings to the /etc/fluent-bit/fluent-bit.conf file:

    [INPUT]
    Name  systemd
    Tag   host.*
    Systemd_Filter  _SYSTEMD_UNIT=logtest.service

[OUTPUT]
    Name            yc-logging
    Match           *
    resource_type   logtest
    folder_id       <folder_ID>
    message_key     MESSAGE
    level_key       SEVERITY
    default_level   WARN
    authorization   instance-service-account

    Where:

  8. Restart fluent-bit:

    sudo systemctl restart fluent-bit

View the logs

  1. In the management console, go to the folder you specified in the fluent-bit settings.
  2. Select Cloud Logging.
  3. Click the row with the default log group.
  4. Navigate to the Logs tab.
  5. The page that opens will show the log group records.

If you do not have the Yandex Cloud (CLI) command line interface yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

To view records in the log group, run this command:

yc logging read --folder-id=<folder_ID>

Where --folder-id is the ID specified in the fluent-bit settings.

To view log group records, use the LogReadingService/Read gRPC API call.

Delete the resources you created

If you no longer need the resources you created, delete them:

  1. Delete the VM.
  2. Delete the log group.
Previous
Getting started with Packer
Next
Building a VM image with infrastructure tools using Packer