Transferring logs from a VM to Yandex Cloud Logging
The Fluent Bit
To set up log transfer:
Getting started
- Create a service account with the
logging.writer
role for the folder. - Create a VM from a public Ubuntu 20.04 image. Under Access, specify the service account you created at the previous step.
- Connect to the VM over SSH.
- On the VM, install:
-
wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz tar -xzf go1.17.6.linux-amd64.tar.gz export PATH=$PWD/go/bin:$PATH
-
Git:
sudo apt install git
-
systemd
service that generates logs
Create a -
Create a directory:
sudo mkdir /usr/local/bin/logtest sudo chown $USER /usr/local/bin/logtest cd /usr/local/bin/logtest
-
Create a file named
logtest.py
:import logging import random import sys import time from systemdlogging.toolbox import check_for_systemd from systemdlogging.toolbox import SystemdFormatter from systemdlogging.toolbox import SystemdHandler # You can replace these lines with the `systemdlogging.toolbox.init_systemd_logging` call. # This code is a sample. # Check whether `systemd` is available. systemd_ok = check_for_systemd() if systemd_ok: handler = SystemdHandler() # syslog_id: Value to use in the `SYSLOG_IDENTIFIER` field. handler.syslog_id = '' handler.setFormatter(SystemdFormatter()) # Get an instance of the logger object. logger = logging.getLogger() logger.addHandler(handler) else: logger = logging.getLogger(__name__) # Output logs to STDOUT. handler = logging.StreamHandler(stream=sys.stdout) handler.setFormatter(logging.Formatter( '[%(levelname)s] %(code)d %(message)s' )) logger.addHandler(handler) # Optionally, configure the default logging level. logger.setLevel(logging.DEBUG) # Generate URL-like values. PATHS = [ '/', '/admin', '/hello', '/docs', ] PARAMS = [ 'foo', 'bar', 'query', 'search', None ] def fake_url(): path = random.choice(PATHS) param = random.choice(PARAMS) if param: val = random.randint(0, 100) param += '=%s' % val code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0] return '?'.join(filter(None, [path, param])), code if __name__ == '__main__': while True: # Create a pair consisting of a code and a URL value. path, code = fake_url() # If the code is 200, write to the `Info` level log. # Cloud Logging uses the textual logging level value; therefore, in `context`, provide # the additional `SEVERITY` value. # This value will end up in `journald`, where you can read it in the `yc-logging` plugin. if code == 200: logger.info( 'Path: %s', path, extra={"code": code, "context": {"SEVERITY": "info"}}, ) # Otherwise, write to the `Error` level log. else: logger.error( 'Error: %s', path, extra={"code": code, "context": {"SEVERITY": "error"}}, ) # To avoid log cluttering, wait for 1 second. time.sleep(1)
-
Upgrade the versions of the installed packages:
sudo apt-get update
-
Create a virtual environment and install the necessary dependencies:
sudo apt install python3-pip python3.8-venv
python3 -m venv venv source venv/bin/activate pip3 install systemd-logging
-
Create a file named
logtest.sh
:#!/bin/bash SCRIPT_PATH=$(dirname "$(realpath "$0")") . "$SCRIPT_PATH/venv/bin/activate" python "$SCRIPT_PATH/logtest.py"
-
systemd
will calllogtest.sh
to run the service; therefore, make the file executable:chmod +x logtest.sh
-
Create a file named
logtest.service
:[Unit] Description=Sample to show logging from a Python application to systemd After=network.target [Service] Type=simple ExecStart=/usr/local/bin/logtest/logtest.sh Restart=on-abort [Install] WantedBy=multi-user.target
-
Create a symbolic link for
logtest.service
:sudo ln -s "$(pwd)/logtest.service" /etc/systemd/system/logtest.service
-
Restart
systemd
:sudo systemctl daemon-reload
-
Check
logtest.service
status:systemctl status logtest.service
Result:
● logtest.service - Sample to show logging from a Python application to systemd Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled) Active: inactive (dead)
-
Start
logtest.service
:sudo systemctl start logtest.service
-
Recheck
logtest.service
status; the service must now be active:systemctl status logtest.service
Result:
● logtest.service - Sample to show logging from a Python application to systemd Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled) Active: active (running) since Wed 2022-02-02 18:49:03 UTC; 13h ago Main PID: 6550 (logtest.sh) Tasks: 2 (limit: 2311) Memory: 8.5M CGroup: /system.slice/logtest.service ├─6550 /bin/bash /usr/local/bin/logtest/logtest.sh └─6568 python /usr/local/bin/logtest/logtest.py
Install and configure Fluent Bit
-
Add the GPG key you used to sign the packages in the Fluent Bit repository:
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
-
Add the following line to the
/etc/apt/sources.list
file:deb https://packages.fluentbit.io/ubuntu/focal focal main
-
Upgrade the versions of the installed packages:
sudo apt-get update
-
Install the
fluent-bit
package:sudo apt-get install fluent-bit
-
Start
fluent-bit
:sudo systemctl start fluent-bit
-
Check
fluent-bit
status; the service must be active:systemctl status fluent-bit
Result:
● fluent-bit.service - Fluent Bit Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2024-04-30 09:00:58 UTC; 3h 35min ago Docs: https://docs.fluentbit.io/manual/ Main PID: 589764 (fluent-bit) Tasks: 9 (limit: 2219) Memory: 18.8M CPU: 2.543s CGroup: /system.slice/fluent-bit.service └─589764 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf
Enable the plugin
-
Clone the repository with the plugin code:
git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git
-
Write the package versions to the environment variables:
cd fluent-bit-plugin-yandex/ export fluent_bit_version=3.0.3 export golang_version=1.22.2 export plugin_version=dev
Where:
fluent_bit_version
: To check the version, use the/opt/fluent-bit/bin/fluent-bit --version
command.golang_version
: Go compiler version. To check the version, use thego version
command.
-
Exit the Python virtual environment and provide permissions for the plugin directory:
deactivate
sudo chown -R $USER:$USER /fluent-bit-plugin-yandex
-
Compile the
yc-logging.so
library:CGO_ENABLED=1 go build -buildmode=c-shared \ -o ./yc-logging.so \ -ldflags "-X main.PluginVersion=${plugin_version}" \ -ldflags "-X main.FluentBitVersion=${fluent_bit_version}"
-
Copy the
yc-logging.so
library to thefluent-bit
library directory:sudo cp yc-logging.so /usr/lib/fluent-bit/plugins/yc-logging.so
-
Add the
yc-logging.so
library path to the/etc/fluent-bit/plugins.conf
plugin settings file:[PLUGINS] Path /usr/lib/fluent-bit/plugins/yc-logging.so
-
Add the
fluent-bit
settings to the/etc/fluent-bit/fluent-bit.conf
file:[INPUT] Name systemd Tag host.* Systemd_Filter _SYSTEMD_UNIT=logtest.service [OUTPUT] Name yc-logging Match * resource_type logtest folder_id <folder_ID> message_key MESSAGE level_key SEVERITY default_level WARN authorization instance-service-account
Where:
folder_id
: ID of the folder whose default log group will get the logs.authorization
: Authorization settings. Specifyinstance-service-account
to log in as a service account you specified under Access when creating the VM.
-
Restart
fluent-bit
:sudo systemctl restart fluent-bit
View the logs
- In the management console
, go to the folder you specified in thefluent-bit
settings. - Select Cloud Logging.
- Click the row with the
default
log group. - Navigate to the Logs tab.
- The page that opens will show the log group records.
If you do not have the Yandex Cloud CLI yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder through the --folder-name
or --folder-id
parameter.
To view records in the log group, run this command:
yc logging read --folder-id=<folder_ID>
Where --folder-id
is the ID specified in the fluent-bit
settings.
To view log group records, use the LogReadingService/Read gRPC API call.
Delete the resources you created
If you no longer need the resources you created, delete them: