Transferring logs from a VM instance to Yandex Cloud Logging
The Fluent Bit
To set up log transfer:
Getting started
- Create a service account with the
logging.writer
role for the folder. - Create a VM from a public Ubuntu 20.04 image. Under Access, specify the service account that you created in the previous step.
- Connect to the VM over SSH.
- On the VM instance, install:
-
wget https://go.dev/dl/go1.17.6.linux-amd64.tar.gz tar -xzf go1.17.6.linux-amd64.tar.gz export PATH=$PWD/go/bin:$PATH
-
Git:
sudo apt install git
-
Create a systemd service that generates logs
-
Create a directory:
sudo mkdir /usr/local/bin/logtest sudo chown $USER /usr/local/bin/logtest cd /usr/local/bin/logtest
-
Create a file named
logtest.py
:import logging import random import sys import time from systemdlogging.toolbox import check_for_systemd from systemdlogging.toolbox import SystemdFormatter from systemdlogging.toolbox import SystemdHandler # You can replace the following few lines with the `systemdlogging.toolbox.init_systemd_logging` call # This is a demo code # Check if `systemd` is available systemd_ok = check_for_systemd() if systemd_ok: handler = SystemdHandler() # syslog_id: Value to use in the `SYSLOG_IDENTIFIER` field handler.syslog_id = '' handler.setFormatter(SystemdFormatter()) # Get an instance of the logger object logger = logging.getLogger() logger.addHandler(handler) else: logger = logging.getLogger(__name__) # Output logs to STDOUT handler = logging.StreamHandler(stream=sys.stdout) handler.setFormatter(logging.Formatter( '[%(levelname)s] %(code)d %(message)s' )) logger.addHandler(handler) # Configure the default logging level (optional) logger.setLevel(logging.DEBUG) # Generate URL-like values PATHS = [ '/', '/admin', '/hello', '/docs', ] PARAMS = [ 'foo', 'bar', 'query', 'search', None ] def fake_url(): path = random.choice(PATHS) param = random.choice(PARAMS) if param: val = random.randint(0, 100) param += '=%s' % val code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0] return '?'.join(filter(None, [path, param])), code if __name__ == '__main__': while True: # Create a code and URL value pair path, code = fake_url() # If the code is 200, write to the log with the `Info` level # Cloud Logging uses the textual logging level value so, in `context`, provide # the additional `SEVERITY` value # It will end up in `journald` where you can read it in the `yc-logging` plugin if code == 200: logger.info( 'Path: %s', path, extra={"code": code, "context": {"SEVERITY": "info"}}, ) # Otherwise, with the `Error` level else: logger.error( 'Error: %s', path, extra={"code": code, "context": {"SEVERITY": "error"}}, ) # Wait for 1 second to avoid log cluttering time.sleep(1)
-
Upgrade the versions of installed packages:
sudo apt-get update
-
Create a virtual environment and install the necessary dependencies:
sudo apt install python3-pip python3.8-venv
python3 -m venv venv source venv/bin/activate pip3 install systemd-logging
-
Create a file named
logtest.sh
:#!/bin/bash SCRIPT_PATH=$(dirname "$(realpath "$0")") . "$SCRIPT_PATH/venv/bin/activate" python "$SCRIPT_PATH/logtest.py"
-
systemd
will calllogtest.sh
to run the service, so make the file executable:chmod +x logtest.sh
-
Create a file named
logtest.service
:[Unit] Description=Sample to show logging from a Python application to systemd After=network.target [Service] Type=simple ExecStart=/usr/local/bin/logtest/logtest.sh Restart=on-abort [Install] WantedBy=multi-user.target
-
Create a symbolic link for
logtest.service
:sudo ln -s "$(pwd)/logtest.service" /etc/systemd/system/logtest.service
-
Restart
systemd
:sudo systemctl daemon-reload
-
Check
logtest.service
status:systemctl status logtest.service
Result:
● logtest.service - Sample to show logging from a Python application to systemd Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled) Active: inactive (dead)
-
Start
logtest.service
:sudo systemctl start logtest.service
-
Recheck
logtest.service
status, it must now be active:systemctl status logtest.service
Result:
● logtest.service - Sample to show logging from a Python application to systemd Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled) Active: active (running) since Wed 2022-02-02 18:49:03 UTC; 13h ago Main PID: 6550 (logtest.sh) Tasks: 2 (limit: 2311) Memory: 8.5M CGroup: /system.slice/logtest.service ├─6550 /bin/bash /usr/local/bin/logtest/logtest.sh └─6568 python /usr/local/bin/logtest/logtest.py
Install and configure Fluent Bit
-
Add the GPG key that you used to sign packages in the Fluent Bit repository:
wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
-
Add the following line to the
/etc/apt/sources.list
file:deb https://packages.fluentbit.io/ubuntu/focal focal main
-
Upgrade the versions of installed packages:
sudo apt-get update
-
Install the
fluent-bit
package:sudo apt-get install fluent-bit
-
Start
fluent-bit
:sudo systemctl start fluent-bit
-
Check
fluent-bit
status, it must be active:systemctl status fluent-bit
Result:
● fluent-bit.service - Fluent Bit Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled) Active: active (running) since Tue 2024-04-30 09:00:58 UTC; 3h 35min ago Docs: https://docs.fluentbit.io/manual/ Main PID: 589764 (fluent-bit) Tasks: 9 (limit: 2219) Memory: 18.8M CPU: 2.543s CGroup: /system.slice/fluent-bit.service └─589764 /opt/fluent-bit/bin/fluent-bit -c //etc/fluent-bit/fluent-bit.conf
Enable the plugin
-
Clone the repository with the plugin code:
git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git
-
Write the package versions to the environment variables:
cd fluent-bit-plugin-yandex/ export fluent_bit_version=3.0.3 export golang_version=1.22.2 export plugin_version=dev
Where:
fluent_bit_version
:fluent-bit
version. To check the version, use the/opt/fluent-bit/bin/fluent-bit --version
command.golang_version
: Go compiler version. To check the version, use thego version
command.
-
Exit the Python virtual environment and provide permissions for the plugin directory:
deactivate
sudo chown -R $USER:$USER /fluent-bit-plugin-yandex
-
Compile the
yc-logging.so
library:CGO_ENABLED=1 go build -buildmode=c-shared \ -o ./yc-logging.so \ -ldflags "-X main.PluginVersion=${plugin_version}" \ -ldflags "-X main.FluentBitVersion=${fluent_bit_version}"
-
Copy the
yc-logging.so
library to thefluent-bit
library directory:sudo cp yc-logging.so /usr/lib/fluent-bit/plugins/yc-logging.so
-
Add the path to the
yc-logging.so
library to the/etc/fluent-bit/plugins.conf
file with plugin settings:[PLUGINS] Path /usr/lib/fluent-bit/plugins/yc-logging.so
-
Add the
fluent-bit
settings to the/etc/fluent-bit/fluent-bit.conf
file:[INPUT] Name systemd Tag host.* Systemd_Filter _SYSTEMD_UNIT=logtest.service [OUTPUT] Name yc-logging Match * resource_type logtest folder_id <folder_ID> message_key MESSAGE level_key SEVERITY default_level WARN authorization instance-service-account
Where:
folder_id
: ID of the folder to whose default log group the logs will be sent.authorization
: Authorization settings. Specifyinstance-service-account
to log in under the service account you specified under Access when creating the VM.
-
Restart
fluent-bit
:sudo systemctl restart fluent-bit
View the logs
- In the management console
, go to the folder you specified in thefluent-bit
settings. - Select Cloud Logging.
- Click the row with the
default
log group. - Go to the Logs tab.
- The page that opens will show the log group records.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To view records in the log group, run the command:
yc logging read --folder-id=<folder_ID>
Wjere --folder-id
is the folder ID specified in the fluent-bit
settings.
To view log group records, use the LogReadingService/Read gRPC API call.
Delete the resources you created
If you no longer need the resources you created, delete them: