Transferring logs from a VM to Yandex Cloud Logging

The Fluent Bit logging processor allows you to transfer logs from VM instances to Yandex Cloud Logging. To transfer logs, you will use the Fluent Bit plugin for Yandex Cloud Logging.

To set up log transfer:

1. Create a systemd service that generates logs.
2. Install and configure Fluent Bit.
3. Enable the plugin.

Getting startedGetting started

1. Create a service account with the logging.writer role for the folder.

2. Create a VM from a public Ubuntu 24.04 image. Under Access, specify the service account you created at the previous step.

3. Connect to the VM over SSH.

4. Install development packages on the VM:

   sudo apt-get update
   sudo apt-get install -y python3-pip python3-venv python3-systemd git build-essential pkg-config libsystemd-dev golang-go
   

   The command installs:

   • python3-pip: Python package manager (pip) for creating a systemd service that generates logs.
   • python3-venv: Module for creating Python virtual environments.
   • python3-systemd: Python library for interacting with systemd.
   • git: Version control system for downloading the Fluent Bit for Yandex Cloud Logging plugin source code from GitHub.
   • build-essential: Software compilers and build tools.
   • pkg-config: Utility that fetches compiler and linker flags for libraries.
   • libsystemd-dev: Header files and development libraries for systemd.
   • golang-go: Go (Golang) compiler and tools for building the Fluent Bit plugin.

5. Check the versions of the packages you installed:

   python3 --version
   pip3 --version
   go version
   

   You need these versions or higher:

   Python 3.10
   pip 22.0.2 from /usr/lib/python3/dist-packages/pip (python 3.10)
   go version go1.17.6 linux/amd64
   

Create a service that generates logsCreate a systemd service that generates logs

1. Create a directory:

   sudo mkdir -p /usr/local/bin/logtest
   sudo chown $USER /usr/local/bin/logtest
   cd /usr/local/bin/logtest
   

2. Create a file named logtest.py:

   import logging
   import random
   import sys
   import time
   from systemd import journal
   
   # Configuring the logger
   logger = logging.getLogger("logtest")
   journald_handler = journal.JournalHandler(SYSLOG_IDENTIFIER="logtest")
   logger.addHandler(journald_handler)
   logger.setLevel(logging.DEBUG)
   
   # Generate URL-like values.
   PATHS = [
       "/",
       "/admin",
       "/hello",
       "/docs",
   ]
   
   PARAMS = [
       "foo",
       "bar",
       "query",
       "search",
       None
   ]
   
   def fake_url():
       path = random.choice(PATHS)
       param = random.choice(PARAMS)
       if param:
           val = random.randint(0, 100)
           param += "={}".format(val)
       code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0]
       return "?".join(filter(None, [path, param])), code
   
   if __name__ == "__main__":
       while True:
           path, code = fake_url()
           if code == 200:
               logger.info(
                   "Path: {}".format(path),
                   extra={
                       "code": code
                   }
               )
           else:
               logger.error(
                   "Error: {}".format(path),
                   extra={
                       "code": code
                   }
               )
           time.sleep(1)
   

3. Create a virtual environment and install the required dependencies:

   python3 -m venv ~/venv
   source ~/venv/bin/activate
   pip install systemd-python
   

4. Make the file executable:

   sudo chmod +x /usr/local/bin/logtest/logtest.py
   

5. Create a file named /etc/systemd/system/logtest.service:

   [Unit]
   Description=Log Test Service
   After=network.target
   
   [Service]
   ExecStart=/home/$USER/venv/bin/python3 /usr/local/bin/logtest/logtest.py
   Environment=PYTHONPATH=/home/$USER/venv/lib/python3.12/site-packages
   Restart=always
   User=$USER
   StandardOutput=journal
   StandardError=journal
   
   [Install]
   WantedBy=multi-user.target
   

6. Restart systemd:

   sudo systemctl daemon-reload
   

7. Run and check the service status:

   sudo systemctl start logtest
   sudo systemctl status logtest
   

   Result:

   ● logtest.service - Sample to show logging from a Python application to systemd
        Loaded: loaded (/etc/systemd/system/logtest.service; linked; vendor preset: enabled)
        Active: active (running) since Thu 2024-05-30 12:34:56 UTC; 5s ago
      Main PID: 12345 (logtest.sh)
         Tasks: 2 (limit: 2311)
        Memory: 18.5M
           CPU: 156ms
        CGroup: /system.slice/logtest.service
                ├─12345 /bin/bash /usr/local/bin/logtest/logtest.sh
                └─12346 python /usr/local/bin/logtest/logtest.py
   

Install and configure Fluent BitInstall and configure Fluent Bit

1. Add the GPG key and Fluent Bit repository:

   wget -qO - https://packages.fluentbit.io/fluentbit.key | sudo apt-key add -
   echo "deb https://packages.fluentbit.io/ubuntu/focal focal main" | sudo tee /etc/apt/sources.list.d/fluent-bit.list
   

2. Install Fluent Bit:

   sudo apt-get update
   sudo apt-get install -y fluent-bit
   

Enable the pluginEnable the plugin

1. Clone the repository with the plugin:

   git clone https://github.com/yandex-cloud/fluent-bit-plugin-yandex.git
   cd fluent-bit-plugin-yandex
   

2. Create and configure the version file:

   cat > versions.sh << 'EOL'
   #!/bin/bash
   export fluent_bit_version=3.0.3
   export golang_version=1.22.2
   export plugin_version=dev
   EOL
   
   chmod +x versions.sh
   source ./versions.sh
   

3. Build the plugin:

   CGO_ENABLED=1 go build -v -buildmode=c-shared -o yc-logging.so yclogging.go
   

4. Install the plugin:

   sudo mkdir -p /usr/local/lib/fluent-bit/
   sudo cp yc-logging.so /usr/local/lib/fluent-bit/
   sudo chmod 644 /usr/local/lib/fluent-bit/yc-logging.so
   

5. Create a file named /etc/fluent-bit/plugins.conf:

   [PLUGINS]
       Path /usr/local/lib/fluent-bit/yc-logging.so
   

6. Create a file named /etc/fluent-bit/fluent-bit.conf:

   [SERVICE]
       Flush        1
       Daemon       Off
       Log_Level    info
       Parsers_File parsers.conf
       Plugins_File plugins.conf
   
   [INPUT]
       Name            systemd
       Tag             host.*
       Systemd_Filter  _SYSTEMD_UNIT=logtest.service
   
   [OUTPUT]
       Name            yc-logging
       Match           *
       resource_type   logtest
       folder_id       <catalog_ID>
       message_key     MESSAGE
       level_key       SEVERITY
       default_level   WARN
       authorization   instance-service-account
   

   Where:

   • folder_id: ID of the Yandex Cloud folder whose default log group will receive the logs.
   • authorization: Authorization settings. The instance-service-account value is used for authorization as the service account specified when creating the VM.
   • level_key: Field indicating the logging level.
   • message_key: Field with the message text.
   • default_level: Default logging level if not specified in the message.

7. Restart Fluent Bit:

   sudo systemctl restart fluent-bit
   

View the logsView the logs

sudo systemctl status logtest
sudo systemctl status fluent-bit

● fluent-bit.service - Fluent Bit
     Loaded: loaded (/lib/systemd/system/fluent-bit.service; disabled; vendor preset: enabled)
     Active: active (running) since Thu 2024-05-30 12:34:56 UTC; 5s ago
       Docs: https://docs.fluentbit.io/manual/
   Main PID: 12347 (fluent-bit)
      Tasks: 4 (limit: 2311)
     Memory: 18.8M
        CPU: 156ms
     CGroup: /system.slice/fluent-bit.service
             └─12347 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf

yc logging read --folder-id=<catalog_ID>

TroubleshootingTroubleshooting

1. File access errors and system issues:

   • Check access permissions for critical files:

     ls -l /usr/local/lib/fluent-bit/yc-logging.so
     ls -l /etc/fluent-bit/plugins.conf
     ls -l /etc/fluent-bit/fluent-bit.conf
     

     The correct result should look like this:

     -rwxr-xr-x 1 root root 8123456 May 30 12:34 /usr/local/lib/fluent-bit/yc-logging.so
     -rw-r--r-- 1 root root 123 May 30 12:34 /etc/fluent-bit/plugins.conf
     -rw-r--r-- 1 root root 456 May 30 12:34 /etc/fluent-bit/fluent-bit.conf
     

   • Check system logs for errors:

     sudo tail -f /var/log/syslog
     

     Example of correct logs:

     May 30 12:34:56 vm-name fluent-bit[12347]: [2024/05/30 12:34:56] [ info] [fluent bit] version=3.0.3
     May 30 12:34:56 vm-name fluent-bit[12347]: [2024/05/30 12:34:56] [ info] [storage] ver=1.4.0, type=memory
     May 30 12:34:56 vm-name fluent-bit[12347]: [2024/05/30 12:34:56] [ info] [cmetrics] version=0.6.3
     

     journalctl -xe
     

     Example of correct logs:

     May 30 12:34:56 vm-name systemd[1]: Started Fluent Bit.
     May 30 12:34:56 vm-name fluent-bit[12347]: Fluent Bit v3.0.3
     May 30 12:34:56 vm-name logtest.sh[12345]: [INFO] 200 Path: /hello?query=42
     

2. Permission denied error:

   • Check the service account permissions:

     yc iam service-account list
     

     The result should display your service account:

     +----------------------+---------------+
     |          ID          |     NAME      |
     +----------------------+---------------+
     | aje5n27q235g8m3...   | service-acc-1 |
     +----------------------+---------------+
     

   • Check the service account roles:

     yc iam service-account get service-acc-1
     

     The result should indicate the logging.writer role:

     id: aje5n27q235g8m3...
     folder_id: b1g4c2a3v000000000000
     name: service-acc-1
     roles:
       - logging.writer
     

3. Logs do not show up in Cloud Logging:

   • Make sure the folder_id in the configuration is correct:

     grep folder_id /etc/fluent-bit/fluent-bit.conf
     

     The result should display your folder_id:

     folder_id b1g4c2a3v000000000000
     

   • Make sure the yc-logging plugin is loaded correctly:

     sudo systemctl status fluent-bit | grep yc-logging
     

     The correct result should be as follows:

     May 30 12:34:56 vm-name fluent-bit[12347]: [2024/05/30 12:34:56] [ info] [output:yc-logging:yc-logging.0] worker #0 started
     

4. Log format issues:

   • Check the settings in the configuration:

     grep -A 5 '\[OUTPUT\]' /etc/fluent-bit/fluent-bit.conf
     

     The correct result should be as follows:

     [OUTPUT]
         Name            yc-logging
         Match           *
         resource_type   logtest
         message_key     MESSAGE
         level_key       SEVERITY
     

   • Check the log format in journald:

     journalctl -u logtest -n 5
     

     The correct log format is as follows:

     May 30 12:34:56 vm-name logtest.sh[12345]: [INFO] 200 Path: /hello?query=42
     May 30 12:34:57 vm-name logtest.sh[12345]: [ERROR] 404 Error: /admin?foo=13
     

5. File access errors and system issues:

   • Check access permissions for critical files:

     ls -l /usr/local/lib/fluent-bit/yc-logging.so
     ls -l /etc/fluent-bit/plugins.conf
     ls -l /etc/fluent-bit/fluent-bit.conf
     

   • Check system logs for errors:

     sudo tail -f /var/log/syslog
     journalctl -xe
     

   • Make sure all files have correct access permissions (644 for configuration files, 755 for libraries).

6. Permission denied error:

   • Check the service account permissions.
   • Make sure the logging.writer role is assigned for the appropriate folder.
   • Make sure the service account token is valid.

7. Logs do not show up in Cloud Logging:

   • Make sure the folder_id in the configuration is correct.

   • Make sure the log format matches the expected one.

   • Make sure the yc-logging plugin is loaded correctly:

     sudo systemctl status fluent-bit | grep yc-logging
     

8. Log format issues:

   • Check the message_key and level_key settings.
   • Make sure the logs contain the required fields.
   • Check the time format in logs.

Useful diagnostic commandsUseful diagnostic commands

1. View extended Fluent Bit logs:

   • Real-time log monitoring:

     sudo journalctl -u fluent-bit -n 100 -f
     

     Output example:

     May 30 12:34:56 vm-name fluent-bit[1234]: [info] [engine] started (pid=1234)
     May 30 12:34:56 vm-name fluent-bit[1234]: [info] [storage] version=1.1.6, initializing...
     May 30 12:34:57 vm-name fluent-bit[1234]: [info] [yc-logging] plugin initialized successfully
     

2. Checking CPU and memory usage:

   • Monitoring the Fluent Bit process:

     ps aux | grep fluent-bit
     top -p $(pgrep fluent-bit)
     

     Example of ps output:

     fluent  1234  0.5  1.2 256892 12644 ?  Ssl  12:34  0:02 /opt/fluent-bit/bin/fluent-bit -c /etc/fluent-bit/fluent-bit.conf
     

3. Monitoring network connections:

   • Checking open ports and connections:

     sudo netstat -tupn | grep fluent-bit
     

     Output example:

     tcp  0  0  0.0.0.0:24224  0.0.0.0:*  LISTEN  1234/fluent-bit
     tcp  0  0  10.0.0.2:52431  logging.api.cloud.yandex.net:443  ESTABLISHED  1234/fluent-bit
     

4. Verifying plugin loading:

   • Viewing open files of the process:

     sudo lsof -p $(pgrep fluent-bit) | grep yc-logging
     

     Output example:

     fluent-bit 1234 fluent-bit  mem REG 8,1 2890144 /usr/local/lib/fluent-bit/yc-logging.so
     

5. Checking configuration files:

   • Getting checksums:

     find /etc/fluent-bit/ -type f -exec md5sum {} \;
     

     Output example:

     a1b2c3d4e5f6g7h8 /etc/fluent-bit/fluent-bit.conf
     b2c3d4e5f6g7h8i9 /etc/fluent-bit/plugins.conf
     c3d4e5f6g7h8i9j0 /etc/fluent-bit/parsers.conf
     

Delete the resources you createdDelete the resources you created

If you no longer need the resources you created, delete them:

1. Delete the VM.
2. Delete the log group.