Transferring logs from Container Optimized Image to Cloud Logging

The Fluent Bit log processor allows you to transfer logs from VM instances created from Container Optimized Image images to Yandex Cloud Logging. The Fluent Bit plugin for Yandex Cloud Logging module is used to transfer logs.

To configure log transfer from a VM instance created from the Container Optimized Image image:

1. Create a log-generating application.
2. Create a Docker image and push it to the registry.
3. Configure Fluent Bit.
4. Create a VM from the Container Optimized Image.

Getting startedGetting started

1. Create a service account with the logging.writer and container-registry.images.puller roles for the folder.
2. Create a registry in Yandex Container Registry.
3. Create a cloud network. Select Create subnets when creating it.

Create a log-generating applicationCreate a log-generating application

Create a file named logs.py:

import logging
import random
import sys
import time

import uuid

logger = logging.getLogger(__name__)

# Set the log format.
formatter = logging.Formatter(
  '[req_id=%(req_id)s] [%(levelname)s] %(code)d %(message)s'
)

handler = logging.StreamHandler(stream=sys.stdout)
handler.setFormatter(formatter)

logger.addHandler(handler)

# Configure the default logging level (optional).
logger.setLevel(logging.DEBUG)

# Generate URL-like values.
PATHS = [
  '/',
  '/admin',
  '/hello',
  '/docs',
]

PARAMS = [
  'foo',
  'bar',
  'query',
  'search',
  None
]

def fake_url():
  path = random.choice(PATHS)
  param = random.choice(PARAMS)
  if param:
    val = random.randint(0, 100)
    param += '=%s' % val
  code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0]
  return '?'.join(filter(None, [path, param])), code

if __name__ == '__main__':
  while True:
    req_id = uuid.uuid4()
    # Create a pair: code and URL value.
    path, code = fake_url()
    extra = {"code": code, "req_id": req_id}
    # If the code is 200, write to the log with the Info level.
    if code == 200:
      logger.info(
        'Path: %s',
        path,
        extra=extra,
      )
    # Otherwise, with the Error level.
    else:
      logger.error(
        'Error: %s',
        path,
        extra=extra,
      )
    # To have multiple messages with the same request ID, in 30% of cases, write the second entry to the log with the Debug level.
    if random.random() > 0.7:
      logger.debug("some additional debug log record %f", random.random(), extra=extra)

    # Wait for 1 second to avoid log cluttering.
    time.sleep(1)

Create a Docker image and push it to the registryCreate a Docker image and push it to the registry

1. Create a file named Dockerfile and add the lines below:

   FROM python:3.10
   
   WORKDIR /usr/src/app
   
   COPY logs.py .
   
   CMD [ "python", "./logs.py" ]
   

   Dockerfile describes a Docker image that contains an application generating logs.

2. Build the Docker image:

   docker build . \
     -t cr.yandex/<registry_ID>/coi:logs
   

3. Log in to the registry and upload a Docker image into it:

   docker push cr.yandex/<registry_ID>/coi:logs
   

Configure Fluent BitConfigure Fluent Bit

1. Create a file named spec.yaml. It describes the specification of two containers: with an application that generates logs, and with a Fluent Bit agent.

   Specify the following in the field:

   • image: Docker image URL. To find it out, in the management console, go to the Overview of Docker image page and copy the value of the Tags field.
   • YC_GROUP_ID: ID of the default log group default.

   In the fluentbit section, the image field shows the image of a container with the Fluent Bit agent, current at the time of this documentation. For a list of all available images, follow the link.

   version: '3.7'
   services:
     logs:
       container_name: logs-app
       image: <Docker_image_URL>
       restart: always
       depends_on:
         - fluentbit
       logging:
         # Fluent Bit understands logs in this format.
         driver: fluentd
         options:
           # Fluent Bit listens to logs on port 24224.
           fluentd-address: localhost:24224
           # Tags are used for routing logs.
           tag: app.logs
   
     fluentbit:
       container_name: fluentbit
       image: cr.yandex/yc/fluent-bit-plugin-yandex:v1.0.3-fluent-bit-1.8.6
       ports:
         - 24224:24224
         - 24224:24224/udp
       restart: always
       environment:
         YC_GROUP_ID: <log_group_ID>
       volumes:
         - /etc/fluentbit/fluentbit.conf:/fluent-bit/etc/fluent-bit.conf
         - /etc/fluentbit/parsers.conf:/fluent-bit/etc/parsers.conf
   

2. Create a file named user-data.yaml. It describes the container log reading rules. If required, change the username and SSH key in the users section. Learn more about how to generate SSH keys here.

   #cloud-config
   write_files:
     - content: |
         [SERVICE]
             Flush         1
             Log_File      /var/log/fluentbit.log
             Log_Level     error
             Daemon        off
             Parsers_File  /fluent-bit/etc/parsers.conf
   
         [FILTER]
             Name parser
             Match app.logs
             Key_Name log
             Parser app_log_parser
             Reserve_Data On
   
         [INPUT]
             Name              forward
             Listen            0.0.0.0
             Port              24224
             Buffer_Chunk_Size 1M
             Buffer_Max_Size   6M
   
         [OUTPUT]
             Name            yc-logging
             Match           *
             group_id        ${YC_GROUP_ID}
             message_key     text
             level_key       severity
             default_level   WARN
             authorization   instance-service-account
       path: /etc/fluentbit/fluentbit.conf
     - content: |
         [PARSER]
             Name   app_log_parser
             Format regex
             Regex  ^\[req_id=(?<req_id>[0-9a-fA-F\-]+)\] \[(?<severity>.*)\] (?<code>\d+) (?<text>.*)$
             Types  code:integer
       path: /etc/fluentbit/parsers.conf
   
   users:
     - name: username
       groups: sudo
       shell: /bin/bash
       sudo: 'ALL=(ALL) NOPASSWD:ALL'
       ssh_authorized_keys:
         - ssh-ed25519 AAAA
   

   The SERVICE section displays Fluent Bit settings. Learn more about the settings.

   The INPUT section displays where and how to retrieve logs. To work with Fluentd and Fluent Bit logs, the forward protocol is used. Fluent Bit listens to logs on port 24224.

   The PARSER section describes the regex parser. It sets a regular expression that processes entries:

   • req_id: Unique request ID
   • severity: Logging level
   • code: HTTP response code
   • text: All remaining text

   The FILTER section shows that only entries tagged app.logs are searched for. The log field of each entry is processed by the regex parser, all other fields are saved in Reserve_Data On.

Create a VM from the Container Optimized ImageCreate a VM from the Container Optimized Image

Specify the following in the field:

• --zone: Select an availability zone, such as ru-central1-a.
• --subnet-name: Name of the subnet in the indicated zone.
• --service-account-name: Service account name.

IMAGE_ID=$(yc compute image get-latest-from-family container-optimized-image --folder-id standard-images --format=json | jq -r .id)

yc compute instance create \
  --name coi-vm \
  --zone=<zone> \
  --network-interface subnet-name=<subnet_name>,nat-ip-version=ipv4 \
  --metadata-from-file user-data=user-data.yaml,docker-compose=spec.yaml \
  --create-boot-disk image-id=${IMAGE_ID} \
  --service-account-name <service_account_name>

View the logsView the logs

yc logging read --group-id=<log_group_ID>

Delete the resources you createdDelete the resources you created

If you no longer need the resources you created, delete them:

1. Delete the cloud network.
2. Delete the Docker image.
3. Delete the registry.
4. Delete the VM.
5. Delete the log group.