Transferring logs from Container Optimized Image to Yandex Cloud Logging
The Fluent Bit
To configure log transfer from a VM instance created from the Container Optimized Image image:
- Create a log-generating application.
- Create a Docker image and push it to the registry.
- Configure Fluent Bit.
- Create a VM from the Container Optimized Image.
Getting started
- Create a service account with the
logging.writer
andcontainer-registry.images.puller
roles for the folder. - Create a registry in Yandex Container Registry.
- Create a cloud network. Select Create subnets when creating it.
Create a log-generating application
Create a file named logs.py
:
import logging
import random
import sys
import time
import uuid
logger = logging.getLogger(__name__)
# Set the log format.
formatter = logging.Formatter(
'[req_id=%(req_id)s] [%(levelname)s] %(code)d %(message)s'
)
handler = logging.StreamHandler(stream=sys.stdout)
handler.setFormatter(formatter)
logger.addHandler(handler)
# Configure the default logging level (optional).
logger.setLevel(logging.DEBUG)
# Generate URL-like values.
PATHS = [
'/',
'/admin',
'/hello',
'/docs',
]
PARAMS = [
'foo',
'bar',
'query',
'search',
None
]
def fake_url():
path = random.choice(PATHS)
param = random.choice(PARAMS)
if param:
val = random.randint(0, 100)
param += '=%s' % val
code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0]
return '?'.join(filter(None, [path, param])), code
if __name__ == '__main__':
while True:
req_id = uuid.uuid4()
# Create a pair: code and URL value.
path, code = fake_url()
extra = {"code": code, "req_id": req_id}
# If the code is 200, write to the log with the Info level.
if code == 200:
logger.info(
'Path: %s',
path,
extra=extra,
)
# Otherwise, with the Error level.
else:
logger.error(
'Error: %s',
path,
extra=extra,
)
# To have multiple messages with the same request ID, in 30% of cases, write the second entry to the log with the Debug level.
if random.random() > 0.7:
logger.debug("some additional debug log record %f", random.random(), extra=extra)
# Wait for 1 second to avoid log cluttering.
time.sleep(1)
Create a Docker image and push it to the registry
-
Create a file named Dockerfile and add the lines below:
FROM python:3.10 WORKDIR /usr/src/app COPY logs.py . CMD [ "python", "./logs.py" ]
Dockerfile describes a Docker image that contains an application generating logs.
-
Build the Docker image:
docker build . \ -t cr.yandex/<registry_ID>/coi:logs
-
Log in to the registry and upload a Docker image into it:
docker push cr.yandex/<registry_ID>/coi:logs
Configure Fluent Bit
-
Create a file named
spec.yaml
. It describes the specification of two containers: with an application that generates logs, and with a Fluent Bit agent.Specify the following in the field:
image
: Docker image URL. To find it out, in the management console , go to the Overview of Docker image page and copy the value of the Tags field.YC_GROUP_ID
: ID of the default log groupdefault
.
In the
fluentbit
section, theimage
field shows the image of a container with the Fluent Bit agent, current at the time of this documentation. For a list of all available images, follow the link .version: '3.7' services: logs: container_name: logs-app image: <Docker_image_URL> restart: always depends_on: - fluentbit logging: # Fluent Bit understands logs in this format. driver: fluentd options: # Fluent Bit listens to logs on port 24224. fluentd-address: localhost:24224 # Tags are used for routing logs. tag: app.logs fluentbit: container_name: fluentbit image: cr.yandex/yc/fluent-bit-plugin-yandex:v1.0.3-fluent-bit-1.8.6 ports: - 24224:24224 - 24224:24224/udp restart: always environment: YC_GROUP_ID: <log_group_ID> volumes: - /etc/fluentbit/fluentbit.conf:/fluent-bit/etc/fluent-bit.conf - /etc/fluentbit/parsers.conf:/fluent-bit/etc/parsers.conf
-
Create a file named
user-data.yaml
. It describes the container log reading rules. If required, change the username and SSH key in theusers
section. Learn more about how to generate SSH keys here.#cloud-config write_files: - content: | [SERVICE] Flush 1 Log_File /var/log/fluentbit.log Log_Level error Daemon off Parsers_File /fluent-bit/etc/parsers.conf [FILTER] Name parser Match app.logs Key_Name log Parser app_log_parser Reserve_Data On [INPUT] Name forward Listen 0.0.0.0 Port 24224 Buffer_Chunk_Size 1M Buffer_Max_Size 6M [OUTPUT] Name yc-logging Match * group_id ${YC_GROUP_ID} message_key text level_key severity default_level WARN authorization instance-service-account path: /etc/fluentbit/fluentbit.conf - content: | [PARSER] Name app_log_parser Format regex Regex ^\[req_id=(?<req_id>[0-9a-fA-F\-]+)\] \[(?<severity>.*)\] (?<code>\d+) (?<text>.*)$ Types code:integer path: /etc/fluentbit/parsers.conf users: - name: username groups: sudo shell: /bin/bash sudo: 'ALL=(ALL) NOPASSWD:ALL' ssh_authorized_keys: - ssh-ed25519 AAAA
The
SERVICE
section displays Fluent Bit settings. Learn more about the settings .The
INPUT
section displays where and how to retrieve logs. To work with Fluentd and Fluent Bit logs, theforward
protocol is used. Fluent Bit listens to logs on port 24224.The
PARSER
section describes theregex
parser. It sets a regular expression that processes entries:req_id
: Unique request IDseverity
: Logging levelcode
: HTTP response codetext
: All remaining text
The
FILTER
section shows that only entries taggedapp.logs
are searched for. Thelog
field of each entry is processed by theregex
parser, all other fields are saved inReserve_Data On
.
Create a VM from the Container Optimized Image
Specify the following in the field:
--zone
: Select an availability zone, such asru-central1-a
.--subnet-name
: Name of the subnet in the indicated zone.--service-account-name
: Service account name.
IMAGE_ID=$(yc compute image get-latest-from-family container-optimized-image --folder-id standard-images --format=json | jq -r .id)
yc compute instance create \
--name coi-vm \
--zone=<zone> \
--network-interface subnet-name=<subnet_name>,nat-ip-version=ipv4 \
--metadata-from-file user-data=user-data.yaml,docker-compose=spec.yaml \
--create-boot-disk image-id=${IMAGE_ID} \
--service-account-name <service_account_name>
View the logs
- In the management console
, go to the folder with thedefault
log group whose ID you specified in thespec.yaml
file. - Select Cloud Logging.
- Select the
default
log group. The page that opens will show the log group records.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To view records in the log group, run this command:
yc logging read --group-id=<log_group_ID>
--group-id
: ID of the default
log group specified in the spec.yaml
file.
You can view the log group records using the LogReadingService/Read gRPC API call.
Delete the resources you created
If you no longer need the resources you created, delete them: