Transferring logs from Container Optimized Image to Yandex Cloud Logging
The Fluent Bit
To configure log transfer from a VM instance created from a Container Optimized Image:
- Create a log-generating application.
- Create a Docker image and push it to the registry.
- Configure Fluent Bit.
- Create a VM from a Container Optimized Image.
Getting started
- Create a service account with the
logging.writer
andcontainer-registry.images.puller
roles for the folder. - Create a registry in Yandex Container Registry.
- Create a cloud network. Select Create subnets when creating it.
Create a log-generating application
Create a file named logs.py
:
import logging
import random
import sys
import time
import uuid
logger = logging.getLogger(__name__)
# Set the log format.
formatter = logging.Formatter(
'[req_id=%(req_id)s] [%(levelname)s] %(code)d %(message)s'
)
handler = logging.StreamHandler(stream=sys.stdout)
handler.setFormatter(formatter)
logger.addHandler(handler)
# Configure the default logging level (optional).
logger.setLevel(logging.DEBUG)
# Generate URL-like values.
PATHS = [
'/',
'/admin',
'/hello',
'/docs',
]
PARAMS = [
'foo',
'bar',
'query',
'search',
None
]
def fake_url():
path = random.choice(PATHS)
param = random.choice(PARAMS)
if param:
val = random.randint(0, 100)
param += '=%s' % val
code = random.choices([200, 400, 404, 500], weights=[10, 2, 2, 1])[0]
return '?'.join(filter(None, [path, param])), code
if __name__ == '__main__':
while True:
req_id = uuid.uuid4()
# Create a pair consisting of a code and a URL value.
path, code = fake_url()
extra = {"code": code, "req_id": req_id}
# If the code is 200, write to the `Info` level log.
if code == 200:
logger.info(
'Path: %s',
path,
extra=extra,
)
# Otherwise, write to the `Error` level log.
else:
logger.error(
'Error: %s',
path,
extra=extra,
)
# To have multiple messages with the same request ID, in 30% of cases, write the second entry to the `Debug` level log.
if random.random() > 0.7:
logger.debug("some additional debug log record %f", random.random(), extra=extra)
# To avoid log cluttering, wait for 1 second.
time.sleep(1)
Create a Docker image and push it to the registry
-
Create a Dockerfile and add the following lines to it:
FROM python:3.10 WORKDIR /usr/src/app COPY logs.py . CMD [ "python", "./logs.py" ]
The Dockerfile describes a Docker image that contains an application generating logs.
-
Build the Docker image:
docker build . \ -t cr.yandex/<registry_ID>/coi:logs
-
Log in to the registry and upload the Docker image to it:
docker push cr.yandex/<registry_ID>/coi:logs
Configure Fluent Bit
-
Create a file named
spec.yaml
. It describes the specification of two containers: with an application that generates logs, and with a Fluent Bit agent.Specify the following in the field:
image
: Docker image URL. To find it out, in the management console , go to the Overview of Docker image page and copy the value of the Tags field.YC_GROUP_ID
: ID of thedefault
log group.
In the
fluentbit
section, theimage
field shows the current image of a container with the Fluent Bit agent. For a list of all available images, follow this link .version: '3.7' services: logs: container_name: logs-app image: <Docker_image_URL> restart: always depends_on: - fluentbit logging: # Fluent Bit parses logs in this format. driver: fluentd options: # Fluent Bit listens to logs on port 24224. fluentd-address: localhost:24224 # Tags are used for routing logs. tag: app.logs fluentbit: container_name: fluentbit image: cr.yandex/yc/fluent-bit-plugin-yandex:v1.0.3-fluent-bit-1.8.6 ports: - 24224:24224 - 24224:24224/udp restart: always environment: YC_GROUP_ID: <log_group_ID> volumes: - /etc/fluentbit/fluentbit.conf:/fluent-bit/etc/fluent-bit.conf - /etc/fluentbit/parsers.conf:/fluent-bit/etc/parsers.conf
-
Create a file named
user-data.yaml
. It describes the container log reading rules. If required, change the username and SSH key in theusers
section. Learn more about generating SSH keys.#cloud-config write_files: - content: | [SERVICE] Flush 1 Log_File /var/log/fluentbit.log Log_Level error Daemon off Parsers_File /fluent-bit/etc/parsers.conf [FILTER] Name parser Match app.logs Key_Name log Parser app_log_parser Reserve_Data On [INPUT] Name forward Listen 0.0.0.0 Port 24224 Buffer_Chunk_Size 1M Buffer_Max_Size 6M [OUTPUT] Name yc-logging Match * group_id ${YC_GROUP_ID} message_key text level_key severity default_level WARN authorization instance-service-account path: /etc/fluentbit/fluentbit.conf - content: | [PARSER] Name app_log_parser Format regex Regex ^\[req_id=(?<req_id>[0-9a-fA-F\-]+)\] \[(?<severity>.*)\] (?<code>\d+) (?<text>.*)$ Types code:integer path: /etc/fluentbit/parsers.conf users: - name: username groups: sudo shell: /bin/bash sudo: 'ALL=(ALL) NOPASSWD:ALL' ssh_authorized_keys: - ssh-ed25519 AAAA
The
SERVICE
section displays the Fluent Bit settings. Learn more about the settings .The
INPUT
section displays where and how to retrieve logs. To work with Fluentd and Fluent Bit logs, theforward
protocol is used. Fluent Bit listens to logs on port 24224.The
PARSER
section describes theregex
parser. It sets a regular expression that processes records:req_id
: Unique request IDseverity
: Logging levelcode
: HTTP response codetext
: All remaining text
The
FILTER
section shows that only records taggedapp.logs
are searched for. Thelog
field of each record is parsed byregex
, all other fields are saved inReserve_Data On
.
Create a VM from a Container Optimized Image
Specify the following in the field:
--zone
: Availability zone, e.g.,ru-central1-a
.--subnet-name
: Name of the subnet in this zone.--service-account-name
: Service account name.
IMAGE_ID=$(yc compute image get-latest-from-family container-optimized-image --folder-id standard-images --format=json | jq -r .id)
yc compute instance create \
--name coi-vm \
--zone=<zone> \
--network-interface subnet-name=<subnet_name>,nat-ip-version=ipv4 \
--metadata-from-file user-data=user-data.yaml,docker-compose=spec.yaml \
--create-boot-disk image-id=${IMAGE_ID} \
--service-account-name <service_account_name>
Note
The commands yc compute instance create
| create-with-container
| update
| add-metadata
support substitution of environment variable values into VM metadata. When you execute a Yandex Cloud CLI command, these values, specified in the user-data
key in $<variable_name>
format, will be substituted into the VM metadata from the environment variables of the environment the command is executed in.
To change such behavior, i.e. to provide a variable name to the VM metadata in $<variable_name>
format rather than take the variable value from the CLI command runtime environment, use the two-dollar syntax, e.g., $$<variable_name>
.
For more information, see Specifics of providing environment variables in metadata via the CLI.
View the logs
- In the management console
, go to the folder with thedefault
log group whose ID you specified in thespec.yaml
file. - Select Cloud Logging.
- Select the
default
log group. The page that opens will show the log group records.
If you do not have the Yandex Cloud CLI installed yet, install and initialize it.
By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID>
command. You can also set a different folder for any specific command using the --folder-name
or --folder-id
parameter.
To view records in the log group, run this command:
yc logging read --group-id=<log_group_ID>
--group-id
: ID of the default
log group specified in the spec.yaml
file.
You can view the log group records using the LogReadingService/Read gRPC API call.
Delete the resources you created
If you no longer need the resources you created, delete them: