Fluent Bit with Monium integration

Fluent Bit is a lightweight open source agent for collecting and processing logs. The product deploys it to a Managed Service for Kubernetes cluster as a DaemonSet with a ready-made configuration: the agent reads container logs on every node in the namespaces you list, enriches them with Kubernetes metadata (namespace, pod, and so on), and delivers them to Monium over the OpenTelemetry protocol — each namespace to a service of the same name.

Logs are buffered on the node disk, so they survive a Monium outage or an agent restart.

1. Create a Kubernetes cluster and a node group.

2. Create a service account required for Fluent Bit:

   yc iam service-account create --name fluent-bit-monium
   

3. Assign the service account the monium.logs.writer role in the folder with the Monium project.

4. Depending on the authentication method:

   • Link the service account to the cluster workload identity federation using subject id system:serviceaccount:<namespace>:fluent-bit-monium, where <namespace> is the namespace you install the application into. This is the recommended method.
     For more information on linking a service account to the federation, see the documentation.
   • Or create an API key for the service account with the yc.monium.logs.write scope.

5. Specify the name of the Monium project to deliver the logs to, for example folder__<folder ID>. Monium automatically creates such a project for every folder.

6. Configure Fluent Bit:

   • Namespace — create a new namespace, e.g., fluent-bit-monium.
   • Application name — enter a name for the application.
   • Monium project — specify the name of the project to deliver the logs to.
   • Monium cluster — specify the destination cluster or leave the default value.
   • Namespaces to collect logs from — list the namespaces whose logs you need. Logs from each one are delivered to a Monium service of the same name. Logs from the other namespaces are not sent.

7. Click Install.

8. Wait for the application to change its status to Deployed.

Logs are collected from all pods in the namespaces you listed, regardless of their format. JSON logs are parsed into record attributes you can search and filter by in Monium. To parse logs in a different format the same way, add the fluentbit.io/parser annotation with the parser name to the pod:

annotations:
  fluentbit.io/parser: logfmt

Standard Fluent Bit parsers are available, e.g., logfmt, nginx, or apache2. See the full list in the Fluent Bit repository.

• Collecting Managed Service for Kubernetes container logs in Monium.
• Migrating log collection from Cloud Logging to Monium.

Yandex Cloud technical support is available 24/7 to respond to requests. The types of requests available and their response time depend on your pricing plan. You can activate paid support in the management console. Learn more about requesting technical support.