Installing Fluent Bit with a plugin for Yandex Cloud Logging

Fluent Bit is an open-source multi-platform tool for collecting and processing logs. Fluent Bit collects, parses, and filters messages from a variety of inputs and saves them to storage. Messages then go to the router, which defines the output to send them to. Plugins are used to work with various inputs and outputs.

The plugin for Cloud Logging enables you to export logs generated in a Managed Service for Kubernetes cluster, including at the pod level, to Yandex Cloud Logging.

Getting startedGetting started

1. If you do not have the Yandex Cloud CLI yet, install and initialize it.

   The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

2. Create a service account, which is required for Fluent Bit.

3. Assign the logging.writer and monitoring.editor roles to the service account.

4. Create an authorized key for the service account and save it to the sa-key.json file.

5. Make sure that the security groups for the Managed Service for Kubernetes cluster and its node groups are configured correctly. If any rule is missing, add it.

   Warning

   The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.

Installation using Yandex Cloud MarketplaceInstallation using Yandex Cloud Marketplace

1. Navigate to the folder dashboard and select Managed Service for Kubernetes.
2. Click the name of the Managed Service for Kubernetes cluster you need and select the Marketplace tab.
3. Under Application available for installation, select Fluent Bit with a plugin for Cloud Logging and click Go to install.
4. Configure the application:
   • Namespace: Create a new namespace, e.g., fluent-bit-space. If you leave the default namespace, Fluent Bit may work incorrectly.
   • Application name: Specify the application name.
   • Target log group ID: Specify the ID of the log group to which Fluent Bit logs will be saved.
   • Cluster ID: Specify the Managed Service for Kubernetes cluster ID.
   • Service account key for Cloud Logging: Paste the contents of the sa-key.json file.
5. Click Install.
6. Wait for the application to change its status to Deployed.

Installation using a Helm chartInstallation using a Helm chart

1. Install Helm v3.8.0 or higher.

2. Install kubect and configure it to work with the new cluster.

3. To install a Helm chart with Fluent Bit, run this command:

   cat sa-key.json | helm registry login cr.yandex --username 'json_key' --password-stdin && \
   helm pull oci://cr.yandex/yc-marketplace/yandex-cloud/fluent-bit/fluent-bit \
     --version 2.1.7-3 \
     --untar && \
   helm install \
     --namespace <namespace> \
     --create-namespace \
     --set loggingGroupId=<log_group_ID> \
     --set loggingFilter=<cluster_ID> \
     --set-file auth.json=sa-key.json \
     fluentbit ./fluent-bit/
   

   This command also creates a new namespace required for Fluent Bit.

   If you set namespace to the default namespace, Fluent Bit may work incorrectly. We recommend you to specify a value different from all existing namespaces (e.g., fluent-bit-space).

   Note

   If you are using a Helm version below 3.8.0, append the export HELM_EXPERIMENTAL_OCI=1 && \ string to the command to enable Open Container Initiative (OCI) support in the Helm client.

Use casesUse cases

• Configuring Fluent Bit for Yandex Cloud Logging.

See alsoSee also

• Fluent Bit documentation
• Cloud Logging documentation