Installing Filebeat OSS

Filebeat OSS is a plugin for collecting and forwarding logs to the OpenSearch ecosystem. Installed in a Managed Service for Kubernetes cluster, Filebeat OSS collects cluster and pods logs and forwards them to Yandex Managed Service for OpenSearch.

Getting startedGetting started

1. If you do not have the Yandex Cloud command line interface yet, install and initialize it.

   The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

2. Make sure that the Managed Service for Kubernetes cluster is located in the same cloud network as the Managed Service for OpenSearch cluster.

3. Make sure that the security groups for the Managed Service for Kubernetes cluster and its node groups are configured correctly. If any rule is missing, add it.

   Warning

   The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.

4. Enable the compatibility mode to support the Filebeat OSS client in OpenSearch. For this, run the following query:

   wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" && \
   curl \
   --user <username>:<password> --cacert CA.pem \
   --request PUT https://<name_of_host_with_DATA_role>:9200/_cluster/settings \
   --header "Content-Type: application/json" \
   --data \
   '{
   "persistent": {
     "compatibility": {
       "override_main_response_version": true
     }
   }
   }'
   

   Where:

   • <username>: OpenSearch username.
   • <password>: OpenSearch user password.
   • <host_name>: Name of the Managed Service for OpenSearch host with the DATA role, e.g., rc1a-6khpaeo31lacqo21.mdb.yandexcloud.net.

   A successful response will have this format:

   {
     "acknowledged":true,
     "persistent":
     {
       "compatibility":
       {
         "override_main_response_version":"true"
       }
     },
     "transient":{}
   }
   

Installation using Yandex Cloud MarketplaceInstallation using Yandex Cloud Marketplace

1. Go to the folder page and select Managed Service for Kubernetes.
2. Click the name of the Managed Service for Kubernetes cluster you need and select the Marketplace tab.
3. Under Application available for installation, select Filebeat OSS and click Go to install.
4. Configure the application:
   • Namespace: Select a namespace or create a new one.
   • Application name: Specify the app name, e.g., filebeat-oss.
   • OpenSearch username: Enter the username that Filebeat OSS will use to connect to the Managed Service for OpenSearch cluster.
   • OpenSearch connection password: Enter the user password for the Managed Service for OpenSearch cluster.
   • OpenSearch FQDN: Enter the URL and port for the Managed Service for OpenSearch cluster host with the DATA role, e.g., https://rc1a-6khpaeo31lacqo21.mdb.yandexcloud.net:9200. For more information about connecting to a cluster, see the service documentation.
5. Click Install.
6. Wait for the application to change its status to Deployed.

Installation using a Helm chartInstallation using a Helm chart

1. Install Helm v3.8.0 or higher.

2. Install kubectl and configure it to work with the created cluster.

3. To install a Helm chart with Filebeat OSS, run the following command:

   helm pull oci://cr.yandex/yc-marketplace/yandex-cloud/filebeat-oss/chart/filebeat-oss \
     --version 7.12.1-1 \
     --untar && \
   helm install \
     --namespace <namespace> \
     --create-namespace \
     --set app.url='<URL_and_port_for_Managed_Service_for_OpenSearch_cluster_with_DATA_role>' \
     --set app.username='<OpenSearch_cluster_username>' \
     --set app.password='<OpenSearch_cluster_user_password>' \
     filebeatoss ./filebeat-oss/
   

   This command will also create a new namespace required for Filebeat OSS.

   Note

   If you are using a Helm version below 3.8.0, append the export HELM_EXPERIMENTAL_OCI=1 && \ string to the command to enable Open Container Initiative (OCI) support in the Helm client.

4. Make sure the Filebeat OSS pod has changed its status to Running:

   kubectl get pods --namespace=<namespace> -l app=filebeatoss-filebeat-oss -w
   

Use casesUse cases

• Monitoring a Managed Service for Kubernetes cluster using Filebeat OSS.

See alsoSee also

• Managed Service for OpenSearch documentation
• Filebeat documentation