Yandex Managed Service for Kubernetes cluster monitoring with the help of Prometheus Operator with Yandex Monitoring support

Managed Service for Kubernetes enables you to export cluster object metrics to monitoring systems.

In this article, you will learn how to configure Yandex Managed Service for Kubernetes cluster monitoring with the help of Prometheus Operator with Yandex Monitoring support.

Prometheus Operator with Monitoring support streamlines the installation of Prometheus Operator, a popular solution for Kubernetes cluster monitoring, providing integration with Yandex Monitoring. You can use it to collect, store, and read metrics from containers, applications, and the infrastructure. The system leverages the Prometheus data model and PromQL.

To set up monitoring for your Managed Service for Kubernetes cluster:

1. Install Prometheus Operator using a Helm chart or Yandex Cloud Marketplace.
2. Set up and test alerts.
3. Visualize metrics.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost for this solution includes:

• Fee for the Managed Service for Kubernetes master: (see Managed Service for Kubernetes pricing).
• Fee for Managed Service for Kubernetes cluster nodes: specifically using computing resources and storage (see Yandex Compute Cloud pricing).
• Fee for public IP addresses for Managed Service for Kubernetes cluster hosts and Managed Service for Kubernetes cluster nodes with public access enabled (see Yandex Virtual Private Cloud pricing).

Getting startedGetting started

1. Create security groups for the Managed Service for Kubernetes cluster and its node groups.

   Warning

   The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.

2. Create a Managed Service for Kubernetes cluster and node group in any suitable configuration with internet access and the security groups you set up earlier.

3. Install kubect and configure it to work with the new cluster.

4. Create a service account with the monitoring.editor role.

5. Create an API key for the service account:

   • If you want to install Prometheus Operator using a Helm chart, run the following command and save the secret key you get:

     yc iam api-key create --service-account-name=<service_account_name>
     

   • If you want to install Prometheus Operator using Cloud Marketplace, create an API key in JSON format and save it to the sa-key.json file:

     yc iam api-key create \
        --service-account-name=<service_account_name> \
        --format=json | jq -c > sa-key.json
     

     Tip

     You can also automatically create a new API key during the app installation.

6. Create a Prometheus workspace:

   Management console

   1. Open the Monitoring home page.
   2. In the left-hand panel, select Prometheus.
   3. Click Create workspace.

Install Prometheus Operator using a Helm chartInstall Prometheus Operator using a Helm chart

1. Install Helm v3.8.0 or higher.

2. Get the Prometheus workspace ID:

   Management console

   1. Open the Monitoring home page.
   2. Select Prometheus on the left.
   3. Select a Prometheus workspace.
   4. This will open the General information tab displaying the workspace ID. Save the ID.

3. To install a Helm chart with Prometheus Operator, run this command:

   helm pull oci://cr.yandex/yc-marketplace/yandex-cloud/prometheus/kube-prometheus-stack \
     --version 72.6.2-1 \
     --untar && \
   helm install \
     --namespace <namespace_for_Prometheus_Operator> \
     --create-namespace \
     --set prometheusWorkspaceId=<Prometheus_workspace_ID> \
     --set iam_api_key_value_generated.secretAccessKey=<secret_part_of_API_key> \
     prometheus ./kube-prometheus-stack/
   

   This command will also create a new namespace for Prometheus Operator.

   If you set namespace to the default namespace, Prometheus Operator may work incorrectly. Use a value different from all the existing namespaces, e.g., prometheus-operator-space.

   Note

   If you are using a Helm version below 3.8.0, append the export HELM_EXPERIMENTAL_OCI=1 && \ string to the command to enable Open Container Initiative (OCI) support in the Helm client.

4. Make sure the Prometheus Operator pods have changed their status to Running:

   kubectl get pods --namespace=<namespace_for_Prometheus_Operator> \
      -l "release=prometheus"
   

Install Prometheus Operator using Cloud MarketplaceInstall Prometheus Operator using Cloud Marketplace

Configure alerts based on metricsConfigure alerts based on metrics

1. Create a notification channel by selecting an available sending method.

2. Set up the Alert Manager responsible for sending notifications:

   1. Open the Monitoring home page.

   2. Select Prometheus on the left.

   3. Select a Prometheus workspace.

   4. Navigate to the Alert manager configuration tab.

   5. Click Upload a configuration file and upload the configuration file in YAML format.

      Sample configuration file

      global:
        resolve_timeout: 5m
      route:
        receiver: telegram
        routes:
          - receiver: "null"
            matchers:
              - severity="none"
            continue: false
      
      receivers:
        - name: telegram
          yandex_monitoring_configs:
            - channel_names:
              - "<notification_channel_name>"
        - name: "null"
          yandex_monitoring_configs:
            - channel_names: []
      

3. Add a test alerting rule:

   cat <<EOF | kubectl apply -f -
   apiVersion: monitoring.coreos.com/v1                                                        
   kind: PrometheusRule
   metadata:
     labels:
       release: kube-prometheus-stack
     name: test
   spec:
     groups:
     - name: test
       rules:
         - alert: Test
           annotations:
             description: this is a test alert
             summary: test alert
           expr: vector(1)
           labels:
             severity: warning
   EOF
   

4. Wait until you get alerted through the selected notification channel or check the alert status in the management console:

   1. Open the Monitoring home page.
   2. Select Prometheus on the left.
   3. Select a Prometheus workspace.
   4. Navigate to the Managing rules tab.
   5. Click the test alert name.
   6. Navigate to the Alerts tab.
   7. Make sure the test alert status changed to FIRING.

Visualize metricsVisualize metrics

Build a chart based on the metrics and add it to the dashboard.

Build a chart based on metricsBuild a chart based on metrics

Add the chart to the dashboardAdd the chart to the dashboard

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

1. Delete the Managed Service for Kubernetes cluster.
2. Delete the public static IP address for your Managed Service for Kubernetes cluster if you reserved one.