Monitoring a cluster using Prometheus and Grafana

Written by

Yandex Cloud

Improved by

Dmitry A.

Updated at November 22, 2024

Getting started
Install Prometheus
Install the Trickster caching proxy
Install Grafana
Set up and check Grafana
Delete the resources you created

Managed Service for Kubernetes enables you to upload cluster object metrics to monitoring systems.

In this article, you will learn how to set up the Prometheus metrics collection system and the Grafana visualization system in a Managed Service for Kubernetes cluster. The Trickster caching proxy will be installed to speed up the transfer of metrics.

To set up the Managed Service for Kubernetes cluster monitoring system:

If you no longer need the resources you created, delete them.

Getting started

Create security groups for the Managed Service for Kubernetes cluster and its node groups.

Warning

The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.
Create a Managed Service for Kubernetes cluster and a node group in any suitable configuration with internet access and the security groups you prepared earlier.
Install kubectl and configure it to work with the created cluster.
Install Helm v3.8.0 or higher.

Install Prometheus

The Prometheus monitoring system scans Managed Service for Kubernetes cluster objects and collects their metrics into its own database. The collected metrics are available within the Managed Service for Kubernetes cluster over HTTP.

Add a repository containing the Prometheus distribution:

helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && \
helm repo update

Install Prometheus:

helm install my-prom prometheus-community/prometheus

Make sure that all pods have entered the Running state:

kubectl get pods -l "app.kubernetes.io/instance=my-prom"

Result:

NAME                                              READY  STATUS   RESTARTS  AGE
my-prom-prometheus-alertmanager-7b********-xt6ws  2/2    Running  0         81s
my-prom-prometheus-node-exporter-*****            1/1    Running  0         81s
my-prom-prometheus-pushgateway-69********-swrfb   1/1    Running  0         81s
my-prom-prometheus-server-7b********-m4v78        2/2    Running  0         81s

Install the Trickster caching proxy

The Trickster caching proxy speeds up reading from a Prometheus database, which enables the display of near real-time Grafana metrics and reduces the load on Prometheus.

Add a repository containing the Trickster distribution:

helm repo add tricksterproxy https://helm.tricksterproxy.io && \
helm repo update

Create a configuration file named trickster.yaml that contains Trickster settings:

trickster.yaml

frontend:
  listenAddress: ""
  tlsListenAddress: ""
  tlsListenPort: ""
  connectionsLimit: "0"
origins:
  - name: default
    originType: prometheus
    originURL: http://my-prom-prometheus-server:80
profiler:
  enabled: false
  port: 6060
prometheusScrape: false
prometheus:
  serviceMonitor:
    enabled: false
    interval: 30s
    labels: {}
replicaCount: 1
image:
  repository: tricksterproxy/trickster
  tag: "1.1"
  pullPolicy: IfNotPresent
service:
  annotations: {}
  labels: {}
  clusterIP: ""
  externalIPs: []
  loadBalancerIP: ""
  loadBalancerSourceRanges: []
  metricsPort: 8481
  servicePort: 8480
  type: ClusterIP
ingress:
  enabled: false
  annotations: {}
  extraLabels: {}
  hosts: []
  tls: []
volumes:
  persistent:
    type: "persistentVolume"
    enabled: false
    mountPath: "/tmp/trickster"
    accessModes:
      - ReadWriteOnce
    annotations: {}
    existingClaim: ""
    size: 15Gi
  generic:
    type: "generic"
    enabled: true
    mountPath: "/tmp/trickster"
podAnnotations: {}
resources: {}
securityContext: {}

You can change the size of the storage allocated to the caching proxy. Specify the storage size you need in the volumes.persistent.size parameter.

Install Trickster:

helm install trickster tricksterproxy/trickster --namespace default -f trickster.yaml

Make sure the Trickster pod has entered the Running state:
```
kubectl get pods -l "app=trickster"
```

The caching proxy is available in the Managed Service for Kubernetes cluster at http://trickster:8480. Grafana will use this URL to collect metrics.

Install Grafana

When deploying the application, the following will be created:

Deployment of the Grafana application.
PersistentVolumeClaim to reserve internal storage.
Service of the LoadBalancer type to enable network access to the Grafana management console.

To install Grafana:

Create a configuration file named grafana.yaml.

grafana.yaml

---
apiVersion: v1
kind: PersistentVolumeClaim
metadata:
  name: grafana-pvc
spec:
  accessModes:
    - ReadWriteOnce
  resources:
    requests:
      storage: 1Gi
---
apiVersion: apps/v1
kind: Deployment
metadata:
  labels:
    app: grafana
  name: grafana
spec:
  selector:
    matchLabels:
      app: grafana
  template:
    metadata:
      labels:
        app: grafana
    spec:
      securityContext:
        fsGroup: 472
        supplementalGroups:
          - 0
      containers:
        - name: grafana
          image: grafana/grafana:latest
          imagePullPolicy: IfNotPresent
          ports:
            - containerPort: 3000
              name: http-grafana
              protocol: TCP
          readinessProbe:
            failureThreshold: 3
            httpGet:
              path: /robots.txt
              port: 3000
              scheme: HTTP
            initialDelaySeconds: 10
            periodSeconds: 30
            successThreshold: 1
            timeoutSeconds: 2
          livenessProbe:
            failureThreshold: 3
            initialDelaySeconds: 30
            periodSeconds: 10
            successThreshold: 1
            tcpSocket:
              port: 3000
            timeoutSeconds: 1
          resources:
            requests:
              cpu: 250m
              memory: 750Mi
          volumeMounts:
            - mountPath: /var/lib/grafana
              name: grafana-pv
      volumes:
        - name: grafana-pv
          persistentVolumeClaim:
            claimName: grafana-pvc
---
apiVersion: v1
kind: Service
metadata:
  name: grafana
spec:
  ports:
    - port: 3000
      protocol: TCP
      targetPort: http-grafana
  selector:
    app: grafana
  sessionAffinity: None
  type: LoadBalancer

If required, change:

Storage size allocated for Grafana in the spec.resources.requests.storage parameter for kind: PersistentVolumeClaim.
Computing resources allocated to the Grafana pod in the spec.containers.resources parameters for kind: Deployment.

Install Grafana:
```
kubectl apply -f grafana.yaml
```
Make sure the Grafana pod has entered the Running state:
```
kubectl get pods -l "app=grafana"
```

Set up and check Grafana

Find the address where Grafana is available and go to it:
```
export GRAFANA_IP=$(kubectl get service/grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}') && \
export GRAFANA_PORT=$(kubectl get service/grafana -o jsonpath='{.spec.ports[0].port}') && \
echo http://$GRAFANA_IP:$GRAFANA_PORT
```
Note

If the resource is unavailable at the specified URL, make sure that the security groups for the Managed Service for Kubernetes cluster and its node groups are configured correctly. If any rule is missing, add it.
In the browser window that opens, enter your admin/admin username and password and then set a new password for the admin user.
Add a data source with the Prometheus type and the following settings:
- Name: Prometheus.
- URL: http://trickster:8480.
Click Save & test and make sure that the data source was successfully connected (Data source is working).
Import the Kubernetes Deployment Statefulset Daemonset metrics dashboard containing the basic Kubernetes metrics. Specify the dashboard ID (8588) when importing.

Tip

To check the scenario, you can use any suitable dashboard from the Grafana catalog.
Open the dashboard and make sure that Grafana receives metrics from the Managed Service for Kubernetes cluster.

Delete the resources you created

Delete the resources you no longer need to avoid paying for them:

Delete the Managed Service for Kubernetes cluster.
If you reserved a public static IP address for your Managed Service for Kubernetes cluster, delete it.

Monitoring a cluster using Prometheus and Grafana

Getting startedGetting started

Install PrometheusInstall Prometheus

Install the Trickster caching proxyInstall the Trickster caching proxy

Install GrafanaInstall Grafana

Set up and check GrafanaSet up and check Grafana

Delete the resources you createdDelete the resources you created