Monitoring a cluster using Prometheus and Grafana
Managed Service for Kubernetes enables you to upload cluster object metrics to monitoring systems.
In this article, you will learn how to set up the Prometheus
To set up the Managed Service for Kubernetes cluster monitoring system:
Getting started
-
Create security groups for the Managed Service for Kubernetes cluster and its node groups.
Warning
The configuration of security groups determines the performance and availability of the cluster and the services and applications running in it.
-
Create a Managed Service for Kubernetes cluster and a node group in any suitable configuration with internet access and the security groups prepared earlier.
-
Install kubectl
and configure it to work with the created cluster. -
Install Helm
v3.8.0 or higher.
Install Prometheus
The Prometheus monitoring system scans Managed Service for Kubernetes cluster objects and collects their metrics into its own database. The collected metrics are available within the Managed Service for Kubernetes cluster over HTTP.
-
Add a repository containing the Prometheus distribution:
helm repo add prometheus-community https://prometheus-community.github.io/helm-charts && \ helm repo update
-
Install Prometheus:
helm install my-prom prometheus-community/prometheus
-
Make sure that all the pods changed their status to
Running
:kubectl get pods -l "app.kubernetes.io/instance=my-prom"
Result:
NAME READY STATUS RESTARTS AGE my-prom-prometheus-alertmanager-7b********-xt6ws 2/2 Running 0 81s my-prom-prometheus-node-exporter-***** 1/1 Running 0 81s my-prom-prometheus-pushgateway-69********-swrfb 1/1 Running 0 81s my-prom-prometheus-server-7b********-m4v78 2/2 Running 0 81s
Install the Trickster caching proxy
The Trickster caching proxy speeds up reading
-
Add a repository containing the Trickster distribution:
helm repo add tricksterproxy https://helm.tricksterproxy.io && \ helm repo update
-
Create a
trickster.yaml
configuration file with the following Trickster settings:trickster.yaml
frontend: listenAddress: "" tlsListenAddress: "" tlsListenPort: "" connectionsLimit: "0" origins: - name: default originType: prometheus originURL: http://my-prom-prometheus-server:80 profiler: enabled: false port: 6060 prometheusScrape: false prometheus: serviceMonitor: enabled: false interval: 30s labels: {} replicaCount: 1 image: repository: tricksterproxy/trickster tag: "1.1" pullPolicy: IfNotPresent service: annotations: {} labels: {} clusterIP: "" externalIPs: [] loadBalancerIP: "" loadBalancerSourceRanges: [] metricsPort: 8481 servicePort: 8480 type: ClusterIP ingress: enabled: false annotations: {} extraLabels: {} hosts: [] tls: [] volumes: persistent: type: "persistentVolume" enabled: false mountPath: "/tmp/trickster" accessModes: - ReadWriteOnce annotations: {} existingClaim: "" size: 15Gi generic: type: "generic" enabled: true mountPath: "/tmp/trickster" podAnnotations: {} resources: {} securityContext: {}
You can change the size of the storage allocated to the caching proxy. Specify the desired storage size in the
volumes.persistent.size
parameter. -
Install Trickster:
helm install trickster tricksterproxy/trickster --namespace default -f trickster.yaml
-
Make sure the Trickster pod status changed to
Running
:kubectl get pods -l "app=trickster"
The caching proxy is available in the Managed Service for Kubernetes cluster at http://trickster:8480
. Grafana will use this URL to collect metrics.
Install Grafana
When deploying the application, the following will be created:
- Grafana application
Deployment
. - PersistentVolumeClaim to reserve internal storage.
LoadBalancer
Service
to enable network access to the Grafana management console.
To install Grafana:
-
Create a
grafana.yaml
configuration file.grafana.yaml
--- apiVersion: v1 kind: PersistentVolumeClaim metadata: name: grafana-pvc spec: accessModes: - ReadWriteOnce resources: requests: storage: 1Gi --- apiVersion: apps/v1 kind: Deployment metadata: labels: app: grafana name: grafana spec: selector: matchLabels: app: grafana template: metadata: labels: app: grafana spec: securityContext: fsGroup: 472 supplementalGroups: - 0 containers: - name: grafana image: grafana/grafana:latest imagePullPolicy: IfNotPresent ports: - containerPort: 3000 name: http-grafana protocol: TCP readinessProbe: failureThreshold: 3 httpGet: path: /robots.txt port: 3000 scheme: HTTP initialDelaySeconds: 10 periodSeconds: 30 successThreshold: 1 timeoutSeconds: 2 livenessProbe: failureThreshold: 3 initialDelaySeconds: 30 periodSeconds: 10 successThreshold: 1 tcpSocket: port: 3000 timeoutSeconds: 1 resources: requests: cpu: 250m memory: 750Mi volumeMounts: - mountPath: /var/lib/grafana name: grafana-pv volumes: - name: grafana-pv persistentVolumeClaim: claimName: grafana-pvc --- apiVersion: v1 kind: Service metadata: name: grafana spec: ports: - port: 3000 protocol: TCP targetPort: http-grafana selector: app: grafana sessionAffinity: None type: LoadBalancer
If required, change:
- The size of the storage allocated for Grafana in the
spec.resources.requests.storage
parameter forkind: PersistentVolumeClaim
. - Computing resources allocated to the Grafana pod in the
spec.containers.resources
parameters forkind: Deployment
.
- The size of the storage allocated for Grafana in the
-
Install Grafana:
kubectl apply -f grafana.yaml
-
Make sure that the Grafana pod status changed to
Running
:kubectl get pods -l "app=grafana"
Set up and check Grafana
-
Find the address where Grafana is available and go to it:
export GRAFANA_IP=$(kubectl get service/grafana -o jsonpath='{.status.loadBalancer.ingress[0].ip}') && \ export GRAFANA_PORT=$(kubectl get service/grafana -o jsonpath='{.spec.ports[0].port}') && \ echo http://$GRAFANA_IP:$GRAFANA_PORT
-
In the browser window that opens, enter your
admin/admin
username and password and then set a new password for theadmin
user. -
Add a data source
with thePrometheus
type and the following settings:- Name:
Prometheus
. - URL:
http://trickster:8480
.
- Name:
-
Click Save & test and make sure that the data source was successfully connected (
Data source is working
). -
Import the
Kubernetes Deployment Statefulset Daemonset metrics
dashboard , which contains basic Kubernetes metrics. Specify the dashboard ID (8588
) when importing.Tip
To check the scenario, you can use any suitable dashboard from the Grafana catalog
. -
Open the dashboard and make sure that Grafana receives metrics from the Managed Service for Kubernetes cluster.
Delete the resources you created
Delete the resources you no longer need to avoid paying for them:
- Delete the Managed Service for Kubernetes cluster.
- If you reserved a public static IP address for your Managed Service for Kubernetes cluster, delete it.