Web service SLO monitoring in Monium

SLO monitoring helps you evaluate service reliability against a target level instead of responding to isolated incidents. In this tutorial, you will set up SLO monitoring for a web service in Monium. The web service can run in Yandex Cloud, other cloud environments, or on your on-premise infrastructure.

An SLO (service level objective) is the target reliability level of your service, e.g., 99.9% of successfully processed requests over a 30-day period. You define SLOs based on your service reliability requirements. Monium uses SLOs to calculate the error budget, which represents the maximum allowed error rate over a certain period. This enables your team to track error budget consumption and proactively address reliability drops.

This guide relies on Prometheus exporters to implement monitoring. You will configure synthetic checks for availability and response time, metric delivery to Monium, two SLOs, and error budget alerts. This tutorial is designed for SREs, DevOps engineers, and developers responsible for maintaining web service reliability.

To collect and deliver metrics, you will need the following components:

• node_exporter: Collects Linux system metrics. These metrics help you diagnose server health after an SLO alert.
• blackbox_exporter: Performs synthetic HTTP, TCP, and ICMP checks. The obtained metrics are used to calculate SLIs for availability and response time.
• OpenTelemetry Collector (OTel Collector): Collects metrics from node_exporter and blackbox_exporter and sends them to Monium.

In this tutorial, node_exporter is configured to run on your application server (app-server), while blackbox_exporter and OTel Collector run on a dedicated monitoring server (monitoring-server). OTel Collector polls both exporters using Prometheus and sends metrics to Monium. This approach does not require any changes in the application and allows you to host services in a private network.

Setup stepsSetup steps

1. Get your cloud ready.
2. Set up system metric collection in Linux.
3. Install and configure blackbox_exporter.
4. Set up metric delivery to Monium.
5. Configure a service availability SLO.
6. Configure error budget alerts.
7. Configure an SLO for service response time.
8. Use system metrics for post-alert diagnostics.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

The infrastructure support cost includes:

• Fee for using Monium (see Monium pricing).
• Fee for continuously running VMs if the web service or monitoring server resides in Yandex Cloud (see Yandex Compute Cloud pricing).

Set up your infrastructureSet up your infrastructure

1. Get your web service ready for SLO monitoring. The service can reside in any infrastructure; simply substitute your actual IP address or domain name in the examples. In this guide, the web service runs on an Ubuntu 20.04 Linux VM named app-server and is accessible via the internal IP address 10.128.0.10.

2. Create a monitoring server in Compute Cloud or another infrastructure. This guide uses an Ubuntu 20.04 Linux Compute Cloud VM named monitoring-server. You will install blackbox_exporter and OTel Collector on this VM to check the web service and deliver metrics to Monium.

3. To enable data delivery to Monium, create a service account and API key. You can do this in the Monium UI: on the left, select Settings → Project settings → Configuring telemetry recording → OpenTelemetry.

   • Click the Create a service account link. Select the monium.metrics.writer or monium.telemetry.writer role.
   • Click the Create an API key link. Select the yc.monium.metrics.write or yc.monium.telemetry.write scope.

   Alternatively, use these guides: Creating a service account and Creating an API key.

4. If you are deploying your web service and monitoring server within Yandex Cloud, create a security group for the monitoring server and allow:

   • Outgoing TCP traffic on port 443 to send metrics to Monium via OTel Collector.
   • Outgoing traffic to your web service endpoints for blackbox_exporter checks.

Set up system metric collection in LinuxSet up system metric collection in Linux

To diagnose incidents, you need data on server state from the exact moment an alert triggers. Install Prometheus node_exporter on the app-server VM. It collects CPU, memory, and disk metrics. Then configure OTel Collector to send these metrics to Monium.

Install Prometheus node_exporterInstall Prometheus node_exporter

1. Create a system user for node_exporter:

   sudo useradd --no-create-home --shell /bin/false node_exporter
   

2. Download and unpack the node_exporter archive:

   wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
   tar zxvf node_exporter-1.11.1.linux-amd64.tar.gz
   

3. Install the binary:

   sudo install -m 0755 ./node_exporter-1.11.1.linux-amd64/node_exporter /usr/local/bin/node_exporter
   sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
   

Create a systemd service for node_exporterCreate a systemd service for node_exporter

1. Create a file named /etc/systemd/system/node_exporter.service:

   [Unit]
   Description=Prometheus Node Exporter
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=node_exporter
   Group=node_exporter
   Type=simple
   ExecStart=/usr/local/bin/node_exporter --web.listen-address=<server_private_IP_address>:9100
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target
   

   Where <server_private_IP_address> is the address of app-server in the private network. If OTel Collector is running on the same server, specify 127.0.0.1.

2. Run node_exporter:

   sudo systemctl daemon-reload
   sudo systemctl enable --now node_exporter
   

3. Make sure the service status has changed to active (running):

   sudo systemctl status node_exporter
   

   Result:

   ● node_exporter.service - Prometheus Node Exporter
        Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; preset: enabled)
        Active: active (running) <...>
   

4. Make sure that metrics are available:

   curl http://<server_private_IP_address>:9100/metrics
   

Instead of node_exporter, you can use Unified Agent on the application server. It collects system metrics and sends them directly to Monium, bypassing OTel Collector. In this setup, your metric configuration will be stored across multiple locations. For more information, see Delivering Linux system metrics.

Install and configure blackbox_exporterInstall and configure blackbox_exporter

Blackbox_exporter runs on your monitoring server to perform synthetic HTTP, TCP, and ICMP checks. It generates the probe_success and probe_duration_seconds metrics, which are used in the availability and response time SLOs.

Install and configure blackbox_exporterInstall and configure blackbox_exporter

1. Create a system user:

   sudo useradd --no-create-home --shell /usr/sbin/nologin blackbox_exporter
   

2. Download and unpack the archive:

   wget -O /tmp/blackbox_exporter.tar.gz https://github.com/prometheus/blackbox_exporter/releases/download/v0.28.0/blackbox_exporter-0.28.0.linux-amd64.tar.gz
   tar -zxvf /tmp/blackbox_exporter.tar.gz
   sudo install -m 0755 ./blackbox_exporter-0.28.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
   sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
   

3. Create a folder for the configuration:

   sudo mkdir -p /etc/blackbox_exporter
   

Configure blackbox_exporterConfigure blackbox_exporter

1. Create a file named /etc/blackbox_exporter/blackbox.yml:

   modules:
     http_2xx:
       prober: http
       timeout: 5s
       http:
         method: GET
         valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
         preferred_ip_protocol: "ip4"
         follow_redirects: true
         fail_if_ssl: false
         fail_if_not_ssl: false
         tls_config:
           insecure_skip_verify: false
   
     http_2xx_tls:
       prober: http
       timeout: 5s
       http:
         method: GET
         follow_redirects: true
         fail_if_not_ssl: true
         preferred_ip_protocol: "ip4"
   
     tcp_connect:
       prober: tcp
       timeout: 3s
   
     icmp:
       prober: icmp
       timeout: 2s
   

   Where:

   • http_2xx: Basic check of HTTP endpoint availability. Check type: HTTP.
   • http_2xx_tls: Check with mandatory TLS enforced via fail_if_not_ssl: true. Check type: HTTP.
   • tcp_connect: TCP port availability check. Check type: TCP.
   • icmp: Network reachability check for the host. Check type: ICMP.

2. Set permissions for the configuration file:

   sudo chown -R blackbox_exporter:blackbox_exporter /etc/blackbox_exporter
   

Create a systemd service for blackbox_exporterCreate a systemd service for blackbox_exporter

1. Create a file named /etc/systemd/system/blackbox_exporter.service:

   [Unit]
   Description=Prometheus Blackbox Exporter
   Wants=network-online.target
   After=network-online.target
   
   [Service]
   User=blackbox_exporter
   Group=blackbox_exporter
   Type=simple
   ExecStart=/usr/local/bin/blackbox_exporter \
     --config.file=/etc/blackbox_exporter/blackbox.yml \
     --web.listen-address=127.0.0.1:9115
   Restart=on-failure
   
   [Install]
   WantedBy=multi-user.target
   

2. Run blackbox_exporter and add it to the auto start:

   sudo systemctl daemon-reload
   sudo systemctl enable --now blackbox_exporter
   

3. Check that the service status is active (running):

   sudo systemctl status blackbox_exporter
   

   Result:

   ● blackbox_exporter.service - Prometheus Blackbox Exporter
        Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; enabled; preset: enabled)
        Active: active (running) <...>
   

Test blackbox_exporterTest blackbox_exporter

Run the following tests:

curl "http://127.0.0.1:9115/probe?target=http://10.128.0.10&module=http_2xx" | grep probe_success
curl "http://127.0.0.1:9115/probe?target=10.128.0.10:80&module=tcp_connect" | grep probe_success
curl "http://127.0.0.1:9115/probe?target=10.128.0.10&module=icmp" | grep probe_success

Where 10.128.0.10 is your web service IP address.

Result:

probe_success 1

Set up metric delivery to MoniumSet up metric delivery to Monium

OTel Collector sends metrics collected by blackbox_exporter and node_exporter to Monium.

Install OTel CollectorInstall OTel Collector

1. Install OTel Collector:

   sudo apt-get update
   sudo apt-get -y install wget
   wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.156.0/otelcol_0.156.0_linux_amd64.deb
   sudo dpkg -i otelcol_0.156.0_linux_amd64.deb
   

2. Edit the /etc/otelcol/otelcol.conf file:

   MONIUM_PROJECT=folder__<folder_ID>
   MONIUM_API_KEY=<API_key>
   OTELCOL_OPTIONS=--config=/etc/otelcol/config.yaml
   

   Where:

   • MONIUM_PROJECT: Monium project name, e.g., folder__b1gg5f45su0k6rjr39s1.
   • MONIUM_API_KEY: API key of the service account with the monium.telemetry.writer role.

Configure OTel CollectorConfigure OTel Collector

Create a file named /etc/otelcol/config.yaml:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: blackbox_http_10_128_0_10
          scrape_interval: 30s
          metrics_path: /probe
          params:
            module: [http_2xx]
          static_configs:
            - targets:
                - http://10.128.0.10
              labels:
                probe_type: http
                target_name: lemp_http
          relabel_configs:
            - source_labels: [__address__]
              target_label: __param_target
            - source_labels: [__param_target]
              target_label: instance
            - target_label: __address__
              replacement: 127.0.0.1:9115

        - job_name: blackbox_tcp_10_128_0_10_80
          scrape_interval: 30s
          metrics_path: /probe
          params:
            module: [tcp_connect]
          static_configs:
            - targets:
                - 10.128.0.10:80
              labels:
                probe_type: tcp
                target_name: lemp_tcp_80
          relabel_configs:
            - source_labels: [__address__]
              target_label: __param_target
            - source_labels: [__param_target]
              target_label: instance
            - target_label: __address__
              replacement: 127.0.0.1:9115

        - job_name: blackbox_icmp_10_128_0_10
          scrape_interval: 30s
          metrics_path: /probe
          params:
            module: [icmp]
          static_configs:
            - targets:
                - 10.128.0.10
              labels:
                probe_type: icmp
                target_name: lemp_icmp
          relabel_configs:
            - source_labels: [__address__]
              target_label: __param_target
            - source_labels: [__param_target]
              target_label: instance
            - target_label: __address__
              replacement: 127.0.0.1:9115

        - job_name: node_exporter_lemp
          scrape_interval: 30s
          static_configs:
            - targets:
                - 10.128.0.10:9100
              labels:
                exporter: node_exporter
                target_name: lemp_node
                host: lemp

processors:
  batch:

exporters:
  otlp/monium:
    endpoint: ingest.monium.yandex.cloud:443
    compression: zstd
    headers:
      Authorization: "Api-Key ${env:MONIUM_API_KEY}"
      x-monium-project: "${env:MONIUM_PROJECT}"
      x-monium-cluster: production
      x-monium-service: lemp

service:
  telemetry:
    logs:
      level: info

  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp/monium]

Where:

• 10.128.0.10: Placeholder IP address of your application server. Replace it with the actual one.
• relabel_configs: Label overwriting rules for Prometheus to poll blackbox_exporter instead of the target.
• x-monium-cluster and x-monium-service: cluster and service labels that Monium assigns to all metrics from this OTel Collector. You will use these labels to find metrics and define your SLOs. The example uses the production and lemp values (Linux, Nginx, MySQL, and PHP stack). Replace these to match your environment.

Run OTel CollectorRun OTel Collector

1. Edit the /etc/systemd/system/multi-user.target.wants/otelcol.service file:

   [Unit]
   Description=OpenTelemetry Collector
   After=network.target
   
   [Service]
   EnvironmentFile=/etc/otelcol/otelcol.conf
   ExecStart=/usr/bin/otelcol $OTELCOL_OPTIONS
   ExecReload=/bin/kill -HUP $MAINPID
   KillMode=mixed
   Restart=on-failure
   Type=simple
   User=otel
   Group=otel
   
   [Install]
   WantedBy=multi-user.target
   

2. Run OTel Collector:

   sudo systemctl daemon-reload
   sudo systemctl enable --now otelcol
   sudo systemctl status otelcol
   

3. Make sure Monium receives the metrics:

   • Open Monium.
   • In the left-hand menu, expand Overview and select Metrics.
   • In the search field, specify service = "lemp", which is the service label value from the x-monium-service header in the OTel Collector configuration. If you have set a different value, specify it instead.

   The list will show the probe_success and probe_duration_seconds metrics as well as node_exporter metrics.

Configure a service availability SLOConfigure a service availability SLO

Blackbox_exporter returns a binary check result: probe_success equals 1 for a successful check and 0 for a failure. Use this metric to create an availability SLO:

1. On the Monium home page, expand Alerts and SLOs in the left menu.

2. Select SLO.

3. Click Create.

4. Specify the SLO parameters:

   • Name: For example, SLO HTTP Availability 30d.
   • Evaluation window: 30d.
   • Evaluation delay: 2m. This value must be greater than your metric collection interval: for scrape_interval: 30s, 2m is sufficient.
   • SLO: 99.9%.
   • Calculation Method: Good Events / Total Events.

5. Under Good Events, specify the following query:

   series_sum(
       {project = "folder__<folder_ID>", cluster = "production", service = "lemp", probe_type = "http", name = "probe_success"}
     )
   

6. Under Total Events, specify the following query:

   series_count(
       {project = "folder__<folder_ID>", cluster = "production", service = "lemp", probe_type = "http", name = "probe_success"}
     )
   

7. Click Create.

The error budget is calculated automatically once the SLO is created.

Configure error budget alertsConfigure error budget alerts

Create alerts to track your error budget consumption:

• Remaining error budget: Indicates that your service reliability is gradually degrading.
• Error budget consumption rate: Indicates spikes in the number of errors.

Create an alert for remaining error budgetCreate an alert for remaining error budget

1. On the Monium home page, expand Alerts and SLOs in the left menu.

2. Select Alerts.

3. Click Create alert → SLO.

4. Specify an alert name and level, e.g., Critical: remaining error budget indicates gradual degradation.

5. Select the SLO you created in the previous step.

6. In the Evaluation method field, select Error Budget Remaining.

7. Set the trigger conditions:

   • Warning: 50%.
   • Alarm: 20%.

8. Click Create.

Create an alert for error budget consumption rateCreate an alert for error budget consumption rate

1. On the Monium home page, expand Alerts and SLOs in the left menu.

2. Select Alerts.

3. Click Create alert → SLO.

4. Specify an alert name and level, e.g., Disaster: a high error budget consumption rate indicates an emergency incident.

5. Select the SLO you created in the previous step.

6. In the Evaluation method field, select Error Budget Exhaustion Rate.

7. Set the trigger conditions:

   • Warning: 1%.
   • Alarm: 2%.
   • Evaluation window: 1h.

8. Click Create.

Configure an SLO for service response timeConfigure an SLO for service response time

Blackbox_exporter measures the duration of synthetic checks using the probe_duration_seconds metric. Use this metric to create an SLO: 99% of HTTP checks must complete in under 300 ms.

The probe_duration_seconds metric shows total check duration but does not isolate where delays occur. Blackbox_exporter breaks down each check into the phases:

To visualize these phases, add these metrics to a dashboard as a stacked area chart. For the SLO and alerts, use the aggregate probe_duration_seconds metric. If latency degrades, the charts will pinpoint exactly where the bottleneck occurred: DNS, TLS, or network routing.

Create a response time SLOCreate a response time SLO

1. On the Monium home page, expand Alerts and SLOs in the left menu.

2. Select SLO.

3. Click Create.

4. Specify the SLO parameters:

   • Name: For example, SLO LATENCY HTTP 30d.
   • Evaluation window: 30d.
   • SLO: 99%.
   • Calculation Method: Good Events / Total Events.

5. Under Good Events, specify the following query:

   series_sum(
     heaviside(
       ({project="folder__<folder_ID>", cluster="production", service="lemp", probe_type="http", name="probe_duration_seconds"} * -1) + 0.3
     )
   )
   

   Where:

   • 0.3: Latency threshold in seconds, 300 ms.
   • heaviside(): Function that filters checks against the threshold. Returns 1 if latency is less than 300 ms or 0 if it is greater.

   Checks with a latency of exactly 300 ms evaluate to 0.5, which does not affect the SLI calculation.

6. Under Total Events, specify the following query:

   series_sum(
     {project="folder__<folder_ID>", cluster="production", service="lemp", probe_type="http", name="probe_duration_seconds"} * 0 + 1
   )
   

7. Click Create.

Use system metrics for post-alert diagnostics .Use system metrics for post-alert diagnostics .

When an SLO alert triggers, you must isolate the failure on the server side. Use node_exporter system metrics to quickly identify the root cause:

If an SLO alert has been triggered by probe_duration_seconds, check network phases first: probe_dns_lookup_time_seconds and probe_tls_handshake_duration_seconds. Then review server metrics: node_disk_io_time_weighted_seconds_total and node_cpu_seconds_total{mode="iowait"}.

Delete the resources you createdDelete the resources you created

To stop paying for the resources you created:

1. Delete the alerts you created in Monium.
2. Delete the Compute Cloud VMs.
3. Delete the Identity and Access Management service account.
4. Delete the Virtual Private Cloud security group.

If you reserved public static IP addresses, delete them.