Yandex Cloud
Documentation
PresentationContact UsTry it for free
© 2026 Direct Cursus Technology L.L.C.
Tutorials
    • All tutorials
    • Architecture and protection of a basic web service
    • Cost analysis by resource using Object Storage
    • Obtaining the information you need to request the Russian Ministry of Digital Development to whitelist a resource
      • Getting started with Terraform
      • Terraform data sources
      • Uploading Terraform states to Object Storage
      • Getting started with Packer
      • Building a VM image with infrastructure tools using Packer
      • Locking Terraform states using Managed Service for YDB
      • Using Yandex Cloud modules in Terraform
      • Managing Kubernetes resources via the Terraform provider
      • Creating a VM and an instance group with a Container Optimized Image using Terraform
      • Transferring logs through Unified Agent HTTP input to Cloud Logging
      • Running the DeepSeek-R1 language model in a Compute Cloud GPU cluster
      • Web service SLO monitoring

In this article:

  • Setup steps
  • Get your cloud ready
  • Required paid resources
  • Set up your infrastructure
  • Set up system metric collection in Linux
  • Install Prometheus node_exporter
  • Create a systemd service for node_exporter
  • Install and configure blackbox_exporter
  • Install and configure blackbox_exporter
  • Set up blackbox_exporter configuration
  • Create a systemd service for blackbox_exporter
  • Test blackbox_exporter
  • Set up metric delivery to Monium
  • Install OTel Collector
  • Set up OTel Collector configuration
  • Run OTel Collector
  • Configure an SLO for service availability
  • Configure error budget alerts
  • Create an alert for error budget remaining
  • Create an alert for error budget consumption
  • Configure an SLO for service response time
  • Create a response time SLO
  • Use system metrics for diagnostics after an alert
  • Delete the resources you created
  1. Basic infrastructure
  2. Tools
  3. Web service SLO monitoring

Web service SLO monitoring in Monium

Written by
Yandex Cloud
Updated at November 11, 2026
View in Markdown
  • Setup steps
  • Get your cloud ready
    • Required paid resources
    • Set up your infrastructure
  • Set up system metric collection in Linux
    • Install Prometheus node_exporter
    • Create a systemd service for node_exporter
  • Install and configure blackbox_exporter
    • Install and configure blackbox_exporter
    • Set up blackbox_exporter configuration
    • Create a systemd service for blackbox_exporter
    • Test blackbox_exporter
  • Set up metric delivery to Monium
    • Install OTel Collector
    • Set up OTel Collector configuration
    • Run OTel Collector
  • Configure an SLO for service availability
  • Configure error budget alerts
    • Create an alert for error budget remaining
    • Create an alert for error budget consumption
  • Configure an SLO for service response time
    • Create a response time SLO
  • Use system metrics for diagnostics after an alert
  • Delete the resources you created

SLO monitoring helps evaluating service reliability against a target level instead of solving faults individually. In this tutorial, you will set up SLO monitoring for a web service in Monium. The web service may run in Yandex Cloud, another cloud, or on your own infrastructure.

An SLO (service level objective) is a target reliability level of a service, such as "99.9% requests have been processed successfully over 30 days". You can set an SLO depending on the service reliability requirements. Based on an SLO, Monium calculates the error budget, the maximum allowed amount of errors over a period. A special command tracks error budget spending and responds to reliability reduction in advance.

This guide uses monitoring based on Prometheus exporters. You will set up synthetic checks for availability and response time, metric delivery to Monium, two SLOs, and error budget alerts. The guide will be useful for SREs, DevOps engineers, and developers in charge of web service reliability.

To collect and deliver metrics, you will need:

  • node_exporter to collect Linux system metrics. They help you diagnose the server’s state after an SLO alert.
  • blackbox_exporter to run synthetic HTTP, TCP, and ICMP checks. Check metrics are used to calculate availability and response time SLIs.
  • OpenTelemetry Collector (OTel Collector) collects metrics from node_exporter and blackbox_exporter and sends them to Monium.

In this guide’s configuration, node_exporter runs on the application server named app-server, while blackbox_exporter and OTel Collector, on a separate monitoring server named monitoring-server. OTel Collector polls both exporters using Prometheus and sends metrics to Monium. This approach does not require any changes in the application and allows hosting servers in a private network.

Note

This guide does not describe installation of the web service. We assume that the web service is already deployed and available over the network. The node_exporter utility is installed on the application server, while blackbox_exporter and OTel Collector, on the monitoring server. For a simple configuration, all components can be installed together with the web service, eliminating the need for a separate monitoring server.

Setup stepsSetup steps

  1. Get your cloud ready.
  2. Set up system metric collection in Linux.
  3. Install and configure blackbox_exporter.
  4. Set up metric delivery to Monium.
  5. Configure an SLO for service availability.
  6. Configure error budget alerts.
  7. Configure an SLO for service response time.
  8. Use system metrics for post-alert diagnostics.

If you no longer need the resources you created, delete them.

Get your cloud readyGet your cloud ready

Sign up for Yandex Cloud and create a billing account:

  1. Navigate to the management console and log in to Yandex Cloud or create a new account.
  2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

The infrastructure support cost includes:

  • Fee for using Monium (see Monium pricing).
  • Fee for continuously running VMs if the web service or monitoring server resides in Yandex Cloud (see Yandex Compute Cloud pricing).

Set up your infrastructureSet up your infrastructure

  1. Get your web service ready for SLO monitoring. It can be deployed on any infrastructure. Specify its IP address and domain name in the examples. In this guide, the web service runs on the app-server VM with Linux Ubuntu 20.04 and is available at internal IP address 10.128.0.10.

  2. Create the monitoring server in Compute Cloud or another infrastructure. This guide uses a Compute Cloud VM named monitoring-server with Linux Ubuntu 20.04. On it, you will install blackbox_exporter and OTel Collector to check the web service and deliver metrics to Monium.

  3. To configure access for sending data to Monium, create a service account and API key. You can do this in the Monium UI: on the left, select Settings → Project settings → Configuring telemetry recording → OpenTelemetry.

    • Click the Create service account link. Select the monium.metrics.writer or monium.telemetry.writer role.
    • Click the Create API key link. Select the yc.monium.metrics.write or yc.monium.telemetry.write scope.

    You can also follow the Creating a service account and Creating an API key guides to create the service account and API key.

  4. In case you deploy the web service and monitoring server in Yandex Cloud, create a security group for the monitoring server and allow the following in it:

    • Outgoing TCP traffic on port 443 to send metrics to Monium via OTel Collector.
    • Incoming traffic to the web service endpoints for blackbox_exporter checks.

Set up system metric collection in LinuxSet up system metric collection in Linux

To diagnose incidents, you need the server state data for the moment the alert was triggered. Install Prometheus node_exporter on the app-server VM. It collects CPU, memory, and disk metrics. Then, configure OTel Collector to send them to Monium.

Install Prometheus node_exporterInstall Prometheus node_exporter

  1. Create a system user for node_exporter:

    sudo useradd --no-create-home --shell /bin/false node_exporter
    
  2. Download and unpack the node_exporter archive:

    wget https://github.com/prometheus/node_exporter/releases/download/v1.11.1/node_exporter-1.11.1.linux-amd64.tar.gz
    tar zxvf node_exporter-1.11.1.linux-amd64.tar.gz
    
  3. Install the binary:

    sudo install -m 0755 ./node_exporter-1.11.1.linux-amd64/node_exporter /usr/local/bin/node_exporter
    sudo chown node_exporter:node_exporter /usr/local/bin/node_exporter
    

Create a systemd service for node_exporterCreate a systemd service for node_exporter

  1. Create a file named /etc/systemd/system/node_exporter.service:

    [Unit]
    Description=Prometheus Node Exporter
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    User=node_exporter
    Group=node_exporter
    Type=simple
    ExecStart=/usr/local/bin/node_exporter --web.listen-address=<server_private_IP_address>:9100
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target
    

    Where <server_private_IP_address> is the address of app-server in the private network. If OTel Collector is running on the same server, specify 127.0.0.1.

  2. Run node_exporter:

    sudo systemctl daemon-reload
    sudo systemctl enable --now node_exporter
    
  3. Make sure the service status has changed to active (running):

    sudo systemctl status node_exporter
    

    Result:

    ● node_exporter.service - Prometheus Node Exporter
         Loaded: loaded (/etc/systemd/system/node_exporter.service; enabled; preset: enabled)
         Active: active (running) <...>
    
  4. Make sure that metrics are available:

    curl http://<server_private_IP_address>:9100/metrics
    

Instead of node_exporter, you can use Unified Agent on the application server. It collects system metrics and sends them directly to Monium, bypassing OTel Collector. In this case, metric configuration will be stored in different places. For more information, see Delivering Linux system metrics.

Install and configure blackbox_exporterInstall and configure blackbox_exporter

Blackbox_exporter on the monitoring server runs synthetic HTTP, TCP, and ICMP checks and generates the probe_success and probe_duration_seconds metrics. You will need these for the availability and response time SLOs.

Install and configure blackbox_exporterInstall and configure blackbox_exporter

  1. Create a system user:

    sudo useradd --no-create-home --shell /usr/sbin/nologin blackbox_exporter
    
  2. Download and unpack the archive:

    wget -O /tmp/blackbox_exporter.tar.gz https://github.com/prometheus/blackbox_exporter/releases/download/v0.28.0/blackbox_exporter-0.28.0.linux-amd64.tar.gz
    tar -zxvf /tmp/blackbox_exporter.tar.gz
    sudo install -m 0755 ./blackbox_exporter-0.28.0.linux-amd64/blackbox_exporter /usr/local/bin/blackbox_exporter
    sudo chown blackbox_exporter:blackbox_exporter /usr/local/bin/blackbox_exporter
    
  3. Create a folder for the configuration:

    sudo mkdir -p /etc/blackbox_exporter
    

Set up blackbox_exporter configurationSet up blackbox_exporter configuration

  1. Create a file named /etc/blackbox_exporter/blackbox.yml:

    modules:
      http_2xx:
        prober: http
        timeout: 5s
        http:
          method: GET
          valid_http_versions: ["HTTP/1.1", "HTTP/2.0"]
          preferred_ip_protocol: "ip4"
          follow_redirects: true
          fail_if_ssl: false
          fail_if_not_ssl: false
          tls_config:
            insecure_skip_verify: false
    
      http_2xx_tls:
        prober: http
        timeout: 5s
        http:
          method: GET
          follow_redirects: true
          fail_if_not_ssl: true
          preferred_ip_protocol: "ip4"
    
      tcp_connect:
        prober: tcp
        timeout: 3s
    
      icmp:
        prober: icmp
        timeout: 2s
    

    Where:

    • http_2xx: Basic check of HTTP endpoint availability. The check type is HTTP.
    • http_2xx_tls: Check that requires TLS, with the fail_if_not_ssl: true parameter. The check type is HTTP.
    • tcp_connect: TCP port availability check. The check type is TCP.
    • icmp: Check that the host is reachable in the network. The check type is ICMP.
  2. Set up permissions for the configuration file:

    sudo chown -R blackbox_exporter:blackbox_exporter /etc/blackbox_exporter
    

Create a systemd service for blackbox_exporterCreate a systemd service for blackbox_exporter

  1. Create a file named /etc/systemd/system/blackbox_exporter.service:

    [Unit]
    Description=Prometheus Blackbox Exporter
    Wants=network-online.target
    After=network-online.target
    
    [Service]
    User=blackbox_exporter
    Group=blackbox_exporter
    Type=simple
    ExecStart=/usr/local/bin/blackbox_exporter \
      --config.file=/etc/blackbox_exporter/blackbox.yml \
      --web.listen-address=127.0.0.1:9115
    Restart=on-failure
    
    [Install]
    WantedBy=multi-user.target
    
  2. Run blackbox_exporter and add it to the auto start:

    sudo systemctl daemon-reload
    sudo systemctl enable --now blackbox_exporter
    
  3. Make sure the service status has changed to active (running):

    sudo systemctl status blackbox_exporter
    

    Result:

    ● blackbox_exporter.service - Prometheus Blackbox Exporter
         Loaded: loaded (/etc/systemd/system/blackbox_exporter.service; enabled; preset: enabled)
         Active: active (running) <...>
    

Test blackbox_exporterTest blackbox_exporter

Run tests:

curl "http://127.0.0.1:9115/probe?target=http://10.128.0.10&module=http_2xx" | grep probe_success
curl "http://127.0.0.1:9115/probe?target=10.128.0.10:80&module=tcp_connect" | grep probe_success
curl "http://127.0.0.1:9115/probe?target=10.128.0.10&module=icmp" | grep probe_success

Where 10.128.0.10 is your web service’s IP address.

Result:

probe_success 1

Set up metric delivery to MoniumSet up metric delivery to Monium

OTel Collector sends metrics collected by blackbox_exporter and node_exporter to Monium.

Install OTel CollectorInstall OTel Collector

  1. Install OTel Collector:

    sudo apt-get update
    sudo apt-get -y install wget
    wget https://github.com/open-telemetry/opentelemetry-collector-releases/releases/download/v0.156.0/otelcol_0.156.0_linux_amd64.deb
    sudo dpkg -i otelcol_0.156.0_linux_amd64.deb
    
  2. Edit the /etc/otelcol/otelcol.conf file:

    MONIUM_PROJECT=folder__<folder_ID>
    MONIUM_API_KEY=<API_key>
    OTELCOL_OPTIONS=--config=/etc/otelcol/config.yaml
    

    Where:

    • MONIUM_PROJECT: Monium project name, e.g., folder__b1gg5f45su0k6rjr39s1.
    • MONIUM_API_KEY: API key of the service account with the monium.telemetry.writer role.

Set up OTel Collector configurationSet up OTel Collector configuration

Create a file named /etc/otelcol/config.yaml:

receivers:
  prometheus:
    config:
      scrape_configs:
        - job_name: blackbox_http_10_128_0_10
          scrape_interval: 30s
          metrics_path: /probe
          params:
            module: [http_2xx]
          static_configs:
            - targets:
                - http://10.128.0.10
              labels:
                probe_type: http
                target_name: lemp_http
          relabel_configs:
            - source_labels: [__address__]
              target_label: __param_target
            - source_labels: [__param_target]
              target_label: instance
            - target_label: __address__
              replacement: 127.0.0.1:9115

        - job_name: blackbox_tcp_10_128_0_10_80
          scrape_interval: 30s
          metrics_path: /probe
          params:
            module: [tcp_connect]
          static_configs:
            - targets:
                - 10.128.0.10:80
              labels:
                probe_type: tcp
                target_name: lemp_tcp_80
          relabel_configs:
            - source_labels: [__address__]
              target_label: __param_target
            - source_labels: [__param_target]
              target_label: instance
            - target_label: __address__
              replacement: 127.0.0.1:9115

        - job_name: blackbox_icmp_10_128_0_10
          scrape_interval: 30s
          metrics_path: /probe
          params:
            module: [icmp]
          static_configs:
            - targets:
                - 10.128.0.10
              labels:
                probe_type: icmp
                target_name: lemp_icmp
          relabel_configs:
            - source_labels: [__address__]
              target_label: __param_target
            - source_labels: [__param_target]
              target_label: instance
            - target_label: __address__
              replacement: 127.0.0.1:9115

        - job_name: node_exporter_lemp
          scrape_interval: 30s
          static_configs:
            - targets:
                - 10.128.0.10:9100
              labels:
                exporter: node_exporter
                target_name: lemp_node
                host: lemp

processors:
  batch:

exporters:
  otlp/monium:
    endpoint: ingest.monium.yandex.cloud:443
    compression: zstd
    headers:
      Authorization: "Api-Key ${env:MONIUM_API_KEY}"
      x-monium-project: "${env:MONIUM_PROJECT}"
      x-monium-cluster: production
      x-monium-service: lemp

service:
  telemetry:
    logs:
      level: info

  pipelines:
    metrics:
      receivers: [prometheus]
      processors: [batch]
      exporters: [otlp/monium]

Where:

  • 10.128.0.10: IP address of your application server. Replace it with the actual one.
  • relabel_configs: Label overwriting rules making Prometheus poll blackbox_exporter, not the target directly.
  • x-monium-cluster and x-monium-service stand for the cluster and service labels that Monium assigns to all metrics from this OTel Collector. You will use them to find metrics and build SLOs. The example uses the production and lemp values (a webstack of Linux, Nginx, MySQL, and PHP). Replace these with the actual ones.

Note

The probe_type label can have the http, tcp, and icmp values. You can use it to conveniently filter metrics in Monium and build separate SLOs for each check type.

Run OTel CollectorRun OTel Collector

  1. Edit the /etc/systemd/system/multi-user.target.wants/otelcol.service file:

    [Unit]
    Description=OpenTelemetry Collector
    After=network.target
    
    [Service]
    EnvironmentFile=/etc/otelcol/otelcol.conf
    ExecStart=/usr/bin/otelcol $OTELCOL_OPTIONS
    ExecReload=/bin/kill -HUP $MAINPID
    KillMode=mixed
    Restart=on-failure
    Type=simple
    User=otel
    Group=otel
    
    [Install]
    WantedBy=multi-user.target
    
  2. Run OTel Collector:

    sudo systemctl daemon-reload
    sudo systemctl enable --now otelcol
    sudo systemctl status otelcol
    
  3. Make sure Monium receives the metrics:

    • Open Monium.
    • In the left-hand menu, expand Overview and select Metrics.
    • In the search field, specify service = "lemp", which is a value of the service label from the x-monium-service header in the OTel Collector configuration. If you have set another value, specify it.

    The list will show the probe_success and probe_duration_seconds metrics as well as node_exporter metrics.

Note

Keep in mind that data in Monium appears with a lag rather than immediately, since the Otel Collector starts sending data after 60 seconds.

Configure an SLO for service availabilityConfigure an SLO for service availability

Blackbox_exporter returns a binary check result: probe_success equals 1 if the check succeeded or 0 if it failed. Based on this metric, create an availability SLO:

Monium UI
  1. On the Monium home page, expand the Alerts and SLOs section on the left.

  2. Select SLO.

  3. Click Create.

  4. Specify the SLO parameters:

    • Name, e.g., SLO HTTP Availability 30d.
    • Evaluation window: 30d.
    • Evaluation delay: 2m. The value must be greater than the metric collection interval: 2m is sufficient for scrape_interval: 30s.
    • SLO: 99.9%.
    • Calculation Method: Good Events / Total Events.
  5. Under Good Events, specify this query:

    series_sum(
        {project = "folder__<folder_ID>", cluster = "production", service = "lemp", probe_type = "http", name = "probe_success"}
      )
    
  6. Under Total Events, specify this query:

    series_count(
        {project = "folder__<folder_ID>", cluster = "production", service = "lemp", probe_type = "http", name = "probe_success"}
    )
    
  7. Click Create.

The error budget is calculated automatically once the SLO is created.

Configure error budget alertsConfigure error budget alerts

Create two Error Budget SLO alerts to monitor the error budget:

  • Error budget remaining: Warns when the error budget is at risk of being exhausted.
  • Error budget consumption: Shows the actual error budget consumption over a specified time window.

Error Budget Exhaustion Rate checks the difference between the remaining budget at the beginning and end of the evaluation window. Burn rate alerts check the ratio of the bad events share to the allowed error share. For more on alert types and general settings, see this guide on creating an SLO alert.

Create an alert for error budget remainingCreate an alert for error budget remaining

Monium UI
  1. On the Monium home page, expand the Alerts and SLOs section on the left.

  2. Select Alerts.

  3. Click Create alert → SLO.

  4. Specify a name for the alert and select its severity level, e.g., Critical.

  5. In the Type field, select Error Budget.

  6. Select the SLO you created in the previous step.

  7. In the Calculation method field, select Error Budget Remaining.

  8. Set the trigger conditions:

    • Warning: 50%.
    • Alarm: 20%.
  9. Click Create.

Create an alert for error budget consumptionCreate an alert for error budget consumption

Monium UI
  1. On the Monium home page, expand the Alerts and SLOs section on the left.

  2. Select Alerts.

  3. Click Create alert → SLO.

  4. Specify a name for the alert and select its severity level, e.g., Disaster.

  5. In the Type field, select Error Budget.

  6. Select the SLO you created in the previous step.

  7. In the Calculation method field, select Error Budget Exhaustion Rate.

  8. Set the trigger conditions:

    • Warning: 1%.
    • Alarm: 2%.
    • Evaluation window: 1h.
  9. Click Create.

Configure an SLO for service response timeConfigure an SLO for service response time

Blackbox_exporter uses the probe_duration_seconds metric to measure the time spent on synthetic checks. Create an SLO based on it: 99% of HTTP checks must be completed within 300 ms or less.

Note

If you need an SLO in the "p95 < 300 ms" format, use the application, ingress controller and load balancer metrics. Blackbox_exporter checks the service from the outside and shows degradations in the DNS, TLS, network, and load balancers.

The probe_duration_seconds metric shows the total check time, but does not specify what causes the delay. Blackbox_exporter breaks the check down into the following phases:

Metric What is measured
probe_dns_lookup_time_seconds DNS resolution time
probe_tcp_connect_duration_seconds Time to establish TCP connection
probe_tls_handshake_duration_seconds TLS handshake time
probe_http_duration_seconds Time of HTTP request phases
probe_duration_seconds Total check time

For visualization, add the phase metrics on your dashboard as a stacked chart. Build the SLOs and alerts based on the total probe_duration_seconds metric. In case of delay degradation, charts will show on which stage the problem has occured: DNS, TLS, or network route.

Create a response time SLOCreate a response time SLO

Monium UI
  1. On the Monium home page, expand the Alerts and SLOs section on the left.

  2. Select SLO.

  3. Click Create.

  4. Specify the SLO parameters:

    • Name, e.g., SLO LATENCY HTTP 30d.
    • Evaluation window: 30d.
    • SLO: 99%.
    • Calculation Method: Good Events / Total Events.
  5. Under Good Events, specify this query:

    series_sum(
      heaviside(
        ({project="folder__<folder_ID>", cluster="production", service="lemp", probe_type="http", name="probe_duration_seconds"} * -1) + 0.3
      )
    )
    

    Where:

    • 0.3: Delay threshold in seconds, 300 ms.
    • heaviside(): Function that selects threshold checks. Returns 1 if the delay is less than 300 ms or 0 if greater.

    Checks with exactly the 300 ms delay are counted as a half event, which does not actually affect the SLI calculation.

  6. Under Total Events, specify this query:

    series_sum(
      {project="folder__<folder_ID>", cluster="production", service="lemp", probe_type="http", name="probe_duration_seconds"} * 0 + 1
    )
    
  7. Click Create.

Use system metrics for diagnostics after an alertUse system metrics for diagnostics after an alert

Once an SLO alert is triggered, you need to understand what failure has happened on the server. Node_exporter system metrics can help you quickly find the cause:

Category Metrics What to check
CPU node_cpu_seconds_total in the context of iowait, idle, and steal modes steal is especially important for virtualization
Memory node_memory_MemAvailable_bytes Memory available for use
Disk node_disk_read_time_seconds_total, node_disk_write_time_seconds_total, node_disk_io_time_weighted_seconds_total Read/write delays, I/O queue
File systems node_filesystem_free_bytes, node_filesystem_files_free Free space and inodes
Network node_network_receive_bytes_total, node_network_transmit_bytes_total, node_network_receive_drop_total Traffic and drops by interfaces

If an SLO alert has been triggered by probe_duration_seconds, first of all, check network phases: probe_dns_lookup_time_seconds and probe_tls_handshake_duration_seconds. Then go to server metrics: node_disk_io_time_weighted_seconds_total and node_cpu_seconds_total{mode="iowait"}.

Delete the resources you createdDelete the resources you created

To stop paying for the resources you created:

  1. Delete the alerts you created in Monium.
  2. Delete the Compute Cloud VMs.
  3. Delete the Identity and Access Management service account.
  4. Delete the Virtual Private Cloud security group.

If you reserved public static IP addresses, delete them.

Was the article helpful?

Previous
Running the DeepSeek-R1 language model in a Compute Cloud GPU cluster
Next
Configuring time synchronization using NTP
© 2026 Direct Cursus Technology L.L.C.