Contact UsGet started

Alerting rules

Written by
Updated at December 4, 2024

Note

Alerting rules are at the Technical Preview stage. To access the feature, contact support.

With Yandex Managed Service for Prometheus® alerting, you can add alert calculation rules and send notifications when these are triggered. To configure alerting in Prometheus, you need to create alerting rules and set up the alert manager to process and deliver notifications.

Requirements for alerting rules

In Prometheus, you can use your existing PromQL-based alerting rule files.

For rules, it supports all the fields described in the YAML file specification. It also supports annotation templates using the $value and $labels variables. Iterations and functions are not supported.

This section describes some aspects related to alerting rules and the alert manager configuration. For how to upload and work with files, see Recording rules.

Alert manager

Alert manager is a Prometheus component responsible for sending notifications. It is configured separately from the rule files. You can use your existing configuration files.

Rule processing highlights:

  • Notification channels from the configuration file are mapped against the Yandex Monitoring notification channels specified in the workspace folder.
  • Notifications to the email and Telegram channels is supported. Other channels will be ignored without error notifications. There are plans to add support for all channels available in Yandex Monitoring later on.
  • The channel is selected according to the specified type; dynamic routing is currently not supported but is planned for upcoming releases.
  • If the configuration has no channels matching the folder channels, the file will not be accepted.

You can use alerting rules without loading a configuration file. In this case, the alerting rules will be calculated and will create the ALERTS and ALERTS_FOR_STATE metrics, but no alert notifications will be sent.

You can manage recording rule files via the management console or API.

Pre-configuration for using the API

The API is represented by REST resources located at https://monitoring.api.cloud.yandex.net/prometheus/workspaces/<workspace_ID>/extensions/v1/rules.

To start executing requests:

  1. Install cURL.
  2. Authenticate in the API.
  3. Create a workspace and copy its ID to use it in the request address.

Adding or replacing an alerting rule file

  1. On the Monitoring page, select Prometheus on the left.
  2. Select or create a workspace.
  3. Go to the Recording rules tab.
  4. If you have not uploaded any files yet, click Add file and select a .yml file with rules.
  5. To add another file, click Add file.
  6. To replace the existing file, click > Replace file to its right.

  1. Encode the file contents as Base64 RFC 4648:

    cat alerting-rule.yaml

# groups:
#   - name: example
#     rules:
#     - alert: HighRequestLatency
#       expr: job:request_latency_seconds:mean5m{job="myjob"} > 0.5
#       for: 10m
#       labels:
#         severity: page
#       annotations:
#         summary: High request latency

base64 -i alerting-rule.yaml

# Z3JvdXBzOgotIG5hbWU6IGV4YW1wbGUKICBydWxlczoKICAtIGFsZXJ0OiBIaW...CBsYXRlbmN5Cg==

  2. Save the result as a JSON file:

    body.json

    {
    "name": "alerting-rules",
    "content" : "Z3JvdXBzOgotIG5hbWU6IGV4YW1wbGUKICBydWxlczoKICAtIGFsZXJ0OiBIaW...CBsYXRlbmN5Cg=="
}

  3. Create or replace an alerting rule file:

    export IAM_TOKEN=<IAM_token>

curl -X PUT \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${IAM_TOKEN}" \
    -d "@body.json"  \
    "https://monitoring.api.cloud.yandex.net/prometheus/workspaces/<workspace_ID>/extensions/v1/rules"

Adding or replacing an alert manager configuration file

  1. On the Monitoring page, select Prometheus on the left.
  2. Select or create a workspace.
  3. Go to the Alert manager configuration tab.
  4. If you have no uploaded configuration files yet, click Upload a configuration file and select a .yml file.
  5. To download the file, click Download.
  6. To replace the file, click Replace file.

  1. Encode the file contents as Base64 RFC 4648:

    cat alert-manager.yaml

# receivers:
#   - name: 'email'
#     email_configs:
#       - to: 'alerts@monitoring.org'
#   - name: 'telegram'
#     telegram_configs:
#     - api_url: https://api.telegram.org

base64 -i alert-manager.yaml

# cmVjZWl2ZXJzOgogIC0gbmFtZTogJ2VtYWlsJwogICA...sOiBodHRwczovL2FwaS50ZWxlZ3JhbS5vcmcKCg==

  2. Save the result as a JSON file:

    body.json

    {
    "content" : "cmVjZWl2ZXJzOgogIC0gbmFtZTogJ2VtYWlsJwogICA...sOiBodHRwczovL2FwaS50ZWxlZ3JhbS5vcmcKCg=="
}

  3. Create or replace a configuration file:

    export IAM_TOKEN=<IAM_token>

curl -X PUT \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer ${IAM_TOKEN}" \
    -d "@body.json"  \
    "https://monitoring.api.cloud.yandex.net/prometheus/workspaces/<workspace_ID>/extensions/v1/alertmanager"

If the request is successful, you will get the 204 HTTP code, if not, the error text. A file without a single match with current notification channels in the folder will not be accepted.

Previous
Recording rules
Next
Reading and writing Kubernetes cluster metrics