Alerting rules

Alerting rules allow you to create PromQL based alerts and send notifications when they are triggered.

In Yandex Managed Service for Prometheus®, you can use your existing recording rule and alerting rule files.

For rules, it supports all the fields described in the YAML file specification. It also supports annotation templates using the $value and $labels variables. Iterations and functions are not supported.

This section describes some aspects related to alerting rules and the alert manager configuration. For how to upload and work with files, see Recording rules.

Alert managerAlert manager

Alert manager is a Prometheus component responsible for sending notifications. It is configured separately from the rule files. You can use your existing configuration files.

Rule processing highlights:

• Notification channels from the configuration file are mapped against the Yandex Monitoring notification channels specified in the workspace folder.
• Notifications to the email and Telegram channels is supported. Other channels will be ignored without error notifications. There are plans to add support for all channels available in Yandex Monitoring later on.
• The channel is selected according to the specified type; dynamic routing is currently not supported but is planned for upcoming releases.
• If the configuration has no channels matching the folder channels, the file will not be accepted.

You can use alerting rules without loading a configuration file. In this case, the alerting rules will be calculated and will create the ALERTS and ALERTS_FOR_STATE metrics, but no alert notifications will be sent.

Creating or replacing an alerting rule fileCreating or replacing an alerting rule file

1. Encode the file contents as Base64 RFC 4648:

   cat alerting-rule.yaml
   
   # groups:
   #   - name: example
   #     rules:
   #     - record: example
   #       expr: up
   
   base64 -i recording-rule.yaml
   
   # Z3JvdXBzOgotIG5hbWU6IGV4YW1wbGUKICBydWxlczoKICAtIGFsZXJ0OiBIaW...CBsYXRlbmN5Cg==
   

2. Save the result as a JSON file:

   body.json

   {
       "name": "alerting-rules",
       "content" : "Z3JvdXBzOgotIG5hbWU6IGV4YW1wbGUKICBydWxlczoKICAtIGFsZXJ0OiBIaW...CBsYXRlbmN5Cg=="
   }
   

3. Create or replace a recording rule file:

   export IAM_TOKEN=<IAM_token>
   
   curl -X PUT \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer ${IAM_TOKEN}" \
       -d "@body.json"  \
       "https://monitoring.api.cloud.yandex.net/prometheus/workspaces/<workspace_ID>/extensions/v1/rules"
   

For more information about rule calculating, see Recording rules.

Creating or replacing an alert manager configuration fileCreating or replacing an alert manager configuration file

1. Create an RFC 4648 Base64 encoding of the file contents:

   cat alert-manager.yaml
   
   # receivers:
   #   - name: 'email'
   #     email_configs:
   #       - to: 'alerts@monitoring.org'
   #   - name: 'telegram'
   #     telegram_configs:
   #     - api_url: https://api.telegram.org
   
   base64 -i alert-manager.yaml
   
   # cmVjZWl2ZXJzOgogIC0gbmFtZTogJ2VtYWlsJwogICA...sOiBodHRwczovL2FwaS50ZWxlZ3JhbS5vcmcKCg==
   

2. Save the result as a JSON file:

   body.json

   {
       "content" : "cmVjZWl2ZXJzOgogIC0gbmFtZTogJ2VtYWlsJwogICA...sOiBodHRwczovL2FwaS50ZWxlZ3JhbS5vcmcKCg=="
   }
   

3. Create or replace a configuration file:

   export IAM_TOKEN=<IAM_token>
   
   curl -X PUT \
       -H "Content-Type: application/json" \
       -H "Authorization: Bearer ${IAM_TOKEN}" \
       -d "@body.json"  \
       "https://monitoring.api.cloud.yandex.net/prometheus/workspaces/<workspace_ID>/extensions/v1/alertmanager"
   

If the request is successful, you will get the 204 HTTP code, if not, the error text. A file without a single match with current notification channels in the folder will not be accepted.