Setting up alerts

Alerting allows you to monitor changes in metrics and notifies you when they hit critical levels using periodic queries known as alerts. When metrics reach threshold values, the system sends a notification via the specified communication channel, e.g., by email or messenger.

To configure alerting:

1. Create a notification channel.
2. Select metrics and labels for monitoring.
3. Create an alert.

Let's consider an example of creating an alert that notifies you when a service becomes unavailable.

The alert will trigger when the number of failed requests reaches 50% or more of the total requests. Such an alert helps detect DDoS attacks or infrastructure failures.

Let's use letters to indicate the number of incoming requests per second:

• A: Total requests.
• B: Failed requests.

Let's set up our alerts:

• B reaches 30% of A: Warning.
• B reaches 50% of A: Alarm (critical level).

Creating a notification channelCreating a notification channel

1. In the management console, select the folder on the left.

2. Select Monitoring.

3. Go to the Notification channels section.

4. Click Create channel.

5. Specify the channel name, notification method, and recipients.

   Note

   To get notifications, the user must:

   • Have the monitoring.viewer role for the folder with the configured alert.
   • In the management console settings:
     • Enable Monitoring.
     • Specify an email address, phone number, and Telegram account or group.

6. Click Create.

Selecting metrics for monitoringSelecting metrics for monitoring

• We will get the A value using the load_balancer.requests_count_per_second metric.
• We will get the B value using the load_balancer.requests_count_per_second metric filtered by the code = 503 label.
• We will calculate the B to A ratio (in percent) using the 100 * B / A formula and save it as C.

Creating an alertCreating an alert

1. On the Monitoring home page, click Create alert.

2. Name your alert, e.g., unavailable_service.

3. Under Alerts config, describe your query to get the A value:

   1. Next to , add the following parameters:
      • service = Application Load Balancer
      • name = load_balancer.requests_count_per_second
      • load_balancer = <load_balancer_name>
   2. Next to , set the replace_nan(0) function to replace missing data with 0, ensuring a continuous chart.

4. Click Add query.

5. Describe your query to get the B value:

   1. Specify the data to collect:
      • service = Application Load Balancer
      • name = load_balancer.requests_count_per_second
      • code = 503
      • load_balancer = <load_balancer_name>
   2. Set the replace_nan(0) function.

6. Click Add query.

7. Describe you query for C to get the B to A ratio in percent:

   1. Click to switch to text mode to edit the query.
   2. Enter 100 * B / A in the query string.

8. Under Alert condition, specify:

   • Query to evaluate: C
   • Aggregation function: All values
   • Warning: 30
   • Alarm: 50 (critical level)
   • Evaluation window: 30 seconds
   • Evaluation delay: 15 seconds

9. Leave the default values under No data policy.

10. Optionally, under Annotations, add the information to log when the alert triggers.

11. Under Notifications, add the notification channel.

12. Click Create.