Creating an SLO alert
SLO alerts monitor metrics for the selected SLO, including the remaining error budget, actual error budget consumption, and burn rate. Use these alerts to promptly detect the risk of SLO breach and assess the incident’s impact on the service reliability.
Before creating an SLO alert, create and configure the SLO. For Burn Rate alerts, make sure to enable the burn rate calculation in the source SLO. Otherwise, the alert will enter the No data status.
Note
To view alerts, you need the viewer role or higher for the relevant cloud. Learn more about roles in Access management in Monium.
-
On the Monium
home page, select Alerts and SLOs → Alerts on the left. -
At the top right, click Create alert → SLO.
-
Specify the main alert parameters:
-
Name: Alert name.
-
ID: Unique alert ID with the project's prefix. If not set, it is generated automatically. Once the alert is created, you cannot change its ID.
-
Description: Alert description or comment.
-
Severity: Alert criticality level. Select it based on the event effect:
Unspecified: Draft or a test without a level.Disaster: Service unavailability or data loss risk.Critical: Service degradation or lack of resources.Info: Scheduled operation or a test alert.
-
-
In the Type field, select:
- Error Budget: Monitors the remaining error budget or its actual consumption over the alert window.
- Burn Rate: Monitors the rate of error budget consumption. It is good for on-duty alerting and fast incident detection.
For more information about the differences between checks, see SLO alerts.
-
Under Alert configuration, select the project that contains the SLO, then select the SLO itself. The SLO and the alert can belong to different projects. You do not need to define custom metric queries.
-
Under Alert conditions, set the conditions for the alert to change its status.
Specify at least one threshold: Warning or Alarm.
For the Error Budget type, select one of the following methods in the Calculation method field:
- Error Budget Remaining: Checks the average remaining budget over the alert calculation window. Set the minimum allowed remaining budget as a percentage. For example, use
20%for Warning and10%for Alarm. If you specify both thresholds, make sure the Alarm value is lower than the Warning value. - Error Budget Exhaustion Rate: Checks the difference between the remaining budget at the beginning and end of the evaluation window. Set the Evaluation window and the allowed consumption thresholds as percentages. For example, for a
1hwindow and a5%Alarm threshold, the alert fires if budget consumption exceeds 5% in an hour. If you specify both thresholds, make sure the Alarm value is greater than the Warning value.
For the Burn Rate type, specify the evaluation window and thresholds:
- Select the long evaluation window. The short window is automatically derived as 1/12 of the long window. For example, if the long window is
1h, the short one is5m. - Choose how to specify the thresholds: as a burn rate or as the percentage of the error budget consumed over the alert window.
- Set the selected thresholds in the specified units. If both thresholds are set, make sure the Alarm value is greater than the Warning value. For example, for an SLO with a 30-day window, a burn rate of
14.4over a1hwindow is equivalent to 2% error budget consumption in an hour.
A burn rate alert fires when the burn rate values for both the long and short windows exceed their respective thresholds. Learn more about evaluation windows and threshold configuration methods.
- Error Budget Remaining: Checks the average remaining budget over the alert calculation window. Set the minimum allowed remaining budget as a percentage. For example, use
-
Configure data decimation or leave the default values.
-
Set up no data policies. To avoid interpreting missing data as SLO compliance, set
No datafor both missing metrics and missing data points within the evaluation window. -
Optionally, specify annotations for the alert.
-
Optionally, to sort alerts and search for alerts, add labels in
key=valueformat.Separate
key=valuepairs with commas. To use labels in another alert or the search bar, click Copy all labels. -
Optionally, under Links, add links to related resources.
-
Configure notifications. If you do not have a notification channel, create one.
If default notification methods are set for the selected alert severity, they apply even if the notification table is empty. To configure the settings, click Edit. Check the values of the Notify about statuses and Re-notification settings.
To avoid receiving notifications every time the status changes, enable Noise suppression. Use this setting for low-traffic SLOs where the status may briefly change due to a small number of events. For more on noise suppression, see Noise suppression.
-
Check the alert calculation result.
In the top right, click Check result. Monium will reevaluate the alert based on the current settings and display the resulting status, chart, and variable values. The check makes it possible to evaluate data availability and the correctness of alert settings.
-
Click Create. The alert will appear in the list.