Monitoring the cluster state Managed Service for Apache Airflow™
Data on the cluster and host state is available in the management console. You can view them on the Monitoring tab of the cluster management page or in Yandex Monitoring.
Diagnostic information about cluster states is presented as graphs.
Charts are updated once a minute.
Note
The most appropriate multiple units (MB, GB, and more) are automatically used in charts.
You can configure alerts in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: Warning and Alarm. If the specified threshold is exceeded, you will receive alerts via the configured notification channels.
Monitoring the cluster state
To view detailed information on the state of a Managed Service for Apache Airflow™ cluster:
-
In the management console, navigate to the relevant folder.
-
In the list of services, select Managed Service for Apache Airflow™.
-
Click the cluster name and select the Monitoring tab.
The page that opens will display the performance charts for the Managed Service for Apache Airflow™ cluster.
-
To get started with Yandex Monitoring metrics, dashboards, or alerts, click Open in Monitoring in the top panel.
Available charts
-
Under Cluster:
- CPU usage: CPU usage by cluster components as a percentage of their limits.
- Memory usage: RAM usage by cluster components as a percentage of their limits.
- Components Restarts: Number of cluster component restarts.
- Components Count: Number of active components in the cluster.
-
Under DAGs:
- DAG import stats: DAG import metrics.
- DAG parse timings: Maximum time for processing DAGs.
-
Under Tasks:
- Tasks completed, 5m window: Number of tasks completed within a five-minute window.
- Tasks duration, p99: 99th percentile of task execution time.
- Tasks duration sum, 5m window: Total time it takes to complete tasks with a five-minute window.
-
Under Resources:
- Pool usage: Number of occupied slots per cluster pool.
Alert settings in Yandex Monitoring
To configure cluster state indicator alerts:
- In the management console, select the folder with the cluster for which you want to configure alerts.
- In the list of services, select Monitoring.
- Under Service dashboards, select Managed Service for Apache Airflow™ — Cluster Overview.
- In the chart you need, click and select Create alert.
- If the chart shows multiple metrics, select a data query to generate a metric and click Continue. You can learn more about the query language in the Yandex Monitoring documentation.
- Set the
AlarmandWarningthreshold values to trigger the alert. - Click Create alert.
To have other cluster health indicators monitored automatically:
- Create an alert.
- Add a status metric.
- In the alert parameters, set the alert thresholds.
For a complete list of supported metrics, see this Monitoring guide.
Cluster state and status
The State of a cluster shows the health of its hosts, while the Status shows whether the cluster is started, stopped, or is at an intermediate stage.
To view a state and status of a cluster:
- Go to the folder page and select Managed Service for Apache Airflow™.
- In the cluster row, hover over the indicator in the Availability column.
Cluster states
| State | Description | Suggested actions |
|---|---|---|
| ALIVE | The cluster is operating normally. | No action is required. |
| DEAD | The cluster is out of order. | Contact support and specify the following:
|
| UNKNOWN | The cluster’s state is unknown. | Contact support and specify the following:
|
Cluster statuses
| Status | Description | Suggested actions |
|---|---|---|
| CREATING | Preparing for the first start | Wait a while and get started. The time it takes to create a cluster depends on the host class. |
| RUNNING | The cluster is operating normally | No action is required. |
| STOPPING | The cluster is stopping | After a while, the cluster status will switch to STOPPED and the cluster will be disabled. No action is required. |
| STOPPED | The cluster is stopped | Start the cluster to get it running again. |
| STARTING | Starting the cluster that was stopped earlier | After a while, the cluster status will switch to RUNNING. Wait a while and get started. |
| UPDATING | Updating the cluster's configuration | Once the update is complete, the cluster will get the status it had prior to the update: RUNNING or STOPPED. |
| ERROR | Error when performing an operation with the cluster or during a maintenance window | If the cluster remains in this status for a long time, contact support. You can see whether a cluster is available by its status. |
| STATUS_UNKNOWN | The cluster is unable to determine its status | If the cluster remains in this status for a long time, contact support. |