Monitoring the health status of a Yandex StoreDoc cluster and its hosts
Data on the cluster and host state is available in the management console
Diagnostic information about cluster states is presented as graphs.
Chart update rate:
- Standard hosts and hosts with an increased RAM to vCPU ratio (
memory-optimized): 15 seconds. - Hosts with a guaranteed vCPU share under 100% (
burstable): 150 seconds.
Note
The most appropriate multiple units (MB, GB, and more) are automatically used in charts.
You can configure alerts in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: Warning and Alarm. If the specified threshold is exceeded, you will receive alerts via the configured notification channels.
Monitoring cluster health status
To view the Yandex StoreDoc cluster health status details:
- Open the folder dashboard
. - Navigate to the Yandex StoreDoc service.
- Click the name of your cluster and select the Monitoring tab.
-
To get started with Yandex Monitoring metrics, dashboards, or alerts, click Open in Monitoring in the top panel.
You will see the following charts:
- Asserts total: Number of asserts triggered in the cluster.
- Average operation time per host: Average time it takes each host to execute operations, in microseconds.
- Average operations time on primary: Average execution time for operations on primary replicas, in microseconds.
- Average operations time on secondaries: Average execution time for operations on secondary replicas, in microseconds.
- CPU usage per host: vCPU utilization rate on each host, as a fraction of the total vCPU cores.
- CPU usage per host, top 5 hosts: 5 hosts with the highest vCPU utilization, in percent.
- Configured oplog size per host: Size of the operation log on each cluster host, in GB.
- Connections per host: Average number of connections to each host.
- Data size on primary, top 5 databases: Size of the five largest databases on the primary replica, in bytes. The chart is based on the raw, uncompressed data.
- Disk read per host, top 5 hosts: 5 hosts with the highest disk read throughput, in bytes per second.
- Disk space usage per host, top 5 hosts: 5 hosts with the highest storage space usage (displayed in two charts: in bytes and in percent). The chart is based on the compressed data.
- Disk usage per host, top 5 hosts: 5 hosts with the highest storage I/O throughput, in bytes per second.
- Disk write per host, top 5 hosts: 5 hosts with the highest disk write throughput, in KB/s.
- Documents affected on primary: Average number of documents affected by queries on the primary replica.
- Documents affected on secondaries: Average number of documents affected by queries on all secondary replicas.
- Documents affected per host: Average number of documents affected by queries on each host.
- Hosts available for read: Number of hosts accepting read queries.
- Hosts available for write: Number of hosts accepting write queries.
- Index size on primary, top 5 indexes: Size of the five largest indexes on the primary replica, in bytes.
- Memory usage per host: Amount of RAM used by each host, in bytes.
- Memory usage per host, top 5 hosts: 5 hosts with the highest RAM usage, in percent.
- Network data received per host, top 5 hosts: 5 hosts with the highest inbound network throughput, in KB/s.
- Network data sent per host, top 5 hosts: 5 hosts with the highest outbound network thoughput, in KB/s.
- Network usage per host, top 5 hosts: 5 hosts with the highest total network throughput, in KB/s.
- Open cursors total: Number of open cursors in the cluster.
- Oplog window: Time interval defining how long replication data is stored in each host's oplog collection.
- Page faults per host: Number of page faults
on each host. - Queries on secondaries: Average number of queries broken down by type, processed on the secondary replicas.
- Queries on primary: Average number of queries broken down by type, processed on the primary replica.
- Read operations time, top 5 collections: Five collections with the longest time spent on read operations.
- Readers/writers active queue per host, top 5: Total size of the 5 largest queues for each host:
- With read queries
- With write queries
- Replicated queries: Average number of replicated queries in the cluster.
- Replication lag per host and write_concern wait: Replication delay on each host and write concern timeout, in seconds.
- Scan and order per host: Number of data sorts without using an index on each host.
- Scanned / returned: Shows the following ratios:
scanned_docs / returned_docs: Scanned documents to returned documents.scanned_keys / returned_docs: Scanned index keys to returned documents.
- TTL indexes activity: Total number of TTL indexes.
- Total operations count on cluster: Total number of operations performed in the cluster.
- Total operations time on cluster: Total execution time of cluster operations, in ms.
- WiredTiger cache pages evicted on primary: Average number of memory pages evicted from the cache on the primary replica.
- WiredTiger cache state on primary: WiredTiger cache usage on the primary replica, in bytes.
- WiredTiger checkpoint time on primary: Time required to create WiredTiger checkpoints on the primary replica, in ms.
- WiredTiger concurrent transactions on primary: Average number of concurrent transactions on the primary replica.
- WiredTiger transactions state on primary: Average number of transactions at each level on the primary replica.
- Write conflicts per host: Number of write conflicts on each host.
- Write operations time, top 5 collections: Five collections with the longest total time spent on write operations.
Host status monitoring
To view the detailed status of individual Yandex StoreDoc hosts:
- Open the folder dashboard
. - Navigate to the Yandex StoreDoc service.
- Click the name of your cluster and select Hosts → Monitoring.
- Select the required host from the drop-down list. Next to the host name, you will see its role, i.e.,
PRIMARYorSECONDARY, and type, i.e.,MONGOCFG,MONGOD,MONGOINFRA, orMONGOS.
This page displays workload charts for an individual cluster host:
- CPU: Processor core workload. With increased workload, the Idle value drops.
- Memory: RAM usage, in bytes. At high loads, the value of the Free space metric decreases, while the others increase.
- Disk bytes: Speed of disk operations, in bytes per second.
- Disk IOPS: Number of disk operations per second.
- Network bytes: Network data transfer rate, in bytes per second.
- Network packets: Network packet exchange rate, in packets per second.
Setting up alerts in Yandex Monitoring
-
In the management console
, select the folder containing the cluster where you want to set up alerts. -
Navigate to the
Monitoring service. -
Under Service dashboards, select:
- Yandex StoreDoc to set up cluster alerts.
- Yandex StoreDoc — Host Overview to set up host alerts.
-
On the relevant chart, click
and select Create alert. -
If the chart displays multiple metrics, select the data query for the relevant metric and click Continue. To learn more about the query language, see this Yandex Monitoring article.
-
Set the
AlarmandWarningalert thresholds. -
Click Create alert.
To have other cluster health indicators monitored automatically:
- Create an alert.
- Add a status metric.
- In the alert parameters, set the alert thresholds.
Recommended threshold values for selected metrics:
| Metric | Internal metric name | Alarm |
Warning |
|---|---|---|---|
| Database write availability | can_write |
0 |
— |
| Replication lag | replset_status-replicationLag |
180 |
30 |
| Storage space used | disk.used_bytes |
90% of the storage size | 70% of the storage size |
The Alarm and Warning thresholds for the disk.used_bytes metric are specified exclusively in bytes. For example, recommended values for a 100 GB disk are as follows:
Alarm:96,636,764,160bytes (90%)Warning:75,161,927,680bytes (70%)
You can check the current storage size in the cluster details. For a complete list of supported metrics, see this Monitoring guide.
Read-only
mode transition monitoring
To track storage fill levels on the cluster hosts and receive notifications when free space is about to run out:
-
Add the
disk.free_bytesmetric.To do this, create a query in the query builder:
service=managed-mongodb→name=disk.free_bytes→host=*→resource_id=*→resource_type=cluster. -
Configure alert notification thresholds:
-
Condition: Set the
Less than or equalscondition for free disk space that will trigger the alert.Recommended thresholds relative to storage size are as follows:
Storage size, GB AlarmWarning⩽ 600 1G: 1 GB1500M: 1.5 GB> 600 6G: 6 GB10G: 10 GB -
Advanced settings → Aggregation function: Select
Minimumthe metric’s minimum value over the period.
-
Cluster health and status
The State of a cluster shows the health of its hosts, while the Status shows whether the cluster is started, stopped, or is at an intermediate stage.
To check the cluster’s health and status:
- Open the folder dashboard
. - Navigate to the Yandex StoreDoc service.
- Locate the cluster you need in the list and hover over the indicator in the Availability column.
Cluster metrics
| State | Description | Suggested actions |
|---|---|---|
| ALIVE | Cluster is operating normally. | No action is required. |
| DEGRADED | Cluster is not running at its full capacity: the state of at least one of the hosts is other than ALIVE. |
Run the diagnostics:
|
| DEAD | The cluster is down: none of its hosts are running. | Make a support request
|
| UNKNOWN | Cluster state is unknown. | Make a support request
|
Cluster statuses
| Status | Description | Suggested actions |
|---|---|---|
| CREATING | Preparing for the first start | Wait a while and get started. The time it takes to create a cluster depends on the host class. |
| RUNNING | The cluster is operating normally | No action is required. |
| STOPPING | The cluster is stopping | After a while, the cluster status will switch to STOPPED and the cluster will be disabled. No action is required. |
| STOPPED | The cluster is stopped | Start the cluster to get it running again. |
| STARTING | Starting the cluster that was stopped earlier | After a while, the cluster status will switch to RUNNING. Wait a while and get started. |
| UPDATING | Updating the cluster's configuration | Once the update is complete, the cluster will get the status it had prior to the update: RUNNING or STOPPED. |
| ERROR | Error when performing an operation with the cluster or during a maintenance window | If the cluster remains in this status for a long time, contact support |
| STATUS_UNKNOWN | The cluster is unable to determine its status | If the cluster remains in this status for a long time, contact support |