Monitoring the state of an Yandex StoreDoc cluster and its hosts
Data on the cluster and host state is available in the management console
Diagnostic information about cluster states is presented as graphs.
Chart update rate:
- Standard hosts and hosts with an increased RAM to vCPU ratio (
memory-optimized): 15 seconds. - Hosts with a guaranteed vCPU share under 100% (
burstable): 150 seconds.
Note
The most appropriate multiple units (MB, GB, and more) are automatically used in charts.
You can configure alerts in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: Warning and Alarm. If the specified threshold is exceeded, you will receive alerts via the configured notification channels.
Monitoring the cluster state
To view detailed information on the state of a Yandex StoreDoc cluster:
-
Navigate to the folder dashboard
and select Yandex StoreDoc. -
Click the name of your cluster and open the Monitoring tab.
-
To get started with Yandex Monitoring metrics, dashboards, or alerts, click Open in Monitoring in the top panel.
The following charts will appear on the page:
- Asserts total: Number of asserts
triggered in the cluster. - Average operation time per host: Average time of operation execution by each host (in microseconds).
- Average operations time on primary: Average operation execution time on primary replicas (in microseconds).
- Average operations time on secondaries: Average operation execution time on secondary replicas (in microseconds).
- CPU usage per host: vCPU utilization per host (as a ratio of the number of vCPU cores).
- CPU usage per host, top 5 hosts: 5 hosts with the highest vCPU utilization (%).
- Configured oplog size per host: Size of the oplog on each cluster host (in GB).
- Connections per host: Average number of connections to each host.
- Data size on primary, top 5 databases: Size of the five largest DBs on the primary replica (in bytes). The chart counts data without compression.
- Disk read per host, top 5 hosts: 5 hosts with the highest load on reading from the disk subsystem (bytes per second).
- Disk space usage per host, top 5 hosts: 5 hosts that take up the most disk space (two charts are displayed: in bytes and %). The chart counts data after compression.
- Disk usage per host, top 5 hosts: 5 hosts with the highest load on the storage I/O subsystem (bytes per second).
- Disk write per host, top 5 hosts: 5 hosts with the highest load on writing to the disk subsystem (kilobytes per second).
- Documents affected on primary: Average number of documents affected by queries on the primary replica.
- Documents affected on secondaries: Average number of documents affected by queries on all secondary replicas.
- Documents affected per host: Average number of documents affected by queries on each host.
- Hosts available for read: Number of hosts that accept read data queries.
- Hosts available for write: Number of hosts that accept write data queries.
- Index size on primary, top 5 indexes: Size of the five largest indexes on the primary replica (in bytes).
- Memory usage per host: Amount of RAM used by each host (in bytes).
- Memory usage per host, top 5 hosts: 5 hosts that use the most RAM (%).
- Network data received per host, top 5 hosts: 5 hosts with the highest network load on reading (kilobytes per second).
- Network data sent per host, top 5 hosts: 5 hosts with the highest network load on writing (kilobytes per second).
- Network usage per host, top 5 hosts: 5 hosts with the highest total network load (kilobytes per second).
- Open cursors total: Number of open cursors in the cluster.
- Oplog window: Time interval, for which replication data is stored in each host's oplog collection.
- Page faults per host: Number of page faults
on each host. - Queries on secondaries: Average number of queries of each type on secondary replicas.
- Queries on primary: Average number of each type of query on primary replicas.
- Read operations time, top 5 collections: Five collections with the longest time spent on read operations.
- Readers/writers active queue per host, top 5: Total size of the 5 largest queues for each host:
- With read requests
- With write requests
- Replicated queries: Average number of replicated queries in the cluster.
- Replication lag per host and write_concern wait: Replication lag on each host and waiting time for write confirmation
(in seconds). - Scan and order per host: Number of data sorts without index usage on each host.
- Scanned / returned: Shows the following ratios:
scanned_docs / returned_docs: Documents scanned to documents returned.scanned_keys / returned_docs: Index keys scanned to documents returned.
- TTL indexes activity: Total number of TTL indexes
. - Total operations count on cluster: Total number of operations performed in the cluster.
- Total operations time on cluster: Total operation execution time in the cluster (in milliseconds).
- WiredTiger cache pages evicted on primary: Average number of memory pages evicted on the primary replica.
- WiredTiger cache state on primary: WiredTiger cache usage on the primary replica (in bytes).
- WiredTiger checkpoint time on primary: Time it takes to create WiredTiger checkpoints on the primary replica (in milliseconds).
- WiredTiger concurrent transactions on primary: Average number of parallel transactions on the primary replica.
- WiredTiger transactions state on primary: Average number of transactions on each level on the primary replica.
- Write conflicts per host: Number of write conflicts on each host.
- Write operations time, top 5 collections: Five collections with the longest time spent on write operations.
Monitoring the state of hosts
To view detailed information on the state of individual Yandex StoreDoc hosts:
- Navigate to the folder dashboard
and select Yandex StoreDoc. - Click the cluster name and select the Hosts → Monitoring tab.
- Select the host from the drop-down list. You will see the host role (
PRIMARYorSECONDARY) and type (MONGOCFG,MONGOD,MONGOINFRA, orMONGOS) next to the host name.
This page displays charts showing the load on an individual host in the cluster:
- CPU: Processor core workload. As the load goes up, the Idle value goes down.
- Memory: Use of RAM, in bytes. At high loads, the value of the Free parameter goes down while those of other parameters go up.
- Disk bytes: Speed of disk operations, in bytes per second.
- Disk IOPS: Number of disk operations per second.
- Network bytes: Speed of network data exchange, in bytes per second.
- Network packets: Number of network packets exchange per second.
Alert settings in Yandex Monitoring
-
In the management console
, select the folder with the cluster for which you want to configure alerts. -
In the list of services, select
Monitoring. -
Under Service dashboards, select:
- Yandex StoreDoc to configure cluster alerts.
- Yandex StoreDoc — Host Overview to configure host alerts.
-
In the chart you need, click
and select Create alert. -
If the chart shows multiple metrics, select a data query to generate a metric and click Continue. You can learn more about the query language in the Yandex Monitoring documentation.
-
Set the
AlarmandWarningthresholds to trigger the alert. -
Click Create alert.
To have other cluster health indicators monitored automatically:
- Create an alert.
- Add a status metric.
- In the alert parameters, set up your alert thresholds.
The recommended thresholds are as follows:
| Metric | Parameter | Alarm |
Warning |
|---|---|---|---|
| DB write availability | can_write |
Equals 0 |
— |
| Replication delay | replset_status-replicationLag |
180 |
30 |
| Storage space used | disk.used_bytes |
90% of the storage size | 70% of the storage size |
For the disk.used_bytes metric, the Alarm and Warning thresholds are only set in bytes. For example, the recommended values for a 100 GB disk are as follows:
Alarm:96,636,764,160bytes (90%)Warning:75,161,927,680bytes (70%)
You can view the current storage size in the detailed information about the cluster. For a complete list of supported metrics, see this Monitoring article.
Monitoring the transition to read-only mode
To monitor storage usage on cluster hosts and get notifications when free space is about to run out:
-
Add the
disk.free_bytesstatus metric.To do this, create a query in the query builder:
service=managed-mongodb→name=disk.free_bytes→host=*→resource_id=*→resource_type=cluster. -
Set the alert threshold values in the alert settings:
-
Condition: Set the
Less than or equalscondition for the size of free disk space to trigger an alert.The recommended threshold values depending on the storage size are as follows:
Storage size, GB AlarmWarning⩽ 600 1G: 1 GB1500M: 1.5 GB> 600 6G: 6 GB10G: 10 GB -
Advanced settings → Aggregation function: Select
Minimum(a minimum metric value for the period).
-
Cluster state and status
The State of a cluster shows the health of its hosts, while the Status shows whether the cluster is started, stopped, or is at an intermediate stage.
To view a state and status of a cluster:
- Navigate to the folder dashboard
and select Yandex StoreDoc. - Hover over the indicator in the cluster row of the Availability column.
Cluster states
| State | Description | Suggested actions |
|---|---|---|
| ALIVE | Cluster is operating normally. | No action is required. |
| DEGRADED | Cluster is not running at its full capacity: the state of at least one of the hosts is other than ALIVE. |
Run the diagnostics:
|
| DEAD | The cluster is down: none of its hosts are running. | Make a support request
|
| UNKNOWN | Cluster state is unknown. | Make a support request
|
Cluster statuses
| Status | Description | Suggested actions |
|---|---|---|
| CREATING | Preparing for the first start | Wait a while and get started. The time it takes to create a cluster depends on the host class. |
| RUNNING | The cluster is operating normally | No action is required. |
| STOPPING | The cluster is stopping | After a while, the cluster status will switch to STOPPED and the cluster will be disabled. No action is required. |
| STOPPED | The cluster is stopped | Start the cluster to get it running again. |
| STARTING | Starting the cluster that was stopped earlier | After a while, the cluster status will switch to RUNNING. Wait a while and get started. |
| UPDATING | Updating the cluster's configuration | Once the update is complete, the cluster will get the status it had prior to the update: RUNNING or STOPPED. |
| ERROR | Error when performing an operation with the cluster or during a maintenance window | If the cluster remains in this status for a long time, contact support |
| STATUS_UNKNOWN | The cluster is unable to determine its status | If the cluster remains in this status for a long time, contact support |