Monitoring the state of a OpenSearch cluster and hosts
Data on cluster and host states are available in the management console. You can view them on the Monitoring tab of the cluster management page or in Yandex Monitoring.
Diagnostic information about cluster states is presented as graphs.
New data for charts is received every 15 seconds.
Note
The most appropriate multiple units (MB, GB, and more) are automatically used in charts.
You can configure alerts in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: Warning
and Alarm
. If the specified threshold is exceeded, you will receive alerts via the configured notification channels.
Monitoring cluster state
To view detailed information about the Managed Service for OpenSearch cluster state:
-
In the management console
, go to the folder page and select Managed Service for OpenSearch. -
Click the name of the cluster you need and select the
Monitoring tab.The page displays the following charts:
-
Health status: Cluster health and technical condition:
0
(red): Cluster is unhealthy or partially functional. At least one of the primary shards is unavailable. If the cluster responds to queries, the search results will be incomplete.1
(yellow): Cluster is functional. There is no access to at least one of the replica shards. The search results in the cluster responses are complete; however, if there are more unavailable shards, cluster performance will be disrupted.2
(green): Cluster is healthy. All cluster shards are available.
-
Active shards: Number of active primary shards and the total number of active shards in the cluster.
-
Other shards: Number of inactive shards in each of the following states:
Delayed unassigned
: Host assignment is delayed.Unassigned
: No host assigned.Relocating
: Moving to another host.Initializing
: Initializing.
-
Nodes: Number of hosts with the
DATA
role. -
Segments: Number of index segments per host.
-
Pending tasks: Number of enqueued tasks.
-
Indexing rate: Number of indexing operations per second, per host.
-
Search rate: Number of search queries per second, per host.
-
Note
To get started with Monitoring metrics, dashboards, or alerts, click Open in Monitoring in the top panel.
Monitoring the state of hosts
To view detailed information about the state of individual Managed Service for OpenSearch hosts:
- In the management console
, go to the folder page and select Managed Service for OpenSearch. - Click the name of the cluster you need and select the
Hosts tab. - Select the Monitoring tab.
- Select the host from the drop-down list.
This page displays charts showing the load on an individual host in the cluster. Which charts are exactly displayed depends on the host type:
- Process CPU: Processor core workload generated by the JVM OpenSearch process.
- Memory usage: Use of RAM, in bytes.
- JVM heap: Use of JVM heap memory, in bytes.
- Disk space usage percent: Use of disk space, in percentage.
- Management thread pool: Number of cluster management requests.
- Generic thread pool: Number of requests for running general operations.
- Thread pool queued: Number of enqueued requests.
- Thread pool rejected: Number of rejected requests.
- Process CPU: Processor core workload generated by the JVM OpenSearch process.
- Memory usage: Use of RAM, in bytes.
- JVM heap percent: Use of JVM heap memory, in percentage
- Disk space usage percent: Use of disk space, in percentage.
- Indexing rate: Number of indexing operations per second.
- Search queries: Number of search queries per second.
- Open file descriptors: Number of open file descriptors.
- Write bytes: Disk write rate, in bytes per second.
- Read bytes: Disk read rate, in bytes per second.
- Write thread pool: Requests for indexing, deleting, or updating documents.
- Write operations: Number of write operations per second.
- Read operations: Number of read operations per second.
- Query time: Time spent to run the queries.
- Thread pool queued: Number of enqueued requests.
- Thread pool rejected: Number of rejected requests.
- Indexing time: Time spent to index the documents.
- Merging time: Time spent to merge the documents.
- Is Alive: Status that shows the host is up and running.
- Requests Total: Total number of host requests.
- Process CPU: Processor core workload generated by the JVM OpenSearch process.
- Memory usage: Use of RAM, in bytes.
- Disk read/write bytes: Speed of disk operations, in bytes per second.
- Disk IOPS: Number of disk operations per second.
- Network packets: Number of packets exchanged over the network, per second.
- Network bytes: Speed of data exchange over the network, in bytes per second.
Monitoring the state of host groups
To view detailed information about the state of a Managed Service for OpenSearch host group:
- In the management console
, go to the folder page and select Managed Service for OpenSearch. - Click the name of the cluster you need and select the
Node groups tab. - Select the Monitoring tab.
- Select the host group you need from the drop-down list.
This page displays charts showing the load on the cluster's host group. Which charts are exactly displayed depends on the type of hosts in the group; this is in line with the charts shown for individual hosts.
Alert settings in Yandex Monitoring
-
In the management console
, select the folder with the cluster you want to configure alerts for. -
In the list of services, select
Monitoring. -
Under Service dashboards, select:
- Managed Service for OpenSearch to configure cluster alerts.
- Managed Service for OpenSearch — Dashboards to configure alerts for hosts with the
DASHBOARDS
role. - Managed Service for OpenSearch — Data to configure alerts for hosts with the
DATA
role. - Managed Service for OpenSearch — Manager to configure alerts for hosts with the
MANAGER
role.
-
In the chart you need, click
and select Create alert. -
If the chart shows multiple metrics, select a data query to generate a metric and click Continue. You can learn more about the query language in the Yandex Monitoring documentation.
-
Set the
Alarm
andWarning
threshold values to trigger the alert. -
Click Create alert.
To have other cluster health indicators monitored automatically:
- Create an alert.
- Add a status metric.
- In the alert parameters, set up your alert thresholds.
The recommended thresholds are as follows:
Metrica | Parameter | Formula | Alarm |
Warning |
---|---|---|---|---|
Cluster status | opensearch_status |
bottom_last(1) |
equal to 0 |
equal to 1 |
Number of unassigned shards | opensearch_unassigned_shards |
top_last(1) |
greater than 0 |
|
Number of relocated shards | opensearch_relocating_shards |
top_last(1) |
greater than 0 |
|
Number of initialized shards | opensearch_initializing_shards |
top_last(1) |
greater than 0 |
|
Number of delayed assignment shards | opensearch_delayed_unassigned_shards |
top_last(1) |
greater than 0 |
|
JVM heap memory used | opensearch_jvm_mem_heap_used_percent |
top_last(1) |
Over 90% of host RAM | |
Storage space used | opensearch_fs_total_used_percent |
top_last(1) |
Over 90% of the storage size | Over 85% of the storage size |
Using the JVM long-lived object pool | opensearch_jvm_mem_heap_pressure |
top_last(1) |
Over 90% of host RAM | Over 75% of host RAM |
Storage space used | disk.used_bytes |
— | 90% of the storage size | 80% of the storage size |
For the disk.used_bytes
metric, the Alarm
and Warning
thresholds are only set in bytes. For example, the recommended values for a 100 GB disk are as follows:
Alarm
:96636764160
bytes (90%).Warning
:85899345920
bytes (80%).
You can view the current storage size and RAM of the hosts in the detailed information about the cluster. For a complete list of supported metrics, see the Monitoring documentation.
Cluster state and status
The State of a cluster shows the health of its hosts, while the Status shows whether the cluster is started, stopped, or is at an intermediate stage.
To view a cluster's state and status:
- In the management console
, go to the folder page and select Managed Service for OpenSearch. - Hover over the indicator in the Availability column in the required cluster row.
If you do not have the Yandex Cloud command line interface yet, install and initialize it.
The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name
or --folder-id
parameter.
To find out the state and status of a cluster, get information about it:
yc managed-opensearch cluster get <cluster_name_or_ID>
The health
parameter gives the cluster state; the status
parameter gives the cluster status.
You can request the cluster name and ID with the list of clusters in the folder.
-
Get an IAM token for API authentication and put it into the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Use the Cluster.Get method and make a request, e.g., via cURL
:curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-opensearch/v1/clusters/<cluster_ID>'
You can get the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
The cluster health and status are shown in the
health
andstatus
parameters, respectively.
-
Get an IAM token for API authentication and put it into the environment variable:
export IAM_TOKEN="<IAM_token>"
-
Clone the cloudapi
repository:cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
Below, we assume the repository contents are stored in the
~/cloudapi/
directory. -
Use the ClusterService.Get call and make a request, e.g., via gRPCurl
:grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/opensearch/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.opensearch.v1.ClusterService.Get
You can get the cluster ID with a list of clusters in the folder.
-
View the server response to make sure the request was successful.
The cluster health and status are shown in the
health
andstatus
parameters, respectively.
Cluster states
State | Description | Suggested actions |
---|---|---|
ALIVE | Cluster is operating normally. | No action is required. |
DEGRADED | Cluster is not running at its full capacity: the state of at least one of the hosts is other than ALIVE . |
Run the diagnostics:
|
DEAD | The cluster is down: none of its hosts are running. | Make a support request
|
UNKNOWN | Cluster state is unknown. | Make a support request
|
Cluster statuses
Status | Description | Suggested actions |
---|---|---|
CREATING | Preparing for the first launch | Wait a while and get started. The time it takes to create a cluster depends on the host class. |
RUNNING | Cluster is operating normally | No action is required. |
STOPPING | Stopping cluster | After a while, the cluster status will change to STOPPED and the cluster will be disabled. No action is required. |
STOPPED | Cluster stopped | Start the cluster to get it running again. |
STARTING | Starting the cluster that was stopped earlier | After a while, the cluster status will change to RUNNING . Wait a while and get started. |
UPDATING | Updating the cluster status | After the update is completed, the cluster status will change to RUNNING . Wait a while and get started. |
ERROR | An error occurred that does not allow the cluster to continue working | Run the initial diagnostics:
|
STATUS_UNKNOWN | Cluster is unable to determine its own status | Run the initial diagnostics:
|