Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Monitoring the state of a OpenSearch cluster and hosts

Written by
Updated at November 18, 2024

Data on cluster and host states are available in the management console. You can view them on the Monitoring tab of the cluster management page or in Yandex Monitoring.

Diagnostic information about cluster states is presented as graphs.

New data for charts is received every 15 seconds.

Note

The most appropriate multiple units (MB, GB, and more) are automatically used in charts.

You can configure alerts in Yandex Monitoring to receive notifications about cluster failures. In Yandex Monitoring, there are two alert thresholds: Warning and Alarm. If the specified threshold is exceeded, you will receive alerts via the configured notification channels.

Monitoring cluster state

To view detailed information about the Managed Service for OpenSearch cluster state:

  1. In the management console, go to the folder page and select Managed Service for OpenSearch.

  2. Click the name of the cluster you need and select the  Monitoring tab.

    The page displays the following charts:

    • Health status: Cluster health and technical condition:

      • 0 (red): Cluster is unhealthy or partially functional. At least one of the primary shards is unavailable. If the cluster responds to queries, the search results will be incomplete.
      • 1 (yellow): Cluster is functional. There is no access to at least one of the replica shards. The search results in the cluster responses are complete; however, if there are more unavailable shards, cluster performance will be disrupted.
      • 2 (green): Cluster is healthy. All cluster shards are available.

    • Active shards: Number of active primary shards and the total number of active shards in the cluster.

    • Other shards: Number of inactive shards in each of the following states:

      • Delayed unassigned: Host assignment is delayed.
      • Unassigned: No host assigned.
      • Relocating: Moving to another host.
      • Initializing: Initializing.

    • Nodes: Number of hosts with the DATA role.

    • Segments: Number of index segments per host.

    • Pending tasks: Number of enqueued tasks.

    • Indexing rate: Number of indexing operations per second, per host.

    • Search rate: Number of search queries per second, per host.

Note

To get started with Monitoring metrics, dashboards, or alerts, click Open in Monitoring in the top panel.

Monitoring the state of hosts

To view detailed information about the state of individual Managed Service for OpenSearch hosts:

  1. In the management console, go to the folder page and select Managed Service for OpenSearch.
  2. Click the name of the cluster you need and select the  Hosts tab.
  3. Select the Monitoring tab.
  4. Select the host from the drop-down list.

This page displays charts showing the load on an individual host in the cluster. Which charts are exactly displayed depends on the host type:

  • Process CPU: Processor core workload generated by the JVM OpenSearch process.
  • Memory usage: Use of RAM, in bytes.
  • JVM heap: Use of JVM heap memory, in bytes.
  • Disk space usage percent: Use of disk space, in percentage.
  • Management thread pool: Number of cluster management requests.
  • Generic thread pool: Number of requests for running general operations.
  • Thread pool queued: Number of enqueued requests.
  • Thread pool rejected: Number of rejected requests.
  • Process CPU: Processor core workload generated by the JVM OpenSearch process.
  • Memory usage: Use of RAM, in bytes.
  • JVM heap percent: Use of JVM heap memory, in percentage
  • Disk space usage percent: Use of disk space, in percentage.
  • Indexing rate: Number of indexing operations per second.
  • Search queries: Number of search queries per second.
  • Open file descriptors: Number of open file descriptors.
  • Write bytes: Disk write rate, in bytes per second.
  • Read bytes: Disk read rate, in bytes per second.
  • Write thread pool: Requests for indexing, deleting, or updating documents.
  • Write operations: Number of write operations per second.
  • Read operations: Number of read operations per second.
  • Query time: Time spent to run the queries.
  • Thread pool queued: Number of enqueued requests.
  • Thread pool rejected: Number of rejected requests.
  • Indexing time: Time spent to index the documents.
  • Merging time: Time spent to merge the documents.
  • Is Alive: Status that shows the host is up and running.
  • Requests Total: Total number of host requests.
  • Process CPU: Processor core workload generated by the JVM OpenSearch process.
  • Memory usage: Use of RAM, in bytes.
  • Disk read/write bytes: Speed of disk operations, in bytes per second.
  • Disk IOPS: Number of disk operations per second.
  • Network packets: Number of packets exchanged over the network, per second.
  • Network bytes: Speed of data exchange over the network, in bytes per second.

Monitoring the state of host groups

To view detailed information about the state of a Managed Service for OpenSearch host group:

  1. In the management console, go to the folder page and select Managed Service for OpenSearch.
  2. Click the name of the cluster you need and select the  Node groups tab.
  3. Select the Monitoring tab.
  4. Select the host group you need from the drop-down list.

This page displays charts showing the load on the cluster's host group. Which charts are exactly displayed depends on the type of hosts in the group; this is in line with the charts shown for individual hosts.

Alert settings in Yandex Monitoring

  1. In the management console, select the folder with the cluster you want to configure alerts for.

  2. In the list of services, select Monitoring.

  3. Under Service dashboards, select:

    • Managed Service for OpenSearch to configure cluster alerts.
    • Managed Service for OpenSearch — Dashboards to configure alerts for hosts with the DASHBOARDS role.
    • Managed Service for OpenSearch — Data to configure alerts for hosts with the DATA role.
    • Managed Service for OpenSearch — Manager to configure alerts for hosts with the MANAGER role.

  4. In the chart you need, click and select Create alert.

  5. If the chart shows multiple metrics, select a data query to generate a metric and click Continue. You can learn more about the query language in the Yandex Monitoring documentation.

  6. Set the Alarm and Warning threshold values to trigger the alert.

  7. Click Create alert.

To have other cluster health indicators monitored automatically:

  1. Create an alert.
  2. Add a status metric.
  3. In the alert parameters, set up your alert thresholds.

The recommended thresholds are as follows:

Metrica Parameter Formula Alarm Warning
Cluster status opensearch_status bottom_last(1) equal to 0 equal to 1
Number of unassigned shards opensearch_unassigned_shards top_last(1) greater than 0
Number of relocated shards opensearch_relocating_shards top_last(1) greater than 0
Number of initialized shards opensearch_initializing_shards top_last(1) greater than 0
Number of delayed assignment shards opensearch_delayed_unassigned_shards top_last(1) greater than 0
JVM heap memory used opensearch_jvm_mem_heap_used_percent top_last(1) Over 90% of host RAM
Storage space used opensearch_fs_total_used_percent top_last(1) Over 90% of the storage size Over 85% of the storage size
Using the JVM long-lived object pool opensearch_jvm_mem_heap_pressure top_last(1) Over 90% of host RAM Over 75% of host RAM
Storage space used disk.used_bytes — 90% of the storage size 80% of the storage size

For the disk.used_bytes metric, the Alarm and Warning thresholds are only set in bytes. For example, the recommended values for a 100 GB disk are as follows:

  • Alarm: 96,636,764,160 bytes (90%)
  • Warning: 85,899,345,920 bytes (80%)

You can view the current storage size and RAM of the hosts in the detailed information about the cluster. For a complete list of supported metrics, see the Monitoring documentation.

Cluster state and status

The State of a cluster shows the health of its hosts, while the Status shows whether the cluster is started, stopped, or is at an intermediate stage.

To view a cluster's state and status:

  1. In the management console, go to the folder page and select Managed Service for OpenSearch.
  2. Hover over the indicator in the Availability column in the required cluster row.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To find out the state and status of a cluster, get information about it:

yc managed-opensearch cluster get <cluster_name_or_ID>

The health parameter gives the cluster state; the status parameter gives the cluster status.

You can request the cluster name and ID with a list of clusters in the folder.

Use the get REST API method for the Cluster resource or the ClusterService/Get gRPC API call, and provide the ID of cluster being deleted in the clusterId request parameter.

The cluster health and status will be shown in the health and status parameters, respectively.

You can get the cluster ID with a list of clusters in the folder.

Cluster states

State Description Suggested actions
ALIVE Cluster is operating normally. No action is required.
DEGRADED Cluster is not running at its full capacity: the state of at least one of the hosts is other than ALIVE. Run the diagnostics:
  • Go to the Hosts tab and see which hosts are not working.
  • Go to the Operations tab and make sure all operations are completed.
  • Make sure the cluster is not under maintenance.
If you cannot find the cause yourself, contact support.
DEAD The cluster is down: none of its hosts are running. Make a support request stating the following:
  • Cluster ID.
  • IDs of the last operations performed on it.
  • Time the cluster entered the DEAD state according to the availability charts.
UNKNOWN Cluster state is unknown. Make a support request stating the following:
  • Cluster ID.
  • IDs of the last operations performed on it.
  • Time the cluster entered the UNKNOWN state according to the availability charts.

Cluster statuses

Status Description Suggested actions
CREATING Preparing for the first launch Wait a while and get started. The time it takes to create a cluster depends on the host class.
RUNNING Cluster is operating normally No action is required.
STOPPING Stopping cluster After a while, the cluster status will change to STOPPED and the cluster will be disabled. No action is required.
STOPPED Cluster stopped Start the cluster to get it running again.
STARTING Starting the cluster that was stopped earlier After a while, the cluster status will change to RUNNING. Wait a while and get started.
UPDATING Updating the cluster status After the update is completed, the cluster status will change to RUNNING. Wait a while and get started.
ERROR An error occurred that does not allow the cluster to continue working Run the initial diagnostics:
  • Analyze the cluster monitoring charts and view the operations performed.
  • Prepare a list of IDs of problem resources.
If you cannot find the cause of the error yourself, contact support.
STATUS_UNKNOWN Cluster is unable to determine its own status Run the initial diagnostics:
  • Analyze the cluster monitoring charts and view the operations performed.
  • Prepare a list of IDs of problem resources.
If you cannot find the cause of the error yourself, contact support.
Previous
Viewing cluster logs
Next
Configuring an index policy in Managed Service for OpenSearch