High availability of a Managed Service for OpenSearch cluster

High availability of a Managed Service for OpenSearch cluster depends on the number of DATA and MANAGER hosts.

Cluster host configurationCluster host configuration

A Managed Service for OpenSearch cluster consists of one or more host groups where each host assumes a specific role.

Each OpenSearch host group can include hosts with the DATA or MANAGER roles. If a cluster has a single OpenSearch group, its hosts will have both roles.

For a cluster to be highly available and covered by a service level agreement (SLA), it must include:

1. Two or more hosts with the DATA role.
2. Three or more hosts with the MANAGER role.

Protecting a cluster from risky changesProtecting a cluster from risky changes

Managed Service for OpenSearch blocks certain requests to the OpenSearch API to protect clusters from accidental downtime or data loss. Blocked requests fail and return an error message describing the reason.

These restrictions apply to all cluster users, including admin.

Basic restrictionsBasic restrictions

These restrictions are enforced across all Managed Service for OpenSearch clusters and block the following operations:

• Updating a cluster’s protected settings. This includes settings with the following prefixes:

  • cluster.routing.allocation.disk.watermark.
  • cluster.fault_detection.follower_check.timeout
  • cluster.fault_detection.leader_check.timeout
  • cluster.follower_lag.timeout
  • cluster.publish.timeout
  • cluster.snapshot.info.max_concurrent_fetches

  To update these settings, submit a request to support.

• Excluding MANAGER hosts from master election voting via the POST /_cluster/voting_config_exclusions request. To switch the master, use this Yandex Cloud CLI command: yc managed-opensearch cluster switch-master.

• Operations involving the yc-automatic-backups and yc-automatic-restore system backup repositories, e.g., creating or deleting the repositories as well as creating or deleting snapshots inside them. The snapshot management policy of the same name, yc-automatic-backups, is also protected.

Validating cluster settingsValidating cluster settings

To find out which cluster and index settings violate high availability requirements, send the following request:

curl \
    --user <username>:<password> \
    --cacert ~/.opensearch/root.crt \
    --request POST \
    --header 'Content-Type: application/json' \
    --url 'https://<FQDN_of_OpenSearch_host_with_public_access>:9200/_plugins/_security/availability_guard/analyze' \
    --data '{}'

Provide an empty request body ({}) to check the current cluster state. To test the changes you plan to make, specify one or more of the following fields in the request body:

• settings: Cluster settings to check. They extend the current cluster configuration and override those settings that match. This allows you to test the results of a PUT /_cluster/settings request before running it.

• excluding_hosts: List of FQDNs of hosts to exclude from the check. Use this to simulate host removal or an availability zone failure.

• excluding_indices: List of index names to exclude from the check. The * wildcard is supported.

• nodes: Cluster host topology to check instead of the actual one. The key is the host name (the name field in the response to GET /_cat/nodes?v), and the value is an object with the following fields:

  • roles: List of OpenSearch host roles: data, cluster_manager, ingest, remote_cluster_client, or warm. This is a required field.
  • attributes: Host attributes used in shard placement rules, e.g., an availability zone or host group. To get the current cluster host attributes, use the GET /_cat/nodeattrs?v request.
  • disk_size: Host storage size, e.g., 100gb.

  If the nodes field is set, the excluding_hosts field is ignored.

For example, to simulate an availability zone failure, exclude all hosts within that zone from the check:

curl \
    --user <username>:<password> \
    --cacert ~/.opensearch/root.crt \
    --request POST \
    --header 'Content-Type: application/json' \
    --url 'https://<FQDN_of_OpenSearch_host_with_public_access>:9200/_plugins/_security/availability_guard/analyze' \
    --data '{
        "excluding_hosts": [
            "<host_1_FQDN>",
            "<host_2_FQDN>"
        ]
    }'

Detected issues are listed in the allocation_impacts array in the response; the reason for each issue is stated in the reason field. An empty array means no issues were found.

Storage settingsStorage settings

When the storage is 95% full, cluster hosts automatically enter read-only mode. In this mode, data write requests fail. To prevent situations where the disk runs out of free space and the cluster becomes unavailable for writes:

• Set up alerts in Yandex Monitoring to monitor storage utilization.
• Configure automatic storage expansion.

Learn more about managing disk space in Managed Service for OpenSearch.

Maintenance settingsMaintenance settings

In multi-host clusters, hosts undergo maintenance one by one. Any host that requires a restart during maintenance will be unavailable until maintenance is complete.

Learn more about maintenance in Managed Service for OpenSearch.

Other parameters and limitationsOther parameters and limitations

Cluster availability may also be affected by:

• Backup settings.

• Storage disk type.

• Host class.

• Quotas and limits.

• Security group settings.