Managed Service for OpenSearch storage

Managed Service for OpenSearch allows you to use network and local storage drives for database clusters. Network drives are based on network blocks, which are virtual disks in the Yandex Cloud infrastructure. Local disks are physically located on the cluster servers.

When creating a cluster, you can select the following disk types for data storage:

• Network HDDs (network-hdd): Most cost-effective option for clusters that do not require high read/write performance.

• Network SSDs (network-ssd): Balanced solution. Such disks are slower than local SSD storage, but, unlike local disks, they ensure data integrity if Yandex Cloud hardware fails.

• Non-replicated SSDs (network-ssd-nonreplicated): Network disks with enhanced performance achieved by eliminating redundancy.

  The storage size can only be increased in 93 GB increments.

• Ultra high-speed network SSDs with three replicas (network-ssd-io-m3): Network disks with the same performance characteristics as non-replicated ones. This disk type provides redundancy.

  Such disks can be increased in size only in 93 GB increments.

• Local SSDs (local-ssd): Disks with the best performance.

  The size of such a storage can be increased:

  • For Intel Cascade Lake: Only in 100 GB increments.
  • For Intel Ice Lake: In 368 GB increments only.

  For clusters with hosts residing in the ru-central1-d availability zone, local SSD storage is not available if using Intel Cascade Lake.

Selecting storage type when creating a clusterSelecting storage type when creating a cluster

The number of hosts with the DATA role you can create together with an OpenSearch cluster depends on the selected storage type:

• You can create a cluster only with three or more hosts when using the following disk types:

  • Local SSDs (local-ssd)
  • Non-replicated SSDs (network-ssd-nonreplicated)

  This cluster will be fault-tolerant.

  Local SSD storage has an effect on how much a cluster will cost: you pay for it even if it is stopped. For more information, refer to the pricing policy.

• You can add any number of hosts within the current quota when using the following disk types:

  • Network HDDs (network-hdd)
  • Network SSDs (network-ssd)
  • Ultra high-speed network SSDs with triple replication (network-ssd-io-m3)

For more information about limits on the number of hosts per cluster, see Quotas and limits.

To improve fault tolerance, you can set up index replication (for multi-host cluster configurations only).

Disk space managementDisk space management

When the storage is 95% full, cluster hosts automatically enter read-only mode. In this mode, data write requests fail. To release hosts from read-only mode, increase the storage size or contact support.

Monitoring the transition to read-only modeMonitoring the transition to read-only mode

To monitor the cluster storage usage, set up alerts in Yandex Monitoring:

1. Navigate to the folder dashboard and select Monitoring.

2. Create a notification channel.

3. Create an alert with the following properties:

   1. Metrics: Set the following metric parameters:

      • Cloud.
      • Folder.
      • Managed Service for OpenSearch.
      • disk.free_bytes label.
      • Managed Service for OpenSearch cluster of interest. You can get the cluster name and ID with the list of clusters in the folder.

   2. Alert condition: Set the condition for free disk space usage to trigger the alert:

      • Aggregation function: Minimum (minimum metric value for the period).
      • Comparison function: Less than or equals.
      • Warning: 90 (90 % of storage size).
      • Alarm: 95 (95 % of storage size).
      • Evaluation window: Preferred metric update period.
      • Evaluation delay: Preferred time shift backward, in seconds. It allows to keep the alert from triggering when multiple metrics are specified and collected at different intervals. To learn more about the calculation delay, see this Yandex Monitoring guide.

   3. Notifications: Add the previously created notification channel.

Automatic increase of storage sizeAutomatic increase of storage size

Automatic increase of storage size prevents situations where free disk space has been used up and hosts have switched to read-only mode. The storage size increases upon reaching the specified threshold percentage of the total capacity. There are two thresholds:

• Scheduled increase threshold: When reached, the storage size increases during the next maintenance window. If this threshold is set, configure the maintenance schedule.
• Immediate increase threshold: When reached, the storage size increases immediately.

You can use either one or both thresholds. If you set both, make sure the immediate increase threshold is higher than the scheduled one.

If the specified threshold is reached, the storage size increases differently depending on disk type:

• For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.

• For non-replicated SSDs, by 93 GB.

• For local SSDs:

  • Intel Cascade Lake cluster, by 100 GB.
  • Intel Ice Lake cluster, by 368 GB.

If the threshold is reached again, the storage size will be automatically increased until it reaches the specified maximum. After that, you can specify a new maximum storage size manually.

You can configure automatic storage increase when creating a cluster, or when creating or modifying a host group.

Disk encryptionDisk encryption

When creating or restoring a cluster from a backup, you can encrypt the storage disk with a custom KMS key. To encrypt a disk of an already created cluster, disable encryption, or encrypt a disk with a different key, create a backup of the cluster and restore it with the new settings.

To create an encrypted disk, you need the kms.keys.user role or higher.

If you deactivate the key used to encrypt a disk, access to the data will be suspended until you reactivate the key.