Storage in Managed Service for OpenSearch

Managed Service for OpenSearch allows you to use network and local storage drives for database clusters. Network disks are based on network blocks, i.e., virtual disks in the Yandex Cloud infrastructure. Local disks are physically located on the cluster servers.

When creating a cluster, you can select the following disk types for data storage:

• Network HDDs (network-hdd): Most cost-effective option for clusters with low requirements for read and write performance.

• Network SSDs (network-ssd): Compromise solution: slower than local SSDs, network SSDs ensure data integrity in the event of Yandex Cloud hardware failure.

• Non-replicated SSDs (network-ssd-nonreplicated): Network disks with higher performance achieved by eliminating redundancy.

  You can only expand this type of storage in 93 GB increments.

• Ultra high-speed network SSDs with three replicas (network-ssd-io-m3): Network disks that deliver performance equivalent to non-replicated SSDs while ensuring redundancy.

  You can only increase the size of these disks in 93 GB increments.

• Local SSDs (local-ssd): Highest-performing disks.

  You can expand such a storage as follows:

  • For Intel Cascade Lake: Only in 100 GB increments.
  • For Intel Ice Lake and AMD Zen 4: Only in 368 GB increments.

  For clusters with hosts residing in the ru-central1-d availability zone, local SSD storage is not available if using Intel Cascade Lake.

Block size for all disk types is 4 KB.

Selecting storage type when creating a clusterSelecting storage type when creating a cluster

The number of hosts with the DATA role you can create together with an OpenSearch cluster depends on the selected storage type:

• You can create a cluster only with three or more hosts when using the following disk types:

  • Local SSDs (local-ssd)
  • Non-replicated SSDs (network-ssd-nonreplicated)

  Storage on local SSDs increases your cluster costs: you pay for the cluster even if it is stopped. For more information, see the pricing policy.

• You can add any number of hosts within the current quota when using the following disk types:

  • Network HDDs (network-hdd)
  • Network SSDs (network-ssd)
  • Ultra high-speed network SSDs with three replicas (network-ssd-io-m3)

For more information about limits on the number of hosts per cluster, see Quotas and limits.

To improve fault tolerance, you can set up index replication (for multi-host cluster configurations only).

Managing disk spaceManaging disk space

When the storage is 95% full, cluster hosts automatically enter read-only mode. In this mode, data write requests fail. To release hosts from read-only mode, increase the storage size or contact support.

Monitoring the transition to read-only modeMonitoring the transition to read-only mode

To monitor the cluster storage usage, set up alerts in Yandex Monitoring:

1. Navigate to the folder dashboard and select Monitoring.

2. Create a notification channel.

3. Create an alert with the following properties:

   1. Metrics: Configure the following metric settings:

      • Cloud.
      • Folder.
      • Managed Service for OpenSearch.
      • disk.free_bytes label.
      • Managed Service for OpenSearch cluster of interest. You can get the cluster name and ID with the list of clusters in the folder.

   2. Alert condition: Set the condition for free disk space usage to trigger the alert:

      • Aggregation function: Minimum (metric’s minimum value over the period).
      • Comparison function: Less than or equals.
      • Warning: 90 (90 % of storage size).
      • Alarm: 95 (95 % of storage size).
      • Evaluation window: Preferred metric update period.
      • Evaluation delay: Preferred time shift backward, in seconds. It allows to keep the alert from triggering when multiple metrics are specified and collected at different intervals. To learn more about the calculation delay, see this Yandex Monitoring guide.

   3. Notifications: Add the notification channel you created earlier.

Automatic storage expansionAutomatic storage expansion

Automatic increase of storage size prevents situations where free disk space has been used up and hosts have switched to read-only mode. The storage size increases upon reaching the specified threshold percentage of the total capacity. There are two thresholds:

• Scheduled expansion threshold: When reached, the storage expands during the next maintenance window. If this threshold is set, configure the maintenance schedule.
• Immediate expansion threshold: When reached, the storage size increases immediately.

You can use either one or both thresholds. If you set both, make sure the immediate increase threshold is higher than the scheduled one.

Upon reaching the specified threshold, the storage expands differently depending on the disk type:

• For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.

• For non-replicated SSDs, by 93 GB.

• For local SSDs:

  • Intel Cascade Lake cluster, by 100 GB.
  • Intel Ice Lake cluster, by 368 GB.

If the threshold is reached again, the storage will be automatically expanded until it reaches the specified maximum. After that, you can set a new maximum storage size manually.

You can configure automatic storage expansion when creating a cluster, or when creating or updating a host group.

Disk encryptionDisk encryption

When creating or restoring a cluster from a backup, you can encrypt the storage disk with a custom KMS key. To encrypt a disk of an already created cluster, disable encryption, or encrypt a disk with a different key, create a backup of the cluster and restore it with the new settings.

To create an encrypted disk, you need the kms.keys.user role or higher.

If you deactivate the key used to encrypt a disk, access to the data will be suspended until you reactivate the key.