Storage in Managed Service for Apache Kafka®

Managed Service for Apache Kafka® allows you to use network and local storage drives for database clusters. Network drives are based on network blocks, which are virtual disks in the Yandex Cloud infrastructure. Local disks are physically located on broker servers.

When creating a cluster, you can select the following disk types for data storage:

• Network HDD storage (network-hdd): Most cost-effective option for clusters that do not require high read/write performance.

• Network SSD storage (network-ssd): Balanced solution. Such disks are slower than local SSD storage, but, unlike local disks, they ensure data integrity in case Yandex Cloud hardware goes down.

• Non-replicated SSD storage (network-ssd-nonreplicated): Network SSD storage with enhanced performance but without redundancy.

  The storage size can only be increased in 93 GB increments.

• Local SSDs (local-ssd): Disks with the fastest performance.

  The size of such a storage can be increased:

  • For Intel Cascade Lake: In 100 GB increments only.

  • For Intel Ice Lake: In 368 GB increments only.

  Note

  For clusters with hosts residing in the ru-central1-d availability zone, local SSD storage is not available if using the Intel Cascade Lake platform.

Selecting the disk type during cluster creationSelecting the disk type during cluster creation

The number of broker hosts you can create together with an Apache Kafka® cluster depends on the selected disk type:

• With local SSDs (local-ssd) or non-replicated SSDs (network-ssd-nonreplicated), you can create a cluster with three or more broker hosts.

  This cluster will be fail-safe only if it meets all fault tolerance conditions.

• With network HDD (network-hdd) or network SSD (network-ssd) storage, you can add any number of broker hosts within the current quota.

For more information about limits on the number of broker hosts per cluster, see Quotas and limits.

Minimum storage sizeMinimum storage size

In order to work, each topic requires space in broker host storage. The amount of such space depends on the replication factor and the number of partitions. If there is not enough available storage space, you will not be able to create a new topic.

You can calculate the minimum storage size for all topics using the formula below:

2 × maximum log segment size × number of partitions in cluster × replication factor

If topic partitions are evenly distributed, divide the value calculated with this formula by the number of broker hosts.

Maximum log segment sizeMaximum log segment size

At least two log segments are required for each replica of a topic partition. You can set the maximum size of such a segment:

• At the topic level using the Segment bytes setting.
• Globally at the cluster level using the Log segment bytes setting.

Thus, the minimum storage size for all topics is: 2 × maximum log segment size × number of partitions in cluster × replication factor. If the cluster partitions are evenly distributed, you can divide the resulting value by the number of brokers to determine the required storage size per broker.

By default, the segment size is 1 GB.

Disk space managementDisk space management

Upon Apache Kafka® logs reaching 97% storage capacity, the host automatically switches to read-only mode. The Managed Service for Apache Kafka® cluster denies any requests for writing messages to a topic.

You can monitor storage utilization on cluster hosts by setting up alerts in Yandex Monitoring:

Recovering a cluster from read-only modeRecovering a cluster from read-only mode

Use one of these methods:

• Increase the storage capacity so that it exceeds the threshold value. The Managed Service for Apache Kafka® cluster will then automatically disable the read-only mode.
• Set up automatic storage size increase.

Automatic increase of storage sizeAutomatic increase of storage size

Automatic storage size increase prevents situations where the disk runs out of free space and the host switches to read-only. The storage size increases upon reaching the specified trigger threshold: a percentage of the total capacity. There are two thresholds:

• Scheduled increase threshold: When reached, the storage size increases during the next maintenance window.
• Immediate increase threshold: When reached, the storage size increases immediately.

You can use either one or both thresholds. If you set both, make sure the immediate increase threshold is higher than the scheduled one.

If the specified threshold is reached, the storage size may increase in different ways depending on the disk type:

• For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.

• For non-replicated SSDs, by 93 GB.

• For local SSDs, in a platform cluster:

  • Intel Cascade Lake, by 100 GB.
  • Intel Ice Lake, by 368 GB.

If the threshold is reached again, the storage size will be automatically increased until it reaches the specified maximum. After that, you can specify a new maximum storage size manually.

You can configure automatic increase of the storage size when creating or updating a cluster. If you set the scheduled increase threshold, you also need to configure the maintenance window schedule.