Storage in Managed Service for Apache Kafka®

Managed Service for Apache Kafka® allows you to use network and local storage drives for database clusters. Network drives are based on network blocks, i.e., virtual disks in the Yandex Cloud infrastructure. Local disks are physically located on broker servers.

When creating a cluster, you can select the following disk types for data storage:

• Network HDDs (network-hdd): Most cost-effective option for clusters with low requirements for read and write performance.

• Network SSDs (network-ssd): Compromise solution: slower than local SSDs, network SSDs ensure data integrity in the event of Yandex Cloud hardware failure.

• Non-replicated SSDs (network-ssd-nonreplicated): Network disks with higher performance achieved by eliminating redundancy.

  You can only expand this type of storage in 93 GB increments.

• Ultra high-speed network SSDs with three replicas (network-ssd-io-m3): Network disks that deliver performance equivalent to non-replicated SSDs while ensuring redundancy.

  You can only increase the size of these disks in 93 GB increments.

• Local SSDs (local-ssd): The highest-performing disks.

  You can expand such a storage as follows:

  • For Intel Cascade Lake: Only in 100 GB increments.
  • For Intel Ice Lake and AMD Zen 4: Only in 368 GB increments.

  For clusters with hosts residing in the ru-central1-d availability zone, local SSD storage is not available if using Intel Cascade Lake.

Selecting disk type when creating a clusterSelecting disk type when creating a cluster

The number of broker hosts you can create together with an Apache Kafka® cluster depends on the selected disk type:

• You can create a cluster of only three or more broker hosts using the following disk types:

  • Local SSDs (local-ssd)
  • Non-replicated SSDs (network-ssd-nonreplicated)

  This cluster will be highly available only if all the conditions are met.

• You can add any number of broker hosts within the current quota when using the following disk types:

  • Network HDDs (network-hdd)
  • Network SSDs (network-ssd)
  • Ultra high-speed network SSDs with three replicas (network-ssd-io-m3)

For more information about limits on the number of broker hosts per cluster, see Quotas and limits.

Disk encryptionDisk encryption

When creating a cluster, you can encrypt the storage disk with a custom KMS key.

To create an encrypted disk, you need the kms.keys.user role or higher.

If you deactivate the key used to encrypt a disk, access to the data will be suspended until you reactivate the key.

Minimum storage sizeMinimum storage size

In order to work, each topic requires space in broker host storage. The amount of such space depends on the replication factor and the number of partitions. If there is not enough available storage space, you will not be able to create a new topic.

You can calculate the minimum storage size for all topics using the formula below:

2 × maximum log segment size × number of partitions in cluster × replication factor.

If topic partitions are evenly distributed, divide the value calculated with this formula by the number of broker hosts.

Maximum log segment sizeMaximum log segment size

At least two log segments are required for each replica of a topic partition. You can set the maximum size of such a segment:

• At the topic level using the Segment bytes setting.
• Globally at the cluster level using the Log segment bytes setting.

Thus, the minimum storage size for all topics is: 2 × maximum log segment size × number of partitions in cluster × replication factor. If the cluster partitions are evenly distributed, you can divide the resulting value by the number of brokers to determine the required storage size per broker.

By default, the segment size is 1 GB.

Managing disk spaceManaging disk space

As soon as Apache Kafka® logs take up 97% of storage capacity, the host automatically enters read-only mode. The Managed Service for Apache Kafka® cluster blocks requests to write to the topic.

You can monitor storage utilization on cluster hosts by setting up alerts in Yandex Monitoring:

Recovering a cluster from read-only modeRecovering a cluster from read-only mode

Use one of these methods:

• Increase the storage capacity to exceed the threshold value. The Managed Service for Apache Kafka® cluster will then automatically clear read-only mode.
• Configure automatic storage expansion.

Automatic storage expansionAutomatic storage expansion

Automatic increase of storage size prevents situations where the disk runs out of free space and the host switches to read-only mode. The storage size increases upon reaching the specified threshold percentage of the total capacity. There are two thresholds:

• Scheduled expansion threshold: When reached, the storage size increases during the next maintenance window.
• Immediate expansion threshold: When reached, the storage size increases immediately.

You can use either one or both thresholds. If you set both, make sure the immediate increase threshold is higher than the scheduled one.

If the specified threshold is reached, the storage size may increase in different ways depending on the disk type:

• For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.

• For non-replicated SSDs, by 93 GB.

• For local SSDs, in a platform cluster:

  • Intel Cascade Lake, by 100 GB.
  • Intel Ice Lake, by 368 GB.

If the threshold is reached again, the storage size will be automatically increased until it reaches the specified maximum. After that, you can specify a new maximum storage size manually.

You can configure automatic storage expansion when creating or updating a cluster. If you set the scheduled increase threshold, you also need to configure the maintenance window schedule.

Use casesUse cases

• Configuring Kafka Connect for Managed Service for Apache Kafka® clusters
• Working with the managed schema registry
• Using Confluent Schema Registry with Managed Service for Apache Kafka®