Contact UsGet started

Managing disk space in a Managed Service for Apache Kafka® cluster

Written by
Updated at May 13, 2025

As soon as the storage is 97% full, the host automatically enters read-only mode. To avoid issues with writing to the topic, use one of the following methods:

Set up alerts in Yandex Monitoring

  1. Navigate to the folder dashboard and select Monitoring.

  2. Create a notification channel.

  3. Create an alert with the following properties:

    1. Metrics: Set the following metric parameters:

      • Cloud.

      • Folder.

      • Managed Service for Kafka

      • Managed Service for Apache Kafka® cluster ID.

        You can get the ID with the list of clusters in the folder.

      • disk.free_bytes label.

    2. Alert condition: Set the condition for free disk space usage to trigger the alert:

      • Aggregation function: Minimum (minimum metric value for the period).
      • Comparison function: Less than or equals.
      • Warning: 95 (95% of the storage size).
      • Alarm: 90 (90% of the storage size).
      • Evaluation window: Preferred metric update period.
      • Evaluation delay: Preferred time shift backward, in seconds. It allows to keep the alert from triggering when multiple metrics are specified and collected at different intervals. To learn more about the calculation delay, see the Yandex Monitoring documentation.

    3. Notifications: Add the previously created notification channel.

Increasing storage size

Make sure the cloud has enough quota to increase the storage size. Open the cloud's Quotas page and check the HDD storage capacity and SSD storage capacity lines under Managed Databases to make sure there is available quota for storage space.

To increase the cluster storage size:

  1. Navigate to the folder dashboard and select Managed Service for Kafka.

  2. In the cluster row, click , then select Edit.

  3. Edit the settings in the Storage section.

    You cannot change the disk type for an Apache Kafka® cluster once you create it.

  4. Click Save.

If you do not have the Yandex Cloud (CLI) command line interface yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

To increase the hosts' storage size:

  1. View the description of the update cluster CLI command:

    yc managed-kafka cluster update --help

  2. To change the size of the broker host storage, run the command:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --disk-size <storage_size>

    If no size units are specified, gigabytes are used.

  3. To change the size of the ZooKeeper host storage, run the command:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --zookeeper-disk-size <disk_size>

    If no size units are specified, gigabytes are used.

You cannot change the disk type for an Apache Kafka® cluster once you create it.

To increase the cluster storage size:

  1. Open the current Terraform configuration file that defines your infrastructure.

    For more information about creating this file, see Creating clusters.

  2. In the Managed Service for Apache Kafka® cluster description, change the value of the disk_size parameter in the kafka.resources and zookeeper.resources sections for Apache Kafka® and ZooKeeper hosts, respectively:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  kafka {
    resources {
      disk_size = <storage_size_in_GB>
      ...
    }
    ...
  }
  zookeeper {
    resources {
      disk_size = <storage_size_in_GB>
      ...
    }
  }
}

    You cannot change the disk type for an Apache Kafka® cluster once you create it.

  3. Make sure the settings are correct.

    1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

    2. Run this command:

      terraform validate

      Terraform will show any errors found in your configuration files.

  4. Confirm updating the resources.

    1. Run this command to view the planned changes:

      terraform plan

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

    2. If everything looks correct, apply the changes:

      1. Run this command:

        terraform apply

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Use the Cluster.update method and send the following request, e.g., via cURL:

    Warning

    The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

    curl \
    --request PATCH \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
    --data '{
              "updateMask": "configSpec.kafka.resources.diskSize,configSpec.zookeeper.resources.diskSize",
              "configSpec": {
                "kafka": {
                  "resources": {
                    "diskSize": "<storage_size_in_bytes>"
                  }
                },
                "zookeeper": {
                  "resources": {
                    "diskSize": "<storage_size_in_bytes>"
                  }
                }
              }
            }'

    Where:

    • updateMask: List of parameters to update as a single string, separated by commas.

      Specify the relevant parameters:

      • configSpec.kafka.resources.diskSize: To change the broker host storage size.
      • configSpec.zookeeper.resources.diskSize: To change the ZooKeeper host storage size. Use only for Apache Kafka® 3.5 clusters.

    • configSpec.kafka.resources.diskSize: Broker host storage size in bytes.

    • configSpec.zookeeper.resources.diskSize: ZooKeeper host storage size in bytes. Use only for Apache Kafka® 3.5 clusters.

    You can request the cluster ID with the list of clusters in the folder.

  3. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Use the ClusterService/Update call and send the following request, e.g., via gRPCurl:

    Warning

    The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.

    Format for listing settings
    "update_mask": {
    "paths": [
        "<setting_1>",
        "<setting_2>",
        ...
        "<setting_N>"
    ]
}

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "cluster_id": "<cluster_ID>",
          "update_mask": {
            "paths": [
              "config_spec.kafka.resources.disk_size",
              "config_spec.zookeeper.resources.disk_size"
            ]
          },
          "config_spec": {
            "kafka": {
              "resources": {
                "disk_size": "<storage_size_in_bytes>"
              }
            },
            "zookeeper": {
              "resources": {
                "disk_size": "<storage_size_in_bytes>"
              }
            }
          }
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.Update

    Where:

    • update_mask: List of parameters to update as an array of paths[] strings.

      Specify the relevant parameters:

      • config_spec.kafka.resources.disk_size: To change the broker host storage size.
      • config_spec.brokers_count: To change the ZooKeeper host storage size. Use only for Apache Kafka® 3.5 clusters.

    • config_spec.kafka.resources.disk_size: Broker host storage size in bytes.

    • config_spec.zookeeper.resources.disk_size: ZooKeeper host storage size in bytes. Use only for Apache Kafka® 3.5 clusters.

    You can request the cluster ID with the list of clusters in the folder. The list of available host classes with their IDs came to you earlier.

  4. View the server response to make sure the request was successful.

Setting up automatic increase of storage size

Set up automatic storage increase to prevent the cluster from running out of disk space and the hosts from switching to read-only mode.

Make sure the cloud has enough quota to increase the storage size. Open the cloud's Quotas page and check the HDD storage capacity and SSD storage capacity lines under Managed Databases to make sure there is available quota for storage space.

Warning

  • You cannot decrease the storage size.
  • While resizing the storage, cluster hosts will be unavailable.

  1. Navigate to the folder dashboard and select Managed Service for Kafka.

  2. In the cluster row, click , then select Edit.

  3. Under Automatic increase of storage size, set the storage utilization thresholds that will trigger an increase in storage size when reached:

    1. In the Increase size field, select one or both thresholds:
      • In the maintenance window when full at more than: Scheduled increase threshold. When reached, the storage size increases during the next maintenance window.
      • Immediately when full at more than: Immediate increase threshold. When reached, the storage size increases immediately.
    2. Specify a threshold value (as a percentage of the total storage size). If you select both thresholds, make sure the immediate increase threshold is higher than the scheduled one.
    3. Set Maximum storage size.

If you do not have the Yandex Cloud (CLI) command line interface yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

To set up automatic increase of storage size:

  1. View the description of the update cluster CLI command:

    yc managed-kafka cluster update --help

  2. Set the maximum storage size and conditions for its increase in the update cluster command:

    yc managed-kafka cluster update <cluster_ID_or_name> \
   --disk-size-autoscaling planned-usage-threshold=<scheduled_increase_percentage>,`
                          `emergency-usage-threshold=<immediate_increase_percentage>,`
                          `disk-size-limit=<maximum_storage_size_in_bytes>

    Where:

    • planned-usage-threshold: Storage utilization percentage to trigger a storage increase in the next maintenance window.

      Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

      If you set this parameter, configure the maintenance schedule.

    • emergency-usage-threshold: Storage utilization percentage to trigger an immediate storage increase.

      Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned-usage-threshold.

    • disk-size-limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

      If the value is 0, automatic increase of storage size will be disabled.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Use the Cluster.update method and send the following request, e.g., via cURL:

    Warning

    The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

    curl \
    --request PATCH \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
    --data '{
              "updateMask": "configSpec.diskSizeAutoscaling.plannedUsageThreshold,configSpec.diskSizeAutoscaling.plannedUsageThreshold,configSpec.diskSizeAutoscaling.plannedUsageThreshold",
              "configSpec": {
                "diskSizeAutoscaling": {
                  "plannedUsageThreshold": "<scheduled_increase_percentage>",
                  "emergencyUsageThreshold": "<immediate_increase_percentage>",
                  "diskSizeLimit": "<maximum_storage_size_in_bytes>"
                }
              }
            }'

    Where:

    • updateMask: List of parameters to update as a single string, separated by commas.

      Specify the relevant parameters:

      • configSpec.diskSizeAutoscaling.plannedUsageThreshold: To change the storage utilization percentage to trigger a scheduled increase.
      • configSpec.diskSizeAutoscaling.emergencyUsageThreshold: To change the storage utilization percentage to trigger a non-scheduled increase.
      • configSpec.diskSizeAutoscaling.diskSizeLimit: To change the maximum storage size during autoscaling.

    • plannedUsageThreshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

      Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

      If you set this parameter, configure the maintenance window schedule.

    • emergencyUsageThreshold: Storage utilization percentage to trigger an immediate storage increase.

      Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to plannedUsageThreshold.

    • diskSizeLimit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

    You can request the cluster ID with the list of clusters in the folder.

  3. View the server response to make sure the request was successful.

  1. Get an IAM token for API authentication and put it into the environment variable:

    export IAM_TOKEN="<IAM_token>"

  2. Clone the cloudapi repository:

    cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

    Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

  3. Use the ClusterService/Update call and send the following request, e.g., via gRPCurl:

    Warning

    The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.

    Format for listing settings
    "update_mask": {
    "paths": [
        "<setting_1>",
        "<setting_2>",
        ...
        "<setting_N>"
    ]
}

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "cluster_id": "<cluster_ID>",
          "update_mask": {
            "paths": [
              "config_spec.disk_size_autoscaling.planned_usage_threshold",
              "config_spec.disk_size_autoscaling.emergency_usage_threshold",
              "config_spec.disk_size_autoscaling.disk_size_limit"
            ]
          },
          "config_spec": {
            "disk_size_autoscaling": {
                "planned_usage_threshold": "<scheduled_increase_percentage>",
                "emergency_usage_threshold": "<immediate_increase_percentage>",
                "disk_size_limit": "<maximum_storage_size_in_bytes>"
            }
          }
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.Update

    Where:

    • update_mask: List of parameters to update as an array of paths[] strings.

      Specify the relevant parameters:

      • config_spec.disk_size_autoscaling.planned_usage_threshold: To change the storage utilization percentage to trigger a scheduled increase.
      • config_spec.disk_size_autoscaling.emergency_usage_threshold: To change the storage utilization percentage to trigger a non-scheduled increase.
      • config_spec.disk_size_autoscaling.disk_size_limit: To change the maximum storage size during autoscaling.

    • planned_usage_threshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

      Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

      If you set this parameter, configure the maintenance window schedule.

    • emergency_usage_threshold: Storage utilization percentage to trigger an immediate storage increase.

      Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned_usage_threshold.

    • disk_size_limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

    You can request the cluster ID with the list of clusters in the folder. The list of available host classes with their IDs came to you earlier.

  4. View the server response to make sure the request was successful.

If the specified threshold is reached, the storage size may increase in different ways depending on the disk type:

  • For network HDDs and SSDs, by the higher of the two values: 20 GB or 20% of the current disk size.

  • For non-replicated SSDs, by 93 GB.

  • For local SSDs, in a platform cluster:

    • Intel Cascade Lake, by 100 GB.
    • Intel Ice Lake, by 368 GB.

If the threshold is reached again, the storage size will be automatically increased until it reaches the specified maximum. After that, you can specify a new maximum storage size manually.

Warning

  • You cannot decrease the storage size.
  • While resizing the storage, cluster hosts will be unavailable.
Previous
Apache Kafka® version upgrade
Next
Stopping and starting a cluster