Creating an Apache Kafka® cluster

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a Managed Service for Apache Kafka® cluster:

1. View the description of the CLI command to create a Managed Service for Apache Kafka® cluster:

   yc managed-kafka cluster create --help
   

2. Specify the Managed Service for Apache Kafka® cluster parameters in the create command (not all parameters are given in the example):

   yc managed-kafka cluster create \
      --name <cluster_name> \
      --environment <environment> \
      --version <version> \
      --schema-registry \
      --network-name <network_name> \
      --subnet-ids <subnet_IDs> \
      --zone-ids <availability_zones> \
      --brokers-count <number_of_broker_hosts_in_zone> \
      --resource-preset <host_class> \
      --disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
      --disk-size <storage_size_in_GB> \
      --assign-public-ip <enable_public_access_to_cluster> \
      --security-group-ids <list_of_security_group_IDs> \
      --deletion-protection \
      --kafka-ui-enabled <true_or_false> \
      --disk-encryption-key-id <KMS_key_ID>
   

   Where:

   • --environment: Cluster environment, prestable or production.

   • --version: Apache Kafka® version, 3.6, 3.7, 3.8, 3.9 or 4.0. Additionally, provide the ZooKeeper host configuration.

   • --schema-registry: Manage data schemas using Managed Schema Registry.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • --zone-ids and --brokers-count: Availability zones and number of broker hosts per zone.

   • --resource-preset: Host class.

   • --disk-type: Disk type.

     Warning

     You cannot change disk type after you create a cluster.

   • --deletion-protection: Cluster protection from accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • --kafka-ui-enabled: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false.

   • --disk-encryption-key-id: ID of the custom KMS key. To encrypt the disk, provide the KMS key ID in this parameter. To learn more about disk encryption, see Storage.

     Warning

     You can enable disk encryption only when creating a cluster.

   Tip

   You can also configure the Apache Kafka® settings here, if required.

3. To use ZooKeeper in your cluster, provide the ZooKeeper host configuration:

   yc managed-kafka cluster create \
       ...
       --zookeeper-resource-preset <host_class> \
       --zookeeper-disk-size <storage_size_in_GB> \
       --zookeeper-disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
       ...
   
   

   Where:

   • --zookeeper-resource-preset: ZooKeeper host class.
   • --zookeeper-disk-size: Storage size.
   • --zookeeper-disk-type: ZooKeeper disk type.

4. To set up a maintenance window (including for disabled Managed Service for Apache Kafka® clusters), provide the required value in the --maintenance-window parameter when creating your cluster:

   yc managed-kafka cluster create \
   ...
   --maintenance-window type=<maintenance_type>,`
                       `day=<day_of_week>,`
                       `hour=<hour> \
   ...
   

   Where type is the maintenance type:

   • anytime: At any time (default).
   • weekly: On a schedule. For this value, also specify the following:
     • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: Hour of day (UTC), from 1 to 24.

5. To prevent the cluster disk space from running out, create a cluster that will increase the storage space automatically.

   yc managed-kafka cluster create \
   ...
   --disk-size-autoscaling disk-size-limit=<maximum_storage_size_in_bytes>,`
                           `planned-usage-threshold=<scheduled_increase_percentage>,`
                           `emergency-usage-threshold=<immediate_increase_percentage> \
   ...
   

   Where:

   • planned-usage-threshold: Storage utilization percentage to trigger a storage increase in the next maintenance window.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

     If you set this parameter, configure the maintenance schedule.

   • emergency-usage-threshold: Storage utilization percentage to trigger an immediate storage increase.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned-usage-threshold.

   • disk-size-limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

     If the value is 0, automatic increase of storage size will be disabled.

   Warning

   • You cannot reduce the storage size.
   • When using local disks (local-ssd), cluster hosts will be unavailable while the storage is being resized.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a Managed Service for Apache Kafka® cluster:

1. In the configuration file, describe the resources you are creating:

   • Managed Service for Apache Kafka® cluster: Description of the cluster and its hosts. You can also configure the Apache Kafka® settings here, if required.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is a configuration file example:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     environment            = "<environment>"
     name                   = "<cluster_name>"
     network_id             = "<network_ID>"
     subnet_ids             = ["<list_of_subnet_IDs>"]
     security_group_ids     = ["<list_of_cluster_security_group_IDs>"]
     deletion_protection    = <protect_cluster_against_deletion>
   
     config {
       version          = "<version>"
       zones            = ["<availability_zones>"]
       brokers_count    = <number_of_broker_hosts>
       assign_public_ip = "<enable_public_access_to_cluster>"
       schema_registry  = "<enable_data_schema_management>"
       kafka_ui {
         enabled = <use_Kafka_UI>
       }
       kafka {
         resources {
           disk_size          = <storage_size_in_GB>
           disk_type_id       = "<disk_type>"
           resource_preset_id = "<host_class>"
         }
         kafka_config {}
       }
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Cluster environment, PRESTABLE or PRODUCTION.

   • version: Apache Kafka® version, 3.6, 3.7, 3.8, 3.9 or 4.0. Additionally, provide the ZooKeeper host configuration.

   • zones and brokers_count: Availability zones and number of broker hosts per zone.

   • deletion_protection: Cluster protection against accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • disk_encryption_key_id: Disk encryption with a custom KMS key. Provide the KMS key ID to encrypt the disk. To learn more about disk encryption, see Storage.

   • assign_public_ip: Public access to the cluster, true or false.

   • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • kafka_ui: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false. The default value is false.

   To use ZooKeeper in the cluster, add the zookeeper section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     zookeeper {
       resources {
         resource_preset_id = "<host_class>"
         disk_type_id       = "<disk_type>"
         disk_size          = <storage_size_in_GB>
       }
     }
   }
   

   To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
     ...
   }
   

   Where:

   • type: Maintenance type. The possible values include:
     • ANYTIME: Anytime
     • WEEKLY: On a schedule
   • day: Day of week for the WEEKLY type, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
   • hour: UTC hour for the WEEKLY type, from 1 to 24.

   To encrypt disks with a custom KMS key, add the disk_encryption_key_id parameter:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
       ...
       disk_encryption_key_id = <KMS_key_ID>
       ...
   }
   

   To learn more about disk encryption, see Storage.

2. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Create a Managed Service for Apache Kafka® cluster.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   This will create all the resources you need in the specified folder, and the terminal will display the FQDNs of the Managed Service for Apache Kafka® cluster hosts. You can check the new resources and their configuration using the management console.

For more information, see this Terraform provider guide.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Call the Cluster.create method, e.g., via the following cURL request:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folderId": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "networkId": "<network_ID>",
        "securityGroupIds": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "configSpec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resourcePresetId": "<Apache Kafka®_host_class>",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"
            }
          },
          "zookeeper": {
            "resources": {
              "resourcePresetId": "<ZooKeeper_host_class>",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"                   
            }
          },
          "zoneId": [
            <list_of_availability_zones>
          ],
          "brokersCount": "<number_of_brokers_in_zone>",
          "assignPublicIp": <enable_public_access_to_cluster>,
          "schemaRegistry": <enable_data_schema_management>,
          "restApiConfig": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "diskSizeAutoscaling": {
            <automatic_storage_expansion_parameters>
          },
          "kafkaUiConfig": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topicSpecs": [
          {
            "name": "<topic_name>",
            "partitions": "<number_of_partitions>",
            "replicationFactor": "<replication_factor>"
          },
          { <similar_settings_for_topic_2> },
          { ... },
          { <similar_settings_for_topic_N> }
        ],
        "userSpecs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topicName": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenanceWindow": {
          "anytime": {},
          "weeklyMaintenanceWindow": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletionProtection": <protect_cluster_against_deletion>,
        "diskEncryptionKeyId": "<KMS_key_ID>"
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • networkId: ID of the network where the cluster will be deployed.

      • securityGroupIds: Security group IDs as an array of strings. Each string is a security group ID.

      • configSpec: Cluster configuration:

        • version: Apache Kafka® version, 3.6, 3.7, 3.8, 3.9 or 4.0. Additionally, provide the ZooKeeper host configuration.

        • kafka: Apache Kafka® configuration:

          • resources.resourcePresetId: Host class ID. You can get the list of available host classes with their IDs by calling the ResourcePreset.list method.
          • resources.diskSize: Disk size, in bytes.
          • resources.diskTypeId: Disk type.

        • zookeeper: ZooKeeper configuration:

          • resources.resourcePresetId: Host class ID. You can get the list of available host classes with their IDs using the ResourcePreset.list method.
          • resources.diskSize: Disk size, in bytes.
          • resources.diskTypeId: Disk type.

        • zoneId and brokersCount: Availability zones and number of broker hosts per zone.

        • assignPublicIp: Access to broker hosts from the internet, true or false.

        • schemaRegistry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create the Managed Service for Apache Kafka® cluster.

        • restApiConfig: Apache Kafka® REST API configuration. To enable sending Apache Kafka® REST API requests, specify enabled: true.

        • diskSizeAutoscaling: Storage utilization thresholds (as a percentage of the total storage size) that will trigger storage expansion when reached:

          • plannedUsageThreshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergencyUsageThreshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to plannedUsageThreshold.

          • diskSizeLimit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafkaUiConfig: Use Kafka UI. For access to Kafka UI, specify enabled: true.

      • topicSpecs: Topic settings as an array of elements, one per topic. Each element has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions.

        • replicationFactor: Replication factor.

      • userSpecs: User settings as an array of elements, one per user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topicName: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allowHosts: (Optional) List of IP addresses the user is allowed to access the topic from.

      • maintenanceWindow: Maintenance window settings, including for stopped clusters. Select one of these options:

        • anytime: Any time (default).
        • weeklyMaintenanceWindow: On schedule:
          • day: Day of week in DDD format, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletionProtection: Cluster protection against accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      • diskEncryptionKeyId: ID of the custom KMS key. Provide the KMS key ID to encrypt the disks. To learn more about disk encryption, see Storage.

        Warning

        You can enable disk encryption only when creating a cluster.

      You can get the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

3. View the server response to make sure your request was successful.

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Call the ClusterService/Create method, e.g., via the following gRPCurl request:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folder_id": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "network_id": "<network_ID>",
        "security_group_ids": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "config_spec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resource_preset_id": "<Apache Kafka®_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"
            }
          },
          "zookeeper": {
            "resources": {
              "resource_preset_id": "<ZooKeeper_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"                   
            }
          },
          "zone_id": [
            <list_of_availability_zones>
          ],
          "brokers_count": {
            "value": "<number_of_brokers_in_zone>"
          },
          "assign_public_ip": <enable_public_access_to_cluster>,
          "schema_registry": <enable_data_schema_management>,
          "rest_api_config": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "disk_size_autoscaling": {
            <automatic_storage_size_increase_parameters>
          },
          "kafka_ui_config": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topic_specs": [
          {
            "name": "<topic_name>",
            "partitions": {
              "value": "<number_of_partitions>"
            },
            "replication_factor": {
              "value": "<replication_factor>"
            }
          },
          { <similar_settings_for_topic_2> },
          { ... },
          { <similar_settings_for_topic_N> }
        ],
        "user_specs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topic_name": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenance_window": {
          "anytime": {},
          "weekly_maintenance_window": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletion_protection": <protect_cluster_against_deletion>,
        "disk_encryption_key_id": "<KMS_key_ID>"
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • network_id: ID of the network where the cluster will be deployed.

      • security_group_ids: Security group IDs as an array of strings. Each string is a security group ID.

      • config_spec: Cluster configuration:

        • version: Apache Kafka® version, 3.6, 3.7, 3.8, 3.9 or 4.0. Additionally, provide the ZooKeeper host configuration.

        • kafka: Apache Kafka® configuration:

          • resources.resource_preset_id: Host class ID. You can get the list of available host classes with their IDs by calling the ResourcePreset.list method.
          • resources.disk_size: Disk size, in bytes.
          • resources.disk_type_id: Disk type.

        • zookeeper: ZooKeeper configuration:

          • resources.resource_preset_id: Host class ID. You can get the list of available host classes with their IDs by calling the ResourcePreset.list method.
          • resources.disk_size: Disk size, in bytes.
          • resources.disk_type_id: Disk type.

        • zone_id and brokers_count: Availability zones and number of broker hosts (provided as an object with the value field) per zone.

        • assign_public_ip: Access to broker hosts from the internet, true or false.

        • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create the Managed Service for Apache Kafka® cluster.

        • rest_api_config: Apache Kafka® REST API configuration. To enable sending Apache Kafka® REST API requests, specify enabled: true.

        • disk_size_autoscaling: To prevent the cluster disk space from running out, set the storage utilization thresholds (as a percentage of the total storage size) that will trigger storage expansion when reached:

          • planned_usage_threshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergency_usage_threshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned_usage_threshold.

          • disk_size_limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafka_ui_config: Use Kafka UI. For access to Kafka UI, specify enabled: true.

      • topic_specs: Topic settings as an array of elements. Each element is for a separate topic and has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions, provided as an object with a field named value.

        • replication_factor: Replication factor. Provided as an object with the value field.

      • user_specs: User settings as an array of elements, one per user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topic_name: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allow_hosts: (Optional) List of IP addresses the user is allowed to access the topic from, as an array of elements.

      • maintenance_window: Maintenance window settings, including for stopped clusters. Select one of these options:

        • anytime: Any time (default).
        • weekly_maintenance_window: On schedule:
          • day: Day of week in DDD format, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletion_protection: Cluster protection against accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      • disk_encryption_key_id: ID of the custom KMS key. Provide the KMS key ID to encrypt the disks. To learn more about disk encryption, see Storage.

        Warning

        You can enable disk encryption only when creating a cluster.

      You can get the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

4. View the server response to make sure your request was successful.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a Managed Service for Apache Kafka® cluster:

1. View the description of the CLI command to create a Managed Service for Apache Kafka® cluster:

   yc managed-kafka cluster create --help
   

2. Specify the Managed Service for Apache Kafka® cluster parameters in the create command (not all parameters are given in the example):

   yc managed-kafka cluster create \
      --name <cluster_name> \
      --environment <environment> \
      --version <version> \
      --schema-registry \
      --network-name <network_name> \
      --subnet-ids <subnet_IDs> \
      --resource-preset <host_class> \
      --disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
      --disk-size <storage_size_in_GB> \
      --assign-public-ip <enable_public_access_to_cluster> \
      --security-group-ids <list_of_security_group_IDs> \
      --deletion-protection \
      --kafka-ui-enabled <true_or_false> \
      --disk-encryption-key-id <KMS_key_ID>
   

   Where:

   • --environment: Cluster environment, prestable or production.

   • --version: Apache Kafka® version. Specify 3.6 or higher.

   • --schema-registry: Manage data schemas using Managed Schema Registry.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • --zone-ids and --brokers-count: Availability zones and number of broker hosts per zone.

     If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

     • --zone-ids=ru-central1-a,ru-central1-b,ru-central1-d --brokers-count=1: Three availability zones with one broker host per zone.
     • --zone-ids=<one_availability_zone> --brokers-count=3: One availability zone with three broker hosts.

   • --resource-preset: Host class.

   • --disk-type: Disk type.

     Warning

     You cannot change disk type after you create a cluster.

   • --deletion-protection: Cluster protection from accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • --kafka-ui-enabled: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false.

   • --disk-encryption-key-id: ID of the custom KMS key. To encrypt the disk, provide the KMS key ID in this parameter. To learn more about disk encryption, see Storage.

     Warning

     You can enable disk encryption only when creating a cluster.

   Tip

   You can also configure the Apache Kafka® settings here, if required.

3. To use KRaft (on separate hosts), provide the KRaft host configuration.

   yc managed-kafka cluster create \
       ...
       --controller-resource-preset <host_class> \
       --controller-disk-size <storage_size_in_GB> \
       --controller-disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
       ...
   
   

   Where:

   • --controller-resource-preset: KRaft host class.
   • --controller-disk-size: Storage size.
   • --controller-disk-type: KRaft disk type.

4. To set up a maintenance window (including for disabled Managed Service for Apache Kafka® clusters), provide the required value in the --maintenance-window parameter when creating your cluster:

   yc managed-kafka cluster create \
   ...
   --maintenance-window type=<maintenance_type>,`
                       `day=<day_of_week>,`
                       `hour=<hour> \
   ...
   

   Where type is the maintenance type:

   • anytime: At any time (default).
   • weekly: On a schedule. For this value, also specify the following:
     • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: Hour of day (UTC), from 1 to 24.

5. To prevent the cluster disk space from running out, create a cluster that will increase the storage space automatically.

   yc managed-kafka cluster create \
   ...
   --disk-size-autoscaling disk-size-limit=<maximum_storage_size_in_bytes>,`
                           `planned-usage-threshold=<scheduled_increase_percentage>,`
                           `emergency-usage-threshold=<immediate_increase_percentage> \
   ...
   

   Where:

   • planned-usage-threshold: Storage utilization percentage to trigger a storage increase in the next maintenance window.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

     If you set this parameter, configure the maintenance schedule.

   • emergency-usage-threshold: Storage utilization percentage to trigger an immediate storage increase.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned-usage-threshold.

   • disk-size-limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

     If the value is 0, automatic increase of storage size will be disabled.

   Warning

   • You cannot reduce the storage size.
   • When using local disks (local-ssd), cluster hosts will be unavailable while the storage is being resized.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a Managed Service for Apache Kafka® cluster:

1. In the configuration file, describe the resources you are creating:

   • Managed Service for Apache Kafka® cluster: Description of the cluster and its hosts. You can also configure the Apache Kafka® settings here, if required.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is a configuration file example:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     environment         = "<environment>"
     name                = "<cluster_name>"
     network_id          = "<network_ID>"
     subnet_ids          = ["<list_of_subnet_IDs>"]
     security_group_ids  = ["<list_of_cluster_security_group_IDs>"]
     deletion_protection = <protect_cluster_against_deletion>
   
     config {
       version          = "<version>"
       zones            = ["<availability_zones>"]
       brokers_count    = <number_of_broker_hosts>
       assign_public_ip = "<enable_public_access_to_cluster>"
       schema_registry  = "<enable_data_schema_management>"
       kafka_ui {
         enabled = <use_Kafka_UI>
       }
       kafka {
         resources {
           disk_size          = <storage_size_in_GB>
           disk_type_id       = "<disk_type>"
           resource_preset_id = "<host_class>"
         }
         kafka_config {}
       }
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Cluster environment, PRESTABLE or PRODUCTION.

   • version: Apache Kafka® version. Specify 3.6 or higher.

   • zones and brokers_count: Availability zones and number of broker hosts per zone.

     If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

     • zones = ["ru-central1-a","ru-central1-b","ru-central1-d"] brokers_count = 1: Three availability zones with one broker host per zone.
     • zones = ["<one_availability_zone>"] brokers_count = 3: One availability zone with three broker hosts.

   • deletion_protection: Cluster protection against accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • assign_public_ip: Public access to the cluster, true or false.

   • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • kafka_ui: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false. The default value is false.

   Add the kraft section to the cluster description to use KRaft (on separate hosts).

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     kraft {
       resources {
         resource_preset_id = "<host_class>"
         disk_type_id       = "<disk_type>"
         disk_size          = <storage_size_in_GB>
       }
     }
   }
   

   To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
     ...
   }
   

   Where:

   • type: Maintenance type. The possible values include:
     • ANYTIME: Anytime
     • WEEKLY: On a schedule
   • day: Day of week for the WEEKLY type, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
   • hour: UTC hour for the WEEKLY type, from 1 to 24.

   To encrypt disks with a custom KMS key, add the disk_encryption_key_id parameter:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
       ...
       disk_encryption_key_id = <KMS_key_ID>
       ...
   }
   

   To learn more about disk encryption, see Storage.

2. Make sure the settings are correct.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Create a Managed Service for Apache Kafka® cluster.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   This will create all the resources you need in the specified folder, and the terminal will display the FQDNs of the Managed Service for Apache Kafka® cluster hosts. You can check the new resources and their configuration using the management console.

For more information, see this Terraform provider guide.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Call the Cluster.create method, e.g., via the following cURL request:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folderId": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "networkId": "<network_ID>",
        "securityGroupIds": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "configSpec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resourcePresetId": "<Apache Kafka®_host_class>",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"
            }
          },
          "kraft": {
            "resources": {
              "resourcePresetId": "<KRaft_host_class>",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"                   
            }
          },
          "zoneId": [
            <list_of_availability_zones>
          ],
          "brokersCount": "<number_of_brokers_in_zone>",
          "assignPublicIp": <enable_public_access_to_cluster>,
          "schemaRegistry": <enable_data_schema_management>,
          "restApiConfig": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "diskSizeAutoscaling": {
            <automatic_storage_expansion_parameters>
          },
          "kafkaUiConfig": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topicSpecs": [
          {
            "name": "<topic_name>",
            "partitions": "<number_of_partitions>",
            "replicationFactor": "<replication_factor>"
          },
          { <similar_settings_for_topic_2> },
          { ... },
          { <similar_settings_for_topic_N> }
        ],
        "userSpecs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topicName": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenanceWindow": {
          "anytime": {},
          "weeklyMaintenanceWindow": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletionProtection": <protect_cluster_against_deletion>,
        "diskEncryptionKeyId": "<KMS_key_ID>"
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • networkId: ID of the network where the cluster will be deployed.

      • securityGroupIds: Security group IDs as an array of strings. Each string is a security group ID.

      • configSpec: Cluster configuration:

        • version: Apache Kafka® version. Specify 3.6 or higher.

        • kafka: Apache Kafka® configuration:

          • resources.resourcePresetId: Host class ID. You can get the list of available host classes with their IDs by calling the ResourcePreset.list method.
          • resources.diskSize: Disk size, in bytes.
          • resources.diskTypeId: Disk type.

        • kraft: KRaft configuration:

          • resources.resourcePresetId: Host class ID. You can get the list of available host classes with their IDs using the ResourcePreset.list method.
          • resources.diskSize: Disk size, in bytes.
          • resources.diskTypeId: Disk type.

          Warning

          If you are creating a KRaft (combined mode) cluster, do not provide the KRaft host configuration.

        • zoneId and brokersCount: Availability zones and number of broker hosts per zone.

          If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

          • "zoneId": ["ru-central1-a","ru-central1-b","ru-central1-d"], "brokersCount": "1": Three availability zones with one broker host per zone.
          • "zoneId": ["<one_availability_zone>"], "brokersCount": "3": One availability zone with three broker hosts.

        • assignPublicIp: Access to broker hosts from the internet, true or false.

        • schemaRegistry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster.

        • restApiConfig: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify enabled: true.

        • diskSizeAutoscaling: Storage utilization thresholds (as a percentage of the total storage size) that will trigger storage expansion when reached:

          • plannedUsageThreshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergencyUsageThreshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to plannedUsageThreshold.

          • diskSizeLimit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafkaUiConfig: Use Kafka UI. For access to Kafka UI, specify enabled: true.

      • topicSpecs: Topic settings as an array of elements, one per topic. Each element has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions.

        • replicationFactor: Replication factor.

      • userSpecs: User settings as an array of elements, one per user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topicName: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allowHosts: (Optional) List of IP addresses the user is allowed to access the topic from.

      • maintenanceWindow: Maintenance window settings, including for stopped clusters. Select one of these options:

        • anytime: At any time (default).
        • weeklyMaintenanceWindow: On schedule:
          • day: Day of week in DDD format, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletionProtection: Cluster protection against accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      • diskEncryptionKeyId: ID of the custom KMS key. Provide the KMS key ID to encrypt the disks. To learn more about disk encryption, see Storage.

        Warning

        You can enable disk encryption only when creating a cluster.

      You can get the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

3. View the server response to make sure your request was successful.

1. Get an IAM token for API authentication and put it into an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

3. Call the ClusterService/Create method, e.g., via the following gRPCurl request:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folder_id": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "network_id": "<network_ID>",
        "security_group_ids": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "config_spec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resource_preset_id": "<Apache Kafka®_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"
            }
          },
          "kraft": {
            "resources": {
              "resource_preset_id": "<KRaft_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"                   
            }
          },
          "zone_id": [
            <list_of_availability_zones>
          ],
          "brokers_count": {
            "value": "<number_of_brokers_in_zone>"
          },
          "assign_public_ip": <enable_public_access_to_cluster>,
          "schema_registry": <enable_data_schema_management>,
          "rest_api_config": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "disk_size_autoscaling": {
            <automatic_storage_size_increase_parameters>
          },
          "kafka_ui_config": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topic_specs": [
          {
            "name": "<topic_name>",
            "partitions": {
              "value": "<number_of_partitions>"
            },
            "replication_factor": {
              "value": "<replication_factor>"
            }
          },
          { <similar_settings_for_topic_2> },
          { ... },
          { <similar_settings_for_topic_N> }
        ],
        "user_specs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topic_name": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenance_window": {
          "anytime": {},
          "weekly_maintenance_window": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletion_protection": <protect_cluster_against_deletion>,
        "disk_encryption_key_id": "<KMS_key_ID>"
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • network_id: ID of the network where the cluster will be deployed.

      • security_group_ids: Security group IDs as an array of strings. Each string is a security group ID.

      • config_spec: Cluster configuration:

        • version: Apache Kafka® version. Specify 3.6 or higher.

        • kafka: Apache Kafka® configuration:

          • resources.resource_preset_id: Host class ID. You can get the list of available host classes with their IDs by calling the ResourcePreset.list method.
          • resources.disk_size: Disk size, in bytes.
          • resources.disk_type_id: Disk type.

        • kraft: KRaft configuration:

          • resources.resource_preset_id: Host class ID. You can get the list of available host classes with their IDs by calling the ResourcePreset.list method.
          • resources.disk_size: Disk size, in bytes.
          • resources.disk_type_id: Disk type.

          Warning

          If you are creating a KRaft (combined mode) cluster, do not provide the KRaft host configuration.

        • zone_id and brokers_count: Availability zones and number of broker hosts (provided as an object with the value field) per zone.

          If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

          • "zone_id": ["ru-central1-a","ru-central1-b","ru-central1-d"], "brokers_count": {"value":"1"}: Three availability zones with one broker host per zone.
          • "zone_id": ["<one_availability_zone>"], "brokers_count": {"value":"3"}: One availability zone with three broker hosts.

        • assign_public_ip: Access to broker hosts from the internet, true or false.

        • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create the Managed Service for Apache Kafka® cluster.

        • rest_api_config: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify enabled: true.

        • disk_size_autoscaling: To prevent the cluster disk space from running out, set the storage utilization thresholds (as a percentage of the total storage size) that will trigger storage expansion when reached:

          • planned_usage_threshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergency_usage_threshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned_usage_threshold.

          • disk_size_limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafka_ui_config: Use Kafka UI. For access to Kafka UI, specify enabled: true.

      • topic_specs: Topic settings as an array of elements. Each element is for a separate topic and has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions, provided as an object with a field named value.

        • replication_factor: Replication factor. Provided as an object with the value field.

      • user_specs: User settings as an array of elements, one per user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topic_name: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allow_hosts: (Optional) List of IP addresses the user is allowed to access the topic from, as an array of elements.

      • maintenance_window: Maintenance window settings, including for stopped clusters. Select one of these options:

        • anytime: At any time (default).
        • weekly_maintenance_window: On schedule:
          • day: Day of week in DDD format, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletion_protection: Cluster protection against accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      • disk_encryption_key_id: ID of the custom KMS key. Provide the KMS key ID to encrypt the disks. To learn more about disk encryption, see Storage.

        Warning

        You can enable disk encryption only when creating a cluster.

      You can get the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

4. View the server response to make sure your request was successful.

Create a Managed Service for Apache Kafka® cluster with the following test specifications:

• Cloud ID: b1gq90dgh25bebiu75o.

• Folder ID: b1gia87mbaomkfvsleds.

• Name: mykf.

• Environment: PRODUCTION.

• Apache Kafka® version: 3.9.

• New network: mynet (new one), subnet: mysubnet, address range: 10.5.0.0/24.

• Security group: mykf-sg allowing internet connections to the Managed Service for Apache Kafka® cluster on port 9091.

• Broker host class: s2.micro (one host), availability zone: ru-central1-a.

• Network SSD storage (network-ssd): 10 GB.

• Public access: Enabled.

• Deletion protection: Enabled.

The configuration file for this Managed Service for Apache Kafka® cluster is as follows:

resource "yandex_mdb_kafka_cluster" "mykf" {
  environment         = "PRODUCTION"
  name                = "mykf"
  network_id          = yandex_vpc_network.mynet.id
  subnet_ids          = [ yandex_vpc_subnet.mysubnet.id ]
  security_group_ids  = [ yandex_vpc_security_group.mykf-sg.id ]
  deletion_protection = true

  config {
    assign_public_ip = true
    brokers_count    = 1
    version          = "3.9"
    kafka {
      resources {
        disk_size          = 10
        disk_type_id       = "network-ssd"
        resource_preset_id = "s2.micro"
      }
      kafka_config {}
    }

    zones = [
      "ru-central1-a"
    ]
  }
}

resource "yandex_vpc_network" "mynet" {
  name = "mynet"
}

resource "yandex_vpc_subnet" "mysubnet" {
  name           = "mysubnet"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.mynet.id
  v4_cidr_blocks = ["10.5.0.0/24"]
}

resource "yandex_vpc_security_group" "mykf-sg" {
  name       = "mykf-sg"
  network_id = yandex_vpc_network.mynet.id

  ingress {
    description    = "Kafka"
    port           = 9091
    protocol       = "TCP"
    v4_cidr_blocks = [ "0.0.0.0/0" ]
  }
}

Create a Managed Service for Apache Kafka® cluster with the following test specifications:

• Name: kafka-kraft.

• Environment: production.

• Apache Kafka® version: 3.9.

• New network: kafka-net, subnets in each availability zone:

  • kafka-subnet-a with the 10.1.0.0/24 address range.
  • kafka-subnet-b with the 10.2.0.0/24 address range.
  • kafka-subnet-d with the 10.3.0.0/24 address range.

• One broker host in each of the following availability zones:

  • ru-central1-a
  • ru-central1-b
  • ru-central1-d

• Host class: s2.micro.

• Network HDD storage (network-hdd): 10 GB.

• Public access: Enabled.

• Security group: kafka-sg allowing all incoming and outgoing traffic.

The configuration file for this Managed Service for Apache Kafka® cluster is as follows:

resource "yandex_mdb_kafka_cluster" "kafka-kraft" {
  name                = "kafka-kraft"
  environment         = "PRODUCTION"
  network_id          = yandex_vpc_network.kafka-net.id
  subnet_ids          = [yandex_vpc_subnet.kafka-subnet-a.id, yandex_vpc_subnet.kafka-subnet-b.id, yandex_vpc_subnet.kafka-subnet-d.id]
  security_group_ids  = [yandex_vpc_security_group.kafka-sg.id]

  config {
    version          = "3.9"
    brokers_count    = 1
    zones            = ["ru-central1-a", "ru-central1-b", "ru-central1-d"]
    assign_public_ip = true
    
    kafka {
      resources {
        disk_size          = 10
        disk_type_id       = "network-hdd"
        resource_preset_id = "s2.micro"
      }

      kafka_config {}
    }
  }
}

resource "yandex_vpc_network" "kafka-net" {
  name = "kafka-net"
}

resource "yandex_vpc_subnet" "kafka-subnet-a" {
  name           = "kafka-subnet-a"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.1.0.0/24"]
}

resource "yandex_vpc_subnet" "kafka-subnet-b" {
  name           = "kafka-subnet-b"
  zone           = "ru-central1-b"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.2.0.0/24"]
}

resource "yandex_vpc_subnet" "kafka-subnet-d" {
  name           = "kafka-subnet-d"
  zone           = "ru-central1-d"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.3.0.0/24"]
}

resource "yandex_vpc_security_group" "kafka-sg" {
  name       = "kafka-sg"
  network_id = yandex_vpc_network.kafka-net.id

  ingress {
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

Create a Managed Service for Apache Kafka® cluster with the following test specifications:

• Name: kafka-kraft-mh.

• Environment: production.

• Apache Kafka® version: 3.9.

• New network: kafka-net, subnets in each availability zone:

  • kafka-subnet-a with the 10.1.0.0/24 address range.
  • kafka-subnet-b with the 10.2.0.0/24 address range.
  • kafka-subnet-d with the 10.3.0.0/24 address range.

• Hosts: Two broker hosts in the ru-central1-a availability zone.

• Host class: s2.micro.

• Network HDD storage (network-hdd): 10 GB.

• KRaft controller host class: s2.micro.

• KRaft controller network SSD storage (network-ssd): 10 GB.

• Public access: Enabled.

• Security group: kafka-sg allowing all incoming and outgoing traffic.

The configuration file for this Managed Service for Apache Kafka® cluster is as follows:

resource "yandex_mdb_kafka_cluster" "kafka-kraft-mh" {
  name                = "kafka-kraft-mh"
  environment         = "PRODUCTION"
  network_id          = yandex_vpc_network.kafka-net.id
  subnet_ids          = [yandex_vpc_subnet.kafka-subnet-a.id,yandex_vpc_subnet.kafka-subnet-b.id,yandex_vpc_subnet.kafka-subnet-d.id]
  security_group_ids  = [yandex_vpc_security_group.kafka-sg.id]

  config {
    version          = "3.9"
    brokers_count    = 2
    zones            = ["ru-central1-a"]
    assign_public_ip = true
    
    kafka {
      resources {
        disk_size          = 10
        disk_type_id       = "network-hdd"
        resource_preset_id = "s2.micro"
      }

      kafka_config {}
    }

    kraft {
      resources {
        resource_preset_id = "s2.micro"
        disk_type_id       = "network-ssd"
        disk_size          = 10
      }
    }
  }
}

resource "yandex_vpc_network" "kafka-net" {
  name = "kafka-net"
}

resource "yandex_vpc_subnet" "kafka-subnet-a" {
  name           = "kafka-subnet-a"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.1.0.0/24"]
}

resource "yandex_vpc_subnet" "kafka-subnet-b" {
  name           = "kafka-subnet-b"
  zone           = "ru-central1-b"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.2.0.0/24"]
}

resource "yandex_vpc_subnet" "kafka-subnet-d" {
  name           = "kafka-subnet-d"
  zone           = "ru-central1-d"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.3.0.0/24"]
}

resource "yandex_vpc_security_group" "kafka-sg" {
  name       = "kafka-sg"
  network_id = yandex_vpc_network.kafka-net.id

  ingress {
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}

Create a Managed Service for Apache Kafka® cluster with the following test specifications:

• Name: kafka-zk-mh.

• Environment: production.

• Apache Kafka® version: 3.9.

• New network: kafka-net, subnets in each availability zone:

  • kafka-subnet-a with the 10.1.0.0/24 address range.
  • kafka-subnet-b with the 10.2.0.0/24 address range.
  • kafka-subnet-d with the 10.3.0.0/24 address range.

• Hosts: Two broker hosts in the ru-central1-a availability zone.

• Host class: s2.micro.

• Network HDD storage (network-hdd): 10 GB.

• ZooKeeper host class: s2.micro.

• ZooKeeper network SSD storage (network-ssd): 10 GB.

• Public access: Enabled.

• Security group: kafka-sg allowing all incoming and outgoing traffic.

The configuration file for this Managed Service for Apache Kafka® cluster is as follows:

resource "yandex_mdb_kafka_cluster" "kafka-zk-mh" {
  name                = "kafka-zk-mh"
  environment         = "PRODUCTION"
  network_id          = yandex_vpc_network.kafka-net.id
  subnet_ids          = [yandex_vpc_subnet.kafka-subnet-a.id,yandex_vpc_subnet.kafka-subnet-b.id,yandex_vpc_subnet.kafka-subnet-d.id]
  security_group_ids  = [yandex_vpc_security_group.kafka-sg.id]

  config {
    version          = "3.9"
    brokers_count    = 2
    zones            = ["ru-central1-a"]
    assign_public_ip = true
    
    kafka {
      resources {
        disk_size          = 10
        disk_type_id       = "network-hdd"
        resource_preset_id = "s2.micro"
      }

      kafka_config {}
    }

    zookeeper {
      resources {
        resource_preset_id = "s2.micro"
        disk_type_id       = "network-ssd"
        disk_size          = 10
      }
    }
  }
}

resource "yandex_vpc_network" "kafka-net" {
  name = "kafka-net"
}

resource "yandex_vpc_subnet" "kafka-subnet-a" {
  name           = "kafka-subnet-a"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.1.0.0/24"]
}

resource "yandex_vpc_subnet" "kafka-subnet-b" {
  name           = "kafka-subnet-b"
  zone           = "ru-central1-b"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.2.0.0/24"]
}

resource "yandex_vpc_subnet" "kafka-subnet-d" {
  name           = "kafka-subnet-d"
  zone           = "ru-central1-d"
  network_id     = yandex_vpc_network.kafka-net.id
  v4_cidr_blocks = ["10.3.0.0/24"]
}

resource "yandex_vpc_security_group" "kafka-sg" {
  name       = "kafka-sg"
  network_id = yandex_vpc_network.kafka-net.id

  ingress {
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }

  egress {
    protocol       = "ANY"
    v4_cidr_blocks = ["0.0.0.0/0"]
  }
}