Creating an Apache Kafka® cluster

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a Managed Service for Apache Kafka® cluster:

1. See the description of the CLI command for creating a Managed Service for Apache Kafka® cluster:

   yc managed-kafka cluster create --help
   

2. Specify the Managed Service for Apache Kafka® cluster parameters in the create command (not all parameters are given in the example):

   yc managed-kafka cluster create \
      --name <cluster_name> \
      --environment <environment> \
      --version <version> \
      --schema-registry \
      --network-name <network_name> \
      --subnet-ids <subnet_IDs> \
      --zone-ids <availability_zones> \
      --brokers-count <number_of_broker_hosts_in_zone> \
      --resource-preset <host_class> \
      --disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
      --disk-size <storage_size_in_GB> \
      --assign-public-ip <enable_public_access_to_cluster> \
      --security-group-ids <list_of_security_group_IDs> \
      --deletion-protection \
      --kafka-ui-enabled <true_or_false>
   

   Where:

   • --environment: Cluster environment, prestable or production.

   • --version: Apache Kafka® version, 3.5 or 3.6. Additionally, provide the ZooKeeper host configuration.

   • --schema-registry: Manage data schemas using Managed Schema Registry.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • --zone-ids and --brokers-count: Availability zones and number of broker hosts per zone.

   • --resource-preset: Host class.

   • --disk-type: Disk type.

     Warning

     You cannot change disk type after you create a cluster.

   • --deletion-protection: Cluster protection from accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • --kafka-ui-enabled: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false.

   Tip

   You can also configure the Apache Kafka® settings here, if required.

3. To use ZooKeeper in the cluster, provide the ZooKeeper host configuration:

   yc managed-kafka cluster create \
       ...
       --zookeeper-resource-preset <host_class> \
       --zookeeper-disk-size <storage_size_in_GB> \
       --zookeeper-disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
       ...
   
   

   Where:

   • --zookeeper-resource-preset: ZooKeeper host class.
   • --zookeeper-disk-size: Storage size.
   • --controller-disk-type: ZooKeeper disk type.

4. To set up a maintenance window (including for disabled Managed Service for Apache Kafka® clusters), provide the required value in the --maintenance-window parameter when creating your cluster:

   yc managed-kafka cluster create \
   ...
   --maintenance-window type=<maintenance_type>,`
                       `day=<day_of_week>,`
                       `hour=<hour> \
   ...
   

   Where type is the maintenance type:

   • anytime: At any time (default).
   • weekly: On a schedule. For this value, also specify the following:
     • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: Hour of day (UTC), from 1 to 24.

5. To prevent the cluster disk space from running out, create a cluster that will increase the storage space automatically.

   yc managed-kafka cluster create \
   ...
   --disk-size-autoscaling disk-size-limit=<maximum_storage_size_in_bytes>,`
                           `planned-usage-threshold=<scheduled_increase_percentage>,`
                           `emergency-usage-threshold=<immediate_increase_percentage> \
   ...
   

   Where:

   • planned-usage-threshold: Storage utilization percentage to trigger a storage increase in the next maintenance window.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

     If you set this parameter, configure the maintenance schedule.

   • emergency-usage-threshold: Storage utilization percentage to trigger an immediate storage increase.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned-usage-threshold.

   • disk-size-limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

     If the value is 0, automatic increase of storage size will be disabled.

   Warning

   • You cannot decrease the storage size.
   • While resizing the storage, cluster hosts will be unavailable.

6. To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, specify the IDs of these hosts as a comma-separated list in the --host-group-ids parameter:

   yc managed-kafka cluster create \
   ...
   --host-group-ids <dedicated_host_group_IDs> \
   ...
   

   Warning

   You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a Managed Service for Apache Kafka® cluster:

1. In the configuration file, describe the resources you are creating:

   • Managed Service for Apache Kafka® cluster: Description of the cluster and its hosts. You can also configure the Apache Kafka® settings here, if required.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     environment         = "<environment>"
     name                = "<cluster_name>"
     network_id          = "<network_ID>"
     subnet_ids          = ["<list_of_subnet_IDs>"]
     security_group_ids  = ["<list_of_cluster_security_group_IDs>"]
     deletion_protection = <protect_cluster_from_deletion>
   
     config {
       version          = "<version>"
       zones            = ["<availability_zones>"]
       brokers_count    = <number_of_broker_hosts>
       assign_public_ip = "<enable_public_access_to_cluster>"
       schema_registry  = "<enable_data_schema_management>"
       kafka_ui {
         enabled = <use_Kafka_UI>
       }
       kafka {
         resources {
           disk_size          = <storage_size_in_GB>
           disk_type_id       = "<disk_type>"
           resource_preset_id = "<host_class>"
         }
         kafka_config {}
       }
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Cluster environment, PRESTABLE or PRODUCTION.

   • version: Apache Kafka® version, 3.5 or 3.6. Additionally, provide the ZooKeeper host configuration.

   • zones and brokers_count: Availability zones and number of broker hosts per zone.

   • deletion_protection: Cluster protection from accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • assign_public_ip: Public access to the cluster, true or false.

   • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • kafka_ui: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false. The default value is false.

   Add the zookeeper section to the cluster description to use ZooKeeper in the cluster:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     zookeeper {
       resources {
         resource_preset_id = "<host_class>"
         disk_type_id       = "<disk_type>"
         disk_size          = <storage_size_in_GB>
       }
     }
   }
   

   To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
     ...
   }
   

   Where:

   • type: Maintenance type. The possible values include:
     • ANYTIME: Anytime
     • WEEKLY: On a schedule
   • day: Day of week for the WEEKLY type, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
   • hour: UTC hour for the WEEKLY type, from 1 to 24.

2. Validate your configuration.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Create a Managed Service for Apache Kafka® cluster.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   After this, all required resources will be created in the specified folder, and the FQDNs of the Managed Service for Apache Kafka® cluster hosts will be displayed in the terminal. You can check the new resources and their configuration in the management console.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.create method and send the following request, e.g., via cURL:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folderId": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "networkId": "<network_ID>",
        "securityGroupIds": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "configSpec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resourcePresetId": "<Apache Kafka®_host_class>",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"
            }
          },
          "zookeeper": {
            "resources": {
              "resourcePresetId": "<ZooKeeper>_host_class",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"                   
            }
          },
          "zoneId": [
            <list_of_availability_zones>
          ],
          "brokersCount": "<number_of_brokers_in_zone>",
          "assignPublicIp": <enable_public_access_to_cluster>,
          "schemaRegistry": <enable_data_schema_management>,
          "restApiConfig": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "diskSizeAutoscaling": {
            <automatic_storage_size_increase_parameters>
          },
          "kafkaUiConfig": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topicSpecs": [
          {
            "name": "<topic_name>",
            "partitions": "<number_of_partitions>",
            "replicationFactor": "<replication_factor>"
          },
          { <similar_list_of_settings_for_topic_2> },
          { ... },
          { <similar_list_of_settings_for_topic_N> }
        ],
        "userSpecs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topicName": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenanceWindow": {
          "anytime": {},
          "weeklyMaintenanceWindow": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletionProtection": <protect_cluster_from_deletion>
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • networkId: ID of the network the cluster will be in.

      • securityGroupIds: Security group IDs as an array of strings. Each string is a security group ID.

      • configSpec: Cluster configuration:

        • version: Apache Kafka® version, 3.5 or 3.6. Additionally, provide the ZooKeeper host configuration.

        • kafka: Apache Kafka® configuration:

          • resources.resourcePresetId: Host class ID. You can get the list of available host classes with their IDs using the ResourcePreset.list method.
          • resources.diskSize: Disk size, in bytes.
          • resources.diskTypeId: Disk type.

        • zookeeper: ZooKeeper configuration:

          • resources.resourcePresetId: Host class ID. You can request the list of available host classes with their IDs using the ResourcePreset.list method.
          • resources.diskSize: Disk size in bytes.
          • resources.diskTypeId: Disk type.

        • zoneId and brokersCount: Availability zones and number of broker hosts per zone.

        • assignPublicIp: Access to broker hosts from the internet, true or false.

        • schemaRegistry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster.

        • restApiConfig: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify enabled: true.

        • diskSizeAutoscaling: Set the storage utilization thresholds (as a percentage of the total storage size), that will trigger an increase in storage size when reached:

          • plannedUsageThreshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergencyUsageThreshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to plannedUsageThreshold.

          • diskSizeLimit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafkaUiConfig: Use Kafka UI. Specify enabled: true to access Kafka UI.

      • topicSpecs: Topic settings as an array of elements. Each element is for a separate topic and has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions.

        • replicationFactor: Replication factor.

      • userSpecs: User settings as an array of elements, one for each user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topicName: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allowHosts: (Optional) List of IP addresses the user is allowed to access the topic from.

      • maintenanceWindow: Maintenance window settings (including for disabled clusters). Select one of these options:

        • anytime: Any time (default).
        • weeklyMaintenanceWindow: On schedule:
          • day: Day of week in DDD format: MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletionProtection: Cluster protection from accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, provide a list of host group IDs in the hostGroupIds parameter.

      Warning

      You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

      You can request the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

3. View the server response to make sure your request was successful.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/Create call and send the following request, e.g., via gRPCurl:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folder_id": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "network_id": "<network_ID>",
        "security_group_ids": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "config_spec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resource_preset_id": "<Apache Kafka®_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"
            }
          },
          "zookeeper": {
            "resources": {
              "resource_preset_id": "<ZooKeeper_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"                   
            }
          },
          "zone_id": [
            <list_of_availability_zones>
          ],
          "brokers_count": {
            "value": "<number_of_brokers_in_zone>"
          },
          "assign_public_ip": <enable_public_access_to_cluster>,
          "schema_registry": <enable_data_schema_management>,
          "rest_api_config": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "disk_size_autoscaling": {
            <automatic_storage_size_increase_parameters>
          },
          "kafka_ui_config": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topic_specs": [
          {
            "name": "<topic_name>",
            "partitions": {
              "value": "<number_of_partitions>"
            },
            "replication_factor": {
              "value": "<replication_factor>"
            }
          },
          { <similar_list_of_settings_for_topic_2> },
          { ... },
          { <similar_list_of_settings_for_topic_N> }
        ],
        "user_specs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topic_name": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenance_window": {
          "anytime": {},
          "weekly_maintenance_window": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletion_protection": <protect_cluster_from_deletion>
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • network_id: ID of the network the cluster will be in.

      • security_group_ids: Security group IDs as an array of strings. Each string is a security group ID.

      • config_spec: Cluster configuration:

        • version: Apache Kafka® version, 3.5 or 3.6. Additionally, provide the ZooKeeper host configuration.

        • kafka: Apache Kafka® configuration:

          • resources.resource_preset_id: Host class ID. You can request the list of available host classes with their IDs using the ResourcePreset.list call.
          • resources.disk_size: Disk size in bytes.
          • resources.disk_type_id: Disk type.

        • zookeeper: ZooKeeper configuration:

          • resources.resource_preset_id: Host class ID. You can request the list of available host classes with their IDs using the ResourcePreset.list call.
          • resources.disk_size: Disk size in bytes.
          • resources.disk_type_id: Disk type.

        • zone_id and brokers_count: Availability zones and number of broker hosts per zone (this number is provided as an object with the value field).

        • assign_public_ip: Access to broker hosts from the internet, true or false.

        • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster.

        • rest_api_config: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify enabled: true.

        • disk_size_autoscaling: To prevent the cluster disk space from running out, set the storage utilization thresholds (as a percentage of the total storage size) that will trigger an increase in storage size when reached:

          • planned_usage_threshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergency_usage_threshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned_usage_threshold.

          • disk_size_limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafka_ui_config: Use Kafka UI. Specify enabled: true to access Kafka UI.

      • topic_specs: Topic settings as an array of elements. Each element is for a separate topic and has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions, provided as an object with a field named value.

        • replication_factor: Replication factor. Provided as an object with the value field.

      • user_specs: User settings as an array of elements, one for each user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topic_name: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allow_hosts: (Optional) List of IP addresses the user is allowed to access the topic from, as an array of elements.

      • maintenance_window: Maintenance window settings (including for disabled clusters). Select one of these options:

        • anytime: Any time (default).
        • weekly_maintenance_window: On schedule:
          • day: Day of week in DDD format: MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletion_protection: Cluster protection from accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, provide a list of host group IDs in the host_group_ids parameter.

      Warning

      You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

      You can request the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

4. View the server response to make sure your request was successful.

If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

To create a Managed Service for Apache Kafka® cluster:

1. See the description of the CLI command for creating a Managed Service for Apache Kafka® cluster:

   yc managed-kafka cluster create --help
   

2. Specify the Managed Service for Apache Kafka® cluster parameters in the create command (not all parameters are given in the example):

   yc managed-kafka cluster create \
      --name <cluster_name> \
      --environment <environment> \
      --version <version> \
      --schema-registry \
      --network-name <network_name> \
      --subnet-ids <subnet_IDs> \
      --resource-preset <host_class> \
      --disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
      --disk-size <storage_size_in_GB> \
      --assign-public-ip <enable_public_access_to_cluster> \
      --security-group-ids <list_of_security_group_IDs> \
      --deletion-protection
      --kafka-ui-enabled <true_or_false>
   

   Where:

   • --environment: Cluster environment, prestable or production.

   • --version: Apache Kafka® version. Specify version 3.6 or higher.

   • --schema-registry: Manage data schemas using Managed Schema Registry.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • --zone-ids and --brokers-count: Availability zones and number of broker hosts per zone.

     If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

     • --zone-ids=ru-central1-a,ru-central1-b,ru-central1-d --brokers-count=1: Three availability zones, one broker host in each availability zone.
     • --zone-ids=<one_availability_zone> --brokers-count=3: One availability zone, three broker hosts.

   • --resource-preset: Host class.

   • --disk-type: Disk type.

     Warning

     You cannot change disk type after you create a cluster.

   • --deletion-protection: Cluster protection from accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • --kafka-ui-enabled: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false.

   Tip

   You can also configure the Apache Kafka® settings here, if required.

3. To use KRaft (on separate hosts), provide the KRaft host configuration.

   yc managed-kafka cluster create \
       ...
       --controller-resource-preset <host_class> \
       --controller-disk-size <storage_size_in_GB> \
       --controller-disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
       ...
   
   

   Where:

   • --controller-resource-preset: KRaft host class.
   • --controller-disk-size: Storage size.
   • --controller-disk-type: KRaft disk type.

4. To set up a maintenance window (including for disabled Managed Service for Apache Kafka® clusters), provide the required value in the --maintenance-window parameter when creating your cluster:

   yc managed-kafka cluster create \
   ...
   --maintenance-window type=<maintenance_type>,`
                       `day=<day_of_week>,`
                       `hour=<hour> \
   ...
   

   Where type is the maintenance type:

   • anytime: At any time (default).
   • weekly: On a schedule. For this value, also specify the following:
     • day: Day of week, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: Hour of day (UTC), from 1 to 24.

5. To prevent the cluster disk space from running out, create a cluster that will increase the storage space automatically.

   yc managed-kafka cluster create \
   ...
   --disk-size-autoscaling disk-size-limit=<maximum_storage_size_in_bytes>,`
                           `planned-usage-threshold=<scheduled_increase_percentage>,`
                           `emergency-usage-threshold=<immediate_increase_percentage> \
   ...
   

   Where:

   • planned-usage-threshold: Storage utilization percentage to trigger a storage increase in the next maintenance window.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

     If you set this parameter, configure the maintenance schedule.

   • emergency-usage-threshold: Storage utilization percentage to trigger an immediate storage increase.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned-usage-threshold.

   • disk-size-limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

     If the value is 0, automatic increase of storage size will be disabled.

   Warning

   • You cannot decrease the storage size.
   • While resizing the storage, cluster hosts will be unavailable.

6. To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, specify the IDs of these hosts as a comma-separated list in the --host-group-ids parameter:

   yc managed-kafka cluster create \
   ...
   --host-group-ids <dedicated_host_group_IDs> \
   ...
   

   Warning

   You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

With Terraform, you can quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. These files store the infrastructure description written in HashiCorp Configuration Language (HCL). If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

Terraform is distributed under the Business Source License. The Yandex Cloud provider for Terraform is distributed under the MPL-2.0 license.

For more information about the provider resources, see the relevant documentation on the Terraform website or its mirror.

If you do not have Terraform yet, install it and configure the Yandex Cloud provider.

To create a Managed Service for Apache Kafka® cluster:

1. In the configuration file, describe the resources you are creating:

   • Managed Service for Apache Kafka® cluster: Description of the cluster and its hosts. You can also configure the Apache Kafka® settings here, if required.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     environment         = "<environment>"
     name                = "<cluster_name>"
     network_id          = "<network_ID>"
     subnet_ids          = ["<list_of_subnet_IDs>"]
     security_group_ids  = ["<list_of_cluster_security_group_IDs>"]
     deletion_protection = <protect_cluster_from_deletion>
   
     config {
       version          = "<version>"
       zones            = ["<availability_zones>"]
       brokers_count    = <number_of_broker_hosts>
       assign_public_ip = "<enable_public_access_to_cluster>"
       schema_registry  = "<enable_data_schema_management>"
       kafka_ui {
         enabled = <use_Kafka_UI>
       }
       kafka {
         resources {
           disk_size          = <storage_size_in_GB>
           disk_type_id       = "<disk_type>"
           resource_preset_id = "<host_class>"
         }
         kafka_config {}
       }
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Cluster environment, PRESTABLE or PRODUCTION.

   • version: Apache Kafka® version. Specify version 3.6 or higher.

   • zones and brokers_count: Availability zones and number of broker hosts per zone.

     If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

     • zones = ["ru-central1-a","ru-central1-b","ru-central1-d"] brokers_count = 1: Three availability zones, one broker host in each availability zone.
     • zones = ["<one_availability_zone>"] brokers_count = 3: One availability zone, three broker hosts.

   • deletion_protection: Cluster protection from accidental deletion, true or false.

     Note

     Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

   • assign_public_ip: Public access to the cluster, true or false.

   • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false.

     Warning

     You cannot disable data schema management using Managed Schema Registry after connecting it.

   • kafka_ui: This setting defines whether to use Kafka UI for Apache Kafka® and can be true or false. The default value is false.

   Add the kraft section to the cluster description to use KRaft (on separate hosts).

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     kraft {
       resources {
         resource_preset_id = "<host_class>"
         disk_type_id       = "<disk_type>"
         disk_size          = <storage_size_in_GB>
       }
     }
   }
   

   To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
     ...
   }
   

   Where:

   • type: Maintenance type. The possible values include:
     • ANYTIME: Anytime
     • WEEKLY: On a schedule
   • day: Day of week for the WEEKLY type, i.e., MON, TUE, WED, THU, FRI, SAT, or SUN.
   • hour: UTC hour for the WEEKLY type, from 1 to 24.

2. Validate your configuration.

   1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

   2. Run this command:

      terraform validate
      

      Terraform will show any errors found in your configuration files.

3. Create a Managed Service for Apache Kafka® cluster.

   1. Run this command to view the planned changes:

      terraform plan
      

      If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

   2. If everything looks correct, apply the changes:

      1. Run this command:

         terraform apply
         

      2. Confirm updating the resources.

      3. Wait for the operation to complete.

   After this, all required resources will be created in the specified folder, and the FQDNs of the Managed Service for Apache Kafka® cluster hosts will be displayed in the terminal. You can check the new resources and their configuration in the management console.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Use the Cluster.create method and send the following request, e.g., via cURL:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folderId": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "networkId": "<network_ID>",
        "securityGroupIds": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "configSpec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resourcePresetId": "<Apache Kafka®_host_class>",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"
            }
          },
          "kraft": {
            "resources": {
              "resourcePresetId": "<KRaft_host_class>",
              "diskSize": "<storage_size_in_bytes>",
              "diskTypeId": "<disk_type>"                   
            }
          },
          "zoneId": [
            <list_of_availability_zones>
          ],
          "brokersCount": "<number_of_brokers_in_zone>",
          "assignPublicIp": <enable_public_access_to_cluster>,
          "schemaRegistry": <enable_data_schema_management>,
          "restApiConfig": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "diskSizeAutoscaling": {
            <automatic_storage_size_increase_parameters>
          },
          "kafkaUiConfig": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topicSpecs": [
          {
            "name": "<topic_name>",
            "partitions": "<number_of_partitions>",
            "replicationFactor": "<replication_factor>"
          },
          { <similar_list_of_settings_for_topic_2> },
          { ... },
          { <similar_list_of_settings_for_topic_N> }
        ],
        "userSpecs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topicName": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenanceWindow": {
          "anytime": {},
          "weeklyMaintenanceWindow": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletionProtection": <protect_cluster_from_deletion>
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • networkId: ID of the network the cluster will be in.

      • securityGroupIds: Security group IDs as an array of strings. Each string is a security group ID.

      • configSpec: Cluster configuration:

        • version: Apache Kafka® version. Specify version 3.6 or higher.

        • kafka: Apache Kafka® configuration:

          • resources.resourcePresetId: Host class ID. You can get the list of available host classes with their IDs using the ResourcePreset.list method.
          • resources.diskSize: Disk size, in bytes.
          • resources.diskTypeId: Disk type.

        • kraft: KRaft configuration:

          • resources.resourcePresetId: Host class ID. You can request the list of available host classes with their IDs using the ResourcePreset.list method.
          • resources.diskSize: Disk size in bytes.
          • resources.diskTypeId: Disk type.

          Warning

          If you are creating a KRaft (combined mode) cluster, do not provide the KRaft host configuration.

        • zoneId and brokersCount: Availability zones and number of broker hosts per zone.

          If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

          • "zoneId": ["ru-central1-a","ru-central1-b","ru-central1-d"], "brokersCount": "1": Three availability zones, one broker host in each zone.
          • "zoneId": ["<one_availability_zone>"], "brokersCount": "3": One availability zone, three broker hosts.

        • assignPublicIp: Access to broker hosts from the internet, true or false.

        • schemaRegistry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster.

        • restApiConfig: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify enabled: true.

        • diskSizeAutoscaling: Set the storage utilization thresholds (as a percentage of the total storage size), that will trigger an increase in storage size when reached:

          • plannedUsageThreshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergencyUsageThreshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to plannedUsageThreshold.

          • diskSizeLimit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafkaUiConfig: Use Kafka UI. Specify enabled: true to access Kafka UI.

      • topicSpecs: Topic settings as an array of elements. Each element is for a separate topic and has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions.

        • replicationFactor: Replication factor.

      • userSpecs: User settings as an array of elements, one for each user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topicName: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allowHosts: (Optional) List of IP addresses the user is allowed to access the topic from.

      • maintenanceWindow: Maintenance window settings (including for disabled clusters). Select one of these options:

        • anytime: At any time (default).
        • weeklyMaintenanceWindow: On schedule:
          • day: Day of week in DDD format: MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletionProtection: Cluster protection from accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, provide a list of host group IDs in the hostGroupIds parameter.

      Warning

      You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

      You can request the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

3. View the server response to make sure your request was successful.

1. Get an IAM token for API authentication and save it as an environment variable:

   export IAM_TOKEN="<IAM_token>"
   

2. Clone the cloudapi repository:

   cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
   

   Below, we assume the repository contents are stored in the ~/cloudapi/ directory.

3. Use the ClusterService/Create call and send the following request, e.g., via gRPCurl:

   1. Create a file named body.json and paste the following code into it:

      Note

      This example does not use all available parameters.

      {
        "folder_id": "<folder_ID>",
        "name": "<cluster_name>",
        "environment": "<environment>",
        "network_id": "<network_ID>",
        "security_group_ids": [
          "<security_group_1_ID>",
          "<security_group_2_ID>",
          ...
          "<security_group_N_ID>"
        ],
        "config_spec": {
          "version": "<Apache Kafka®_version>",
          "kafka": {
            "resources": {
              "resource_preset_id": "<Apache Kafka®_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"
            }
          },
          "kraft": {
            "resources": {
              "resource_preset_id": "<KRaft_host_class>",
              "disk_size": "<storage_size_in_bytes>",
              "disk_type_id": "<disk_type>"                   
            }
          },
          "zone_id": [
            <list_of_availability_zones>
          ],
          "brokers_count": {
            "value": "<number_of_brokers_in_zone>"
          },
          "assign_public_ip": <enable_public_access_to_cluster>,
          "schema_registry": <enable_data_schema_management>,
          "rest_api_config": {
            "enabled": <enable_sending_requests_to_Apache Kafka®_API>
          },
          "disk_size_autoscaling": {
            <automatic_storage_size_increase_parameters>
          },
          "kafka_ui_config": {
            "enabled": <use_Kafka_UI>
          }
        },
        "topic_specs": [
          {
            "name": "<topic_name>",
            "partitions": {
              "value": "<number_of_partitions>"
            },
            "replication_factor": {
              "value": "<replication_factor>"
            }
          },
          { <similar_list_of_settings_for_topic_2> },
          { ... },
          { <similar_list_of_settings_for_topic_N> }
        ],
        "user_specs": [
          {
            "name": "<username>",
            "password": "<user_password>",
            "permissions": [
              {
                "topic_name": "<topic_name>",
                "role": "<user's_role>"
              }
            ]
          },
          { <similar_settings_for_user_2> },
          { ... },
          { <similar_settings_for_user_N> }
        ],
        "maintenance_window": {
          "anytime": {},
          "weekly_maintenance_window": {
            "day": "<day_of_week>",
            "hour": "<hour_UTC>"
          }
        },
        "deletion_protection": <protect_cluster_from_deletion>
      }
      

      Where:

      • name: Cluster name.

      • environment: Cluster environment, PRODUCTION or PRESTABLE.

      • network_id: ID of the network the cluster will be in.

      • security_group_ids: Security group IDs as an array of strings. Each string is a security group ID.

      • config_spec: Cluster configuration:

        • version: Apache Kafka® version. Specify version 3.6 or higher.

        • kafka: Apache Kafka® configuration:

          • resources.resource_preset_id: Host class ID. You can request the list of available host classes with their IDs using the ResourcePreset.list call.
          • resources.disk_size: Disk size in bytes.
          • resources.disk_type_id: Disk type.

        • kraft: KRaft configuration.

          • resources.resource_preset_id: Host class ID. You can request the list of available host classes with their IDs using the ResourcePreset.list call.
          • resources.disk_size: Disk size in bytes.
          • resources.disk_type_id: Disk type.

          Warning

          If you are creating a KRaft (combined mode) cluster, do not provide the KRaft host configuration.

        • zone_id and brokers_count: Availability zones and number of broker hosts per zone (this number is provided as an object with the value field).

          If you are creating a cluster with KRaft (combined mode), specify one of the available configurations:

          • "zone_id": ["ru-central1-a","ru-central1-b","ru-central1-d"], "brokers_count": {"value":"1"}: Three availability zones, one broker host in each zone.
          • "zone_id": ["<one_availability_zone>"], "brokers_count": {"value":"3"}: One availability zone, three broker hosts.

        • assign_public_ip: Access to broker hosts from the internet, true or false.

        • schema_registry: Manage data schemas using Managed Schema Registry, true or false. The default value is false. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster.

        • rest_api_config: Apache Kafka® REST API configuration. For access to sending requests to the Apache Kafka® REST API, specify enabled: true.

        • disk_size_autoscaling: To prevent the cluster disk space from running out, set the storage utilization thresholds (as a percentage of the total storage size) that will trigger an increase in storage size when reached:

          • planned_usage_threshold: Storage utilization percentage to trigger a storage increase during the next maintenance window.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

            If you set this parameter, configure the maintenance window schedule.

          • emergency_usage_threshold: Storage utilization percentage to trigger an immediate storage increase.

            Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned_usage_threshold.

          • disk_size_limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

        • kafka_ui_config: Use Kafka UI. Specify enabled: true to access Kafka UI.

      • topic_specs: Topic settings as an array of elements. Each element is for a separate topic and has the following structure:

        • name: Topic name.

          Note

          Use the Apache Kafka® Admin API if you need to create a topic that starts with _. You cannot create such a topic using the Yandex Cloud interfaces.

        • partitions: Number of partitions, provided as an object with a field named value.

        • replication_factor: Replication factor. Provided as an object with the value field.

      • user_specs: User settings as an array of elements, one for each user. Each element has the following structure:

        • name: Username. It may contain Latin letters, numbers, hyphens, and underscores, and must start with a letter or underscore.

        • password: User password. The password must be from 8 to 128 characters long.

        • permissions: List of topics the user must have access to.

          The list is arranged as an array of elements. Each element is for a separate topic and has the following structure:

          • topic_name: Topic name or name template:
            • * to allow access to any topics.
            • Full topic name to allow access to a specific topic.
            • <prefix>* to grant access to topics whose names start with the prefix. Let's assume you have topics named topic_a1, topic_a2, and a3. If you put topic*, access will be granted to topic_a1 and topic_a2. To include all the cluster's topics, use the * mask.
          • role: User’s role, ACCESS_ROLE_CONSUMER, ACCESS_ROLE_PRODUCER, or ACCESS_ROLE_ADMIN. The ACCESS_ROLE_ADMIN role is only available if all topics are selected (topicName: "*").
          • allow_hosts: (Optional) List of IP addresses the user is allowed to access the topic from, as an array of elements.

      • maintenance_window: Maintenance window settings (including for disabled clusters). Select one of these options:

        • anytime: At any time (default).
        • weekly_maintenance_window: On schedule:
          • day: Day of week in DDD format: MON, TUE, WED, THU, FRI, SAT, or SUN.
          • hour: Time of day (UTC) in HH format, from 1 to 24.

      • deletion_protection: Cluster protection from accidental deletion, true or false. The default value is false.

        Note

        Even with cluster deletion protection enabled, one can still delete a user or topic or connect manually and delete the data.

      To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, provide a list of host group IDs in the host_group_ids parameter.

      Warning

      You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

      You can request the folder ID with the list of folders in the cloud.

   2. Run this request:

      curl \
        --request POST \
        --header "Authorization: Bearer $IAM_TOKEN" \
        --header "Content-Type: application/json" \
        --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters' \
        --data '@body.json'
      

4. View the server response to make sure your request was successful.

Create a Managed Service for Apache Kafka® cluster with the following test specifications:

• Cloud ID: b1gq90dgh25bebiu75o.

• Folder ID: b1gia87mbaomkfvsleds.

• Name: mykf.

• Environment: PRODUCTION.

• Apache Kafka® version: 3.5.

• New network: mynet, subnet: mysubnet.

• Security group: mykf-sg (allow ingress connections to the Managed Service for Apache Kafka® cluster on port 9091).

• Host class: s2.micro, availability zone: ru-central1-a.

• With one broker host.

• Network SSD storage (network-ssd): 10 GB.

• Public access: Allowed.

• Deletion protection: Enabled.

The configuration file for this Managed Service for Apache Kafka® cluster is as follows:

resource "yandex_mdb_kafka_cluster" "mykf" {
  environment         = "PRODUCTION"
  name                = "mykf"
  network_id          = yandex_vpc_network.mynet.id
  subnet_ids          = [ yandex_vpc_subnet.mysubnet.id ]
  security_group_ids  = [ yandex_vpc_security_group.mykf-sg.id ]
  deletion_protection = true

  config {
    assign_public_ip = true
    brokers_count    = 1
    version          = "3.5"
    kafka {
      resources {
        disk_size          = 10
        disk_type_id       = "network-ssd"
        resource_preset_id = "s2.micro"
      }
      kafka_config {}
    }

    zones = [
      "ru-central1-a"
    ]
  }
}

resource "yandex_vpc_network" "mynet" {
  name = "mynet"
}

resource "yandex_vpc_subnet" "mysubnet" {
  name           = "mysubnet"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.mynet.id
  v4_cidr_blocks = ["10.5.0.0/24"]
}

resource "yandex_vpc_security_group" "mykf-sg" {
  name       = "mykf-sg"
  network_id = yandex_vpc_network.mynet.id

  ingress {
    description    = "Kafka"
    port           = 9091
    protocol       = "TCP"
    v4_cidr_blocks = [ "0.0.0.0/0" ]
  }
}