Creating an Apache Kafka® cluster

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To create a Managed Service for Apache Kafka® cluster:

1. View the description of the create Managed Service for Apache Kafka® cluster CLI command:

   yc managed-kafka cluster create --help
   

2. Specify the Managed Service for Apache Kafka® cluster parameters in the create command (the example below does not list all possible parameters):

   yc managed-kafka cluster create \
      --name <cluster_name> \
      --environment <environment> \
      --version <version> \
      --network-name <network_name> \
      --subnet-ids <subnet_IDs> \
      --brokers-count <number_of_brokers_in_zone> \
      --resource-preset <host_class> \
      --disk-type <network-hdd|network-ssd|network-ssd-nonreplicated|local-ssd> \
      --disk-size <storage_size_in_GB> \
      --assign-public-ip <public_access> \
      --security-group-ids <list_of_security_group_IDs> \
      --deletion-protection
   

   Where:

   • --environment: Cluster environment, prestable or production.

   • --version: Apache Kafka® version, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4 or 3.5.

   • --resource-preset: Host class.

   • --disk-type: Disk type.

     Warning

     You cannot change disk type after you create a cluster.

   • --deletion-protection: Protection of the cluster, its databases, and users against accidental deletion.

     Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

   Tip

   You can also configure the Apache Kafka® settings here, if required.

3. To set up a maintenance window (including for disabled Managed Service for Apache Kafka® clusters), provide the required value in the --maintenance-window parameter when creating your cluster:

   yc managed-kafka cluster create \
      ...
      --maintenance-window type=<maintenance_type>,`
                          `day=<day_of_week>,`
                          `hour=<hour> \
   

   Where type is the maintenance type:

   • anytime (default): Any time.
   • weekly: On a schedule. If setting this value, specify the day of week and the hour:
     • day: Day of week in DDD format: MON, TUE, WED, THU, FRI, SAT, or SUN.
     • hour: Hour (UTC) in HH format: 1 to 24.

4. To prevent the cluster disk space from running out, create a cluster that will increase the storage space automatically.

   yc managed-kafka cluster create \
      ...
      --disk-size-autoscaling disk-size-limit=<maximum_storage_size_in_bytes>,`
                             `planned-usage-threshold=<scheduled_increase_percentage>,`
                             `emergency-usage-threshold=<immediate_increase_percentage>
   

   Where:

   • planned-usage-threshold: Storage utilization percentage to trigger a storage increase in the next maintenance window.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled).

     If you set this parameter, configure the maintenance schedule.

   • emergency-usage-threshold: Storage utilization percentage to trigger an immediate storage increase.

     Use a percentage value between 0 and 100. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to planned-usage-threshold.

   • disk-size-limit: Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages.

     If the value is 0, automatic increase of storage size will be disabled.

   Warning

   • You cannot decrease the storage size.
   • While resizing the storage, cluster hosts will be unavailable.

5. To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, specify their IDs as a comma-separated list in the --host-group-ids parameter when creating the cluster:

   yc managed-kafka cluster create \
      ...
      --host-group-ids <dedicated_host_group_IDs>
   

   Alert

   You cannot edit this setting after you create a cluster. The use of dedicated hosts significantly affects cluster pricing.

Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

For more information about the provider resources, see the documentation on the Terraform website or the mirror.

If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

To create a Managed Service for Apache Kafka® cluster:

1. In the configuration file, describe the resources you are creating:

   • Managed Service for Apache Kafka® cluster: Description of a cluster and its hosts. You can also configure the Apache Kafka® settings here, if required.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     environment         = "<environment>"
     name                = "<cluster_name>"
     network_id          = "<network_ID>"
     subnet_ids          = ["<list_of_subnet_IDs>"]
     security_group_ids  = ["<list_of_cluster_security_group_IDs>"]
     deletion_protection = <deletion_protection>
   
     config {
       assign_public_ip = "<public_access>"
       brokers_count    = <number_of_brokers>
       version          = "<version>"
       schema_registry  = "<data_schema_management>"
       kafka {
         resources {
           disk_size          = <storage_size_in_GB>
           disk_type_id       = "<disk_type>"
           resource_preset_id = "<host_class>"
         }
         kafka_config {}
       }
   
       zones = [
         "<availability_zones>"
       ]
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: Cluster environment, PRESTABLE or PRODUCTION.
   • deletion_protection: Cluster deletion protection, true or false.
   • assign_public_ip: Public access to the cluster, true or false.
   • version: Apache Kafka® version, 2.8, 3.0, 3.1, 3.2, 3.3, 3.4 or 3.5.
   • schema_registry: Managing data schemas, true or false.

   Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

   To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
     ...
   }
   

   Where:

   • type: Maintenance type. The possible values include:
     • anytime: Anytime.
     • weekly: By schedule.
   • day: Day of the week for the weekly type in DDD format, e.g., MON.
   • hour: Hour of the day for the weekly type in the HH format, e.g., 21.

2. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

3. Create a Managed Service for Apache Kafka® cluster.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

   After this, all required resources will be created in the specified folder, and the FQDNs of the Managed Service for Apache Kafka® cluster hosts will be displayed in the terminal. You can check the new resources and their settings using the management console.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To create a Managed Service for Apache Kafka® cluster, use the create REST API method for the Cluster resource or the ClusterService/Create gRPC API call and provide the following in the request:

• ID of the folder to host the Managed Service for Apache Kafka® cluster, in the folderId parameter.

• Managed Service for Apache Kafka® cluster name in the name parameter.

• Apache Kafka® version (2.8, 3.0, 3.1, 3.2, 3.3, 3.4 or 3.5) in the configSpec.version parameter.

• Security group IDs in the securityGroupIds parameter.

• Maintenance window settings (including for disabled Managed Service for Apache Kafka® clusters) in the maintenanceWindow parameter.

• Managed Service for Apache Kafka® cluster deletion protection settings in the deletionProtection parameter.

  Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

To manage data schemas using Managed Schema Registry, set the configSpec.schemaRegistry parameter to true. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster.

To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, provide a list of host group IDs in the hostGroupIds parameter.

To prevent the cluster disk space from running out, create a cluster that will increase the storage space automatically. For this, provide the following in the request:

• Storage utilization percentage to trigger a storage increase in the next maintenance window, in the configSpec.diskSizeAutoscaling.plannedUsageThreshold parameter.

  Use a value between 0 and 100%. The default value is 0 (automatic increase is disabled).

  If you set this parameter, configure the maintenance window schedule.

• Storage utilization percentage to trigger an immediate storage increase, in the configSpec.diskSizeAutoscaling.emergencyUsageThreshold parameter.

  Use a value between 0 and 100%. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to configSpec.diskSizeAutoscaling.plannedUsageThreshold.

• Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages, in the configSpec.diskSizeAutoscaling.diskSizeLimit parameter.

Terraform allows you to quickly create a cloud infrastructure in Yandex Cloud and manage it using configuration files. Configuration files store the infrastructure description in the HashiCorp Configuration Language (HCL). Terraform and its providers are distributed under the Business Source License.

For more information about the provider resources, see the documentation on the Terraform website or the mirror.

If you change the configuration files, Terraform automatically detects which part of your configuration is already deployed, and what should be added or removed.

If you don't have Terraform, install it and configure the Yandex Cloud provider.

To create a Managed Service for Apache Kafka® cluster:

1. In the configuration file, describe the resources you are creating:

   • Managed Service for Apache Kafka® cluster: Description of a cluster and its hosts. You can also configure the Apache Kafka® settings here, if required.

   • Network: Description of the cloud network where a cluster will be located. If you already have a suitable network, you don't have to describe it again.

   • Subnets: Description of the subnets to connect the cluster hosts to. If you already have suitable subnets, you don't have to describe them again.

   Here is an example of the configuration file structure:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     environment         = "PRESTABLE"
     name                = "<cluster_name>"
     network_id          = "<network_ID>"
     subnet_ids          = ["<subnet_IDs>"]
     security_group_ids  = ["<security_group_IDs>"]
     deletion_protection = <deletion_protection>
   
     config {
       version          = "3.6"
       zones            = ["<availability_zones>"]
       brokers_count    = <number_of_brokers>
       assign_public_ip = "<public_access>"
       schema_registry  = "<data_schema_management>"
       kafka {
         resources {
           disk_size          = <storage_size_in_GB>
           disk_type_id       = "<disk_type>"
           resource_preset_id = "<host_class>"
         }
         kafka_config {}
       }
     }
   }
   
   resource "yandex_vpc_network" "<network_name>" {
     name = "<network_name>"
   }
   
   resource "yandex_vpc_subnet" "<subnet_name>" {
     name           = "<subnet_name>"
     zone           = "<availability_zone>"
     network_id     = "<network_ID>"
     v4_cidr_blocks = ["<range>"]
   }
   

   Where:

   • environment: PRESTABLE cluster environment. Only this environment supports Apache Kafka® 3.6.

   • deletion_protection: Cluster deletion protection, true or false.

   • version: Apache Kafka® version, 3.6.

   • zones and brokers_count: Availability zones and number of brokers per zone. Specify one of the available configurations:

     • zones = ["ru-central1-a","ru-central1-b","ru-central1-d"] brokers_count = 1
     • zones = ["<one_availability_zone>"] brokers_count = 3

   • assign_public_ip: Public access to the cluster, true or false.

   • schema_registry: Managing data schemas, true or false.

   Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

   To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
     ...
   }
   

   Where:

   • type: Maintenance type. The possible values include:
     • anytime: Anytime.
     • weekly: By schedule.
   • day: Day of the week for the weekly type in DDD format, e.g., MON.
   • hour: Hour of the day for the weekly type in the HH format, e.g., 21.

2. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

3. Create a Managed Service for Apache Kafka® cluster.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

   After this, all required resources will be created in the specified folder, and the FQDNs of the Managed Service for Apache Kafka® cluster hosts will be displayed in the terminal. You can check the new resources and their settings using the management console.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To create a Managed Service for Apache Kafka® cluster, use the create REST API method for the Cluster resource or the ClusterService/Create gRPC API call and provide the following in the request:

• ID of the folder to host the Managed Service for Apache Kafka® cluster, in the folderId parameter.

• Managed Service for Apache Kafka® cluster name in the name parameter.

• PRESTABLE cluster environment in the environment parameter.

• Apache Kafka® version 3.6 in the configSpec.version parameter.

• Availability zones in the configSpec.zoneId parameter. You can specify either one or three availability zones.

• Number of broker hosts in the configSpec.brokersCount parameter. If you specify one availability zone, set the number of brokers to three. In case of three availability zones, set the number of brokers to one.

• Security group IDs in the securityGroupIds parameter.

• Maintenance window settings (including for disabled Managed Service for Apache Kafka® clusters) in the maintenanceWindow parameter.

• Managed Service for Apache Kafka® cluster deletion protection settings in the deletionProtection parameter.

  Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

To manage data schemas using Managed Schema Registry, set the configSpec.schemaRegistry parameter to true. You will not be able to edit this setting once you create a Managed Service for Apache Kafka® cluster.

To create a Managed Service for Apache Kafka® cluster based on dedicated host groups, provide a list of host group IDs in the hostGroupIds parameter.

To prevent the cluster disk space from running out, create a cluster that will increase the storage space automatically. For this, provide the following in the request:

• Storage utilization percentage to trigger a storage increase in the next maintenance window, in the configSpec.diskSizeAutoscaling.plannedUsageThreshold parameter.

  Use a value between 0 and 100%. The default value is 0 (automatic increase is disabled).

  If you set this parameter, configure the maintenance window schedule.

• Storage utilization percentage to trigger an immediate storage increase, in the configSpec.diskSizeAutoscaling.emergencyUsageThreshold parameter.

  Use a value between 0 and 100%. The default value is 0 (automatic increase is disabled). This parameter value must be greater than or equal to configSpec.diskSizeAutoscaling.plannedUsageThreshold.

• Maximum storage size, in bytes, that can be set when utilization reaches one of the specified percentages, in the configSpec.diskSizeAutoscaling.diskSizeLimit parameter.

Create a Managed Service for Apache Kafka® cluster with the following test specifications:

• Cloud ID: b1gq90dgh25bebiu75o.

• Folder ID: b1gia87mbaomkfvsleds.

• Name: mykf.

• Environment: PRODUCTION.

• Apache Kafka® version: 3.5.

• New network: mynet, subnet: mysubnet.

• Security group: mykf-sg (allow ingress connections to the Managed Service for Apache Kafka® cluster on port 9091).

• Host class: s2.micro, availability zone: ru-central1-a.

• Number of brokers: 1.

• Network SSD storage (network-ssd): 10 GB.

• Public access: Allowed.

• Protection against accidental Managed Service for Apache Kafka® cluster deletion: Enabled.

The configuration file for this Managed Service for Apache Kafka® cluster is as follows:

resource "yandex_mdb_kafka_cluster" "mykf" {
  environment         = "PRODUCTION"
  name                = "mykf"
  network_id          = yandex_vpc_network.mynet.id
  subnet_ids          = [ yandex_vpc_subnet.mysubnet.id ]
  security_group_ids  = [ yandex_vpc_security_group.mykf-sg.id ]
  deletion_protection = true

  config {
    assign_public_ip = true
    brokers_count    = 1
    version          = "3.5"
    kafka {
      resources {
        disk_size          = 10
        disk_type_id       = "network-ssd"
        resource_preset_id = "s2.micro"
      }
      kafka_config {}
    }

    zones = [
      "ru-central1-a"
    ]
  }
}

resource "yandex_vpc_network" "mynet" {
  name = "mynet"
}

resource "yandex_vpc_subnet" "mysubnet" {
  name           = "mysubnet"
  zone           = "ru-central1-a"
  network_id     = yandex_vpc_network.mynet.id
  v4_cidr_blocks = ["10.5.0.0/24"]
}

resource "yandex_vpc_security_group" "mykf-sg" {
  name       = "mykf-sg"
  network_id = yandex_vpc_network.mynet.id

  ingress {
    description    = "Kafka"
    port           = 9091
    protocol       = "TCP"
    v4_cidr_blocks = [ "0.0.0.0/0" ]
  }
}