Updating Apache Kafka® cluster settings

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the name and description of a cluster:

1. View the description of the update cluster CLI command:

   yc managed-kafka cluster update --help
   

2. Specify a new name and description in the cluster update command:

   yc managed-kafka cluster update <cluster_name_or_ID> \
     --new-name <new_cluster_name> \
     --description <new_cluster_description>
   

   To find out the cluster name or ID, get a list of clusters in the folder.

To update the cluster description:

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

2. In the Managed Service for Apache Kafka® cluster description, change the description parameter value:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     name        = "<cluster_name>"
     description = "<new_cluster_description>"
     ...
   }
   

3. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

4. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_redis_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To change additional cluster settings, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• New cluster name in the name parameter.
• New cluster description in the description parameter.
• List of fields to update (in this case, name and description) in the updateMask parameter.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the class and number of hosts:

1. Get information about the cluster:

   yc managed-kafka cluster list
   yc managed-kafka cluster get <cluster_name_or_ID>
   

2. View the description of the update cluster CLI command:

   yc managed-kafka cluster update --help
   

3. To increase the number of broker hosts, run this command:

   yc managed-kafka cluster update <cluster_name_or_ID> --brokers-count <number>
   

4. To change the broker host class, run this command:

   yc managed-kafka cluster update <cluster_name_or_ID> --resource-preset <host_class>
   

To find out the cluster name or ID, get a list of clusters in the folder.

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

2. In the Managed Service for Apache Kafka® cluster description, change the brokers_count parameter to increase the number of broker hosts:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     config {
       brokers_count = <number_of_broker_hosts>
       ...
     }
     ...
   }
   

3. In the Managed Service for Apache Kafka® cluster description, edit the value of the resource_preset_id parameter under kafka.resources to specify a new broker host class:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     kafka {
       resources {
         resource_preset_id = "<broker_host_class>"
         ...
       }
     }
   }
   

4. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

5. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To change the class and number of broker hosts, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• Broker host class in the configSpec.kafka.resources.resourcePresetId parameter.
• Number of broker hosts in the configSpec.brokersCount parameter.
• List of settings to update in the updateMask parameter.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the class of ZooKeeper hosts:

1. Get information about the cluster:

   yc managed-kafka cluster list
   yc managed-kafka cluster get <cluster_name_or_ID>
   

2. View the description of the update cluster CLI command:

   yc managed-kafka cluster update --help
   

3. To change the ZooKeeper host class, run this command:

   yc managed-kafka cluster update <cluster_name_or_ID> \
     --zookeeper-resource-preset <host_class>
   

To find out the cluster name or ID, get a list of clusters in the folder.

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

2. In the Managed Service for Apache Kafka® cluster description, edit the value of the resource_preset_id parameter under zookeeper.resources to specify a new ZooKeeper host class:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     zookeeper {
       resources {
         resource_preset_id = "<ZooKeeper_host_class>"
         ...
       }
     }
    }
   

3. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

4. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To change the class of ZooKeeper hosts, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• ZooKeeper host class in the configSpec.zookeeper.resources.resourcePresetId parameter.
• List of settings to update in the updateMask parameter.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To edit the list of security groups for your cluster:

1. View the description of the update cluster CLI command:

   yc managed-kafka cluster update --help
   

2. Specify the security groups and public access settings in the update cluster command:

   yc managed-kafka cluster update <cluster_name_or_ID> \
      --security-group-ids <list_of_security_groups> \
      --assign-public-ip=<public_access>
   

   Where:

   • --security-group-ids: List of cluster security group IDs.
   • --assign-public-ip: Public access to the cluster, true or false.

   To find out the cluster name or ID, get a list of clusters in the folder.

Restart the cluster for the new public access settings to take effect.

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

2. Change the values of the security_group_ids and assign_public_ip parameters in the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     security_group_ids = [ <list_of_security_groups> ]
     ...
     config {
       assign_public_ip = "<public_access>"
       ...
       }
   }
   

   Where:

   • security_group_ids: List of cluster security group IDs.
   • assign_public_ip: Public access to the cluster, true or false.

3. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

4. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

Restart the cluster for the new public access settings to take effect.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To change security group and public access settings, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• List of security group IDs in the securityGroupIds parameter.
• Public access settings in the configSpec.assignPublicIp parameter.
• List of settings to update in the updateMask parameter.

Restart the cluster for the new public access settings to take effect.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To change additional cluster settings:

1. View the description of the update cluster CLI command:

   yc managed-kafka cluster update --help
   

2. Run the following command with a list of settings to update:

   yc managed-kafka cluster update <cluster_name_or_ID> \
      --maintenance-window type=<maintenance_type>,`
                          `day=<day_of_week>,`
                          `hour=<hour> \
      --deletion-protection \
      --schema-registry=<data_schema_management>
   

You can change the following settings:

• --maintenance-window: Maintenance window settings (including for disabled clusters), where type is the maintenance type:

  • anytime (default): Any time.
  • weekly: On a schedule. If setting this value, specify the day of week and the hour:
    • day: Day of week in DDD format: MON, TUE, WED, THU, FRI, SAT, or SUN.
    • hour: Hour (UTC) in HH format: 1 to 24.

• --deletion-protection: Protection of the cluster, its databases, and users against accidental deletion.

  Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

• --schema-registry: Enable this option to manage data schemas using Managed Schema Registry.

  Warning

  Once enabled, the Schema registry setting cannot be disabled.

To find out the cluster name or ID, get a list of clusters in the folder.

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

2. To set up the maintenance window (for disabled clusters as well), add the maintenance_window section to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     maintenance_window {
       type = <maintenance_type>
       day  = <day_of_week>
       hour = <hour>
     }
     ...
   }
   

   Where:

   • type: Maintenance type. The possible values include:
     • anytime: Anytime.
     • weekly: By schedule.
   • day: Day of the week for the weekly type in DDD format, e.g., MON.
   • hour: Hour of the day for the weekly type in the HH format, e.g., 21.

3. To enable cluster protection against accidental deletion by a user of your cloud, add the deletion_protection field set to true to your cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     deletion_protection = <deletion_protection>
   }
   

   Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

4. To enable data schema management using Managed Schema Registry, add the config.schema_registry field set to true to the cluster description:

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     config {
       ...
       schema_registry  = <data_schema_management>
     }
   }
   

   Warning

   Once enabled, the Schema registry setting cannot be disabled.

5. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

6. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To change additional cluster settings, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.

• Maintenance window settings (including for disabled clusters) in the maintenanceWindow parameter.

• Cluster deletion protection settings in the deletionProtection parameter.

  Cluster deletion protection will not prevent a manual connection to a cluster to delete data.

• Settings for data schema management using Managed Schema Registry in the configSpec.schemaRegistry parameter.

  Warning

  Once enabled, the Schema registry setting cannot be disabled.

• List of cluster configuration fields to update in the updateMask parameter.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

To update the Apache Kafka® settings:

1. View the description of the CLI update cluster settings command:

   yc managed-kafka cluster update --help
   

2. Change the Apache Kafka® settings in the cluster update command (the example below does not list all possible parameters):

   yc managed-kafka cluster update <cluster_name_or_ID> \
      --compression-type <compression_type> \
      --log-flush-interval-messages <number_of_messages_in_log> \
      --log-flush-interval-ms <maximum_time_to_store_messages>
   

   Where:

   • --log-flush-interval-messages: Number of messages in the log to trigger flushing to disk.
   • --log-flush-interval-ms: Maximum time a message can be stored in memory before flushing to disk.

   To find out the cluster name or ID, get a list of clusters in the folder.

1. Open the current Terraform configuration file with an infrastructure plan.

   For more information about creating this file, see Creating clusters.

2. In the Managed Service for Apache Kafka® cluster description, modify the values of the parameters in the kafka.kafka_config section (the example below does not list all possible settings):

   resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
     ...
     config {
       kafka {
         ...
         kafka_config {
           compression_type            = "<compression_type>"
           log_flush_interval_messages = <maximum_number_of_messages_in_memory>
           ...
         }
       }
     }
   }
   

3. Make sure the settings are correct.

   1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

   2. Run the command:

      terraform validate
      

      If there are errors in the configuration files, Terraform will point to them.

4. Confirm updating the resources.

   1. Run the command to view planned changes:

      terraform plan
      

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

   2. If you are happy with the planned changes, apply them:

      1. Run the command:

         terraform apply
         

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To change Apache Kafka® settings, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.

• New values of the Apache Kafka® settings in:

  • configSpec.kafka.kafkaConfig_2_8: If you are using Apache Kafka® 2.8.
  • configSpec.kafka.kafkaConfig_3: If you are using Apache Kafka® 3.x.

• List of settings to update in the updateMask parameter.

To move a cluster, use the move REST API method for the Cluster resource or the ClusterService/Move gRPC API call and provide the following in the request:

• Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
• ID of the destination folder in the destinationFolderId parameter.