Contact UsGet started

Apache Kafka® version upgrade

Written by
Updated at November 26, 2024

You can upgrade a Managed Service for Apache Kafka® cluster to version 3.5 or lower. Upgrading to 3.6 or higher is not supported; however, you can create a cluster with one of these versions.

We recommend updating Apache Kafka® step by step, without skipping any versions. For example, upgrading the version from 2.8 to 3.1 is performed in the following sequence: 2.8 → 3.0 → 3.1.

To learn more about updates within a single version and host maintenance, see Maintenance.

Before a version upgrade

Make sure this does not affect your applications: review the Apache Kafka® change log.

Upgrading a cluster

Alert

Once upgraded, you cannot roll an Apache Kafka® cluster back to the previous version.

During an upgrade, topics may be unavailable if their replication factor is 1.

  1. Go to the folder page and select Managed Service for Kafka.
  2. In the cluster row, click , then select Edit.
  3. In the Version field, select a new version number.
  4. Click Save.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

  1. Get a list of your Managed Service for Apache Kafka® clusters:

    yc managed-kafka cluster list

  2. Get information about the cluster you need and check the version in the config.version property:

    yc managed-kafka cluster get <cluster_name_or_ID>

  3. Run the Apache Kafka® upgrade:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --version=<new_version_number>

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about creating this file, see Creating clusters.

  2. In the config section of the required Managed Service for Apache Kafka® cluster, add the version field (the Apache Kafka® version) or edit the existing one:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  config {
    version = "<version>"
  }
}

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To update a cluster, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

  • Cluster ID, in the clusterId parameter. You can get it with a list of clusters in the folder.
  • Apache Kafka® version number in the configSpec.version parameter.
  • List of cluster configuration fields to update in the updateMask parameter.

Warning

The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

Examples

Let's assume you need to upgrade your cluster from version 2.8 to version 3.0.

  1. To get a list of clusters and find out their IDs and names, run this command:

    yc managed-kafka cluster list

    +----------------------+---------------+---------------------+--------+---------+
|          ID          |     NAME      |     CREATED AT      | HEALTH | STATUS  |
+----------------------+---------------+---------------------+--------+---------+
| c9q8p8j2gaih******** |    kafka35    | 2021-10-23 12:44:17 | ALIVE  | RUNNING |
+----------------------+---------------+---------------------+--------+---------+

  2. To get information about a cluster named kafka35, run the following command:

    yc managed-kafka cluster get kafka35

      id: c9q8p8j2gaih********
  ...
  config:
    version: "2.8"
    ...

  3. To upgrade the kafka35 cluster to version 3.0, run this command:

    yc managed-kafka cluster update kafka35 --version=3.0
Previous
Updating cluster settings
Next
Managing disk space