Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Upgrading Apache Kafka® version

Written by
Updated at October 17, 2024

You can upgrade a Managed Service for Apache Kafka® cluster to version 3.5 or lower. Upgrading to 3.6 or higher is not supported; however, you can create a cluster with one of these versions.

We recommend updating Apache Kafka® step by step, without skipping any versions. For example, to upgrade from 2.8 to 3.1, follow these steps: 2.8 → 3.0 → 3.1.

To learn more about updates within a single version and host maintenance, see Maintenance.

Before a version upgrade

Make sure this does not affect your applications: review the Apache Kafka® change log.

Upgrading a cluster

Alert

Once upgraded, you cannot roll an Apache Kafka® cluster back to the previous version.

During an upgrade, topics may be unavailable if their replication factor is 1.

  1. Go to the folder page and select Managed Service for Kafka.
  2. In the appropriate cluster row, click and select Edit.
  3. In the Version field, select a new version number.
  4. Click Save.

If you do not have the Yandex Cloud command line interface yet, install and initialize it.

The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

  1. Get a list of your Managed Service for Apache Kafka® clusters:

    yc managed-kafka cluster list

  2. Get information about a cluster and check the version in the config.version parameter:

    yc managed-kafka cluster get <cluster_name_or_ID>

  3. Run the Apache Kafka® upgrade:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --version=<new_version_ID>

  1. Open the current Terraform configuration file with an infrastructure plan.

    For more information about how to create this file, see Creating clusters.

  2. Add a version (Apache Kafka® version) field to the config section of the appropriate Managed Service for Apache Kafka® cluster or edit the field value if the field already exists:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  config {
    version = "<version>"
  }
}

  3. Make sure the settings are correct.

    1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

    2. Run the command:

      terraform validate

      If there are errors in the configuration files, Terraform will point to them.

  4. Confirm updating the resources.

    1. Run the command to view planned changes:

      terraform plan

      If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

    2. If you are happy with the planned changes, apply them:

      1. Run the command:

        terraform apply

      2. Confirm the update of resources.

      3. Wait for the operation to complete.

For more information, see the Terraform provider documentation.

Time limits

The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?

Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

To update a cluster, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

  • Cluster ID in the clusterId parameter. You can get it with a list of clusters in the folder.
  • Apache Kafka® version number in the configSpec.version parameter.
  • List of cluster configuration fields to update in the UpdateMask parameter.

Warning

This API method overrides all parameters of the object being modified that were not explicitly passed in the request to the default values. To avoid this, list the settings you want to change in the updateMask parameter (one line separated by commas).

Examples

Let's assume you need to upgrade your cluster from version 2.8 to version 3.0.

  1. To get a list of clusters and find out their IDs and names, run this command:

    yc managed-kafka cluster list

    +----------------------+---------------+---------------------+--------+---------+
|          ID          |     NAME      |     CREATED AT      | HEALTH | STATUS  |
+----------------------+---------------+---------------------+--------+---------+
| c9q8p8j2gaih******** |    kafka35    | 2021-10-23 12:44:17 | ALIVE  | RUNNING |
+----------------------+---------------+---------------------+--------+---------+

  2. To get information about the cluster kafka35, run the following command:

    yc managed-kafka cluster get kafka35

      id: c9q8p8j2gaih********
  ...
  config:
    version: "2.8"
    ...

  3. To upgrade the kafka35 cluster to version 3.0, run the following command:

    yc managed-kafka cluster update kafka35 --version=3.0
Previous
Updating cluster settings
Next
Managing disk space