Contact UsGet started
© 2024 Iron Hive LLC Belgrade

Migrating Apache Kafka® cluster hosts to a different availability zone

Written by
Updated at November 18, 2024

Managed Service for Apache Kafka® cluster hosts reside in Yandex Cloud availability zones. You can move Apache Kafka® hosts from one availability zone to another. The migration process is different for single-host and multi-host clusters.

Note

Not available for clusters with hosts residing in the ru-central1-d availability zone:

  • Intel Broadwell platform
  • Local SSD storage if using Intel Cascade Lake

If a Managed Service for Apache Kafka® cluster is an endpoint in Yandex Data Transfer, restart the transfer to make it work correctly. For more information on which transfers to restart and how to do it, see Specifics of migration in Yandex Data Transfer.

Migrating a single-host cluster

There are several ways to perform the migration:

  • Use the Yandex Cloud interfaces. In this case, the availability zone in the cluster's configuration will be updated. You do not need to create an additional cluster.

    This option is easier to implement; however, the cluster will remain idle during migration while its availability zone is updating. This may take several minutes.

  • Use auxiliary tools offered by MirrorMaker or Yandex Data Transfer. In this case, a new cluster will be created, and the data from the initial cluster will be transferred into it.

    This is a more difficult option, as you will need to create and configure the infrastructure. However, it allows you to avoid idle time. You can maintain two clusters with up-to-date data until you delete the original one.

Migrating a single-host cluster using Yandex Cloud interfaces

To move a Managed Service for Apache Kafka® host to a different availability zone in an Apache Kafka® cluster:

  1. Create a subnet in the availability zone to which you are migrating the cluster.

  2. If the cluster security group is set up for a subnet in the availability zone from which you are migrating the cluster, reconfigure the group for the new subnet. To do so, replace the source subnet CIDR with the new subnet CIDR in the security group rules.

  3. Change the availability zone of the cluster and its Apache Kafka® host:

    1. Go to the folder page and select Managed Service for Kafka.
    2. In the cluster row, click , then select Edit.
    3. Under Network settings, specify a new availability zone.
    4. Specify a subnet in the new availability zone if the zone has more than one subnet.
    5. Click Save.

    If you do not have the Yandex Cloud command line interface yet, install and initialize it.

    The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

    To change the availability zone of the cluster and its Apache Kafka® host, run this command:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zone> \
   --subnet-ids <subnet_ID>

    If there is only one subnet in the new availability zone, you do not need to specify the --subnet-ids parameter.

    1. Open the current Terraform configuration file with an infrastructure plan.

      For more information about creating this file, see Creating an Apache Kafka® cluster.

    2. In the Managed Service for Apache Kafka® cluster description, specify the new subnet under subnet_ids and the new availability zone under zones:

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  subnet_ids = ["<subnet>"]
  config {
    ...
    zones = ["<availability_zone>"]
  }
  ...
}

      If there is only one subnet in the new availability zone, you do not need to specify the subnet_ids parameter.

    3. Make sure the settings are correct.

      1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

      2. Run the command:

        terraform validate

        If there are errors in the configuration files, Terraform will point to them.

    4. Confirm updating the resources.

      1. Run the command to view planned changes:

        terraform plan

        If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

        1. Run the command:

          terraform apply

        2. Confirm the update of resources.

        3. Wait for the operation to complete.

    For more information, see the Terraform provider documentation.

    Time limits

    The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

    Operations exceeding the set timeout are interrupted.

    How do I change these limits?

    Add the timeouts block to the cluster description, for example:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

    To edit the availability zone of a cluster and its Apache Kafka® host, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

    • Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.
    • New availability zone in the configSpec.zoneId parameter.
    • New subnet in the subnetIds parameter if the new availability zone has more than one subnet.
    • List of settings to update in the updateMask parameter.

    Warning

    The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

  4. To successfully connect to topics after the migration is complete, specify the FQDN of the new broker in your backend or client (e.g., in the code or graphical IDE). Delete the FQDN of the original broker in the initial availability zone.

    To find out the FQDN, get a list of hosts in the cluster:

    1. In the management console, go to the relevant folder.
    2. In the list of services, select Managed Service for Kafka.
    3. Click the name of the cluster you need and select the Hosts tab.
    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    The FQDN is specified in the command output under NAME.

    Use the listHosts REST API method for the Cluster resource or the ClusterService/listHosts gRPC API call and provide the cluster ID in the clusterId request parameter. To find out the cluster ID, get a list of clusters in the folder.

    The FQDN is specified in the response under hosts[].name.

Migrating a single-host cluster with the help of auxiliary tools

To move a Managed Service for Apache Kafka® host to a different availability zone in an Apache Kafka® cluster:

  1. Create a subnet in the availability zone to which you are migrating the cluster.

  2. If the cluster security group is set up for a subnet in the availability zone from which you are migrating the cluster, reconfigure the group for the new subnet. To do so, replace the source subnet CIDR with the new subnet CIDR in the security group rules.

  3. Create a Managed Service for Apache Kafka® cluster with a configuration different from the source cluster's configuration only by subnet and security group.

  4. Migrate data from the initial cluster to the new one using one of the following tools:

    • MirrorMaker: You can use either the MirrorMaker connector embedded in Managed Service for Apache Kafka® or the MirrorMaker utility.
    • Data Transfer​.

  5. To successfully connect to topics after the migration is complete, specify the FQDN of the new cluster's broker in your backend or client (e.g., in the code or graphical IDE). Delete the FQDN of the original cluster's broker in the initial availability zone.

    To find out the FQDN, get a list of hosts in the cluster:

    1. In the management console, go to the relevant folder.
    2. In the list of services, select Managed Service for Kafka.
    3. Click the name of the cluster you need and select the Hosts tab.
    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    The FQDN is specified in the command output under NAME.

    Use the listHosts REST API method for the Cluster resource or the ClusterService/listHosts gRPC API call and provide the cluster ID in the clusterId request parameter. To find out the cluster ID, get a list of clusters in the folder.

    The FQDN is specified in the response under hosts[].name.

  6. Delete the initial Managed Service for Apache Kafka® cluster.

Migrating a multi-host cluster

If you create a cluster out of more than one Apache Kafka® host, three dedicated ZooKeeper hosts will be automatically added to the cluster. Each host is assigned a subnet from different availability zones. After you create a cluster, you cannot change its subnet in an availability zone.

The migration process depends on which availability zones Apache Kafka® and ZooKeeper hosts initially reside in and the number of subnets in each availability zone. See these examples to better understand how migration works.

To move Apache Kafka® hosts to a different availability zone in a cluster:

  1. Find out in which availability zones Apache Kafka® and ZooKeeper hosts reside:

    1. In the management console, go to the relevant folder.
    2. In the list of services, select Managed Service for Kafka.
    3. Click the name of the cluster you need and select the Hosts tab. The Availability zone column lists the availability zones for each host.
    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    The availability zone is specified in the command output under ZONE ID.

    Use the listHosts REST API method for the Cluster resource or the ClusterService/listHosts gRPC API call and provide the cluster ID in the clusterId query parameter. To find out the cluster ID, get a list of clusters in the folder.

    The availability zone is specified in the response under hosts[].zoneId.

  2. If the target availability zone for Apache Kafka® hosts is not on the list, create a subnet in this zone.

    If the target availability zone is on the list, during migration, Apache Kafka® hosts will be moved to a subnet where Apache Kafka® or ZooKeeper hosts already reside in this zone.

  3. Check the cluster security group. If it is set up for a subnet in the source availability zone, reconfigure the group for the subnet in the target availability zone. To do so, replace the source subnet CIDR with the target subnet CIDR in the security group rules.

  4. Change the availability zone of the cluster and its Apache Kafka® host:

    1. Go to the folder page and select Managed Service for Kafka.

    2. In the cluster row, click , then select Edit.

    3. Under Network settings, specify a new set of availability zones. Their number must not decrease.

      Warning

      After adding a new availability zone, deselect one of the old zones. Otherwise, you will not be able to delete an old availability zone after you save the settings.

    4. Specify a subnet in the new availability zone if:

      • Apache Kafka® or ZooKeeper hosts were not previously placed in the target availability zone.
      • The target availability zone has more than one subnet.

    5. Click Save.

    If you do not have the Yandex Cloud command line interface yet, install and initialize it.

    The folder specified in the CLI profile is used by default. You can specify a different folder using the --folder-name or --folder-id parameter.

    To edit availability zones for a cluster and its Apache Kafka® hosts, run this command:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zones> \
   --subnet-ids <subnet_IDs>

    In the --zone-ids parameter, list the availability zones separated by commas. Their number must not decrease.

    Warning

    After adding a new availability zone, delete one of the old zones from the list. Otherwise, you will not be able to delete an old availability zone after you run the command.

    In the --subnet-ids parameter, list the subnets in the ru-central1-a, ru-central1-b, and ru-central1-d availability zones, separated by commas. You must specify subnets for these zones even if the Apache Kafka® hosts are placed in a smaller number of availability zones. You need all three availability zones for ZooKeeper hosts.

    1. Open the current Terraform configuration file with an infrastructure plan.

      For more information about creating this file, see Creating an Apache Kafka® cluster.

    2. In the Managed Service for Apache Kafka® cluster description, change the list of availability zones under zones:

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  config {
    ...
    zones = ["<availability_zones>"]
  }
  ...
}

      The number of availability zones must not decrease.

      Warning

      After adding a new availability zone, delete one of the old zones from the list. Otherwise, you will not be able to delete an old availability zone after you save the settings.

    3. Change the list of subnets under subnet_ids if these two conditions are met:

      • You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
      • The target availability zone has more than one subnet.

      If the cluster hosts are placed in the ru-central1-a, ru-central1-b, and ru-central1-c availability zones and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if there are multiple subnets in the ru-central1-d zone. Otherwise, you do not need to specify a subnet.

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  subnet_ids = ["<subnets>"]
  ...
}

    4. Make sure the settings are correct.

      1. Using the command line, navigate to the folder that contains the up-to-date Terraform configuration files with an infrastructure plan.

      2. Run the command:

        terraform validate

        If there are errors in the configuration files, Terraform will point to them.

    5. Confirm updating the resources.

      1. Run the command to view planned changes:

        terraform plan

        If the resource configuration descriptions are correct, the terminal will display a list of the resources to modify and their parameters. This is a test step. No resources are updated.

      2. If you are happy with the planned changes, apply them:

        1. Run the command:

          terraform apply

        2. Confirm the update of resources.

        3. Wait for the operation to complete.

    For more information, see the Terraform provider documentation.

    Time limits

    The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

    Operations exceeding the set timeout are interrupted.

    How do I change these limits?

    Add the timeouts block to the cluster description, for example:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

    To edit the availability zones of a cluster and its Apache Kafka® host, use the update REST API method for the Cluster resource or the ClusterService/Update gRPC API call and provide the following in the request:

    • Cluster ID in the clusterId parameter. To find out the cluster ID, get a list of clusters in the folder.

    • New set of availability zones in the configSpec.zoneId parameter. Their number must not decrease.

    • New set of subnets in the subnetIds parameter if these two conditions are met:

      • You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
      • The target availability zone has more than one subnet.

      If the cluster hosts are placed in the ru-central1-a, ru-central1-b, and ru-central1-c availability zones and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if there are multiple subnets in the ru-central1-d zone. Otherwise, you do not need to specify a subnet.

    • List of settings to update in the updateMask parameter.

    Warning

    The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

  5. To successfully connect to topics after the migration is complete, specify the FQDN of the new broker in your backend or client (e.g., in the code or graphical IDE). Delete the FQDN of the original broker in the initial availability zone.

    To find out the FQDN, get a list of hosts in the cluster:

    1. In the management console, go to the relevant folder.
    2. In the list of services, select Managed Service for Kafka.
    3. Click the name of the cluster you need and select the Hosts tab.
    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    The FQDN is specified in the command output under NAME.

    Use the listHosts REST API method for the Cluster resource or the ClusterService/listHosts gRPC API call and provide the cluster ID in the clusterId request parameter. To find out the cluster ID, get a list of clusters in the folder.

    The FQDN is specified in the response under hosts[].name.

Examples of migrating a multi-host cluster

The examples below discuss the situations in which you need to specify a subnet to migrate a Managed Service for Apache Kafka® multi-host cluster to a different availability zone.

A ZooKeeper host already resides in the target availability zone

Let's assume your cluster comprises the hosts:

  • Apache Kafka® Host 1 in the ru-central1-a availability zone
  • Apache Kafka® Host 2 in the ru-central1-c availability zone
  • ZooKeeper Host 1 in the ru-central1-a availability zone
  • ZooKeeper Host 2 in the ru-central1-b availability zone
  • ZooKeeper Host 3 in the ru-central1-d availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a and ru-central1-d availability zones.

ZooKeeper Host 3 already resides in the ru-central1-d zone, which means it belongs to one of the subnets in this availability zone. This subnet will be used for migration, since you cannot change a cluster's subnets after you create it. As a result, you need not specify a subnet when performing the migration.

There are no hosts and there is one subnet in the target availability zone

Let's assume your cluster comprises the hosts:

  • Apache Kafka® Host 1 in the ru-central1-a availability zone
  • Apache Kafka® Host 2 in the ru-central1-b availability zone
  • Apache Kafka® Host 3 in the ru-central1-c availability zone
  • ZooKeeper Host 1 in the ru-central1-a availability zone
  • ZooKeeper Host 2 in the ru-central1-b availability zone
  • ZooKeeper Host 3 in the ru-central1-c availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.

No hosts initially reside in the ru-central1-d zone. But there is only one subnet in it, so you need not specify a subnet when performing the migration.

There are no hosts and there are multiple subnets in the target availability zone

Let's assume your cluster comprises the hosts:

  • Apache Kafka® Host 1 in the ru-central1-a availability zone
  • Apache Kafka® Host 2 in the ru-central1-b availability zone
  • Apache Kafka® Host 3 in the ru-central1-c availability zone
  • ZooKeeper Host 1 in the ru-central1-a availability zone
  • ZooKeeper Host 2 in the ru-central1-b availability zone
  • ZooKeeper Host 3 in the ru-central1-c availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.

No hosts initially reside in the ru-central1-d zone, but there are multiple subnets, so you need to specify a subnet when performing the migration.

Specifics of migration in Yandex Data Transfer

If your cluster is used as an endpoint when transferring data with Data Transfer, and the transfer type is Replication or Snapshot and increment, restart the transfer after migrating the cluster. This way, the transfer will get data about the cluster's new topology.

You do not need to restart Snapshot transfers, as information about the new topology is provided automatically while activating them.

To restart a transfer, choose one of the two methods:

For more information, see Migrating a Data Transfer transfer and endpoints to a different availability zone.

For an example of data transfer using Data Transfer, see this tutorial.

Previous
Getting a list of cluster hosts
Next
Deleting a cluster