Migrating Apache Kafka® cluster hosts to a different availability zone

Written by

Updated at May 5, 2025

Migrating a single-host cluster
- Migrating a single-host cluster using Yandex Cloud interfaces
- Migrating a single-host cluster with the help of auxiliary tools
Migrating a multi-host cluster
- Examples of migrating a multi-host cluster
Specifics of migration in Yandex Data Transfer

Managed Service for Apache Kafka® cluster hosts reside in Yandex Cloud availability zones. You can move Apache Kafka® hosts from one availability zone to another. The migration process is different for single-host and multi-host clusters.

Note

Not available for clusters with hosts residing in the ru-central1-d availability zone:

Intel Broadwell platform
Local SSD storage if using Intel Cascade Lake

If a Managed Service for Apache Kafka® cluster is an endpoint in Yandex Data Transfer, restart the transfer to make it work correctly. For more information on which transfers to restart and how to do it, see Specifics of migration in Yandex Data Transfer.

Migrating a single-host cluster

There are several ways to perform the migration:

Use the Yandex Cloud interfaces. In this case, the availability zone in the cluster's configuration will be updated. You do not need to create an additional cluster.

This option is easier to implement; however, the cluster will remain idle during migration while its availability zone is updating. This may take several minutes.
Use auxiliary tools offered by MirrorMaker or Yandex Data Transfer. In this case, a new cluster will be created, and the data from the initial cluster will be transferred into it.

This is a more difficult option, as you will need to create and configure the infrastructure. However, it allows you to avoid idle time. You can maintain two clusters with up-to-date data until you delete the original one.

Migrating a single-host cluster using Yandex Cloud interfaces

To move a Managed Service for Apache Kafka® host to a different availability zone in an Apache Kafka® cluster:

Create a subnet in the availability zone to which you are migrating the cluster.
If the cluster security group is set up for a subnet in the availability zone from which you are migrating the cluster, reconfigure the group for the new subnet. To do so, replace the source subnet CIDR with the new subnet CIDR in the security group rules.
Change the availability zone of the cluster and its Apache Kafka® host:
Management console

CLI

Terraform

REST API

gRPC API
1. Go to the folder page and select Managed Service for Kafka.
2. In the cluster row, click , then select Edit.
3. Under Network settings, specify a new availability zone.
4. Specify a subnet in the new availability zone if the zone has more than one subnet.
5. Click Save.
If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.

To change the availability zone for a cluster and its Apache Kafka® host, run this command:
```
yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zone> \
   --subnet-ids <subnet_ID>
```
If there is only one subnet in the new availability zone, you do not need to specify the --subnet-ids parameter.
1. Open the current Terraform configuration file with an infrastructure plan.
  
  For more information about creating this file, see Creating an Apache Kafka® cluster.
2. In the Managed Service for Apache Kafka® cluster description, specify the new subnet under subnet_ids and the new availability zone under zones:
  resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... subnet_ids = ["<subnet>"] config { ... zones = ["<availability_zone>"] } ... }
  If there is only one subnet in the new availability zone, you do not need to specify the subnet_ids parameter.
3. Make sure the settings are correct.
  1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
  2. Run this command:
    
    terraform validate
    
    Terraform will show any errors found in your configuration files.
4. Confirm updating the resources.
  1. Run this command to view the planned changes:
    
    terraform plan
    
    If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
  2. If everything looks correct, apply the changes:
    
    Run this command:
    
    terraform apply
    
    Confirm updating the resources.
    
    Wait for the operation to complete.
For more information, see the Terraform provider documentation.
Time limits
The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?
Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } }
1. Get an IAM token for API authentication and put it into the environment variable:
  export IAM_TOKEN="<IAM_token>"
2. Use the Cluster.update method and send the following request, e.g., via cURL:
  
  Warning
  
  The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.
  curl \ --request PATCH \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \ --data '{ "updateMask": "configSpec.zoneId,subnetIds", "subnetIds": [ "<subnet>" ] "configSpec": { "zoneId": [ "<availability_zone>" ] } }'
  Where:
  - updateMask: List of parameters to update as a single string, separated by commas.
    
    Specify the relevant parameters:
    
    subnetIds: To change the list of subnets.
    
    configSpec.zoneId: To change an availability zone.
  - subnetIds: Array of strings. Each string is a subnet ID. If there is only one subnet in the new availability zone, you do not need to specify the subnetIds parameter.
  - zoneId: New cluster availability zone.
  You can request the cluster ID with the list of clusters in the folder.
3. View the server response to make sure the request was successful.
1. Get an IAM token for API authentication and put it into the environment variable:
  export IAM_TOKEN="<IAM_token>"
2. Clone the cloudapi repository:
  cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
  Below, we assume the repository contents are stored in the ~/cloudapi/ directory.
3. Use the ClusterService/Update call and send the following request, e.g., via gRPCurl:
  Warning
  
  The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.
  
  Format for listing settings
  
  "update_mask": { "paths": [ "<setting_1>", "<setting_2>", ... "<setting_N>" ] }
  grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "update_mask": { "paths": [ "subnet_ids", "config_spec.zone_id" ] }, "subnet_ids": [ "<subnet>" ] "config_spec": { "zone_id": [ "<availability_zone>" ] } }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.kafka.v1.ClusterService.Update
  Where:
  - update_mask: List of parameters to update as an array of paths[] strings.
    
    Specify the relevant parameters:
    
    subnet_ids: To change the list of subnets.
    
    config_spec.zone_id: To change an availability zone.
  - subnet_ids: Array of strings. Each string is a subnet ID. If there is only one subnet in the new availability zone, you do not need to specify the subnet_ids parameter.
  - zone_id: New cluster availability zone.
  You can request the cluster ID with the list of clusters in the folder.
4. View the server response to make sure the request was successful.
To successfully connect to topics after the migration is complete, specify the FQDN of the new broker in your backend or client (e.g., in the code or graphical IDE). Delete the FQDN of the original broker in the initial availability zone.

To find out the FQDN, get a list of hosts in the cluster:
Management console

CLI

REST API

gRPC API
1. In the management console, go to the relevant folder.
2. In the list of services, select Managed Service for Kafka.
3. Click the name of the cluster you need and select the Hosts tab.
```
yc managed-kafka cluster list-hosts <cluster_name_or_ID>
```
The FQDN is specified in the command output under NAME.
1. Use the Cluster.listHosts method and send the following request, e.g., via cURL:
  curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'
  You can get the cluster ID with a list of clusters in the folder.
2. View the server response to make sure the request was successful. The FQDN is specified in the response under hosts[].name.
1. Use the ClusterService/ListHosts call and send the following request, e.g., via gRPCurl:
  grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts
  You can get the cluster ID with a list of clusters in the folder.
2. View the server response to make sure the request was successful. The FQDN is specified in the response under hosts[].name.

Migrating a single-host cluster with the help of auxiliary tools

To move a Managed Service for Apache Kafka® host to a different availability zone in an Apache Kafka® cluster:

Create a subnet in the availability zone to which you are migrating the cluster.
If the cluster security group is set up for a subnet in the availability zone from which you are migrating the cluster, reconfigure the group for the new subnet. To do so, replace the source subnet CIDR with the new subnet CIDR in the security group rules.
Create a Managed Service for Apache Kafka® cluster with a configuration different from the source cluster's configuration only by subnet and security group.
Migrate data from the initial cluster to the new one using one of the following tools:
- MirrorMaker: You can use either the MirrorMaker connector embedded in Managed Service for Apache Kafka® or the MirrorMaker utility.
- Data Transfer.
To successfully connect to topics after the migration is complete, specify the FQDN of the new cluster's broker in your backend or client (e.g., in the code or graphical IDE). Delete the FQDN of the original cluster's broker in the initial availability zone.

To find out the FQDN, get a list of hosts in the cluster:
Management console

CLI

REST API

gRPC API
1. In the management console, go to the relevant folder.
2. In the list of services, select Managed Service for Kafka.
3. Click the name of the cluster you need and select the Hosts tab.
```
yc managed-kafka cluster list-hosts <cluster_name_or_ID>
```
The FQDN is specified in the command output under NAME.
1. Use the Cluster.listHosts method and send the following request, e.g., via cURL:
  curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'
  You can get the cluster ID with a list of clusters in the folder.
2. View the server response to make sure the request was successful. The FQDN is specified in the response under hosts[].name.
1. Use the ClusterService/ListHosts call and send the following request, e.g., via gRPCurl:
  grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts
  You can get the cluster ID with a list of clusters in the folder.
2. View the server response to make sure the request was successful. The FQDN is specified in the response under hosts[].name.
Delete the initial Managed Service for Apache Kafka® cluster.

Migrating a multi-host cluster

Note

Clusters running Apache Kafka® 3.6 or higher do not use ZooKeeper hosts. Migration is only possible for configurations with three brokers in the same availability zone.

If you create an Apache Kafka® 3.5 cluster out of more than one broker host, three dedicated ZooKeeper hosts will be automatically added to the cluster. Each host is assigned a subnet from different availability zones. After you create a cluster, you cannot change its subnet in an availability zone.

The migration process of an Apache Kafka® 3.5 cluster depends on what availability zones Apache Kafka® and ZooKeeper hosts reside in prior to migration and how many subnets there are in each availability zone. See these examples to better understand how migration works.

To move Apache Kafka® hosts to a different availability zone in an Apache Kafka® 3.5 cluster:

Find out in which availability zones Apache Kafka® and ZooKeeper hosts reside:
Management console

CLI

REST API

gRPC API
1. In the management console, go to the relevant folder.
2. From the list of services, select Managed Service for Kafka.
3. Click the name of the cluster you need and select the Hosts tab. The Availability zone column lists the availability zones for each host.
If you do not have the Yandex Cloud CLI yet, install and initialize it.

The folder specified when creating the CLI profile is used by default. To change the default folder, use the yc config set folder-id <folder_ID> command. You can specify a different folder using the --folder-name or --folder-id parameter.
```
yc managed-kafka cluster list-hosts <cluster_name_or_ID>
```
The availability zone is specified in the command output under ZONE ID.
1. Get an IAM token for API authentication and put it into the environment variable:
  export IAM_TOKEN="<IAM_token>"
2. Use the Cluster.listHosts method and send the following request, e.g., via cURL:
  curl \ --request GET \ --header "Authorization: Bearer $IAM_TOKEN" \ --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'
  You can request the cluster ID with the list of clusters in the folder.
3. View the server response to make sure the request was successful. The availability zone is specified in the response under hosts[].zoneId.
1. Get an IAM token for API authentication and put it into the environment variable:
  export IAM_TOKEN="<IAM_token>"
2. Clone the cloudapi repository:
  cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi
  Below, we assume the repository contents are stored in the ~/cloudapi/ directory.
3. Use the ClusterService/ListHosts call and send the following request, e.g., via gRPCurl:
  grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>" }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts
  You can request the cluster ID with the list of clusters in the folder.
4. View the server response to make sure the request was successful. The availability zone is specified in the response under hosts[].zone_id.
If the target availability zone for Apache Kafka® hosts is not on the list, create a subnet in this zone.

If the target availability zone is on the list, during migration, Apache Kafka® hosts will be moved to a subnet where Apache Kafka® or ZooKeeper hosts already reside in this zone.
Check the cluster security group. If it is set up for a subnet in the source availability zone, reconfigure the group for the subnet in the target availability zone. To do so, replace the source subnet CIDR with the target subnet CIDR in the security group rules.
Change the availability zone of the cluster and its Apache Kafka® host:
Management console

CLI

Terraform

REST API

gRPC API
1. Go to the folder page and select Managed Service for Kafka.
2. In the cluster row, click , then select Edit.
3. Under Network settings, specify a new set of availability zones. Their number must not decrease.
  
  Warning
  
  After adding a new availability zone, deselect one of the old zones. Otherwise, you will not be able to delete an old availability zone after you save the settings.
4. Specify a subnet in the new availability zone if:
  - Apache Kafka® or ZooKeeper hosts were not previously placed in the target availability zone.
  - The target availability zone has more than one subnet.
5. Click Save.
To edit availability zones for a cluster and its Apache Kafka® hosts, run this command:
```
yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zones> \
   --subnet-ids <subnet_IDs>
```
In the --zone-ids parameter, list the availability zones separated by commas. Their number must not decrease.

Warning

After adding a new availability zone, delete one of the old zones from the list. Otherwise, you will not be able to delete an old availability zone after you run the command.

In the --subnet-ids parameter, list the subnets in the ru-central1-a, ru-central1-b, and ru-central1-d availability zones, separated by commas. You must specify subnets for these zones even if the Apache Kafka® hosts are placed in a smaller number of availability zones. You need all three availability zones for ZooKeeper hosts.
1. Open the current Terraform configuration file with an infrastructure plan.
  
  For more information about creating this file, see Creating an Apache Kafka® cluster.
2. In the Managed Service for Apache Kafka® cluster description, change the list of availability zones under zones:
  resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... config { ... zones = ["<availability_zones>"] } ... }
  The number of availability zones must not decrease.
  
  Warning
  
  After adding a new availability zone, delete one of the old zones from the list. Otherwise, you will not be able to delete an old availability zone after you save the settings.
3. Change the list of subnets under subnet_ids if these two conditions are met:
  - You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
  - The target availability zone has more than one subnet.
  If the cluster hosts are placed in the ru-central1-a and ru-central1-b availability zones, and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if there are multiple subnets in the ru-central1-d zone. Otherwise, you do not need to specify a subnet.
  resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... subnet_ids = ["<subnets>"] ... }
4. Make sure the settings are correct.
  1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
  2. Run this command:
    
    terraform validate
    
    Terraform will show any errors found in your configuration files.
5. Confirm updating the resources.
  1. Run this command to view the planned changes:
    
    terraform plan
    
    If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
  2. If everything looks correct, apply the changes:
    
    Run this command:
    
    terraform apply
    
    Confirm updating the resources.
    
    Wait for the operation to complete.
For more information, see the Terraform provider documentation.
Time limits
The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?
Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } }
1. Use the Cluster.update method and send the following request, e.g., via cURL:
  
  Warning
  
  The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.
  curl \ --request PATCH \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \ --data '{ "updateMask": "configSpec.zoneId,subnetIds", "subnetIds": [ <subnet_list> ], "configSpec": { "zoneId": [ <list_of_availability_zones> ] } }'
  Where:
  - updateMask: List of parameters to update as a single string, separated by commas.
    
    Specify the relevant parameters:
    
    subnetIds: To change the list of subnets.
    
    configSpec.zoneId: To change the list of availability zones.
  - subnetIds: Array of strings. Each string is a subnet ID. Change the list of subnets if these two conditions are met:
    
    You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
    
    The target availability zone has more than one subnet.
    
    If the cluster hosts are placed in the ru-central1-a, ru-central1-b, and ru-central1-c availability zones, and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if there are multiple subnets in the ru-central1-d zone. Otherwise, you do not need to specify a subnet.
  - zoneId: New set of cluster availability zones. Their number must not decrease.
  You can request the cluster ID with the list of clusters in the folder.
2. View the server response to make sure the request was successful.
1. Use the ClusterService/Update call and send the following request, e.g., via gRPCurl:
  Warning
  
  The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.
  
  Format for listing settings
  
  "update_mask": { "paths": [ "<setting_1>", "<setting_2>", ... "<setting_N>" ] }
  grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "update_mask": { "paths": [ "subnet_ids", "config_spec.zone_id" ] }, "subnet_ids": [ <subnet_list> ], "config_spec": { "zone_id": [ <list_of_availability_zones> ] } }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.kafka.v1.ClusterService.Update
  Where:
  - update_mask: List of parameters to update as an array of paths[] strings.
    
    Specify the relevant parameters:
    
    subnet_ids: To change the list of subnets.
    
    config_spec.zone_id: To change the list of availability zones.
  - subnet_ids: Array of strings. Each string is a subnet ID. Change the list of subnets if these two conditions are met:
    
    You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
    
    The target availability zone has more than one subnet.
    
    If the cluster hosts are placed in the ru-central1-a, ru-central1-b, and ru-central1-c availability zones, and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if there are multiple subnets in the ru-central1-d zone. Otherwise, you do not need to specify a subnet.
  - zone_id: New set of cluster availability zones. Their number must not decrease.
  You can request the cluster ID with the list of clusters in the folder.
2. View the server response to make sure the request was successful.

To move Apache Kafka® hosts to a different availability zone in an Apache Kafka® cluster version 3.6 or higher:

Create a subnet in the target availability zone you are migrating Apache Kafka® hosts to.
Check the cluster security group. If it is set up for a subnet in the source availability zone, reconfigure the group for the subnet in the target availability zone. To do so, replace the source subnet CIDR with the target subnet CIDR in the security group rules.
Change the availability zone of the cluster and its Apache Kafka® host:
CLI

Terraform

REST API

gRPC API
To change the availability zone for a cluster and its Apache Kafka® hosts, run this command:
```
yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zone> \
   --subnet-ids <subnet_ID>
```
1. Open the current Terraform configuration file with an infrastructure plan.
  
  For more information about creating this file, see Creating an Apache Kafka® cluster.
2. In the Managed Service for Apache Kafka® cluster description, change the availability zone in the zones parameter:
  resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... config { ... zones = ["<availability_zone>"] } ... }
3. In the subnet_ids parameter, specify the subnet in the target availability zone.
  resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... subnet_ids = ["<subnet>"] ... }
4. Make sure the settings are correct.
  1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.
  2. Run this command:
    
    terraform validate
    
    Terraform will show any errors found in your configuration files.
5. Confirm updating the resources.
  1. Run this command to view the planned changes:
    
    terraform plan
    
    If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.
  2. If everything looks correct, apply the changes:
    
    Run this command:
    
    terraform apply
    
    Confirm updating the resources.
    
    Wait for the operation to complete.
For more information, see the Terraform provider documentation.
Time limits
The Terraform provider limits the amount of time for all Managed Service for Apache Kafka® cluster operations to complete to 60 minutes.

Operations exceeding the set timeout are interrupted.

How do I change these limits?
Add the timeouts block to the cluster description, for example:

resource "yandex_mdb_kafka_cluster" "<cluster_name>" { ... timeouts { create = "1h30m" # 1 hour 30 minutes update = "2h" # 2 hours delete = "30m" # 30 minutes } }
1. Use the Cluster.update method and send the following request, e.g., via cURL:
  
  Warning
  
  The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.
  curl \ --request PATCH \ --header "Authorization: Bearer $IAM_TOKEN" \ --header "Content-Type: application/json" \ -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \ --data '{ "updateMask": "configSpec.zoneId,subnetIds", "subnetIds": [ <subnet_in_target_availability_zone> ], "configSpec": { "zoneId": [ <target_availability_zone> ] } }'
  Where:
  - updateMask: List of parameters to update as a single string, separated by commas.
    
    In this case, specify the subnetIds and configSpec.zoneId parameters.
  - subnetIds: Array of strings. Each string is a subnet ID. Specify only the subnet in the target availability zone.
  - zoneId: New set of cluster availability zones. Specify only the target availability zone.
  You can request the cluster ID with the list of clusters in the folder.
2. View the server response to make sure the request was successful.
1. Use the ClusterService/Update call and send the following request, e.g., via gRPCurl:
  Warning
  
  The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.
  
  Format for listing settings
  
  "update_mask": { "paths": [ "<setting_1>", "<setting_2>", ... "<setting_N>" ] }
  grpcurl \ -format json \ -import-path ~/cloudapi/ \ -import-path ~/cloudapi/third_party/googleapis/ \ -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \ -rpc-header "Authorization: Bearer $IAM_TOKEN" \ -d '{ "cluster_id": "<cluster_ID>", "update_mask": { "paths": [ "subnet_ids", "config_spec.zone_id" ] }, "subnet_ids": [ <subnet_in_target_availability_zone> ], "config_spec": { "zone_id": [ <target_availability_zone> ] } }' \ mdb.api.cloud.yandex.net:443 \ yandex.cloud.mdb.kafka.v1.ClusterService.Update
  Where:
  - update_mask: List of parameters to update as an array of paths[] strings.
    
    In this case, the array consists of the subnetIds and configSpec.zoneId strings.
  - subnetIds: Array of strings. Each string is a subnet ID. Specify only the subnet in the target availability zone.
  - zoneId: New set of cluster availability zones. Specify only the target availability zone.
  You can request the cluster ID with the list of clusters in the folder.
2. View the server response to make sure the request was successful.

To successfully connect to topics after the migration is complete, specify the FQDN of the new broker in your backend or client (e.g., in the code or graphical IDE). Delete the FQDN of the original broker in the initial availability zone.

To find out the FQDN, get a list of hosts in the cluster:

Management console

CLI

REST API

gRPC API

In the management console, go to the relevant folder.
In the list of services, select Managed Service for Kafka.
Click the name of the cluster you need and select the Hosts tab.

yc managed-kafka cluster list-hosts <cluster_name_or_ID>

The FQDN is specified in the command output under NAME.

Use the Cluster.listHosts method and send the following request, e.g., via cURL:

curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'

You can get the cluster ID with a list of clusters in the folder.

View the server response to make sure the request was successful. The FQDN is specified in the response under hosts[].name.

Use the ClusterService/ListHosts call and send the following request, e.g., via gRPCurl:

grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
            "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts

You can get the cluster ID with a list of clusters in the folder.

View the server response to make sure the request was successful. The FQDN is specified in the response under hosts[].name.

Examples of migrating a multi-host cluster

The examples below provide insight into situations where you need to specify a subnet to migrate a Managed Service for Apache Kafka® 3.5 multi-host cluster to a different availability zone.

A ZooKeeper host already resides in the target availability zone

Let's assume your cluster comprises the hosts:

Apache Kafka® Host 1 in the ru-central1-a availability zone
Apache Kafka® Host 2 in the ru-central1-c availability zone
ZooKeeper Host 1 in the ru-central1-a availability zone
ZooKeeper Host 2 in the ru-central1-b availability zone
ZooKeeper Host 3 in the ru-central1-d availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a and ru-central1-d availability zones.

ZooKeeper Host 3 already resides in the ru-central1-d zone, which means it belongs to one of the subnets in this availability zone. This subnet will be used for migration, since you cannot change a cluster's subnets after you create it. As a result, you need not specify a subnet when performing the migration.

There are no hosts and there is one subnet in the target availability zone

Let's assume your cluster comprises the hosts:

Apache Kafka® Host 1 in the ru-central1-a availability zone
Apache Kafka® Host 2 in the ru-central1-b availability zone
Apache Kafka® Host 3 in the ru-central1-c availability zone
ZooKeeper Host 1 in the ru-central1-a availability zone
ZooKeeper Host 2 in the ru-central1-b availability zone
ZooKeeper Host 3 in the ru-central1-c availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.

No hosts initially reside in the ru-central1-d zone. But there is only one subnet in it, so you need not specify a subnet when performing the migration.

There are no hosts and there are multiple subnets in the target availability zone

Let's assume your cluster comprises the hosts:

Apache Kafka® Host 1 in the ru-central1-a availability zone
Apache Kafka® Host 2 in the ru-central1-b availability zone
Apache Kafka® Host 3 in the ru-central1-c availability zone
ZooKeeper Host 1 in the ru-central1-a availability zone
ZooKeeper Host 2 in the ru-central1-b availability zone
ZooKeeper Host 3 in the ru-central1-c availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.

No hosts initially reside in the ru-central1-d zone. However, there are multiple subnets, so you need to specify a subnet to perform the migration.

Specifics of migration in Yandex Data Transfer

If your cluster is used as an endpoint when transferring data with Data Transfer, and the transfer type is Replication or Snapshot and increment, restart the transfer after migrating the cluster. This way, the transfer will get data about the cluster's new topology.

You do not need to restart Snapshot transfers, as information about the new topology is provided automatically while activating them.

To restart a transfer, choose one of the two methods:

Deactivate the transfer and wait for its status to change to Stopped. Next, reactivate the transfer and wait for its status to change to Replicating.
Update any setting for the transfer or endpoint.

For more information, see Migrating a Data Transfer transfer and endpoints to a different availability zone.

For an example of data transfer using Data Transfer, see this tutorial.

Migrating Apache Kafka® cluster hosts to a different availability zone

Migrating a single-host clusterMigrating a single-host cluster

Migrating a single-host cluster using Yandex Cloud interfacesMigrating a single-host cluster using Yandex Cloud interfaces

Migrating a single-host cluster with the help of auxiliary toolsMigrating a single-host cluster with the help of auxiliary tools

Migrating a multi-host clusterMigrating a multi-host cluster

Examples of migrating a multi-host clusterExamples of migrating a multi-host cluster

A ZooKeeper host already resides in the target availability zoneA ZooKeeper host already resides in the target availability zone

There are no hosts and there is one subnet in the target availability zoneThere are no hosts and there is one subnet in the target availability zone

There are no hosts and there are multiple subnets in the target availability zoneThere are no hosts and there are multiple subnets in the target availability zone

Specifics of migration in Yandex Data TransferSpecifics of migration in Yandex Data Transfer

Was the article helpful?

Migrating a single-host cluster

Migrating a single-host cluster using Yandex Cloud interfaces

Migrating a single-host cluster with the help of auxiliary tools

Migrating a multi-host cluster

Examples of migrating a multi-host cluster

A ZooKeeper host already resides in the target availability zone

There are no hosts and there is one subnet in the target availability zone

There are no hosts and there are multiple subnets in the target availability zone

Specifics of migration in Yandex Data Transfer