Contact UsTry it for free
© 2026 Direct Cursus Technology L.L.C.

Migrating Apache Kafka® cluster hosts to a different availability zone

Written by
Updated at February 6, 2026

Managed Service for Apache Kafka® cluster hosts reside in Yandex Cloud availability zones. You can move Apache Kafka® hosts from one availability zone to another. The migration process is different for single-host and multi-host clusters.

Note

Clusters with hosts residing in the ru-central1-d availability zone do not support:

  • Intel Broadwell
  • Local SSD storage if using Intel Cascade Lake

If a Managed Service for Apache Kafka® cluster is an endpoint in Yandex Data Transfer, restart the transfer to make it work correctly. To find out which transfers to restart and how to do it, see Specifics of migration in Yandex Data Transfer.

Migrating a single-host cluster

You can migrate your cluster in the following ways:

  • Use the Yandex Cloud interfaces. In this case, the availability zone in the cluster configuration will change. You do not need to create an additional cluster.

    This option is easier to implement; however, the cluster will remain idle during migration while changing its availability zone. This may take a few minutes.

  • Use auxiliary tools offered by MirrorMaker or Yandex Data Transfer. In this case, you need to create a new cluster and transfer data from the source cluster into it.

    This option is more challenging to implement since it requires building and configuring the infrastructure. However, it allows you to avoid downtime: you can maintain two clusters with up-to-date data until you delete the source one.

Migrating a single-host cluster using Yandex Cloud interfaces

To move an Apache Kafka® host to a different availability zone in a Managed Service for Apache Kafka® cluster:

  1. Create a subnet in the target availability zone.

  2. If the cluster security group is set up for a subnet in the availability zone you are moving your cluster from, reconfigure the group for the new subnet. To do so, replace the source subnet CIDR with the new subnet CIDR in the security group rules.

  3. Change the availability zone of your cluster and its Apache Kafka® host:

    1. In the management console, navigate to the relevant folder.
    2. Navigate to the Managed Service for Kafka service.
    3. In the cluster row, click and select Edit.
    4. Under Network settings, specify the new availability zone.
    5. If the new availability zone has multiple subnets, specify the one you need.
    6. Click Save.

    If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

    By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

    To change the availability zone for a cluster and its Apache Kafka® host, run this command:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zone> \
   --subnet-ids <subnet_ID>

    If there is only one subnet in the new availability zone, you do not need to specify --subnet-ids.

    1. Open the current Terraform configuration file describing your infrastructure.

      Learn how to create this file in Creating an Apache Kafka® cluster.

    2. In the Managed Service for Apache Kafka® cluster description, specify the new subnet in subnet_ids and the new availability zone in zones:

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  subnet_ids = ["<subnet>"]
  config {
    ...
    zones = ["<availability_zone>"]
  }
  ...
}

      If there is only one subnet in the new availability zone, you do not need to specify subnet_ids.

    3. Make sure the settings are correct.

      1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

      2. Run this command:

        terraform validate

        Terraform will show any errors found in your configuration files.

    4. Confirm updating the resources.

      1. Run this command to view the planned changes:

        terraform plan

        If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

        1. Run this command:

          terraform apply

        2. Confirm updating the resources.

        3. Wait for the operation to complete.

    For more information, see this Terraform provider guide.

    Timeouts

    The Terraform provider limits the time for all operations with the Managed Service for Apache Kafka® cluster to 60 minutes.

    Operations exceeding the timeout are aborted.

    How do I change these limits?

    Add the timeouts section to your cluster description, such as the following:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

    1. Get an IAM token for API authentication and put it into an environment variable:

      export IAM_TOKEN="<IAM_token>"

    2. Call the Cluster.update method, e.g., via the following cURL request:

      Warning

      The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

      curl \
    --request PATCH \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
    --data '{
              "updateMask": "configSpec.zoneId,subnetIds",
              "subnetIds": [
                "<subnet>"
              ]
              "configSpec": {
                "zoneId": [
                  "<availability_zone>"
                ]
              }
            }'

      Where:

      • updateMask: Comma-separated string of settings you want to update.

        Specify the relevant parameters:

        • subnetIds: To change the list of subnets.
        • configSpec.zoneId: To change the availability zone.

      • subnetIds: Array of strings. Each string is a subnet ID. If there is only one subnet in the new availability zone, you do not need to specify subnetIds.

      • zoneId: New availability zone for the cluster.

      You can get the cluster ID with the list of clusters in the folder.

    3. Check the server response to make sure your request was successful.

    1. Get an IAM token for API authentication and put it into an environment variable:

      export IAM_TOKEN="<IAM_token>"

    2. Clone the cloudapi repository:

      cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

      Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

    3. Call the ClusterService/Update method, e.g., via the following gRPCurl request:

      Warning

      The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.

      Format for listing settings
      "update_mask": {
    "paths": [
        "<setting_1>",
        "<setting_2>",
        ...
        "<setting_N>"
    ]
}

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "cluster_id": "<cluster_ID>",
          "update_mask": {
            "paths": [
              "subnet_ids",
              "config_spec.zone_id"
            ]
          },
          "subnet_ids": [
            "<subnet>"
          ]
          "config_spec": {
            "zone_id": [
              "<availability_zone>"
            ]
          }
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.Update

      Where:

      • update_mask: List of settings you want to update as an array of strings (paths[]).

        Specify the relevant parameters:

        • subnet_ids: To change the list of subnets.
        • config_spec.zone_id: To change the availability zone.

      • subnet_ids: Array of strings. Each string is a subnet ID. If there is only one subnet in the new availability zone, you do not need to specify subnet_ids.

      • zone_id: New availability zone for the cluster.

      You can get the cluster ID with the list of clusters in the folder.

    4. Check the server response to make sure your request was successful.

  4. To successfully connect to topics after migration, specify the new broker’s FQDN in your backend or client, e.g., in your application code or graphical IDE. Delete the FQDN of the original broker in your source availability zone.

    To find out the FQDN, get the list of hosts in the cluster:

    1. In the management console, navigate to the relevant folder.
    2. Go to Managed Service for Kafka.
    3. Click the name of your cluster and select the Hosts tab.
    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    You will see the FQDN under NAME in the command output.

    1. Call the Cluster.listHosts method, e.g., via the following cURL request:

      curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful. You will find the FQDN in the hosts[].name field of the response.

    1. Call the ClusterService/ListHosts method, e.g., via the following gRPCurl request:

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
            "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful. You will find the FQDN in the hosts[].name field of the response.

Migrating a single-host cluster using auxiliary tools

To move an Apache Kafka® host to a different availability zone in a Managed Service for Apache Kafka® cluster:

  1. Create a subnet in the target availability zone.

  2. If the cluster security group is set up for a subnet in the availability zone you are moving your cluster from, reconfigure the group for the new subnet. To do so, replace the source subnet CIDR with the new subnet CIDR in the security group rules.

  3. Create a Managed Service for Apache Kafka® cluster with the same configuration as the source cluster, except for a different subnet and security group.

  4. Migrate data from your source cluster to the new one using one of the following tools:

    • MirrorMaker: You can use either the MirrorMaker connector embedded in Managed Service for Apache Kafka® or the MirrorMaker utility.
    • Data Transfer.

  5. To successfully connect to topics after migration, specify the new cluster’s broker FQDN in your backend or client, e.g., in your application code or graphical IDE. Delete the FQDN of your source cluster's broker in the source availability zone.

    To find out the FQDN, get the list of hosts in the cluster:

    1. In the management console, navigate to the relevant folder.
    2. Go to Managed Service for Kafka.
    3. Click the name of your cluster and select the Hosts tab.
    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    You will see the FQDN under NAME in the command output.

    1. Call the Cluster.listHosts method, e.g., via the following cURL request:

      curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful. You will find the FQDN in the hosts[].name field of the response.

    1. Call the ClusterService/ListHosts method, e.g., via the following gRPCurl request:

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
            "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful. You will find the FQDN in the hosts[].name field of the response.

  6. Delete the source Managed Service for Apache Kafka® cluster.

Migrating a multi-host cluster

Note

Clusters running Apache Kafka® 3.6 or higher do not use ZooKeeper hosts. Migration is only possible for configurations with three brokers in the same availability zone.

If you create an Apache Kafka® 3.5 cluster out of more than one broker host, three dedicated ZooKeeper hosts will be automatically added to the cluster. Each host is assigned a subnet from different availability zones. After you create a cluster, you cannot change its subnet in the availability zone.

The migration process of an Apache Kafka® 3.5 cluster depends on which availability zones Apache Kafka® and ZooKeeper hosts reside in prior to migration and how many subnets each zone contains. See these examples to better understand how migration works.

To move Apache Kafka® hosts to a different availability zone in an Apache Kafka® 3.5 cluster:

  1. Find out in which availability zones Apache Kafka® and ZooKeeper hosts reside:

    1. In the management console, navigate to the relevant folder.
    2. Navigate to the Managed Service for Kafka service.
    3. Click the name of your cluster and select the Hosts tab. The Availability zone column lists the availability zones for each host.

    If you do not have the Yandex Cloud CLI installed yet, install and initialize it.

    By default, the CLI uses the folder specified when creating the profile. To change the default folder, use the yc config set folder-id <folder_ID> command. You can also set a different folder for any specific command using the --folder-name or --folder-id parameter.

    yc managed-kafka cluster list-hosts <cluster_name_or_ID>

    The availability zone is specified in the command output under ZONE ID.

    1. Get an IAM token for API authentication and put it into an environment variable:

      export IAM_TOKEN="<IAM_token>"

    2. Call the Cluster.listHosts method, e.g., via the following cURL request:

      curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'

      You can get the cluster ID with the list of clusters in the folder.

    3. Check the server response to make sure your request was successful. You will find the availability zone in the hosts[].zoneId field of the response.

    1. Get an IAM token for API authentication and put it into an environment variable:

      export IAM_TOKEN="<IAM_token>"

    2. Clone the cloudapi repository:

      cd ~/ && git clone --depth=1 https://github.com/yandex-cloud/cloudapi

      Below, we assume that the repository contents reside in the ~/cloudapi/ directory.

    3. Call the ClusterService/ListHosts method, e.g., via the following gRPCurl request:

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
            "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts

      You can get the cluster ID with the list of clusters in the folder.

    4. Check the server response to make sure your request was successful. You will find the availability zone in the hosts[].zone_id field of the response.

  2. If the target availability zone for Apache Kafka® hosts is not on the list, create a subnet in this zone.

    If the target availability zone is on the list, Apache Kafka® hosts will migrate to the subnet that already contains Apache Kafka® or ZooKeeper hosts in that zone.

  3. Check the cluster security group. If it is set up for a subnet in the source availability zone, reconfigure the group for the subnet in the target availability zone. To do so, replace the source subnet CIDR with the target subnet CIDR in the security group rules.

  4. Change the availability zone of your cluster and its Apache Kafka® hosts:

    1. In the management console, navigate to the relevant folder.

    2. Navigate to the Managed Service for Kafka service.

    3. In the cluster row, click and select Edit.

    4. Under Network settings, set the new availability zones. Their total number must not decrease.

      Warning

      After adding a new availability zone, deselect one of the old zones. Otherwise, you will not be able to delete an old availability zone after you save the settings.

    5. Specify a subnet in the new availability zone if:

      • Apache Kafka® or ZooKeeper hosts were not previously placed in the target availability zone.
      • The target availability zone has more than one subnet.

    6. Click Save.

    To update availability zones for a cluster and its Apache Kafka® hosts, run this command:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zones> \
   --subnet-ids <subnet_IDs>

    In --zone-ids, list the availability zones separated by commas. Their total number must not decrease.

    Warning

    After adding a new availability zone, remove one of the old zones from the list. Otherwise, you will not be able to delete an old availability zone after you run the command.

    In --subnet-ids, list the subnets for the ru-central1-a, ru-central1-b, and ru-central1-d availability zones, separated by commas. You must specify subnets for these zones even if the Apache Kafka® hosts are deployed in fewer zones. You need all three availability zones for ZooKeeper hosts.

    1. Open the current Terraform configuration file describing your infrastructure.

      Learn how to create this file in Creating an Apache Kafka® cluster.

    2. In the Managed Service for Apache Kafka® cluster description, change the list of availability zones under zones:

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  config {
    ...
    zones = ["<availability_zones>"]
  }
  ...
}

      The number of availability zones must not decrease.

      Warning

      After adding a new availability zone, remove one of the old zones from the list. Otherwise, you will not be able to delete an old availability zone after you save the settings.

    3. Change the list of subnets under subnet_ids if these two conditions are met:

      • You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
      • The target availability zone has more than one subnet.

      If the cluster hosts are placed in the ru-central1-a and ru-central1-b availability zones, and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if the ru-central1-d zone has multiple subnets. Otherwise, you do not need to specify it.

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  subnet_ids = ["<subnets>"]
  ...
}

    4. Make sure the settings are correct.

      1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

      2. Run this command:

        terraform validate

        Terraform will show any errors found in your configuration files.

    5. Confirm updating the resources.

      1. Run this command to view the planned changes:

        terraform plan

        If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

        1. Run this command:

          terraform apply

        2. Confirm updating the resources.

        3. Wait for the operation to complete.

    For more information, see this Terraform provider guide.

    Timeouts

    The Terraform provider limits the time for all operations with the Managed Service for Apache Kafka® cluster to 60 minutes.

    Operations exceeding the timeout are aborted.

    How do I change these limits?

    Add the timeouts section to your cluster description, such as the following:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

    1. Call the Cluster.update method, e.g., via the following cURL request:

      Warning

      The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

      curl \
    --request PATCH \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
    --data '{
              "updateMask": "configSpec.zoneId,subnetIds",
              "subnetIds": [
                <subnet_list>
              ],
              "configSpec": {
                "zoneId": [
                  <list_of_availability_zones>
                ]
              }
            }'

      Where:

      • updateMask: Comma-separated string of settings you want to update.

        Specify the relevant parameters:

        • subnetIds: To change the list of subnets.
        • configSpec.zoneId: To change the list of availability zones.

      • subnetIds: Array of strings. Each string is a subnet ID. Change the list of subnets if these two conditions are met:

        • You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
        • The target availability zone has more than one subnet.

        If the cluster hosts are placed in the ru-central1-a, ru-central1-b, and ru-central1-c availability zones, and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if the ru-central1-d zone has multiple subnets. Otherwise, you do not need to specify it.

      • zoneId: New availability zones for the cluster. Their total number must not decrease.

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful.

    1. Call the ClusterService/Update method, e.g., via the following gRPCurl request:

      Warning

      The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.

      Format for listing settings
      "update_mask": {
    "paths": [
        "<setting_1>",
        "<setting_2>",
        ...
        "<setting_N>"
    ]
}

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "cluster_id": "<cluster_ID>",
          "update_mask": {
            "paths": [
              "subnet_ids",
              "config_spec.zone_id"
            ]
          },
          "subnet_ids": [
            <subnet_list>
          ],
          "config_spec": {
            "zone_id": [
              <list_of_availability_zones>
            ]
          }
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.Update

      Where:

      • update_mask: List of settings you want to update as an array of strings (paths[]).

        Specify the relevant parameters:

        • subnet_ids: To change the list of subnets.
        • config_spec.zone_id: To change the list of availability zones.

      • subnet_ids: Array of strings. Each string is a subnet ID. Change the list of subnets if these two conditions are met:

        • You are migrating Apache Kafka® hosts to an availability zone where ZooKeeper hosts were not previously placed.
        • The target availability zone has more than one subnet.

        If the cluster hosts are placed in the ru-central1-a, ru-central1-b, and ru-central1-c availability zones, and you change the availability zones to ru-central1-a, ru-central1-b, and ru-central1-d, specify a subnet only if the ru-central1-d zone has multiple subnets. Otherwise, you do not need to specify it.

      • zone_id: New availability zones for the cluster. Their total number must not decrease.

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful.

To move Apache Kafka® hosts to a different availability zone in an Apache Kafka® cluster version 3.6 or higher:

  1. Create a subnet in the target availability zone you are migrating Apache Kafka® hosts to.

  2. Check the cluster security group. If it is set up for a subnet in the source availability zone, reconfigure the group for the subnet in the target availability zone. To do so, replace the source subnet CIDR with the target subnet CIDR in the security group rules.

  3. Change the availability zone of your cluster and its Apache Kafka® hosts:

    To change the availability zone for a cluster and its Apache Kafka® hosts, run this command:

    yc managed-kafka cluster update <cluster_name_or_ID> \
   --zone-ids <availability_zone> \
   --subnet-ids <subnet_ID>

    1. Open the current Terraform configuration file describing your infrastructure.

      Learn how to create this file in Creating an Apache Kafka® cluster.

    2. In the Managed Service for Apache Kafka® cluster description, change the availability zone in the zones parameter:

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  config {
    ...
    zones = ["<availability_zone>"]
  }
  ...
}

    3. In the subnet_ids parameter, specify a subnet in the target availability zone.

      resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  subnet_ids = ["<subnet>"]
  ...
}

    4. Make sure the settings are correct.

      1. In the command line, navigate to the directory that contains the current Terraform configuration files defining the infrastructure.

      2. Run this command:

        terraform validate

        Terraform will show any errors found in your configuration files.

    5. Confirm updating the resources.

      1. Run this command to view the planned changes:

        terraform plan

        If you described the configuration correctly, the terminal will display a list of the resources to update and their parameters. This is a verification step that does not apply changes to your resources.

      2. If everything looks correct, apply the changes:

        1. Run this command:

          terraform apply

        2. Confirm updating the resources.

        3. Wait for the operation to complete.

    For more information, see this Terraform provider guide.

    Timeouts

    The Terraform provider limits the time for all operations with the Managed Service for Apache Kafka® cluster to 60 minutes.

    Operations exceeding the timeout are aborted.

    How do I change these limits?

    Add the timeouts section to your cluster description, such as the following:

    resource "yandex_mdb_kafka_cluster" "<cluster_name>" {
  ...
  timeouts {
    create = "1h30m" # 1 hour 30 minutes
    update = "2h"    # 2 hours
    delete = "30m"   # 30 minutes
  }
}

    1. Call the Cluster.update method, e.g., via the following cURL request:

      Warning

      The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the updateMask parameter as a single comma-separated string.

      curl \
    --request PATCH \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --header "Content-Type: application/json" \
    -url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>' \
    --data '{
              "updateMask": "configSpec.zoneId,subnetIds",
              "subnetIds": [
                <subnet_in_target_availability_zone>
              ],
              "configSpec": {
                "zoneId": [
                  <target_availability_zone>
                ]
              }
            }'

      Where:

      • updateMask: Comma-separated string of settings you want to update.

        In this case, specify subnetIds and configSpec.zoneId.

      • subnetIds: Array of strings. Each string is a subnet ID. Specify only the subnet in the target availability zone.

      • zoneId: New availability zones for the cluster. Specify only the target availability zone.

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful.

    1. Call the ClusterService/Update method, e.g., via the following gRPCurl request:

      Warning

      The API method will assign default values to all the parameters of the object you are modifying unless you explicitly provide them in your request. To avoid this, list the settings you want to change in the update_mask parameter as an array of paths[] strings.

      Format for listing settings
      "update_mask": {
    "paths": [
        "<setting_1>",
        "<setting_2>",
        ...
        "<setting_N>"
    ]
}

      grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
          "cluster_id": "<cluster_ID>",
          "update_mask": {
            "paths": [
              "subnet_ids",
              "config_spec.zone_id"
            ]
          },
          "subnet_ids": [
            <subnet_in_target_availability_zone>
          ],
          "config_spec": {
            "zone_id": [
              <target_availability_zone>
            ]
          }
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.Update

      Where:

      • update_mask: List of settings you want to update as an array of strings (paths[]).

        In this case, the array consists of the subnetIds and configSpec.zoneId strings.

      • subnetIds: Array of strings. Each string is a subnet ID. Specify only the subnet in the target availability zone.

      • zoneId: New availability zones for the cluster. Specify only the target availability zone.

      You can get the cluster ID with the list of clusters in the folder.

    2. Check the server response to make sure your request was successful.

To successfully connect to topics after migration, specify the new broker’s FQDN in your backend or client, e.g., in your application code or graphical IDE. Delete the FQDN of the original broker in your source availability zone.

To find out the FQDN, get the list of hosts in the cluster:

  1. In the management console, navigate to the relevant folder.
  2. Go to Managed Service for Kafka.
  3. Click the name of your cluster and select the Hosts tab.
yc managed-kafka cluster list-hosts <cluster_name_or_ID>

You will see the FQDN under NAME in the command output.

  1. Call the Cluster.listHosts method, e.g., via the following cURL request:

    curl \
    --request GET \
    --header "Authorization: Bearer $IAM_TOKEN" \
    --url 'https://mdb.api.cloud.yandex.net/managed-kafka/v1/clusters/<cluster_ID>/hosts'

    You can get the cluster ID with the list of clusters in the folder.

  2. Check the server response to make sure your request was successful. You will find the FQDN in the hosts[].name field of the response.

  1. Call the ClusterService/ListHosts method, e.g., via the following gRPCurl request:

    grpcurl \
    -format json \
    -import-path ~/cloudapi/ \
    -import-path ~/cloudapi/third_party/googleapis/ \
    -proto ~/cloudapi/yandex/cloud/mdb/kafka/v1/cluster_service.proto \
    -rpc-header "Authorization: Bearer $IAM_TOKEN" \
    -d '{
            "cluster_id": "<cluster_ID>"
        }' \
    mdb.api.cloud.yandex.net:443 \
    yandex.cloud.mdb.kafka.v1.ClusterService.ListHosts

    You can get the cluster ID with the list of clusters in the folder.

  2. Check the server response to make sure your request was successful. You will find the FQDN in the hosts[].name field of the response.

Examples of migrating a multi-host cluster

The examples below provide insight into situations where you need to specify a subnet to migrate a Managed Service for Apache Kafka® 3.5 multi-host cluster to a different availability zone.

A ZooKeeper host already resides in the target availability zone

Let’s assume your cluster comprises the following hosts:

  • Apache Kafka® host 1 in the ru-central1-a availability zone
  • Apache Kafka® host 2 in the ru-central1-c availability zone
  • ZooKeeper host 1 in the ru-central1-a availability zone
  • ZooKeeper host 2 in the ru-central1-b availability zone
  • ZooKeeper host 3 in the ru-central1-d availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a and ru-central1-d availability zones.

ZooKeeper host 3 already resides in the ru-central1-d zone, which means it belongs to one of the subnets in this availability zone. This subnet will be used for migration, since you cannot change subnets after you create the cluster. As a result, you do not need to specify a subnet during migration.

The target availability zone contains no hosts and has one subnet

Let’s assume your cluster comprises the following hosts:

  • Apache Kafka® host 1 in the ru-central1-a availability zone
  • Apache Kafka® host 2 in the ru-central1-b availability zone
  • Apache Kafka® host 3 in the ru-central1-c availability zone
  • ZooKeeper host 1 in the ru-central1-a availability zone
  • ZooKeeper host 2 in the ru-central1-b availability zone
  • ZooKeeper host 3 in the ru-central1-c availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.

No hosts initially reside in the ru-central1-d zone. But there is only one subnet in it, so you need not specify a subnet when performing the migration.

The target availability zone contains no hosts and has multiple subnets

Let’s assume your cluster comprises the following hosts:

  • Apache Kafka® host 1 in the ru-central1-a availability zone
  • Apache Kafka® host 2 in the ru-central1-b availability zone
  • Apache Kafka® host 3 in the ru-central1-c availability zone
  • ZooKeeper host 1 in the ru-central1-a availability zone
  • ZooKeeper host 2 in the ru-central1-b availability zone
  • ZooKeeper host 3 in the ru-central1-c availability zone

You need to migrate Apache Kafka® hosts to the ru-central1-a, ru-central1-b, and ru-central1-d availability zones.

No hosts initially reside in the ru-central1-d zone. However, there are multiple subnets, so you need to specify a subnet to perform the migration.

Specifics of migration in Yandex Data Transfer

If your cluster is used as an endpoint when transferring data with Data Transfer, and the transfer type is Replication or Snapshot and increment, restart the transfer after migrating the cluster. This way, the transfer will get data about the cluster's new topology.

You do not need to restart Snapshot transfers, as information about the new topology is provided automatically while activating them.

To restart a transfer, choose one of the two methods:

For more information, see Migrating a Data Transfer transfer and endpoints to a different availability zone.

For an example of transferring data using Data Transfer, see this tutorial.

Previous
Getting a list of cluster hosts
Next
Maintenance