Migrating a database from a third-party Apache Kafka® cluster to Yandex Managed Service for Apache Kafka®

There are two ways to migrate topics from an Apache Kafka® source cluster to a Managed Service for Apache Kafka® target cluster:

• Using the built-in Yandex Managed Service for Apache Kafka® MirrorMaker connector.

  This method is easy to configure and does not require creating an intermediate VM.

• Using MirrorMaker 2.0.

  This requires setting up the utility manually on an intermediate virtual machine. Use this method only if it is not possible to migrate data using the built-in MirrorMaker connector for whatever reason.

Both methods are also suitable for migrating a single-host Managed Service for Apache Kafka® cluster to a different availability zone.

Migrating data using Yandex Managed Service for Apache Kafka® ConnectorMigrating data using Yandex Managed Service for Apache Kafka® Connector

To transfer data using Yandex Managed Service for Apache Kafka® Connector:

1. Prepare the source cluster.
2. Create a target cluster and a connector.
3. Check the target cluster topic for data.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

• Managed Service for Apache Kafka® cluster, which includes the use of computing resources allocated to hosts, storage and backup size (see Managed Service for Apache Kafka® pricing).
• Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Prepare the source clusterPrepare the source cluster

1. Create the admin-source user and assign them the ACCESS_ROLE_ADMIN role for all topics (*).
2. Make sure the source cluster’s network settings allow cluster connections from the internet.

Create a target cluster and a connectorCreate a target cluster and a connector

Check the target cluster topic for dataCheck the target cluster topic for data

1. In the management console, open the target cluster.

2. Make sure the migrated topic is displayed on the Topics tab.

   A prefix (source by default) will be added to the topic name. For example, a topic named mytopic will be moved to the target cluster as source.mytopic.

3. Connect to the target cluster topic using kafkacat. Add the prefix to the source cluster topic name.

4. Make sure the console displays messages from the source cluster topic.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them:

Migrating data via MirrorMakerMigrating data via MirrorMaker

To transfer data using MirrorMaker:

1. Set up your infrastructure.
2. Configure the source cluster and VM.
3. Configure MirrorMaker.
4. Start replication.
5. Check the target cluster topic for data.

If you no longer need the resources you created, delete them.

Getting startedGetting started

Sign up for Yandex Cloud and create a billing account:

1. Navigate to the management console and log in to Yandex Cloud or create a new account.
2. On the Yandex Cloud Billing page, make sure you have a billing account linked and it has the ACTIVE or TRIAL_ACTIVE status. If you do not have a billing account, create one and link a cloud to it.

If you have an active billing account, you can create or select a folder for your infrastructure on the cloud page.

Learn more about clouds and folders here.

Required paid resourcesRequired paid resources

• Managed Service for Apache Kafka® cluster, which includes the use of computing resources allocated to hosts, storage and backup size (see Managed Service for Apache Kafka® pricing).
• Public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• VM instance: use of computing resources, storage, public IP address, and OS (see Compute Cloud pricing).

Set up your infrastructureSet up your infrastructure

Configure the source cluster and VMConfigure the source cluster and VM

1. Prepare the source cluster:

   1. In the source cluster, create the admin-source user and assign them the ACCESS_ROLE_ADMIN role for all topics (*).
   2. Enable the Auto create topics enable setting.

2. Connect to the VM over SSH.

   1. Install the JDK:

      sudo apt update && sudo apt install --yes default-jdk
      

   2. Download and unpack the Apache Kafka® archive with the same version as installed on the target cluster, e.g., for version 3.9:

      wget https://archive.apache.org/dist/kafka/3.9.0/kafka_2.12-3.9.0.tgz && \
      tar -xvf kafka_2.12-3.9.0.tgz
      

   3. Download an SSL certificate for connecting to the Managed Service for Apache Kafka® cluster:

      sudo mkdir -p /usr/local/share/ca-certificates/Yandex && \
      sudo wget "https://storage.yandexcloud.net/cloud-certs/CA.pem" \
          --output-document /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt && \
      sudo chmod 0655 /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt
      

   4. Install kafkacat:

      sudo apt update && sudo apt install --yes kafkacat
      

3. Configure a firewall and security groups if required for MirrorMaker connection to the target and source clusters.

Configure MirrorMakerConfigure MirrorMaker

1. Connect to the MirrorMaker VM over SSH.

2. In the home directory, create a folder named mirror-maker to store Java Keystore certificates and MirrorMaker configuration files:

   mkdir --parents /home/<home_directory>/mirror-maker
   

3. Choose a password of at least 6 characters for a certificate store, create the store, and add the SSL certificate for cluster connection:

   sudo keytool --noprompt -importcert -alias YandexCA \
      -file /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt \
      -keystore /home/<home_directory>/mirror-maker/keystore \
      -storepass <certificate_store_password>
   

4. Create a MirrorMaker configuration file named mm2.properties in the mirror-maker folder:

   # Kafka clusters
   clusters=source, cloud
   source.bootstrap.servers=<source_cluster_broker_FQDN>:9091
   cloud.bootstrap.servers=<source_cluster_broker_1_FQDN>:9091, ..., <source_cluster_broker_N_FQDN>:9091
   
   # Source and target cluster settings
   source->cloud.enabled=true
   cloud->source.enabled=false
   source.cluster.alias=source
   cloud.cluster.alias=cloud
   
   # Internal topics settings
   source.config.storage.replication.factor=<R>
   source.status.storage.replication.factor=<R>
   source.offset.storage.replication.factor=<R>
   source.offsets.topic.replication.factor=<R>
   source.errors.deadletterqueue.topic.replication.factor=<R>
   source.offset-syncs.topic.replication.factor=<R>
   source.heartbeats.topic.replication.factor=<R>
   source.checkpoints.topic.replication.factor=<R>
   source.transaction.state.log.replication.factor=<R>
   cloud.config.storage.replication.factor=<R>
   cloud.status.storage.replication.factor=<R>
   cloud.offset.storage.replication.factor=<R>
   cloud.offsets.topic.replication.factor=<R>
   cloud.errors.deadletterqueue.topic.replication.factor=<R>
   cloud.offset-syncs.topic.replication.factor=<R>
   cloud.heartbeats.topic.replication.factor=<R>
   cloud.checkpoints.topic.replication.factor=<R>
   cloud.transaction.state.log.replication.factor=<R>
   
   # Topics
   topics=.*
   groups=.*
   topics.blacklist=.*[\-\.]internal, .*\replica, __consumer_offsets
   groups.blacklist=console-consumer-.*, connect-.*, __.*
   replication.factor=<M>
   refresh.topics.enable=true
   sync.topic.configs.enabled=true
   refresh.topics.interval.seconds=10
   
   # Tasks
   tasks.max=<T>
   
   # Source cluster authentication parameters. Comment out if no authentication required
   source.client.id=mm2_consumer_test
   source.group.id=mm2_consumer_group
   source.security.protocol=SASL_PLAINTEXT
   source.sasl.mechanism=SCRAM-SHA-512
   source.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-source" password="<password>";
   
   # Target cluster authentication parameters
   cloud.client.id=mm2_producer_test
   cloud.group.id=mm2_producer_group
   cloud.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
   cloud.ssl.truststore.location=/home/<home_directory>/mirror-maker/keystore
   cloud.ssl.truststore.password=<certificate_store_password>
   cloud.ssl.protocol=TLS
   cloud.security.protocol=SASL_SSL
   cloud.sasl.mechanism=SCRAM-SHA-512
   cloud.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-cloud" password="<password>";
   
   # Enable heartbeats and checkpoints
   source->target.emit.heartbeats.enabled=true
   source->target.emit.checkpoints.enabled=true
   

   MirrorMaker configuration notes:

   • It performs one-way replication (source->cloud.enabled = true, cloud->source.enabled = false).
   • In the topics parameter, list the topics you want to migrate. You can also specify a regular expression for selecting topics. To migrate all topics, specify .*. This configuration replicates all topics.
   • Topic names in the target cluster cluster match those in the source cluster.
   • <R> stands for the replication factor for MirrorMaker service topics. Its value should not exceed the lesser of the broker counts in the source and target clusters.
   • <M> stands for the default replication factor defined for topics in the target cluster.
   • <T> stands for the number of concurrent MirrorMaker processes. To distribute replication load evenly, we recommend a value of at least 2. For more information, see this Apache Kafka® guide.

   You can get the Managed Service for Apache Kafka® broker FQDNs with the list of hosts in the cluster.

Start replicationStart replication

1. Connect to the MirrorMaker VM over SSH.

2. Run MirrorMaker on the VM as follows:

   <Apache_Kafka_installation_path>/bin/connect-mirror-maker.sh /home/<home_directory>/mirror-maker/mm2.properties
   

Check the target cluster topic for dataCheck the target cluster topic for data

1. In the management console, open the target cluster.

2. Make sure the migrated topic is displayed on the Topics tab.

   A prefix (source by default) will be added to the topic name. For example, a topic named mytopic will be moved to the target cluster as source.mytopic.

3. Connect to the target cluster topic using kafkacat. Add the prefix to the source cluster topic name.

4. Make sure the console displays messages from the source cluster topic.

To learn more about MirrorMaker 2.0, see this Apache Kafka® article.

Delete the resources you createdDelete the resources you created

Some resources are not free of charge. Delete the resources you no longer need to avoid paying for them: