Migrating a database from a third-party Apache Kafka® cluster to Yandex Managed Service for Apache Kafka®

There are two ways to migrate topics from an Apache Kafka® source cluster to a Managed Service for Apache Kafka® target cluster:

• Using the built-in Yandex Managed Service for Apache Kafka® MirrorMaker connector.

  This method is easy to configure and does not require creating an intermediate VM.

• Using MirrorMaker 2.0.

  This requires setting up the utility manually on an intermediate virtual machine. Use this method only if it is not possible to migrate data using the built-in MirrorMaker connector for whatever reason.

Both methods are also suitable for migrating a single-host Managed Service for Apache Kafka® cluster to a different availability zone.

Migrating data using Yandex Managed Service for Apache Kafka® ConnectorMigrating data using Yandex Managed Service for Apache Kafka® Connector

1. Create a connector.
2. Check the target cluster topic for data.

Required paid resourcesRequired paid resources

The support cost for this solution includes:

• Managed Service for Apache Kafka® cluster fee, which covers the use of computing resources allocated to hosts (including ZooKeeper hosts) and disk space (see Apache Kafka® pricing).
• Fee for public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Create a cluster and a connectorCreate a cluster and a connector

Check the target cluster topic for dataCheck the target cluster topic for data

1. Connect to the target cluster topic using kafkacat. Add the source prefix to the source cluster topic name: for example, the mytopic topic is migrated to the target cluster as source.mytopic.
2. Make sure the console displays messages from the source cluster topic.

Migrating data using MirrorMakerMigrating data using MirrorMaker

1. Configure MirrorMaker.
2. Start replication.
3. Check the target cluster topic for data.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost for this solution includes:

• Managed Service for Apache Kafka® cluster fee, which covers the use of computing resources allocated to hosts (including ZooKeeper hosts) and disk space (see Apache Kafka® pricing).
• Fee for public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• VM fee, which covers the use of computing resources, storage, and, optionally, public IP address (see Compute Cloud pricing).

Getting startedGetting started

Set up your infrastructureSet up your infrastructure

Configure additional settingsConfigure additional settings

1. In the source cluster, create the admin-source user authorized to manage topics via the Admin API.

2. Connect to the VM over SSH.

   1. Install the JDK:

      sudo apt update && sudo apt install --yes default-jdk
      

   2. Download and unpack the Apache Kafka® archive with the same version as installed on the target cluster, e.g., Apache Kafka® 2.8:

      wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.12-2.8.0.tgz && \
      tar -xvf kafka_2.12-2.8.0.tgz
      

   3. Install kafkacat:

      sudo apt update && sudo apt install --yes kafkacat
      

      Make sure you can use it to connect to the source and target clusters over SSL.

3. Configure a firewall and security groups if required for MirrorMaker connection to the target and source clusters.

Configure MirrorMakerConfigure MirrorMaker

1. Connect to the MirrorMaker VM over SSH.

2. Download an SSL certificate for connecting to the Managed Service for Apache Kafka® cluster.

3. In the home directory, create a folder named mirror-maker to store Java Keystore certificates and MirrorMaker configuration files:

   mkdir --parents /home/<home_directory>/mirror-maker
   

4. Choose a password of at least 6 characters for a certificate store, create the store, and add the SSL certificate for cluster connection:

   sudo keytool --noprompt -importcert -alias YandexCA \
      -file /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt \
      -keystore /home/<home_directory>/mirror-maker/keystore \
      -storepass <certificate_store_password>
   

5. Create a MirrorMaker configuration file named mm2.properties in the mirror-maker folder:

   # Kafka clusters
   clusters=cloud, source
   source.bootstrap.servers=<source_cluster_broker_FQDN>:9092
   cloud.bootstrap.servers=<source_cluster_broker_1_FQDN>:9091, ..., <source_cluster_broker_N_FQDN>:9091
   
   # Source and target cluster settings
   source->cloud.enabled=true
   cloud->source.enabled=false
   source.cluster.alias=source
   cloud.cluster.alias=cloud
   
   # Internal topics settings
   source.config.storage.replication.factor=<R>
   source.status.storage.replication.factor=<R>
   source.offset.storage.replication.factor=<R>
   source.offsets.topic.replication.factor=<R>
   source.errors.deadletterqueue.topic.replication.factor=<R>
   source.offset-syncs.topic.replication.factor=<R>
   source.heartbeats.topic.replication.factor=<R>
   source.checkpoints.topic.replication.factor=<R>
   source.transaction.state.log.replication.factor=<R>
   cloud.config.storage.replication.factor=<R>
   cloud.status.storage.replication.factor=<R>
   cloud.offset.storage.replication.factor=<R>
   cloud.offsets.topic.replication.factor=<R>
   cloud.errors.deadletterqueue.topic.replication.factor=<R>
   cloud.offset-syncs.topic.replication.factor=<R>
   cloud.heartbeats.topic.replication.factor=<R>
   cloud.checkpoints.topic.replication.factor=<R>
   cloud.transaction.state.log.replication.factor=<R>
   
   # Topics
   topics=.*
   groups=.*
   topics.blacklist=.*[\-\.]internal, .*\replica, __consumer_offsets
   groups.blacklist=console-consumer-.*, connect-.*, __.*
   replication.factor=<M>
   refresh.topics.enable=true
   sync.topic.configs.enabled=true
   refresh.topics.interval.seconds=10
   
   # Tasks
   tasks.max=<T>
   
   # Source cluster authentication parameters. Comment out if no authentication required
   source.client.id=mm2_consumer_test
   source.group.id=mm2_consumer_group
   source.security.protocol=SASL_PLAINTEXT
   source.sasl.mechanism=SCRAM-SHA-512
   source.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-source" password="<password>";
   
   # Target cluster authentication parameters
   cloud.client.id=mm2_producer_test
   cloud.group.id=mm2_producer_group
   cloud.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
   cloud.ssl.truststore.location=/home/<home_directory>/mirror-maker/keystore
   cloud.ssl.truststore.password=<certificate_store_password>
   cloud.ssl.protocol=TLS
   cloud.security.protocol=SASL_SSL
   cloud.sasl.mechanism=SCRAM-SHA-512
   cloud.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-cloud" password="<password>";
   
   # Enable heartbeats and checkpoints
   source->target.emit.heartbeats.enabled=true
   source->target.emit.checkpoints.enabled=true
   

   MirrorMaker configuration notes:

   • It performs one-way replication (source->cloud.enabled = true, cloud->source.enabled = false).
   • In the topics parameter, list the topics you want to migrate. You can also specify a regular expression for selecting topics. To migrate all topics, specify .*. This configuration replicates all topics.
   • Topic names in the target cluster cluster match those in the source cluster.
   • <R> stands for the replication factor for MirrorMaker service topics. Its value should not exceed the lesser of the broker counts in the source and target clusters.
   • <M> stands for the default replication factor defined for topics in the target cluster.
   • <T> stands for the number of concurrent MirrorMaker processes. To distribute replication load evenly, we recommend a value of at least 2. For more information, see this Apache Kafka® guide.

   You can get the Managed Service for Apache Kafka® broker FQDNs with the list of hosts in the cluster.

Start replicationStart replication

Run MirrorMaker on the VM as follows:

<Apache_Kafka_installation_path>/bin/connect-mirror-maker.sh /home/<home_directory>/mirror-maker/mm2.properties

Check the target cluster topic for dataCheck the target cluster topic for data

1. Connect to the target cluster topic using kafkacat. Add the source prefix to the source cluster topic name: for example, the mytopic topic is migrated to the target cluster as source.mytopic.
2. Make sure the console displays messages from the source cluster topic.

To learn more about using MirrorMaker 2.0, see this Apache Kafka® article.

Delete the resources you createdDelete the resources you created

Delete the resources you no longer need to avoid paying for them: