Migrating a database from a third-party Apache Kafka® cluster to Yandex Managed Service for Apache Kafka®

There are two ways to migrate topics from an Apache Kafka® source cluster to a Managed Service for Apache Kafka® target cluster:

• Using the built-in Yandex Managed Service for Apache Kafka® MirrorMaker connector.

  This method is easy to configure and does not require you to create an intermediate VM.

• Using the MirrorMaker 2.0 utility.

  To use this method, first install and configure the utility on an intermediate VM. Use this method only if it is not possible to migrate data using the built-in MirrorMaker connector for whatever reason.

Both methods are also suitable for migrating a Managed Service for Apache Kafka® cluster with one host to a different availability zone.

Data migration using Yandex Managed Service for Apache Kafka® ConnectorData migration using Yandex Managed Service for Apache Kafka® Connector

1. Create a connector.
2. Check the target cluster topic for data.

Required paid resourcesRequired paid resources

The support cost includes:

• Managed Service for Apache Kafka® cluster fee: Using computing resources allocated to hosts (including ZooKeeper hosts) and disk space (see Apache Kafka® pricing).
• Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).

Create a cluster and a connectorCreate a cluster and a connector

Check the target cluster topic for dataCheck the target cluster topic for data

1. Connect to the target cluster topic using kafkacat. Add the source prefix to the source cluster topic name: for example, the mytopic topic is migrated to the target cluster as source.mytopic.
2. Make sure the console displays messages from the source cluster topic.

Migrating data using MirrorMakerMigrating data using MirrorMaker

1. Configure MirrorMaker.
2. Start replication.
3. Check the target cluster topic for data.

If you no longer need the resources you created, delete them.

Required paid resourcesRequired paid resources

The support cost includes:

• Managed Service for Apache Kafka® cluster fee: using computing resources allocated to hosts (including ZooKeeper hosts) and disk space (see Apache Kafka® pricing).
• Fee for using public IP addresses if public access is enabled for cluster hosts (see Virtual Private Cloud pricing).
• VM fee: using computing resources, storage, and, optionally, public IP address (see Compute Cloud pricing).

Getting startedGetting started

Prepare the infrastructurePrepare the infrastructure

Configure additional settingsConfigure additional settings

1. In the source cluster, create the admin-source authorized to manage topics via the Admin API.

2. Connect to a virtual machine over SSH.

   1. Install the JDK:

      sudo apt update && sudo apt install --yes default-jdk
      

   2. Download and unpack the Apache Kafka® archive with the same version number as the version installed in the target cluster. For example, for version 2.8:

      wget https://archive.apache.org/dist/kafka/2.8.0/kafka_2.12-2.8.0.tgz && \
      tar -xvf kafka_2.12-2.8.0.tgz
      

   3. Install the kafkacat utility:

      sudo apt update && sudo apt install --yes kafkacat
      

      Make sure that you can use it to connect to the source and target clusters via SSL.

3. Configure a firewall and security groups if it is required to connect MirrorMaker to the target and the source clusters.

Configure MirrorMakerConfigure MirrorMaker

1. Connect to the MirrorMaker VM over SSH.

2. Download an SSL certificate for connecting to the Managed Service for Apache Kafka® cluster.

3. In the home directory, create a folder named mirror-maker to store Java Keystore certificates and MirrorMaker configuration files:

   mkdir --parents /home/<home_directory>/mirror-maker
   

4. Choose a password at least 6 characters long for a certificate store, create a store, and add there an SSL certificate for connecting to the cluster:

   sudo keytool --noprompt -importcert -alias YandexCA \
      -file /usr/local/share/ca-certificates/Yandex/YandexInternalRootCA.crt \
      -keystore /home/<home_directory>/mirror-maker/keystore \
      -storepass <certificate_store_password>
   

5. Create a MirrorMaker configuration file named mm2.properties in the mirror-maker folder:

   # Kafka clusters
   clusters=cloud, source
   source.bootstrap.servers=<source_cluster_broker_FQDN>:9092
   cloud.bootstrap.servers=<source_cluster_broker_1_FQDN>:9091, ..., <source_cluster_broker_N_FQDN>:9091
   
   # Source and target cluster settings
   source->cloud.enabled=true
   cloud->source.enabled=false
   source.cluster.alias=source
   cloud.cluster.alias=cloud
   
   # Internal topics settings
   source.config.storage.replication.factor=<R>
   source.status.storage.replication.factor=<R>
   source.offset.storage.replication.factor=<R>
   source.offsets.topic.replication.factor=<R>
   source.errors.deadletterqueue.topic.replication.factor=<R>
   source.offset-syncs.topic.replication.factor=<R>
   source.heartbeats.topic.replication.factor=<R>
   source.checkpoints.topic.replication.factor=<R>
   source.transaction.state.log.replication.factor=<R>
   cloud.config.storage.replication.factor=<R>
   cloud.status.storage.replication.factor=<R>
   cloud.offset.storage.replication.factor=<R>
   cloud.offsets.topic.replication.factor=<R>
   cloud.errors.deadletterqueue.topic.replication.factor=<R>
   cloud.offset-syncs.topic.replication.factor=<R>
   cloud.heartbeats.topic.replication.factor=<R>
   cloud.checkpoints.topic.replication.factor=<R>
   cloud.transaction.state.log.replication.factor=<R>
   
   # Topics
   topics=.*
   groups=.*
   topics.blacklist=.*[\-\.]internal, .*\replica, __consumer_offsets
   groups.blacklist=console-consumer-.*, connect-.*, __.*
   replication.factor=<M>
   refresh.topics.enable=true
   sync.topic.configs.enabled=true
   refresh.topics.interval.seconds=10
   
   # Tasks
   tasks.max=<T>
   
   # Source cluster authentication parameters. Comment out if no authentication required
   source.client.id=mm2_consumer_test
   source.group.id=mm2_consumer_group
   source.security.protocol=SASL_PLAINTEXT
   source.sasl.mechanism=SCRAM-SHA-512
   source.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-source" password="<password>";
   
   # Target cluster authentication parameters
   cloud.client.id=mm2_producer_test
   cloud.group.id=mm2_producer_group
   cloud.ssl.enabled.protocols=TLSv1.2,TLSv1.1,TLSv1
   cloud.ssl.truststore.location=/home/<home_directory>/mirror-maker/keystore
   cloud.ssl.truststore.password=<certificate_store_password>
   cloud.ssl.protocol=TLS
   cloud.security.protocol=SASL_SSL
   cloud.sasl.mechanism=SCRAM-SHA-512
   cloud.sasl.jaas.config=org.apache.kafka.common.security.scram.ScramLoginModule required username="admin-cloud" password="<password>";
   
   # Enable heartbeats and checkpoints
   source->target.emit.heartbeats.enabled=true
   source->target.emit.checkpoints.enabled=true
   

   Notes on MirrorMaker configuration:

   • It performs one-way replication (source->cloud.enabled = true, cloud->source.enabled = false).
   • In the topics parameter, list the topics you want to migrate. You can also specify a regular expression for selecting topics. To migrate all topics, put .*. In this configuration, all the topics will be replicated.
   • Topic names in the target cluster are the same as in the source.
   • <R> is the parameter that sets the replication factor for MirrorMaker service topics. The value of this parameter should not exceed the smaller of the number of brokers in the source cluster or the number of brokers in the target cluster.
   • <M> is the default replication factor defined for topics in the target cluster.
   • <T> is the number of concurrent MirrorMaker processes. To distribute replication load evenly, we recommend a value of at least 2. For more information, see the relevant Apache Kafka® documentation.

   You can request Managed Service for Apache Kafka® broker FQDNs with the list of hosts in the cluster.

Start replicationStart replication

Launch MirrorMaker on the VM as follows:

<Apache_Kafka_installation_path>/bin/connect-mirror-maker.sh /home/<home_directory>/mirror-maker/mm2.properties

Check the target cluster topic for dataCheck the target cluster topic for data

1. Connect to the target cluster topic using kafkacat. Add the source prefix to the source cluster topic name: for example, the mytopic topic is migrated to the target cluster as source.mytopic.
2. Make sure the console displays messages from the source cluster topic.

To learn more about MirrorMaker 2.0, see the Apache Kafka® documentation.

Delete the resources you createdDelete the resources you created

Delete the resources you no longer need to avoid paying for them: