Apache Kafka® Raft protocol in Managed Service for Apache Kafka® clusters
Apache Kafka® Raft (short name: KRaft) is an approval protocol allowing you to store metadata and manage it in Apache Kafka®. KRaft makes Apache Kafka® independent of the external metadata synchronization service, ZooKeeper.
KRaft is supported by Managed Service for Apache Kafka® clusters with Apache Kafka® 3.6 or higher. ZooKeeper is used only in older versions.
Reasons for replacing ZooKeeper with KRaft:
- ZooKeeper has become obsolete in Apache Kafka® 3.5 or higher.
- ZooKeeper support will be discontinued starting from Apache Kafka® 4.0.
Managed Service for Apache Kafka® cluster topology
In Managed Service for Apache Kafka® clusters with the KRaft protocol, only the combined mode is used, where a broker and the KRaft metadata controller reside on the same Apache Kafka® host at the same time. In this case, only three Apache Kafka® hosts are created in the cluster in one of these configurations:
- All three are in the same availability zone.
- Each host is located in a separate availability zone.
For more information on creating a Managed Service for Apache Kafka® cluster with KRaft, see this guide.
Apache Kafka® Raft pros and cons
KRaft advantages stem from the fact that Apache Kafka® no longer depends on the ZooKeeper external system:
- You do not need to pay extra for ZooKeeper hosts.
- There is a single mechanism in place for information security instead of multiple ones.
- A cluster with KRaft is easier to manage, as it has no ZooKeeper hosts.
- The cluster becomes more fault-tolerant and recovers almost instantly in case of a failure.
- You can create a greater number of partitions within the cluster.
For more information, see the Confluent website and documentation.
KRaft cons arise from the cluster topology specifics:
- In the combined mode, KRaft metadata controllers are not isolated from Apache Kafka® as effectively as ZooKeeper hosts. Therefore, Apache Kafka® does not recommend using KRaft in critical development environments. KRaft is more suitable for local development and testing.
- In a cluster with KRaft, you cannot flexibly manage the number of hosts: you can have exactly three hosts.