Resource relationships in Managed Service for Apache Kafka®

Apache Kafka® is a distributed cross-application communication system that uses the publication-subscription principle.

Yandex Managed Service for Apache Kafka® helps you deploy and maintain clusters of Apache Kafka®server versions 2.8, 3.0, 3.1, 3.2, 3.3, 3.4, 3.5, and 3.6 in the Yandex Cloud infrastructure.

Each Managed Service for Apache Kafka® cluster consists of one or more broker hosts, where topics and topic partitions are located. Cluster hosts may reside in different availability zones. You can learn more about Yandex Cloud availability zones in Platform overview.

If all broker hosts are created in the same availability zone, you cannot change their number.

Cluster hosts accept connections from clients, such as data producer and consumer applications. Producers send messages to specific cluster topics and consumers read messages from them. Thus, you will not need to reconfigure consumers if a producer changes.

Host configuration depending on Apache Kafka® versionHost configuration depending on Apache Kafka® version

Different Apache Kafka® versions use different tools to store cluster metadata, state, and configuration:

• Versions 3.5 or lower use ZooKeeper.
• Versions 3.6 or higher use Apache Kafka® Raft.

The number and type of hosts are different between the two cases.

Host configuration in clusters with ZooKeeperHost configuration in clusters with ZooKeeper

ZooKeeper is automatically configured and placed in a cluster based on the number of broker hosts:

• If the Apache Kafka® cluster consists of one broker host, ZooKeeper is hosted on the same host.
• If the Apache Kafka® cluster consists of two or more hosts, ZooKeeper is hosted separately from brokers on three additional hosts. These hosts are added to the cluster automatically.

You cannot delete ZooKeeper hosts. The number of ZooKeeper hosts is fixed.

For more information on creating a Managed Service for Apache Kafka® cluster with ZooKeeper, see this guide.

Host configuration in clusters with Apache Kafka® RaftHost configuration in clusters with Apache Kafka® Raft

In Apache Kafka® 3.6 or higher, Apache Kafka® Raft protocol (abbreviated as KRaft) is used to store metadata instead of ZooKeeper.

In Managed Service for Apache Kafka® clusters with the KRaft protocol, only the combined mode is used, where a broker and the KRaft metadata controller reside on the same Apache Kafka® host at the same time. In this case, only three Apache Kafka® hosts are created in the cluster in one of these configurations:

• All three are in the same availability zone.
• Each host is located in a separate availability zone.

For more information on creating a Managed Service for Apache Kafka® cluster with KRaft, see this guide.

Hosting cluster hostsHosting cluster hosts

VMs for cluster hosts can be hosted on:

• Yandex Cloud regular hosts.

  These are physical servers for hosting cluster VMs. They are randomly selected from a pool of available hosts that meet the selected cluster configuration.

• Yandex Cloud dedicated hosts.

  These are physical servers that only host your VMs. Such VMs ensure the operation of both the cluster and your other services that support dedicated hosts. These hosts are selected from dedicated host groups specified when creating a cluster.

  A cluster with multiple broker hosts needs at least three groups of dedicated hosts for its operation. This is required for the deployment of ZooKeeper hosts.

  This placement option ensures physical isolation of the VMs. A Managed Service for Apache Kafka® cluster using dedicated hosts includes all features of a regular cluster.

  For more information, see Dedicated hosts.

For more information, see the Apache Kafka® documentation.

Cluster fault toleranceCluster fault tolerance

To make sure the cluster data is stored in a reliable way and continuously available:

• Your cluster must have at least three broker hosts.

• Each of the three availability zones must host at least one broker host.

• Cluster topics must have the following parameters:

  • Replication factor: 3
  • Minimum number of in-sync replicas: 2

For more information, see the Service Level Agreement (SLA).