Resource relationships in Managed Service for Apache Kafka®

Apache Kafka® is a distributed cross-application communication system that uses the publication-subscription principle.

Yandex Managed Service for Apache Kafka® helps you deploy and maintain clusters of 3.5 or 3.6 Apache Kafka® servers in the Yandex Cloud infrastructure.

Each Managed Service for Apache Kafka® cluster consists of one or more broker hosts, where topics and topic partitions are located. Cluster hosts may reside in different availability zones. You can learn more about Yandex Cloud availability zones in Platform overview.

If all broker hosts are created in the same availability zone, you cannot change their number.

Cluster hosts accept connections from clients, such as data producer and consumer applications. Producers send messages to specific cluster topics and consumers read messages from them. Thus, you will not need to reconfigure consumers if a producer changes.

Be mindful of what is what is controlled by the service, and what by the Yandex Cloud customer. Understanding these control zones will help you use your cloud resources effectively and avoid potential database-related problems. For more information, see Zones of control between managed database (MDB) service users and Yandex Cloud.

Host configuration depending on Apache Kafka® versionHost configuration depending on Apache Kafka® version

Different Apache Kafka® versions use different tools to store cluster metadata, state, and configuration:

• Versions 3.5 and lower support ZooKeeper.
• Versions 3.6 to (and including) 3.9 support ZooKeeper and Apache Kafka® Raft.
• Versions 4.0 or higher support Apache Kafka® Raft only.

Host configuration in clusters with ZooKeeperHost configuration in clusters with ZooKeeper

ZooKeeper is automatically configured and placed in a cluster based on the number of broker hosts:

• If the Apache Kafka® cluster consists of one broker host, ZooKeeper is hosted on the same host.
• If the Apache Kafka® cluster consists of two or more hosts, ZooKeeper is hosted separately from brokers on three additional hosts. These hosts are added to the cluster automatically.

You cannot delete ZooKeeper hosts. The number of ZooKeeper hosts is fixed.

For more information on creating a Managed Service for Apache Kafka® cluster with ZooKeeper, see this guide.

Host configuration in clusters with Apache Kafka® RaftHost configuration in clusters with Apache Kafka® Raft

The Apache Kafka® Raft protocol (abbreviated KRaft) is used to store metadata instead of ZooKeeper.

KRaft is automatically configured and placed in a cluster based on the selected mode and the number of broker hosts:

• KRaft (combined mode): One Apache Kafka® host accommodates a broker and a KRaft metadata controller at the same time. In the cluster, only three Apache Kafka® hosts get created in one of these configurations:

  • Three hosts in the same availability zone.
  • Each host in a separate availability zone.

  You cannot set the number of broker hosts manually.

• KRaft (on separate hosts): A broker and a KRaft metadata controller are on separate hosts. When you create a multiple-host cluster, three dedicated KRaft hosts are added to it.

  The number of broker hosts is set manually.

You cannot delete KRaft hosts. The number of KRaft hosts is fixed.

For more information on creating a Managed Service for Apache Kafka® cluster with KRaft, see this guide.

Hosting cluster hostsHosting cluster hosts

VMs for cluster hosts can be hosted on:

• Yandex Cloud regular hosts.

  These are physical servers for hosting cluster VMs. They are randomly selected from a pool of available hosts that meet the selected cluster configuration.

• Yandex Cloud dedicated hosts.

  These are physical servers reserved exclusively for your VMs. Such VMs ensure the operation of both the cluster and your other services that support dedicated hosts. The hosts are selected from dedicated host groups specified when creating a cluster.

  A cluster with multiple broker hosts needs at least three groups of dedicated hosts to operate. This enables the placement of ZooKeeper and KRaft hosts.

  This placement configuration ensures physical isolation of VMs. A Managed Service for Apache Kafka® cluster using dedicated hosts includes all features of a regular cluster.

  For more information, see Dedicated host.

For more information, see the Apache Kafka® documentation.