Topics and partitions

TopicsTopics

A topic is a way to group message streams into categories. Producers publish messages of a certain category to a topic, while consumers subscribe to this topic and read messages from it. For each topic, Apache Kafka® keeps a message log that can be divided into multiple partitions.

For example, if the data producer is an online store, you can create separate topics for logging user actions, storing data about their shopping cart, keeping records about transactions, etc.

Managed Service for Apache Kafka® manages the storage of topic messages and enables:

• Partition replication, in case a cluster consists of at least two brokers and the replication factor for topics is greater than one.
• Message compression.
• Log purging in accordance with the policy when partition messages become outdated or the specified log size is reached.

You can learn more about topics in the Apache Kafka® documentation.

Service topicsService topics

While running, Managed Service for Apache Kafka® is able to create and use service topics. You cannot write user data to such topics.

The __schema_registry service topic is used to support Managed Schema Registry.

The __connect-configs, __connect-offsets, and __connect-status service topics are used to support Managed Kafka Connect.

PartitionsPartitions

A partition is a sequence of topic messages stored in the order they are received. The message sequence number is called an offset. Consumers read messages from the beginning to the end of a partition, that is, messages that were received earlier are consumed first. To start reading data from a specific message, consumers must provide the broker with its offset.

Using partitions, you can:

• Distribute the message storage and request processing load among multiple Apache Kafka® brokers.
• Provide fault tolerance: a partition can be replicated to a specified number of brokers.

You can learn more about partitions in the Apache Kafka® documentation.

Managing topics and partitionsManaging topics and partitions

A Managed Service for Apache Kafka® cluster provides two ways for you to manage topics and partitions:

• Using native Yandex Cloud interfaces, such as the management console, CLI, Terraform, or API. Use this method if you want to create, delete, and configure topics and partitions using Managed Service for Apache Kafka® features.

• Using the Apache Kafka® Admin API. The method will suit you if you already have your own solution for managing topics and partitions. When using the Admin API, the following restrictions apply:

  • Topics can be managed by an admin user with the ACCESS_ROLE_ADMIN role effective for all the cluster topics. If there is no such user in the cluster, create one.

  • You need to configure user permissions for producers and consumers using native Yandex Cloud interfaces rather than the Apache Kafka® Admin API.

You can use these methods separately or combined.