Producers and consumers
Apache Kafka® is a distributed system for exchanging messages between data producers and consumers.
- Producer is a data source application. It connects to an Apache Kafka® cluster and writes messages with data to specific topics and topic partitions. To write messages to a topic and partition, the user must have relevant permissions which come from roles.
- Consumer is an application that receives and processes data from producers. Instead of getting data directly from a producer, it connects to the Apache Kafka® cluster, subscribes to relevant topics, and reads messages from them. To subscribe to a topic, the user must have relevant permissions which come from roles.
Consumer groups
Apache Kafka® supports two consumer management modes:
-
Autonomous mode: Each consumer selects which Apache Kafka® topic sections to read all by themselves.
-
Group mode: The Apache Kafka® topic sections are automatically distributed among the group's consumers. The consumers thus read messages in a coordinated manner.
A consumer group consists of one or more consumers who collectively read messages from a topic. To keep track of messages already read by the consumer group, each Apache Kafka® topic section is mapped to a sequence number called group offset. As a result, each one of the topic's messages is delivered to only one consumer from the group.
The Apache Kafka® topic sections are distributed among the group's consumers:
- Each section is assigned to one consumer within a specific group. At the same time, one consumer can process multiple sections.
- If there are changes in group membership or the number of sections, rebalancing takes place: the sections get redistributed among the group's consumers.
For more information, see the Apache Kafka® documentation for producers and consumers.