Kafka

Kafka - Work Flow

Kafka - Work Flow

Kafka offers a fast and persisted workflow in both pub-sub and queue-based messaging systems. Simply put, Kafka corresponds to a collection of topics with partitions consisting of sequenced messages where each message is indexed. Producers send the message to the consumer based on a certain topic and consumers are free to choose the type of messaging system based on their preference and requirement. 

 

The workflow of pub-sub type of messaging system is enlisted below -

  • The first step begins with the producer sending message belonging to a topic.
  • The message is stored in partitions that belong to a particular topic and messages are equally shared between these partitions.  
  • In the case of two messages, there are two corresponding partitions. Kafka will simply allocate the first and the second message in their respective partitions.
  • As a consumer subscribes to a topic, the latest offset of that topic will be provided to the consumer. The offset will also be saved in the Zookeeper ensemble of Kafka.
  • When Kafka receives messages from producers at regular intervals, these are broadcast to the consumers who requested them.
  • After receiving the message, the consumer will share the acknowledgment with Kafka brokers.
  • As soon as Kafka brokers receive the acknowledgment, they change and update the offset value in the Zookeeper so that consumers can read the messages whenever they want even in the event of a server outage.
  • The entire workflow will be repeated in cycles until the message request is put to a halt by the consumer.
  • The consumer can choose to skip the desired topic and read the next messages on another topic as they wish. 

 

The workflow of consumer group or queue messaging system can be explained in the following steps -

  • Producers send messages on one or more topics.
  • All the messages are stored in the partitions corresponding to a particular topic assigned by Kafka.
  • Let us say a single consumer subscribed to the topic denoted as ‘Topic A’ with group id ‘Group -1.’
  • Till only one consumer subscribes, Kafka will follow the pub-sub messaging system work flow to interact with the consumer until more consumers join in.
  • Kafka will switch its operational mode to shared to interact with more than one consumer. All the consumers subscribing to the same topic with the same group id will be considered as a single group. This process will continue till the number of consumers is equal to the number of partitions assigned for the particular topic.
  • In case the number of consumers exceeds the number of partitions, no new consumer will receive messages till an existing consumer unsubscribes. The new consumers will have to wait.
  • The feature of Kafka is also referred to as ‘Consumer Group.’