Kafka

Consumer Group Example

Consumer Group Example

 

The consumer is the one who consumes data from a certain partition from the Kafka cluster. It knows from which broker to read or consume data on a specific topic.

 

Consumer Groups, on the other hand, consist of a group of more than one consumer and each customer reads or consumes data from the assigned partitions. If the number of consumers will exceed the number of partitions, the new consumers will go into the inactive state until an existing customer unsubscribes.    

 

Consider this scenario with two consumers -

 

Let us assume there are two groups of consumers- Consumer Group A and Consumer Group B. If the consumers from Consumer Group A are reading data from different partitions, it implies that they are reading it parallel under Topic - T and therefore will remain in an active state.

Now, if one of the consumers of Group B reads data from Partition - 1 under Topic - T, then also the consumer remains in an active state as it belongs to Group B. 

 

Consider another scenario with three consumers -

 

There are three consumers with two partitions available. Two consumers - Consumer A and Consumer B are in an active state with consumer A reading data from Partition 0 and consumer B reading from Partition 1. Consumer C will remain in an inactive state unless consumer A or consumer B leaves. 

Now let us see examples of how to send and receive messages through Java client.

 

Here’s how you can create a sample group with two consumers -

 

>>java -cp “/path/to/kafka/kafka_2.11-0.9.0.0/libs/*":. 

ConsumerGroup <topic-name> my-group

>>java -cp "/home/bala/Workspace/kafka/kafka_2.11-0.9.0.0/libs/*":. 

ConsumerGroup <topic-name> my-group

 

Now send messages using Producer CLI -

 

Test consumer group 01

Test consumer group 02

 

It will give the output of the first and the second process respectively as -

 

Subscribed to topic Hello-kafka

offset = 3, key = null, value = Test consumer group 01

 

And 

 

Subscribed to topic Hello-kafka

offset = 3, key = null, value = Test consumer group 02

 

Top course recommendations for you

    Blockchain Process
    1 hrs
    Beginner
    7.1K+ Learners
    4.58  (719)
    GO Programming Language
    1 hrs
    Beginner
    6.9K+ Learners
    4.49  (578)
    Conditional Formatting in Excel
    1 hrs
    Beginner
    18.1K+ Learners
    4.52  (1770)
    Fibonacci Series in Python
    1 hrs
    Beginner
    2.2K+ Learners
    4.62  (98)
    Design App
    1 hrs
    Beginner
    18.5K+ Learners
    4.45  (1277)
    Pivot Tables in Excel
    1 hrs
    Beginner
    22.8K+ Learners
    4.57  (3005)
    Divide and Conquer Algorithms
    1 hrs
    Beginner
    2.4K+ Learners
    4.59  (416)
    QR Code Generator in Python
    1 hrs
    Beginner
    4.9K+ Learners
    4.39  (225)
    Backtracking Algorithm
    1 hrs
    Beginner
    2.8K+ Learners
    4.56  (308)
    Bitcoin for Beginners
    1 hrs
    Beginner
    7K+ Learners
    4.54  (461)