Kafka

How to integrate with Storm?

How to integrate with Storm?

 

Both Apache Kafka and Storm complement each other for accurate real-time streaming analysis of the big data in very little time. Kafka-Storm integration allows developers to easily publish streams of data using Storm topologies and run continuously. 

 

Three main classes that oversee the integration of Kafka-Storm API are -

 

  • BrokerHosts 

 

The BrokerHosts interface consists of two implementations - ZkHosts and StaticHosts. ZkHosts offers a fast way for tracking and maintaining the details of Kafka brokers 9n ZooKeeper while StaticHosts manually select the Kafka brokers. 

The signature of ZkHosts is -

 

public ZkHosts(String brokerZkStr, String brokerZkPath)

public ZkHosts(String brokerZkStr)

 

where brokerZkStr refers to the ZooKeeper host and brokerZkPath is the ZooKeeper path for maintaining the Kafka broker and its details.

 

  • KafkaConfig API 

This API defines the configuration settings of the Kafka cluster and its signature is -

 

public KafkaConfig(BrokerHosts hosts, string topic)

 

where hosts are the BrokerHosts (ZkHosts / StaticHosts) and the topic corresponds to the name of the topic.

 

  • SpoutConfig API 

This API is an extension of KafkaConfigAPI and is responsible for storing additional ZooKeeper information. Its signature is -

 

public SpoutConfig(BrokerHosts hosts, string topic, string zkRoot, string id)

 

where zkRoot is the ZooKeeper root path and id corresponds to the state of the offsets that identifies the spout. 

 

Top course recommendations for you

    Beautiful Soup
    2 hrs
    Beginner
    1.3K+ Learners
    4.59  (49)
    Java Applications
    1 hrs
    Beginner
    6K+ Learners
    4.47  (125)
    Python List
    1 hrs
    Beginner
    4.3K+ Learners
    4.6  (145)
    Python Classes
    1 hrs
    Beginner
    5.9K+ Learners
    4.51  (166)
    OOPs in Python
    1 hrs
    Beginner
    13.2K+ Learners
    4.39  (752)
    Docker Swarm Project
    1 hrs
    Beginner
    691 Learners
    4.73  (15)
    Collections in Java
    3 hrs
    Beginner
    3.8K+ Learners
    4.61  (97)
    PyCharm for Beginners
    1 hrs
    Beginner
    2.3K+ Learners
    4.39  (97)
    Git Tutorial
    2 hrs
    Beginner
    8.6K+ Learners
    4.53  (499)
    Create a IPL theme Landing page with CSS and HTML
    1 hrs
    Beginner
    5.9K+ Learners
    4.29  (243)