Kafka

How to integrate with Storm?

How to integrate with Storm?

 

Both Apache Kafka and Storm complement each other for accurate real-time streaming analysis of the big data in very little time. Kafka-Storm integration allows developers to easily publish streams of data using Storm topologies and run continuously. 

 

Three main classes that oversee the integration of Kafka-Storm API are -

 

  • BrokerHosts 

 

The BrokerHosts interface consists of two implementations - ZkHosts and StaticHosts. ZkHosts offers a fast way for tracking and maintaining the details of Kafka brokers 9n ZooKeeper while StaticHosts manually select the Kafka brokers. 

The signature of ZkHosts is -

 

public ZkHosts(String brokerZkStr, String brokerZkPath)

public ZkHosts(String brokerZkStr)

 

where brokerZkStr refers to the ZooKeeper host and brokerZkPath is the ZooKeeper path for maintaining the Kafka broker and its details.

 

  • KafkaConfig API 

This API defines the configuration settings of the Kafka cluster and its signature is -

 

public KafkaConfig(BrokerHosts hosts, string topic)

 

where hosts are the BrokerHosts (ZkHosts / StaticHosts) and the topic corresponds to the name of the topic.

 

  • SpoutConfig API 

This API is an extension of KafkaConfigAPI and is responsible for storing additional ZooKeeper information. Its signature is -

 

public SpoutConfig(BrokerHosts hosts, string topic, string zkRoot, string id)

 

where zkRoot is the ZooKeeper root path and id corresponds to the state of the offsets that identifies the spout. 

 

Top course recommendations for you

    Java Applications
    1 hrs
    Beginner
    8.8K+ Learners
    4.38  (452)
    Python List
    1 hrs
    Beginner
    6.4K+ Learners
    4.21  (666)
    Python Classes
    1 hrs
    Beginner
    11.7K+ Learners
    4.4  (882)
    OOPs in Python
    1 hrs
    Beginner
    16.7K+ Learners
    4.32  (1413)
    Docker Swarm Project
    1 hrs
    Beginner
    879 Learners
    4.56  (36)
    Collections in Java
    3 hrs
    Beginner
    5.1K+ Learners
    4.42  (478)
    PyCharm for Beginners
    1 hrs
    Beginner
    3K+ Learners
    4.41  (175)
    Git Tutorial
    2 hrs
    Beginner
    10.9K+ Learners
    4.54  (919)
    Create a IPL theme Landing page with CSS and HTML
    1 hrs
    Beginner
    7.1K+ Learners
    4.31  (347)