{"id":18668,"date":"2020-08-23T12:39:27","date_gmt":"2020-08-23T07:09:27","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/"},"modified":"2024-12-03T17:56:56","modified_gmt":"2024-12-03T12:26:56","slug":"apache-kafka","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/","title":{"rendered":"Apache Kafka Use Cases, APIs and How Kafka Works?"},"content":{"rendered":"\n<p>Kafka is a distributed streaming platform. Distributed streaming platform is nothing but assuming you have a lot of applications running in your company, so there can be applications like web applications, mobile applications, and web server logs so there is \u2018n\u2019 number of applications which are writing logs or embedding data. We need to store all this data in a very efficient manner.<\/p>\n\n\n\n<p>Check out this free <a href=\"https:\/\/www.mygreatlearning.com\/academy\/learn-for-free\/courses\/kafka\" target=\"_blank\" rel=\"noreferrer noopener\">Kafka course<\/a> to learn more. <\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"what-is-kafka\"><strong>What is Kafka?<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kafka is a distributed streaming platform: <br>- publish-subscribe messaging system<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A messaging system lets you send messages between processes, applications, and servers.<br>- Store streams of records in a fault-tolerant durable way.<br>- Process streams of records as they occur.<\/li>\n<\/ul>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kafka is used for building real-time data pipelines and streaming apps<\/li>\n\n\n\n<li>It is horizontally scalable, fault-tolerant, fast and runs in production in thousands of companies.<\/li>\n\n\n\n<li>Originally started by LinkedIn, later open sourced Apache in 2011.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"use-cases-of-kafka\"><strong>Use cases of Kafka<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li><strong>Metrics<\/strong> \u2212 Apache Kafka is often used for operational monitoring data. This involves aggregating statistics from distributed applications to produce centralized feeds of operational data.<\/li>\n\n\n\n<li><strong>Log Aggregation Solution<\/strong> \u2212 Apache Kafka can be used across an organization to collect logs from multiple services and make them available in a standard format to multiple consumers.<\/li>\n\n\n\n<li><strong>Stream Processing<\/strong> \u2212 Popular frameworks such as Storm and Spark Streaming read data from a topic, process it, and write processed data to a new topic where it becomes available for users and applications. Apache Kafka\u2019s strong durability is also very useful in the context of stream processing.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"key-components-of-kafka\"><strong>Key components of Kafka<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Broker<\/li>\n\n\n\n<li>Producers<\/li>\n\n\n\n<li>Consumers<\/li>\n\n\n\n<li>Topic<\/li>\n\n\n\n<li>Partitions<\/li>\n\n\n\n<li>Offset<\/li>\n\n\n\n<li>Consumer Group<\/li>\n\n\n\n<li>Replication<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"broker\"><strong>Broker:<\/strong> <\/h4>\n\n\n\n<p>Apache Kafka runs as a cluster on one or more servers that can span multiple data centers. An instance of the cluster is broker.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"producer-consumer\"><strong>Producer &amp; Consumer<\/strong>:<\/h4>\n\n\n\n<p>Producer: It writes data to the brokers.<br>Consumer: It consumes data from brokers.<\/p>\n\n\n\n<p>Apache Kafka clusters can be running in multiple nodes.<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"kafka-topic\"><strong>Kafka Topic:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A Topic is a category\/feed name to which messages are stored and published.<\/li>\n\n\n\n<li>If you wish to send a message you send it to a specific topic and if you wish to read a message you read it from a specific topic.<\/li>\n\n\n\n<li>Why we need a topic: In the same Kafka Cluster data from many different sources can be coming at the same time. Ex. logs, web activities, metrics etc. So Topics are useful to identify that this data is stored in a particular topic.<\/li>\n\n\n\n<li>Producer applications write data to topics and consumer applications read from topics.&nbsp;<\/li>\n<\/ul>\n\n\n\n<p>Also Read: <a href=\"https:\/\/www.mygreatlearning.com\/blog\/top-hadoop-interview-questions\/\" target=\"_blank\" rel=\"noreferrer noopener\" aria-label=\"Top 40 Hadoop Interview Questions  (opens in a new tab)\">Top 40 Hadoop Interview Questions <\/a><\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"partitions\"><strong>Partitions:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Kafka topics are divided into a number of partitions, which contains messages in an unchangeable sequence(immutable).<\/li>\n\n\n\n<li>Each message in a partition is assigned and identified by its unique offset.<\/li>\n\n\n\n<li>A topic can also have multiple partition logs.This allows for multiple consumers to read from a topic in parallel.<\/li>\n\n\n\n<li>Partitions allow you to parallelize a topic by splitting the data in a particular topic across multiple brokers<\/li>\n<\/ul>\n\n\n\n<p>Anatomy of a topic:<\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"partition-offset\"><strong>Partition Offset:<\/strong><\/h4>\n\n\n\n<p>Offset:&nbsp; Messages in the partitions are each assigned a unique (per partition) and sequential id called the offset. Consumers track their pointers via (offset, partition, topic) tuples. <\/p>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"consumer-and-consumer-group\"><strong>Consumer and Consumer Group:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Consumers can read messages starting from a specific offset and are allowed to read from any offset point they choose.<\/li>\n\n\n\n<li>This allows consumers to join the cluster at any point in time.<\/li>\n\n\n\n<li>Consumers can join a group called a consumer group.<\/li>\n\n\n\n<li>A consumer group includes the set of consumer processes that are subscribing to a specific topic.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"replication\"><strong>Replication:<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>In Kafka, replication is implemented at the partition level. Helps to prevent data loss.<\/li>\n\n\n\n<li>The redundant unit of a topic partition is called a replica.<\/li>\n\n\n\n<li>Each partition usually has one or more replicas meaning that partitions contain messages that are replicated over a few Kafka brokers in the cluster. As we can see in the pictures - the <em>click-topic<\/em> is replicated to Kafka node 2 and Kafka node 3.<\/li>\n<\/ul>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"kafka-apis\"><strong>Kafka APIs<\/strong><\/h2>\n\n\n\n<p>Kafka has four core APIs:<\/p>\n\n\n\n<p>\u25cf The <strong>Producer API<\/strong> allows an application to publish a stream of records to one or more Kafka topics.<\/p>\n\n\n\n<p>\u25cf The <strong>Consumer API<\/strong> allows an application to subscribe to one or more topics and process the stream of records.<\/p>\n\n\n\n<p>\u25cf The <strong>Streams API<\/strong> allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.<\/p>\n\n\n\n<p>\u25cf The <strong>Connector API<\/strong> allows building and running reusable producers or consumers that connect Kafka topics to existing applications or data systems. For example, a connector to a relational database might capture every change to a table.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"how-kafka-works\"><strong>How Kafka Works?<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Producers writes data to the topic<\/li>\n\n\n\n<li>As a message record is written to a partition of the topic, it\u2019s offset is increased by 1.<\/li>\n\n\n\n<li>Consumers consume data from the topic. Each consumer read data based on the offset value.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"real-world-example\"><strong>Real world example<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Website activity tracking.<\/li>\n\n\n\n<li>Let\u2019s take an example of Flipkart, when you visit flipkart &amp; perform any action like search, login, click on a product etc all of these events are captured.<\/li>\n\n\n\n<li>Tracking event will create a message stream for this based on the kind of event it\u2019ll go to a specific topic by Kafka Producer.<\/li>\n\n\n\n<li>This kind of activity tracking often requires a very high volume of throughput, messages are generated for each action.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"steps\"><strong>Steps<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>A user clicks on a button on a website.<\/li>\n\n\n\n<li>The web application publishes a message to partition 0 in topic \"click\".<\/li>\n\n\n\n<li>The message is appended to its commit log and the message offset is incremented.<\/li>\n\n\n\n<li>The consumer can pull messages from the click-topic and show monitoring usage in real-time or for any other use case.<\/li>\n<\/ul>\n\n\n\n<h4 class=\"wp-block-heading\" id=\"zookeeper\"><strong>Zookeeper<\/strong><\/h4>\n\n\n\n<ul class=\"wp-block-list\">\n<li>ZooKeeper is used for managing and coordinating Kafka broker.<\/li>\n\n\n\n<li>ZooKeeper service is mainly used to notify producers and consumers about the presence of any new broker in the Kafka system or failure of the broker in the Kafka system.<\/li>\n\n\n\n<li>As per the notification received by the Zookeeper regarding presence or failure of the broker then producer and consumer takes decision and starts coordinating their task with some other broker.<\/li>\n\n\n\n<li>The ZooKeeper framework was originally built at Yahoo!<\/li>\n<\/ul>\n\n\n\n<p>Also <a href=\"https:\/\/www.mygreatlearning.com\/apache\/free-courses\" target=\"_blank\" rel=\"noreferrer noopener\">learn apache<\/a> for free.<\/p>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"how-to-install-and-get-started\"><strong>How to install and get started?<\/strong><\/h2>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Download Apache kafka &amp; zookeeper<\/li>\n<\/ol>\n\n\n\n<p>2. Start Zookeeper server then kafka &amp; run a single broker<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; bin\/zookeeper-server-start.sh config\/zookeeper.properties\n&gt; bin\/kafka-server-start.sh config\/server.properties\n<\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Create a topic named test<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; bin\/kafka-topics.sh --create --zookeeper localhost:2181 --replication-factor 1 --partitions 1 --topic test\n&gt; bin\/kafka-topics.sh --list --zookeeper localhost:2181\ntest\n<\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Run the producer &amp; send some messages<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; bin\/kafka-console-producer.sh --broker-list localhost:9092 --topic test\nThis is a message\nThis is another message<\/code><\/pre>\n\n\n\n<ol class=\"wp-block-list\">\n<li>Start a consumer<\/li>\n<\/ol>\n\n\n\n<pre class=\"wp-block-code\"><code>&gt; bin\/kafka-console-consumer.sh --bootstrap-server localhost:9092 --topic test --from-beginning\nThis is a message\nThis is another message\n<\/code><\/pre>\n\n\n\n<h2 class=\"wp-block-heading\" id=\"live-demo-code\"><strong>Live Demo code:<\/strong><\/h2>\n\n\n\n<ul class=\"wp-block-list\">\n<li>Live Demo of Getting Tweets in Real Time by Calling Twitter API<\/li>\n\n\n\n<li>Pushing all the Tweets to a Kafka Topic by Creating Kafka Producer in Real Time<\/li>\n<\/ul>\n\n\n\n<p>Please see the code in Jupyter below.<\/p>\n\n\n\n<pre class=\"wp-block-code\"><code>from tweepy.streaming import StreamListener\nfrom tweepy import OAuthHandler\nfrom tweepy import Stream\nfrom kafka import SimpleProducer, KafkaClient\nimport time\n\naccess_token = \"748200460067045376-zYxRRyxiPIywcw2IV50IQiIxzQVN5FZ\"\naccess_token_secret =  \"c6dRkeRbgPqtbWAOTz0OfOMpBvhZS6KqWFjtEqHBEv7me\"\nconsumer_key =  \"ukUrCrJdd6MQQd0HQBzCDwcLq\"\nconsumer_secret =  \"VCf2wU1MhedUFnQeCwffzstdVkF7rbURzoNNDAdPPvbWfDtggP\"\nkafka_endpoint = \"ip-20-0-32-4.ap-south-1.compute.internal:9092\"\nkafka_topic = \"rk_hadoop\"\ntwitter_hash_tag = \"RamNavami\"\ntime_limit = 10\n\nclass StdOutListener(StreamListener):\ndef __init__(self, time_limit=time_limit):\nself.start_time = time.time()\nself.limit = time_limit\nsuper(StdOutListener, self).__init__()\ndef on_data(self, data):\nif (time.time() - self.start_time) &lt; self.limit:\n      #msg = json.loads(data)\nproducer.send_messages(kafka_topic, data.encode('utf-8'))\nprint (data)\nreturn True\nexit(0)\ndef on_error(self, status):\nprint (status)\n\nkafka = KafkaClient(kafka_endpoint)\nproducer = SimpleProducer(kafka)\nl = StdOutListener()\nauth = OAuthHandler(consumer_key, consumer_secret)\nauth.set_access_token(access_token, access_token_secret)\nstream = Stream(auth, l)\nstream.filter(track=twitter_hash_tag)<\/code><\/pre>\n\n\n\n<p>This brings us to the end of the blog. If you found this helpful and wish to learn more about such concepts, embark on a learning adventure like no other with our extensive collection of <a href=\"https:\/\/www.mygreatlearning.com\/academy\" target=\"_blank\" rel=\"noreferrer noopener\">free online courses<\/a>. Whether you're interested in diving into the world of Cybersecurity, mastering the art of Management, exploring the wonders of Cloud Computing, or delving into the intricate realm of IT and Software, we have courses that cater to your interests and goals.<\/p>\n\n\n\n<p>Have a look at this <a href=\"https:\/\/www.mygreatlearning.com\/academy\/learn-for-free\/courses\/introduction-to-apache-hive\" target=\"_blank\" rel=\"noreferrer noopener\">Apache Hive Tutorial<\/a> while you are at it. All the best!<\/p>\n","protected":false},"excerpt":{"rendered":"<p>Kafka is a distributed streaming platform. Distributed streaming platform is nothing but assuming you have a lot of applications running in your company, so there can be applications like web applications, mobile applications, and web server logs so there is \u2018n\u2019 number of applications which are writing logs or embedding data. We need to store [&hellip;]<\/p>\n","protected":false},"author":41,"featured_media":19047,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[9],"tags":[],"content_type":[],"class_list":["post-18668","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-data-science"],"acf":[],"yoast_head":"<!-- This site is optimized with the Yoast SEO Premium plugin v27.3 (Yoast SEO v27.3) - https:\/\/yoast.com\/product\/yoast-seo-premium-wordpress\/ -->\n<title>What is Apache Kafka, Use Cases, APIs and Real World Examples<\/title>\n<meta name=\"description\" content=\"What is Apache Kafka: Apache Kafka is a distributed streaming platform. Learn more about how it works, real-world use cases, how to install it and it&#039;s implementation.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Apache Kafka Use Cases, APIs and How Kafka Works?\" \/>\n<meta property=\"og:description\" content=\"What is Apache Kafka: Apache Kafka is a distributed streaming platform. Learn more about how it works, real-world use cases, how to install it and it&#039;s implementation.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/\" \/>\n<meta property=\"og:site_name\" content=\"Great Learning Blog: Free Resources what Matters to shape your Career!\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/GreatLearningOfficial\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-23T07:09:27+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-12-03T12:26:56+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1254\" \/>\n\t<meta property=\"og:image:height\" content=\"836\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Great Learning Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/Great_Learning\" \/>\n<meta name=\"twitter:site\" content=\"@Great_Learning\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Great Learning Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"6 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/\"},\"author\":{\"name\":\"Great Learning Editorial Team\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\"},\"headline\":\"Apache Kafka Use Cases, APIs and How Kafka Works?\",\"datePublished\":\"2020-08-23T07:09:27+00:00\",\"dateModified\":\"2024-12-03T12:26:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/\"},\"wordCount\":1212,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/iStock-880810940.jpg\",\"articleSection\":[\"Data Science and Analytics\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/\",\"name\":\"What is Apache Kafka, Use Cases, APIs and Real World Examples\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/iStock-880810940.jpg\",\"datePublished\":\"2020-08-23T07:09:27+00:00\",\"dateModified\":\"2024-12-03T12:26:56+00:00\",\"description\":\"What is Apache Kafka: Apache Kafka is a distributed streaming platform. Learn more about how it works, real-world use cases, how to install it and it's implementation.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/iStock-880810940.jpg\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/iStock-880810940.jpg\",\"width\":1254,\"height\":836,\"caption\":\"apache kafka\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/apache-kafka\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Data Science and Analytics\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/data-science\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Apache Kafka Use Cases, APIs and How Kafka Works?\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"name\":\"Great Learning Blog\",\"description\":\"Learn, Upskill &amp; Career Development Guide and Resources\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"alternateName\":\"Great Learning\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\",\"name\":\"Great Learning\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"width\":900,\"height\":900,\"caption\":\"Great Learning\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/GreatLearningOfficial\\\/\",\"https:\\\/\\\/x.com\\\/Great_Learning\",\"https:\\\/\\\/www.instagram.com\\\/greatlearningofficial\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/in.pinterest.com\\\/greatlearning12\\\/\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/beaconelearning\\\/\"],\"description\":\"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.\",\"email\":\"info@mygreatlearning.com\",\"legalName\":\"Great Learning Education Services Pvt. Ltd\",\"foundingDate\":\"2013-11-29\",\"numberOfEmployees\":{\"@type\":\"QuantitativeValue\",\"minValue\":\"1001\",\"maxValue\":\"5000\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\",\"name\":\"Great Learning Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"caption\":\"Great Learning Editorial Team\"},\"description\":\"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.\",\"sameAs\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/\",\"https:\\\/\\\/in.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/x.com\\\/https:\\\/\\\/twitter.com\\\/Great_Learning\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCObs0kLIrDjX2LLSybqNaEA\"],\"award\":[\"Best EdTech Company of the Year 2024\",\"Education Economictimes Outstanding Education\\\/Edtech Solution Provider of the Year 2024\",\"Leading E-learning Platform 2024\"],\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/author\\\/greatlearning\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"What is Apache Kafka, Use Cases, APIs and Real World Examples","description":"What is Apache Kafka: Apache Kafka is a distributed streaming platform. Learn more about how it works, real-world use cases, how to install it and it's implementation.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/","og_locale":"en_US","og_type":"article","og_title":"Apache Kafka Use Cases, APIs and How Kafka Works?","og_description":"What is Apache Kafka: Apache Kafka is a distributed streaming platform. Learn more about how it works, real-world use cases, how to install it and it's implementation.","og_url":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/","og_site_name":"Great Learning Blog: Free Resources what Matters to shape your Career!","article_publisher":"https:\/\/www.facebook.com\/GreatLearningOfficial\/","article_published_time":"2020-08-23T07:09:27+00:00","article_modified_time":"2024-12-03T12:26:56+00:00","og_image":[{"width":1254,"height":836,"url":"http:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg","type":"image\/jpeg"}],"author":"Great Learning Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/Great_Learning","twitter_site":"@Great_Learning","twitter_misc":{"Written by":"Great Learning Editorial Team","Est. reading time":"6 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#article","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/"},"author":{"name":"Great Learning Editorial Team","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad"},"headline":"Apache Kafka Use Cases, APIs and How Kafka Works?","datePublished":"2020-08-23T07:09:27+00:00","dateModified":"2024-12-03T12:26:56+00:00","mainEntityOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/"},"wordCount":1212,"commentCount":0,"publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg","articleSection":["Data Science and Analytics"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/","url":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/","name":"What is Apache Kafka, Use Cases, APIs and Real World Examples","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#primaryimage"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg","datePublished":"2020-08-23T07:09:27+00:00","dateModified":"2024-12-03T12:26:56+00:00","description":"What is Apache Kafka: Apache Kafka is a distributed streaming platform. Learn more about how it works, real-world use cases, how to install it and it's implementation.","breadcrumb":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#primaryimage","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg","width":1254,"height":836,"caption":"apache kafka"},{"@type":"BreadcrumbList","@id":"https:\/\/www.mygreatlearning.com\/blog\/apache-kafka\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https:\/\/www.mygreatlearning.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Data Science and Analytics","item":"https:\/\/www.mygreatlearning.com\/blog\/data-science\/"},{"@type":"ListItem","position":3,"name":"Apache Kafka Use Cases, APIs and How Kafka Works?"}]},{"@type":"WebSite","@id":"https:\/\/www.mygreatlearning.com\/blog\/#website","url":"https:\/\/www.mygreatlearning.com\/blog\/","name":"Great Learning Blog","description":"Learn, Upskill &amp; Career Development Guide and Resources","publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"alternateName":"Great Learning","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.mygreatlearning.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization","name":"Great Learning","url":"https:\/\/www.mygreatlearning.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","width":900,"height":900,"caption":"Great Learning"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/GreatLearningOfficial\/","https:\/\/x.com\/Great_Learning","https:\/\/www.instagram.com\/greatlearningofficial\/","https:\/\/www.linkedin.com\/school\/great-learning\/","https:\/\/in.pinterest.com\/greatlearning12\/","https:\/\/www.youtube.com\/user\/beaconelearning\/"],"description":"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.","email":"info@mygreatlearning.com","legalName":"Great Learning Education Services Pvt. Ltd","foundingDate":"2013-11-29","numberOfEmployees":{"@type":"QuantitativeValue","minValue":"1001","maxValue":"5000"}},{"@type":"Person","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad","name":"Great Learning Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","caption":"Great Learning Editorial Team"},"description":"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.","sameAs":["https:\/\/www.mygreatlearning.com\/","https:\/\/in.linkedin.com\/school\/great-learning\/","https:\/\/x.com\/https:\/\/twitter.com\/Great_Learning","https:\/\/www.youtube.com\/channel\/UCObs0kLIrDjX2LLSybqNaEA"],"award":["Best EdTech Company of the Year 2024","Education Economictimes Outstanding Education\/Edtech Solution Provider of the Year 2024","Leading E-learning Platform 2024"],"url":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"}]}},"uagb_featured_image_src":{"full":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg",1254,836,false],"thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940-150x150.jpg",150,150,true],"medium":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940-300x200.jpg",300,200,true],"medium_large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940-768x512.jpg",768,512,true],"large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940-1024x683.jpg",1024,683,true],"1536x1536":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg",1254,836,false],"2048x2048":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg",1254,836,false],"web-stories-poster-portrait":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg",640,427,false],"web-stories-publisher-logo":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg",96,64,false],"web-stories-thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/iStock-880810940.jpg",150,100,false]},"uagb_author_info":{"display_name":"Great Learning Editorial Team","author_link":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"},"uagb_comment_info":0,"uagb_excerpt":"Kafka is a distributed streaming platform. Distributed streaming platform is nothing but assuming you have a lot of applications running in your company, so there can be applications like web applications, mobile applications, and web server logs so there is \u2018n\u2019 number of applications which are writing logs or embedding data. We need to store&hellip;","_links":{"self":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/18668","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/users\/41"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/comments?post=18668"}],"version-history":[{"count":33,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/18668\/revisions"}],"predecessor-version":[{"id":101965,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/18668\/revisions\/101965"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media\/19047"}],"wp:attachment":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media?parent=18668"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/categories?post=18668"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/tags?post=18668"},{"taxonomy":"content_type","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/content_type?post=18668"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}