{"id":8826,"date":"2019-10-11T18:05:07","date_gmt":"2019-10-11T12:35:07","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/"},"modified":"2024-09-03T14:54:40","modified_gmt":"2024-09-03T09:24:40","slug":"aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/","title":{"rendered":"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3"},"content":{"rendered":"

A Capstone Project by Amit Bajaj and Sathya Guruprasad<\/em><\/p>\n

Introduction<\/b><\/h3>\n

Cloud Computing has become very popular due to the multiple benefits it provides and is being adopted by businesses worldwide. Flexibility to scale up or down as per the business needs, faster and efficient disaster recovery, subscription-based models which reduce the high cost of hardware, and flexible working for employees are some of the benefits of cloud that attracts businesses. Similar to cloud, Data Analytics is another crucial area which businesses are exploring for their growth. With the exponential rise in the amount of data available on the internet is a result of the boom in the usage of social media, mobile apps, IoT devices, sensors and so on. It has become imperative for the organisations to analyse this data to get insights into their businesses and take appropriate action.<\/span>
\nAWS provides a reliable platform for solving complex problems where cost-effective infrastructure can be built with great ease at low cost. AWS provides a wide range of managed services, including computing, storage, networking, database, analytics, application services <\/span>and many more.\u00a0<\/span><\/p>\n

Problem Statement:<\/b><\/h3>\n

We have analysed multiple software solutions which perform analysis on data collected from the market and provide information as well as suggestions and provide better customer experience. This includes trade application providing stock price, taxi companies providing locations of nearby taxis, journey plan applications providing live updates on the different transport media and many more.<\/span>
\nWe have considered a \u201c<\/span>server-less<\/b>\u201d platform \/ \u201cServer-less Computing Execution Model\u201d to build the real-time data-processing app. Architecture is based on managed services provided by AWS.<\/span><\/p>\n

What is \u201cServer-less\u201d?<\/b><\/h3>\n

A cloud-based execution model in which the cloud provider dynamically allocates and runs the server. This is a consumption-based model where pricing is directly proportional to consumer use. AWS takes complete ownership of operational responsibilities eliminating infrastructure management and availability with higher uptime.\u00a0<\/span><\/p>\n

Services Consumed:<\/b><\/h3>\n
    \n
  1. Kinesis\u00a0<\/span>- Kinesis Data Stream- Kinesis Data Analytics- Kinesis Firehose<\/li>\n
  2. Athena<\/span><\/li>\n
  3. Lambda<\/span><\/li>\n
  4. Dynamo DB<\/span><\/li>\n
  5. Amazon S3<\/span><\/li>\n
  6. AWS CLI<\/span><\/li>\n<\/ol>\n

    Without building a sizable infrastructure, how to receive data from different sources for cloud-based infrastructure?<\/b><\/em>
    \nKinesis, a managed service by AWS, Amazon Kinesis makes it easy to collect, process, and analyse real-time, streaming data so you can get timely insights and react quickly to new information. Kinesis Datastream allows user to receive data from data generation source. We have created amazon kinesis data stream using AWS CLI commands which is expected to consume data from the data source.<\/span><\/p>\n

    Technical + Functional Flow\u00a0<\/b><\/h3>\n

    Create Kinesis data streams:\u00a0<\/span><\/p>\n

      \n
    1. \n
        \n
      1. \n
          \n
        1. Create a stream in Kinesis using AWS Console or AWS CLI Commands; one to receive data from Data generator and another to write post processing. Data generator will produce the data which will be read and written to input\/source data stream. Kinesis Analytics App will process and write data to Output\/destination stream.<\/span><\/li>\n
        2. We have created a program to generate data, and with the help of AWS SDKs and AWS CLI commands transmitted to Kinesis Data Streams. Data can be generated in various fashion:<\/span>\n
            \n
          1. Using IoT devices<\/span><\/li>\n
          2. Live trackers<\/span><\/li>\n
          3. GPS trackers<\/span><\/li>\n
          4. API<\/span><\/li>\n
          5. Data generator tools (in case of Analysis)<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

            Create a Kinesis Analytics App to Aggregate data<\/span>:\u00a0<\/b><\/p>\n

              \n
            1. \n
                \n
              1. \n
                  \n
                1. Build a Kinesis Data Analytics application to read from the input\/source data stream\u00a0and write to output\/destination data stream in formatted fashion in a specified time interval.<\/span><\/li>\n
                2. It is very important to stop the application when not in use to save unwanted cost.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

                  Data Storage and Processing<\/span>:<\/p>\n

                    \n
                  1. \n
                      \n
                    1. \n
                        \n
                      1. Lambda, another managed service by AWS processes data from trigger data stream and write to dynamo DB<\/span><\/li>\n
                      2. Lambda function works on trigger basis and cost model is strictly driven by consumption. No cost is incurred from user when function is not running. Data is stored in Dynamo DB and can be accessed in standard fashion.<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

                        Kinesis Firehose, S3 and Athena<\/span>:<\/p>\n

                          \n
                        1. \n
                            \n
                          1. Kinesis Firehose acts as mediator between Kinesis Datastream and S3 where Data received from Kinesis Datastream will be predefined S3 bucket in specified format<\/span><\/li>\n
                          2. Amazon Athena is server-less interactive query service which enables user to glorify data stored in S3 Bucket for analysis.\u00a0<\/span><\/li>\n<\/ol>\n<\/li>\n<\/ol>\n

                            Amazon CLI, AWS Cloud formation and AWS IAM also plays a very important role in building Cloud based infrastructure and ensure secure connectivity within and outside AWS cloud world.\u00a0<\/span><\/p>\n

                            Conclusion:<\/b><\/h3>\n

                            Using AWS services, we were able to create a real-time data processing application based on serverless architecture which is capable of accepting data through Kinesis data streams, processing through Kinesis Data Analytics, triggering Lambda Function and storing in DynamoDB. The architecture can be reused for multiple data types from various data sources and formats with minor modifications. We have used all the managed services provided by AWS which led to zero infrastructure management efforts.\u00a0<\/span>
                            \nCapstone project has helped us in building practical expertise on AWS services like Kinesis, Lambda, Dynamo DB, Athena, S3, Identity and Access Management, Serverless Architecture and Managed Services. We have also learnt the Go\u00a0(programming language)<\/span> <\/a>to build pseudo data producer programs. AWS CLI has helped us to connect on-premise infrastructure with cloud services.\u00a0\u00a0<\/span>
                            \nThis project is a part of Great Learning's                             post-graduate program in Cloud Computing<\/a>.
                            \nAuthors<\/strong>
                            \nAmit Bajaj - Project Manager at Cognizant
                            \nSathya Guruprasad - Infrastructure Specialist at IBM Pvt Ltd<\/p>\n","protected":false},"excerpt":{"rendered":"

                            A Capstone Project by Amit Bajaj and Sathya Guruprasad Introduction Cloud Computing has become very popular due to the multiple benefits it provides and is being adopted by businesses worldwide. Flexibility to scale up or down as per the business needs, faster and efficient disaster recovery, subscription-based models which reduce the high cost of hardware, […]<\/p>\n","protected":false},"author":41,"featured_media":8139,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[6],"tags":[],"content_type":[],"class_list":["post-8826","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-cloud-computing"],"acf":[],"yoast_head":"\nAWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3<\/title>\n<meta name=\"description\" content=\"A cloud computing project on using AWS to build real-time data processing application using Kinesis, Lambda, DynamoDB, S3. Read to know about implementation\" \/>\n<meta name=\"robots\" content=\"noindex, follow\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3\" \/>\n<meta property=\"og:description\" content=\"A cloud computing project on using AWS to build real-time data processing application using Kinesis, Lambda, DynamoDB, S3. Read to know about implementation\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/\" \/>\n<meta property=\"og:site_name\" content=\"Great Learning Blog: Free Resources what Matters to shape your Career!\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/GreatLearningOfficial\/\" \/>\n<meta property=\"article:published_time\" content=\"2019-10-11T12:35:07+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-03T09:24:40+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1337\" \/>\n\t<meta property=\"og:image:height\" content=\"515\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Great Learning Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/Great_Learning\" \/>\n<meta name=\"twitter:site\" content=\"@Great_Learning\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Great Learning Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"4 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/\"},\"author\":{\"name\":\"Great Learning Editorial Team\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\"},\"headline\":\"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3\",\"datePublished\":\"2019-10-11T12:35:07+00:00\",\"dateModified\":\"2024-09-03T09:24:40+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/\"},\"wordCount\":875,\"commentCount\":1,\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg\",\"articleSection\":[\"Cloud Computing\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/\",\"name\":\"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg\",\"datePublished\":\"2019-10-11T12:35:07+00:00\",\"dateModified\":\"2024-09-03T09:24:40+00:00\",\"description\":\"A cloud computing project on using AWS to build real-time data processing application using Kinesis, Lambda, DynamoDB, S3. Read to know about implementation\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2019\\\/10\\\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg\",\"width\":1337,\"height\":515},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"Cloud Computing\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/cloud-computing\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"name\":\"Great Learning Blog\",\"description\":\"Learn, Upskill & Career Development Guide and Resources\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"alternateName\":\"Great Learning\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\",\"name\":\"Great Learning\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"width\":900,\"height\":900,\"caption\":\"Great Learning\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/GreatLearningOfficial\\\/\",\"https:\\\/\\\/x.com\\\/Great_Learning\",\"https:\\\/\\\/www.instagram.com\\\/greatlearningofficial\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/in.pinterest.com\\\/greatlearning12\\\/\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/beaconelearning\\\/\"],\"description\":\"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.\",\"email\":\"info@mygreatlearning.com\",\"legalName\":\"Great Learning Education Services Pvt. Ltd\",\"foundingDate\":\"2013-11-29\",\"numberOfEmployees\":{\"@type\":\"QuantitativeValue\",\"minValue\":\"1001\",\"maxValue\":\"5000\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\",\"name\":\"Great Learning Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"caption\":\"Great Learning Editorial Team\"},\"description\":\"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.\",\"sameAs\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/\",\"https:\\\/\\\/in.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/x.com\\\/https:\\\/\\\/twitter.com\\\/Great_Learning\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCObs0kLIrDjX2LLSybqNaEA\"],\"award\":[\"Best EdTech Company of the Year 2024\",\"Education Economictimes Outstanding Education\\\/Edtech Solution Provider of the Year 2024\",\"Leading E-learning Platform 2024\"],\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/author\\\/greatlearning\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3","description":"A cloud computing project on using AWS to build real-time data processing application using Kinesis, Lambda, DynamoDB, S3. Read to know about implementation","robots":{"index":"noindex","follow":"follow"},"og_locale":"en_US","og_type":"article","og_title":"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3","og_description":"A cloud computing project on using AWS to build real-time data processing application using Kinesis, Lambda, DynamoDB, S3. Read to know about implementation","og_url":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/","og_site_name":"Great Learning Blog: Free Resources what Matters to shape your Career!","article_publisher":"https:\/\/www.facebook.com\/GreatLearningOfficial\/","article_published_time":"2019-10-11T12:35:07+00:00","article_modified_time":"2024-09-03T09:24:40+00:00","og_image":[{"width":1337,"height":515,"url":"http:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg","type":"image\/jpeg"}],"author":"Great Learning Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/Great_Learning","twitter_site":"@Great_Learning","twitter_misc":{"Written by":"Great Learning Editorial Team","Est. reading time":"4 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#article","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/"},"author":{"name":"Great Learning Editorial Team","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad"},"headline":"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3","datePublished":"2019-10-11T12:35:07+00:00","dateModified":"2024-09-03T09:24:40+00:00","mainEntityOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/"},"wordCount":875,"commentCount":1,"publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg","articleSection":["Cloud Computing"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/","url":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/","name":"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#primaryimage"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg","datePublished":"2019-10-11T12:35:07+00:00","dateModified":"2024-09-03T09:24:40+00:00","description":"A cloud computing project on using AWS to build real-time data processing application using Kinesis, Lambda, DynamoDB, S3. Read to know about implementation","breadcrumb":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#primaryimage","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg","width":1337,"height":515},{"@type":"BreadcrumbList","@id":"https:\/\/www.mygreatlearning.com\/blog\/aws-solution-to-build-real-time-data-processing-application-using-kinesis-lambda-dynamodb-s3\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https:\/\/www.mygreatlearning.com\/blog\/"},{"@type":"ListItem","position":2,"name":"Cloud Computing","item":"https:\/\/www.mygreatlearning.com\/blog\/cloud-computing\/"},{"@type":"ListItem","position":3,"name":"AWS solution to build Real-time Data processing Application using Kinesis, Lambda, DynamoDB, S3"}]},{"@type":"WebSite","@id":"https:\/\/www.mygreatlearning.com\/blog\/#website","url":"https:\/\/www.mygreatlearning.com\/blog\/","name":"Great Learning Blog","description":"Learn, Upskill & Career Development Guide and Resources","publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"alternateName":"Great Learning","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.mygreatlearning.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization","name":"Great Learning","url":"https:\/\/www.mygreatlearning.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","width":900,"height":900,"caption":"Great Learning"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/GreatLearningOfficial\/","https:\/\/x.com\/Great_Learning","https:\/\/www.instagram.com\/greatlearningofficial\/","https:\/\/www.linkedin.com\/school\/great-learning\/","https:\/\/in.pinterest.com\/greatlearning12\/","https:\/\/www.youtube.com\/user\/beaconelearning\/"],"description":"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.","email":"info@mygreatlearning.com","legalName":"Great Learning Education Services Pvt. Ltd","foundingDate":"2013-11-29","numberOfEmployees":{"@type":"QuantitativeValue","minValue":"1001","maxValue":"5000"}},{"@type":"Person","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad","name":"Great Learning Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","caption":"Great Learning Editorial Team"},"description":"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.","sameAs":["https:\/\/www.mygreatlearning.com\/","https:\/\/in.linkedin.com\/school\/great-learning\/","https:\/\/x.com\/https:\/\/twitter.com\/Great_Learning","https:\/\/www.youtube.com\/channel\/UCObs0kLIrDjX2LLSybqNaEA"],"award":["Best EdTech Company of the Year 2024","Education Economictimes Outstanding Education\/Edtech Solution Provider of the Year 2024","Leading E-learning Platform 2024"],"url":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"}]}},"uagb_featured_image_src":{"full":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg",1337,515,false],"thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1-150x150.jpg",150,150,true],"medium":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1-300x116.jpg",300,116,true],"medium_large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1-768x296.jpg",768,296,true],"large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1-1024x394.jpg",1024,394,true],"1536x1536":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg",1337,515,false],"2048x2048":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg",1337,515,false],"web-stories-poster-portrait":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg",640,247,false],"web-stories-publisher-logo":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg",96,37,false],"web-stories-thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2019\/10\/AWS-solution-to-build-Real-time-Data-processing-Application-1-1.jpg",150,58,false]},"uagb_author_info":{"display_name":"Great Learning Editorial Team","author_link":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"},"uagb_comment_info":1,"uagb_excerpt":"A Capstone Project by Amit Bajaj and Sathya Guruprasad Introduction Cloud Computing has become very popular due to the multiple benefits it provides and is being adopted by businesses worldwide. Flexibility to scale up or down as per the business needs, faster and efficient disaster recovery, subscription-based models which reduce the high cost of hardware,…","_links":{"self":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/8826","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/users\/41"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/comments?post=8826"}],"version-history":[{"count":7,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/8826\/revisions"}],"predecessor-version":[{"id":104703,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/8826\/revisions\/104703"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media\/8139"}],"wp:attachment":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media?parent=8826"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/categories?post=8826"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/tags?post=8826"},{"taxonomy":"content_type","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/content_type?post=8826"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}