{"id":17311,"date":"2020-08-01T14:52:01","date_gmt":"2020-08-01T09:22:01","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/apache-hadoop-tutorial\/"},"modified":"2024-09-03T15:15:20","modified_gmt":"2024-09-03T09:45:20","slug":"apache-hadoop-tutorial","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/apache-hadoop-tutorial\/","title":{"rendered":"Apache Hadoop Tutorial |What is Apache Hadoop?"},"content":{"rendered":"\n

Big data\u2013 Introduction<\/strong><\/h2>\n\n\n\n

Before we jump into our Hadoop Tutorial, lets understand Big Data. Will start with questions like What is Big data, Why big data, What big data signifies so that the companies\/industries are moving to big data from legacy systems, is it worth it to learn big data technologies and will professional get paid high? Learn Apache<\/a> today!<\/p>\n\n\n\n

What is Big data?<\/strong><\/h2>\n\n\n\n

By name implies, big data is data with huge size. We get a large amount of data in different forms from different sources and in huge volume, velocity, variety and etc which can be derived from human or machine sources.<\/p>\n\n\n\n

We are talking about data and let us see what are the types of data to understand the logic behind big data. <\/p>\n\n\n\n

Types of data:<\/strong><\/h2>\n\n\n\n

Three types of data can be classified as:<\/p>\n\n\n\n

Structured data:<\/strong>  Data which is represented in a tabular form. The data can be stored, accessed and processed in the form of fixed format. Ex: databases, tables.<\/p>\n\n\n\n

Semi structured data:<\/strong>  Data which does not have a formal data model. Ex: XML files.<\/p>\n\n\n\n

Unstructured data:<\/strong> Data which does not have a pre-defined data model. Ex: Text files, web logs.<\/p>\n\n\n\n

Learn more about Structured, Unstructured, and Semi-Structured Data<\/a><\/p>\n\n\n\n

Let us look at the 6 V's of Big Data<\/strong><\/h2>\n\n\n\n

Volume:<\/strong> The amount of data from various sources like in TB, PB, ZB etc. It is a rise of bytes we are nowhere in GBs now.<\/p>\n\n\n\n

Velocity:<\/strong> High frequency data like in stocks. The speed at which big data is generated.<\/p>\n\n\n\n

Veracity:<\/strong> Refers to the biases, noises and abnormality in data.<\/p>\n\n\n\n

Variety:<\/strong> Refers to the different forms of data. Data can come in various forms and shapes, like visuals data like pictures, and videos, log data etc. This can be the biggest problem to handle for most businesses.<\/p>\n\n\n\n

Variability:<\/strong> to what extent, and how fast, is the structure of your data changing? And how often does the meaning or shape of your data change?<\/p>\n\n\n\n

Value:<\/strong> This describes what value you can get from which data, how big data will get better results from stored data.<\/p>\n\n\n\n

Challenges in Big data<\/strong><\/h2>\n\n\n\n

Complex:<\/strong> No proper understanding of the underlying data<\/p>\n\n\n\n

Storage:<\/strong> How to accommodate large amounts of data in a single physical machine.<\/p>\n\n\n\n

Performance:<\/strong> How to process large amounts of data efficiently and effectively so as to increase the performance.<\/p>\n\n\n\n

Big Data Technologies<\/strong><\/h2>\n\n\n\n

Big Data is broad and surrounded by many trends and new technology developments, the top emerging technologies given below are helping users cope with and handle Big Data in a cost-effective manner.<\/p>\n\n\n\n

    \n
  1. Apache Hadoop<\/li>\n\n\n\n
  2. Apache Spark<\/li>\n\n\n\n
  3. Apache Hive<\/li>\n<\/ol>\n\n\n\n

    There are many other technologies. But we will learn about the above 3 technologies in detail.<\/p>\n\n\n\n

    Also Read: Introduction to JavaScript <\/a><\/p>\n\n\n\n

    Hadoop Tutorial Introduction<\/strong><\/h2>\n\n\n\n

    Hadoop is a distributed parallel processing framework, which facilitates distributed computing.<\/p>\n\n\n\n

    Now to dig more on Hadoop Tutorial, we need to have understanding on \u201cDistributed Computing\u201d. This will actually give us a root cause of the Hadoop and understand this Hadoop Tutorial. To learn more, you can also take up Free Hadoop Courses<\/a> learn gain comprehensive knowledge about this in-demand skill. <\/p>\n\n\n\n

    Distributed Computing<\/strong><\/h2>\n\n\n\n

    In simple English, distributed computing is also called parallel processing. Let's take an example, let's say we have a task of painting a room in our house, and we will hire a painter to paint and may approximately take 2 hours to paint one surface. Let's say we have 4 walls and 1 ceiling to be painted and this may take one day(~10 hours) for one man to finish, if he does this non stop.<\/p>\n\n\n\n

    The same thing to be done by 4 or 5 more people can take half a day to finish the same task. This is the simple real time problem to understand the logic behind distributed computing<\/p>\n\n\n\n

    Now let's take an actual data related problem and analyse the same.<\/p>\n\n\n\n

    We have an input file of lets say 1 GB and we need to calculate the sum of these numbers together and the operation may take 50secs to produce a sum of numbers<\/p>\n\n\n\n

    Then let's take the same example by dividing the dataset into 2 parts and give the input to 2 different machines, then the operation may take 25 secs to produce the same sum results.<\/p>\n\n\n\n

    This is the fundamental idea of parallel processing.<\/p>\n\n\n\n

    Why Hadoop?<\/strong><\/h2>\n\n\n\n

    The idea of parallel processing was not something new!<\/p>\n\n\n\n

    The  idea ws existing since long back in the time of Super computers (back in 1970s)<\/p>\n\n\n\n

    There we used to have army of network engineers and cables required in manufacturing supercomputers and there are still few research organizations which use these kind of infrastructures which is called as \u201csuper Computers\u201d.<\/p>\n\n\n\n

    Lets see what were the challenges of SuperComputing<\/strong><\/h4>\n\n\n\n