{"id":16749,"date":"2020-07-29T23:53:02","date_gmt":"2020-07-29T18:23:02","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/anomaly-detection-using-machine-learning\/"},"modified":"2024-09-03T17:26:25","modified_gmt":"2024-09-03T11:56:25","slug":"anomaly-detection-using-machine-learning","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/anomaly-detection-using-machine-learning\/","title":{"rendered":"Anomaly Detection using Machine Learning | How Machine Learning Can Enable Anomaly Detection?"},"content":{"rendered":"\n

Anomaly detection is something similar to how our human brains are always trying to recognize something abnormal or out of the \u201cnormal\u201d or the \u201cusual stuff.\u201d Anomaly is basically  that does not fit with the usual pattern. Abnormal growth of data or some pattern which is not similar with the other part of the data and our data science concepts and tools are also trying to look for anomalies that do not maintain the normal data flow.<\/p>\n\n\n\n

So to understand more intuitively let me give you an example like an \u201cunusually high\u201d number of login attempts to some particular account may point to a potential cyberattack, or a huge amount of transaction with credit card can be a reason for fraud transaction. Also, check put this Fake news detection using machine learning<\/a> course today.<\/p>\n\n\n\n

So In this blog we will discuss about:<\/p>\n\n\n\n

    \n
  1. What is Anomaly detection? <\/a><\/strong><\/li>\n\n\n\n
  2. Why do we need Anomaly detection?<\/strong><\/a><\/li>\n\n\n\n
  3. Types of Anomaly detection<\/strong><\/a><\/li>\n\n\n\n
  4. Benefits and applications of Anomaly detection <\/strong><\/a><\/li>\n\n\n\n
  5. Machine learning methods to do anomaly detection<\/strong><\/a><\/li>\n\n\n\n
  6. Machine Learning and Outlier Analysis<\/strong><\/a><\/li>\n\n\n\n
  7. Project with anomaly detection<\/a><\/strong><\/li>\n<\/ol>\n\n\n\n

    What is Anomaly Detection?<\/h2>\n\n\n\n

    Anomaly Detection is basically a technique to identify rare events or observations. These events or the observations  can be a cause of  suspicious activity. Because these observations are statistically different from the rest of the observations. Examples of such activities are known as anomaly in the dataset and these behaviours are known as   \u201canomalous\u201d behaviour and  typically the reason for some kind of a problem like a credit card fraud, failing machine in a server, a cyber attack, or some other kind of serious issues.<\/p>\n\n\n\n

    Why do we need Anomaly Detection?<\/h2>\n\n\n\n

    We all know that Machine Learning has four parts or classes of applications: <\/p>\n\n\n\n

      \n
    1. classification, <\/li>\n\n\n\n
    2. predicting the next upcoming value, <\/li>\n\n\n\n
    3. anomaly detection, <\/li>\n\n\n\n
    4. and discovering structure.<\/li>\n<\/ol>\n\n\n\n

      From these four classes Anomaly detection helps to detect data points from the dataset  that do not fit well or not behaving normally with the rest of the data. The applications for this particular class are fraud detection, surveillance, diagnosis, data cleanup, and predictive maintenance etc.<\/p>\n\n\n\n

      The advent of IoT and  anomaly detection is now playing a  key role in IoT applications as well  such as monitoring and predictive maintenance.<\/p>\n\n\n\n

      Modern businesses are now becoming dependent on the data and they try to forecast their sales with the base of these new technologies and they have started understanding  the importance of interconnected operations to get the full schema of their business.At the same time they need to respond and take actions  to fast-moving changes in data promptly. Especially if we talk about the cybersecurity threats and rapid changes in their domain. To solve these heart throbbing problems anomaly detection can be a key or a bright side  for solving such intrusions. And unfortunately, we have no other effective way to handle and analyze constantly growing datasets manually. In constantly changing data \u201cnormal\u201d behavior is redefining each and every moment, and a new effective, proactive approach to identify anomalous behaviour can be find using these concepts.<\/p>\n\n\n\n

      So to get and use all the features anomaly detection is needed to identify the anomaly of the data.<\/p>\n\n\n\n

      Types of Anomaly detection<\/h2>\n\n\n\n

      These anomalies can be divided into three categories:<\/p>\n\n\n\n

        \n
      1. Point Anomaly: An individual data instance in a dataset is considered to be a Point Anomaly if it belongs to far away from the rest of the data.<\/li>\n<\/ol>\n\n\n\n

        Example: Sudden transaction of huge amount from a credit card<\/p>\n\n\n\n

          \n
        1. Contextual Anomaly: If a particular observation is anomalous from other data points then this is called a Contextual Anomaly. Or if the data is  an anomaly because of the context of the observation.<\/li>\n\n\n\n
        2. Collective Anomaly: Collective Anomaly are known as a set of data instances that help in finding an anomaly. This type of anomaly  has two variations.<\/li>\n<\/ol>\n\n\n\n
            \n
          1. Events are in unexpected order <\/li>\n\n\n\n
          2. Unexpected value combinations <\/li>\n<\/ol>\n\n\n\n

            Benefits and application of anomaly detection:<\/h2>\n\n\n\n

            Artificial Intelligence helps our human resources to handle the elastic environment of cloud infrastructure, microservices and containers. We use artificial intelligence concepts everywhere to overcome these challenges.<\/p>\n\n\n\n

            So let me give you some examples of anomaly detection,<\/p>\n\n\n\n

            Suppose in the automation Industry they start using AI-driven anomaly detection algorithms that can automatically analyze and understand the datasets. These algorithms are too efficient for that they can dynamically fine-tune the parameters of normal behavior and identify breaches in the patterns.<\/p>\n\n\n\n

            Real-time analysis: In the real time analysis industry also AI solutions can interpret data activity in real time. Basically here AI tries to check the pattern and compare them. At any moment if a pattern is not recognized by the system, then the system will send a signal as soon as possible.<\/p>\n\n\n\n

            Scrupulousness: In this application of anomaly detection basically helps to provide an end-to-end gap-free monitoring which can go through minutiae of data. By this application we can identify the smallest anomalies in the data which is almost impossible for a human eye  to find out.<\/p>\n\n\n\n

            Accuracy:When it comes to comparing the accuracy between AI and the human resource we always notice that Artificial Intelligence is far better to deal with anomaly detection.  Artificial Intelligence helps to enhance the accuracy of anomaly detection with avoiding nuisance alerts and false positives\/negatives triggered by static thresholds. <\/p>\n\n\n\n

            Self-learning: We all have the idea about self driving  cars and Tesla Self driving cars are also very famous so this industry is also using Artificial Intelligence concepts. So the heart of this industry is based on AI-driven algorithms which constitute the core of self-learning systems. These systems are able to learn from data patterns and deliver predictions or answers as required.<\/p>\n\n\n\n

            \n
            \n
            \n
            <\/div>\n Advance Your Data Science Career<\/span>\n <\/div>\n

            \n Post Graduate Program in Data Science<\/a>\n <\/p>\n

            Master data science skills with a focus on in-demand Gen AI through real-world projects and expert-led learning.<\/p>\n

            \n
            \n
            <\/div>\n Duration: 12 months<\/span>\n <\/div>\n
            \n
            <\/div>\n Ratings: 4.78<\/span>\n <\/div>\n <\/div>\n \n Explore the Course\n
            <\/div>\n <\/a>\n <\/div>\n <\/div>\n\n\n\n

            Machine learning methods to do anomaly detection:<\/h2>\n\n\n\n

            What is Machine Learning?<\/p>\n\n\n\n

            Machine learning<\/a> is a sub-set of artificial intelligence (AI) that allows the system to automatically learn and improve from experience without being explicitly programmed<\/p>\n\n\n\n

            Three types are there in machine learning:<\/p>\n\n\n\n

              \n
            1. Supervised<\/li>\n\n\n\n
            2. Unsupervised<\/li>\n\n\n\n
            3. Reinforcement learning<\/li>\n<\/ol>\n\n\n\n

              What is supervised learning?<\/p>\n\n\n\n

              From the name itself, we can understand supervised learning works as a supervisor or teacher. Basically, In supervised learning, we teach or train the machine with labeled data (that means data is already tagged with some predefined class). Then we test our model with some unknown new set of data and predict the level for them.<\/p>\n\n\n\n

              What is unsupervised learning?<\/p>\n\n\n\n

              Unsupervised learning is a machine learning technique, where you do not need to supervise the model. Instead, you need to allow the model to work on its own to discover information. It mainly deals with the unlabeled data.<\/p>\n\n\n\n

              What is Reinforcement Learning?<\/p>\n\n\n\n

              Reinforcement learning is about taking suitable action to maximize reward in a particular situation. It is used to define the best sequence of decisions that allow the agent to solve a problem while maximizing a long-term reward.<\/p>\n\n\n\n

              \n
              \n
              \n
              <\/div>\n Texas McCombs, UT Austin<\/span>\n <\/div>\n

              \n Post Graduate Program in AI & Machine Learning: Business Applications<\/a>\n <\/p>\n

              Master in-demand AI and machine learning skills with this executive-level AI course\u2014designed to transform professionals into strategic tech leaders.<\/p>\n

              \n
              \n
              <\/div>\n Duration: 7 months<\/span>\n <\/div>\n
              \n
              <\/div>\n 4.72\/5 Rating<\/span>\n <\/div>\n <\/div>\n \n Take your First Step\n
              <\/div>\n <\/a>\n <\/div>\n <\/div>\n\n\n\n

              Machine Learning and Outlier Analysis<\/h2>\n\n\n\n

              What is an outlier?<\/p>\n\n\n\n

              An outlier in the data basically stands for identifying any  data object or point that significantly deviates from the remaining data points. In the concept of data mining, outliers are commonly considered as an exception or simply noise to the data. But the same process cannot be applied in anomaly detection, hence the emphasis on outlier analysis.<\/p>\n\n\n\n

              Let me give you an example about performing anomaly detection using machine learning. This method is the K-means clustering method.<\/p>\n\n\n\n

              What is K means clustering?<\/p>\n\n\n\n

               K-means clustering is a part of unsupervised learning in the machine learning algorithm. This algorithm is used for the unlabeled data (i.e.,The  data without defined categories, class, or groups). The goal of this K-means algorithm is to find the specific groups in the data, with the number of groups represented by the variable K. The algorithm normally works iteratively to put  and classify each data point to one of K groups based on the features the data points have. These  data points are clustered based on feature similarity. <\/p>\n\n\n\n

              The results we got from the  K-means clustering algorithm are:<\/p>\n\n\n\n

                \n
              1.  The centroids of the K clusters, which are used to label new data <\/li>\n\n\n\n
              2. Labels for the each data points in the training data (each data point is assigned to a single cluster)<\/li>\n<\/ol>\n\n\n\n

                So this algorithm K-means clustering method is applied to detect the outlier based on their plotted distance from the closest cluster.<\/p>\n\n\n\n

                K-means clustering method helps the form the different types of  clusters for each  data point in the dataset with a mean value. Normally we observe that objects within a cluster have the closest mean value. Any object if it has the threshold value greater than the nearest cluster mean value then that is identified as an outlier. <\/p>\n\n\n\n

                Step-by-step method implementation to use K-means clustering:<\/p>\n\n\n\n

                  \n
                1. First calculate the mean value of each cluster<\/li>\n\n\n\n
                2. Then try to set an initial threshold value<\/li>\n\n\n\n
                3. When the testing process is going in that time try to  determine the distance of each data point from the mean value<\/li>\n\n\n\n
                4. Find or Identify the cluster which  is nearest to the test data point<\/li>\n\n\n\n
                5. If we observe that the \u201cDistance\u201d value is more than the \u201cthreshold\u201d value, then we can conclude that it is an outlier.<\/li>\n<\/ol>\n\n\n\n
                    \n
                  1. Supervised Anomaly Detection:<\/li>\n<\/ol>\n\n\n\n

                    As we are already familiar with that, the  supervised learning method needs a labeled dataset. This dataset can  contain both the  normal and the anomalous samples to construct a predictive model. This dataset helps us  to classify future data points. The most commonly and famous  used algorithms for this purpose are supervised learning processes like, Support Vector Machine learning<\/a>, Machine learning modelling<\/a>, K-Nearest Neighbors Classifier, etc.<\/p>\n\n\n\n

                      \n
                    1. Unsupervised Anomaly Detection:<\/li>\n<\/ol>\n\n\n\n

                      So in the beginning of this tutorial we have learned about unsupervised learning as well. <\/p>\n\n\n\n

                      Unsupervised anomaly detection algorithms are divided into some parts like: <\/p>\n\n\n\n

                      (1) Nearest-neighbor based techniques,<\/p>\n\n\n\n

                       (2) Clustering-based methods and<\/p>\n\n\n\n

                       (3) Statistical algorithms.<\/p>\n\n\n\n

                      This learning process does not require any training data and instead of that it automatically assumes two things about the data:<\/p>\n\n\n\n

                       A very small percentage of data is anomalous in the whole dataset<\/p>\n\n\n\n

                      And the second one is  that any anomaly is statistically different from the normal samples. <\/p>\n\n\n\n

                      Based on these two assumptions, the data is then process to cluster using the similarity measurement and the data points in the dataset which are far away from the cluster they are considered as the  anomalies.<\/p>\n\n\n\n

                      Project with anomaly detection:<\/h2>\n\n\n\n

                      Credit card Fraud Analysis:<\/p>\n\n\n\n

                      Project: Credit card Fraud Analysis using Data mining techniques<\/strong><\/h5>\n\n\n\n
                      \"\"<\/figure>\n\n\n\n

                      IN today\u2019s world, we are literally sitting on the express train to become a cashless society. As per the World Payments Report<\/a>, in 2016 total non-cash transactions increased by 10.1% from 2015 for a total of 482.6 billion transactions! That\u2019s huge! Also, it\u2019s expected that in future years there will be a steady growth of non-cash transactions.<\/p>\n\n\n\n

                      As this is a blessing on the other hand it becomes a curse for this cashless society because of the immense number of fraud transactions even if EMV smart chips are also implemented.<\/p>\n\n\n\n

                      So our data scientists are trying to come up with one of the best solutions to make a model for predicting fraud transactions.<\/p>\n\n\n\n

                      Import the libraries:<\/h5>\n\n\n\n
                      import pandas as pd \n\nimport numpy as np\n\nimport matplotlib\n\nimport matplotlib.pyplot as plt\n\n%matplotlib inline \n\nimport matplotlib.pyplot as plt\n\nfrom sklearn.model_selection import train_test_split \n\nfrom sklearn.metrics import classification_report, accuracy_score  \n\nfrom sklearn.metrics import confusion_matrix \n\nfrom sklearn.linear_model import LogisticRegression\n\nfrom sklearn import metrics\n\nimport seaborn as sns<\/pre>\n\n\n\n
                      Collect the Data<\/strong><\/h5>\n\n\n\n
                      I collected the data from Kaggle dataset<\/a>.<\/p>\n\n\n\n
                      Dataset description:<\/strong> <\/p>\n\n\n\n
                        \n
                      1. It contains 285,000 rows of data and 31 columns.<\/li>\n\n\n\n
                      2.  The most important columns are<\/li>\n<\/ol>\n\n\n\n