{"id":6057,"date":"2023-01-10T12:06:03","date_gmt":"2023-01-10T06:36:03","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/linear-regression-in-machine-learning\/"},"modified":"2024-09-02T15:21:57","modified_gmt":"2024-09-02T09:51:57","slug":"linear-regression-in-machine-learning","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/linear-regression-in-machine-learning\/","title":{"rendered":"A Guide to Linear Regression in Machine Learning"},"content":{"rendered":"\n

What is Linear Regression?<\/strong><\/h2>\n\n\n\n

Linear Regression is the basic form of regression analysis. It assumes that there is a linear relationship between the dependent variable and the predictor(s). In regression, we try to calculate the best fit line, which describes the relationship between the predictors and predictive\/dependent variables.<\/p>\n\n\n\n

There are four assumptions associated with a linear regression model:<\/p>\n\n\n\n

    \n
  1. Linearity<\/strong>: The relationship between independent variables and the mean of the dependent variable is linear. <\/li>\n\n\n\n
  2. Homoscedasticity<\/strong>: The variance of residuals should be equal.<\/li>\n\n\n\n
  3. Independence<\/strong>: Observations are independent of each other.<\/li>\n\n\n\n
  4. Normality<\/strong>: The dependent variable is normally distributed for any fixed value of an independent variable.<\/li>\n<\/ol>\n\n\n\n

    Isn\u2019t Linear Regression from Statistics?<\/b><\/h2>\n\n\n\n

    Before we dive into the details of linear regression, you may be asking yourself why we are looking at this algorithm.<\/span><\/p>\n\n\n\n

    Isn\u2019t it a technique from statistics? Machine learning<\/a>, more specifically the field of predictive modeling<\/a>,<\/span> is primarily concerned with minimizing the error of a model or making the most accurate predictions possible at the expense of explainability. In applied machine learning, we will borrow and reuse algorithms from many different fields, including statistics and use them towards these ends.<\/span><\/p>\n\n\n\n

    As such, linear regression was developed in the field of statistics and is studied as a model for understanding the relationship between input and output numerical variables. However, it has been borrowed by machine learning, and it is both a statistical algorithm and a machine learning algorithm.<\/span><\/p>\n\n\n\n

    Linear Regression Model Representation<\/b><\/h2>\n\n\n\n

    Linear regression is an attractive model because the representation is so simple.<\/span>
    The representation is a linear equation that combines a specific set of input values (x), the solution to which is the predicted output for that set of input values (y). As such, both the input values (x) and the output value are numeric.<\/span><\/p>\n\n\n\n

    The linear equation assigns one scale factor to each input value or column, called a coefficient and represented by the capital Greek letter Beta (B). One additional coefficient is <\/span>added, giving the line an additional degree of freedom (e.g., moving up and down on a two-dimensional plot) and is often called the intercept or the bias coefficient.<\/span><\/p>\n\n\n\n

    For example, in a simple regression problem (a single x and a single y), the form of the model would be:<\/span>
    Y= \u03b20 + \u03b21x<\/span><\/p>\n\n\n\n

    In higher dimensions, the line is called a plane or a hype<\/span>r-plane when we have more than one input (x). The representation, therefore, is in the form of the equation and the specific values used for the coefficients (e.g., \u03b20and \u03b21 in the above example).<\/span><\/p>\n\n\n\n

    Performance of Regression<\/strong><\/h2>\n\n\n\n

    The regression model's performance can be evaluated using various metrics like MAE, MAPE, RMSE, R-squared, etc.<\/p>\n\n\n\n

    Mean Absolute Error (MAE)<\/h3>\n\n\n\n

    By using MAE, we calculate the average absolute difference between the actual values and the predicted values. <\/p>\n\n\n\n

    Mean Absolute Percentage Error (MAPE) <\/h3>\n\n\n\n

    MAPE is defined as the average of the absolute deviation of the predicted value from the actual value. It is the average of the ratio of the absolute difference between actual & predicted values and actual values. <\/p>\n\n\n\n

    <\/p>\n\n\n\n

    Root Mean Square Error (RMSE)<\/h3>\n\n\n\n

    RMSE calculates the square root average of the sum of the squared difference between the actual and the predicted values.<\/p>\n\n\n\n

    <\/p>\n\n\n\n

    R-squared values<\/h3>\n\n\n\n

    R-square value depicts the percentage of the variation in the dependent variable explained by the independent variable in the model. <\/p>\n\n\n\n

    <\/p>\n\n\n\n

    RSS = Residual sum of squares<\/strong>: It measures the difference between the expected and the actual output. A small RSS indicates a tight fit of the model to the data. It is also defined as follows: <\/p>\n\n\n\n

    <\/p>\n\n\n\n

    TSS = Total sum of squares<\/strong>: It is the sum of data points' errors from the response variable's mean. <\/p>\n\n\n\n

    <\/p>\n\n\n\n

    R2<\/sup> value ranges from 0 to 1. The higher the R-square value better the model. The value of R2 increases if we add more variables to the model, irrespective of whether the variable contributes to the model or not. This is the disadvantage of using R2<\/sup>.<\/p>\n\n\n\n

    Adjusted R-squared values<\/h3>\n\n\n\n

    The Adjusted R2 value fixes the disadvantage of R2. The adjusted R2 value will improve only if the added variable contributes significantly to the model, and the adjusted R2<\/sup> value adds a penalty to the model.<\/p>\n\n\n\n

    <\/p>\n\n\n\n

    where R2<\/sup> is the R-square value, n = the total number of observations, and k = the total number of variables used in the model, if we increase the number of variables, the denominator becomes smaller, and the overall ratio will be high. Subtracting from 1 will reduce the overall Adjusted R2<\/sup>. So to increase the Adjusted R2<\/sup>, the contribution of additive features to the model should be significantly high.<\/p>\n\n\n\n

    Simple Linear Regression Example<\/strong><\/h3>\n\n\n\n

    For the given equation for the Linear Regression,<\/p>\n\n\n\n

    <\/p>\n\n\n\n

    If there is only 1 predictor available, then it is known as Simple Linear Regression. <\/p>\n\n\n\n

    <\/p>\n\n\n\n

    While executing the prediction, there is an error term that is associated with the equation.<\/p>\n\n\n\n

    <\/p>\n\n\n\n

    The SLR model aims to find the estimated values of \u03b21 <\/sub>& \u03b20<\/sub> by keeping the error term (\u03b5) minimum.<\/p>\n\n\n\n

    Multiple Linear Regression Example<\/strong><\/h3>\n\n\n\n

    Contributed by: Rakesh Lakalla
    LinkedIn profile:     https:\/\/www.linkedin.com\/in\/lakkalarakesh\/ <\/a><\/em><\/strong><\/p>\n\n\n\n

    For the given equation of Linear Regression, <\/p>\n\n\n

    \"\"<\/figure>\n\n\n\n

    if there is more than 1 predictor available, then it is known as Multiple Linear Regression. <\/p>\n\n\n\n

    The equation for MLR will be:<\/p>\n\n\n

    \"\"<\/figure>\n\n\n\n

    \u03b21<\/sub> = coefficient for X1<\/sub> variable<\/p>\n\n\n\n

    \u03b22<\/sub> = coefficient for X2<\/sub> variable<\/p>\n\n\n\n

    \u03b23<\/sub> = coefficient for X3<\/sub> variable and so on\u2026<\/p>\n\n\n\n

    \u03b20<\/sub> is the intercept (constant term). While making the prediction, there is an error term that is associated with the equation.<\/p>\n\n\n

    \"\"<\/figure>\n\n\n\n

    The goal of the MLR model is to find the estimated values of \u03b20, <\/sub>\u03b21, <\/sub>\u03b22,<\/sub> \u03b23\u2026<\/sub> by keeping the error term (i) minimum.<\/p>\n\n\n\n

    Broadly speaking, supervised machine learning algorithms are classified into two types-<\/p>\n\n\n\n

      \n
    1. Regression: Used to predict a continuous variable<\/li>\n\n\n\n
    2. Classification: Used to predict discrete variable <\/li>\n<\/ol>\n\n\n\n

      In this post, we will discuss one of the regression techniques, \u201cMultiple Linear Regression,\u201d and its implementation using Python.<\/p>\n\n\n\n

      Linear regression is one of the statistical methods of predictive analytics to predict the target variable (dependent variable). When we have one independent variable, we call it Simple Linear Regression. If the number of independent variables is more than one, we call it Multiple Linear Regression.<\/p>\n\n\n\n

      Assumptions for Multiple Linear Regression<\/strong><\/h4>\n\n\n\n
        \n
      1. Linearity: <\/strong>There should be a linear relationship between dependent and independent variables, as shown in the below example graph.<\/li>\n<\/ol>\n\n\n\n

        2. Multicollinearity: <\/strong>There should not be a high correlation between two or more independent variables. Multicollinearity can be checked using a correlation matrix, Tolerance and Variance Influencing Factor (VIF).<\/p>\n\n\n\n

        3. Homoscedasticity: <\/strong>If Variance of errors is constant across independent variables, then it is called Homoscedasticity. The residuals should be homoscedastic. Standardized residuals versus predicted values are used to check homoscedasticity, as shown in the below figure. Breusch-Pagan and White tests are the famous tests used to check Homoscedasticity. Q-Q plots are also used to check homoscedasticity.<\/p>\n\n\n\n

        4. Multivariate Normality: <\/strong>Residuals should be normally distributed.<\/p>\n\n\n\n

        5. Categorical Data: <\/strong>Any categorical data present should be converted into dummy variables.<\/p>\n\n\n\n

        6. Minimum records: <\/strong>There should be at least 20 records of independent variables.<\/p>\n\n\n\n

        A mathematical formulation of Multiple Linear Regression<\/strong><\/h4>\n\n\n\n

        In Linear Regression, we try to find a linear relationship between independent and dependent variables by using a linear equation on the data.<\/p>\n\n\n\n

        The equation for a linear line is-<\/p>\n\n\n\n

        \t<\/strong>Y=mx + c<\/strong><\/p>\n\n\n\n

        Where m is slope and c is the intercept.<\/p>\n\n\n\n

        In Linear Regression, we are actually trying to predict the best m and c values for dependent variable Y and independent variable x. We fit as many lines and take the best line that gives the least possible error. We use the corresponding m and c values to predict the y value.<\/p>\n\n\n\n

        The same concept can be used in multiple Linear Regression where we have multiple independent variables, x1, x2, x3\u2026xn.<\/p>\n\n\n\n

        Now the equation changes to- <\/p>\n\n\n\n

        Y=M1X1 + M2X2 + M3M3 + \u2026MnXn+C<\/strong><\/p>\n\n\n\n

        The above equation is not a line but a plane of multi-dimensions.<\/p>\n\n\n\n

        Model Evaluation:<\/strong><\/p>\n\n\n\n

        A model can be evaluated by using the below methods-<\/p>\n\n\n\n

          \n
        1. Mean absolute error: <\/strong>It is the mean of absolute values of the errors, formulated as- <\/li>\n<\/ol>\n\n\n\n
            \n
          1. Mean squared error: <\/strong>It is the mean of the square of errors.<\/li>\n<\/ol>\n\n\n\n
              \n
            1. Root mean squared error: <\/strong>It is just the square root of MSE.<\/li>\n<\/ol>\n\n\n\n

              Applications<\/strong><\/p>\n\n\n\n

                \n
              1. The effect of the independent variable on the dependent variable can be calculated.<\/li>\n\n\n\n
              2. Used to predict trends.<\/li>\n\n\n\n
              3. Used to find how much change can be expected in a dependent variable with change in an independent variable.<\/li>\n<\/ol>\n\n\n\n

                Polynomial Regression<\/strong><\/h2>\n\n\n\n

                Polynomial regression is a non-linear regression. In Polynomial regression, the relationship of the dependent variable is fitted to the nth degree of the independent variable. <\/p>\n\n\n\n

                Equation of polynomial regression: <\/p>\n\n\n

                \"\"<\/figure>\n\n\n\n

                Underfitting and Overfitting<\/strong><\/h2>\n\n\n\n

                When we fit a model, we try to find the optimized, best-fit line, which can describe the impact of the change in the independent variable on the change in the dependent variable by keeping the error term minimum. While fitting the model, there can be 2 events that will lead to the bad performance of the model. These events are<\/p>\n\n\n\n

                  \n
                1. Underfitting <\/a><\/li>\n\n\n\n
                2. Overfitting<\/a><\/li>\n<\/ol>\n\n\n\n

                  Underfitting <\/strong><\/h3>\n\n\n\n

                  Underfitting is the condition where the model cannot fit the data well enough. The under-fitted model leads to low accuracy of the model. Therefore, the model is unable to capture the relationship, trend, or pattern in the training data. Underfitting of the model could be avoided by using more data or by optimizing the parameters of the model.<\/p>\n\n\n\n

                  Overfitting<\/strong><\/h3>\n\n\n\n

                  Overfitting is the opposite case of underfitting, i.e., when the model predicts very well on training data and is not able to predict well on test data or validation data. The main reason for overfitting could be that the model is memorizing the training data and is unable to generalize it on a test\/unseen dataset. Overfitting can be reduced by making feature selection or by using regularisation techniques. <\/p>\n\n\n\n

                  <\/p>\n\n\n\n

                  The above graphs depict the 3 cases of the model performance. <\/p>\n\n\n\n

                  Implementing Linear Regression in Python<\/strong><\/h2>\n\n\n\n

                  Contributed by: Ms. Manorama Yadav
                  LinkedIn:                   https:\/\/www.linkedin.com\/in\/manorama-3110\/<\/a> <\/em><\/strong><\/p>\n\n\n\n

                  Dataset Introduction<\/h3>\n\n\n\n

                  The data concerns city-cycle fuel consumption in miles per gallon(mpg) to be predicted. There are a total of 392 rows, 5 independent variables, and 1 dependent variable. All 5 predictors are continuous variables.<\/p>\n\n\n\n

                   Attribute Information:<\/strong><\/h4>\n\n\n\n
                    \n
                  1. mpg:                   continuous (Dependent Variable<\/strong>)<\/li>\n\n\n\n
                  2. cylinders:           multi-valued discrete<\/li>\n\n\n\n
                  3. displacement:   Continuous<\/li>\n\n\n\n
                  4. horsepower:      continuous<\/li>\n\n\n\n
                  5. weight:               Continuous<\/li>\n\n\n\n
                  6. acceleration:     Continuous<\/li>\n<\/ol>\n\n\n\n

                    The objective of the problem statement is to predict the miles per gallon using the Linear Regression model.<\/strong><\/p>\n\n\n\n

                    Python Packages for Linear Regression<\/strong><\/h3>\n\n\n\n

                    Import the necessary Python package<\/a> to perform various steps like data reading, plotting the data, and performing linear regression. Import the following packages:<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    Read the data<\/h3>\n\n\n\n
                    \"\"\/<\/figure>\n\n\n\n

                    Download the data and save it in the data directory of the project folder. <\/p>\n\n\n\n

                    Simple Linear Regression With scikit-learn<\/strong><\/h3>\n\n\n\n

                    Simple Linear regression has only 1 predictor variable and 1 dependent variable. From the above dataset, let\u2019s consider the effect of horsepower on the \u2018mpg\u2019 of the vehicle.<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    Let\u2019s take a look at what the data looks like:<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    <\/p>\n\n\n\n

                    From the above graph, we can infer a negative linear relationship between horsepower and miles per gallon (mpg). With horsepower increasing, mpg is decreasing.<\/p>\n\n\n\n

                    Now, let\u2019s perform the Simple linear regression. <\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    From the output of the above SLR model, the equation of the best fit line of the model is <\/p>\n\n\n\n

                    mpg = 39.94 + (-0.16)*(horsepower)<\/strong><\/p>\n\n\n\n

                    By comparing the above equation to the SLR model equation Yi= \u03b2iXi + \u03b20 , \u03b20=39.94, \u03b21=-0.16<\/span><\/p>\n\n\n\n

                    Now, check for the model relevancy by looking at its R2<\/sup> and RMSE Values<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    R2<\/sup> and RMSE (Root mean square) values are 0.6059 <\/span>and 4.89,<\/span> respectively. It means that 60% of the variance in mpg is explained by horsepower. For a simple linear regression model, this result is okay but not so good since there could be an effect of other variables like cylinders, acceleration, etc. RMSE value is also very less. <\/p>\n\n\n\n

                    Let\u2019s check how the line fits the data.<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    <\/p>\n\n\n\n

                    From the graph, we can infer that the best fit line is able to explain the effect of horsepower on mpg.<\/p>\n\n\n\n

                    Multiple Linear Regression With scikit-learn<\/strong><\/h3>\n\n\n\n

                    Since the data is already loaded in the system, we will start performing multiple linear regression.<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    The actual data has 5 independent variables and 1 dependent variable (mpg)<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    The best fit line for Multiple Linear Regression is <\/p>\n\n\n\n

                    Y = 46.26 + -0.4cylinders + -8.313e-05displacement + -0.045horsepower + -0.01weight + -0.03acceleration<\/strong><\/p>\n\n\n\n

                    By comparing the best fit line equation with<\/p>\n\n\n

                    \"\"<\/figure>\n\n\n\n

                    \u03b20 (Intercept)= 46.25, \u03b21 = -0.4, \u03b22 = -8.313e-05, \u03b23= -0.045, \u03b24= 0.01, \u03b25 = -0.03<\/span><\/p>\n\n\n\n

                    Now, let\u2019s check the R2<\/sup> and RMSE values.<\/p>\n\n\n\n

                    \"\"\/<\/figure>\n\n\n\n

                    R2<\/sup> and RMSE (Root mean square) values are 0.707<\/span> and 4.21,<\/span> respectively. It means that ~71% of the variance in mpg is explained by all the predictors. This depicts a good model. Both values are less than the results of Simple Linear Regression, which means that adding more variables to the model will help in good model performance. However, the more the value of R2<\/sup> and the least RMSE, the better the model will be.<\/p>\n\n\n\n

                    Multiple Linear Regression- Implementation using Python<\/strong><\/h4>\n\n\n\n

                    Let us take a small data set and try out a building model using python.<\/p>\n\n\n\n

                    import pandas as pd\nimport numpy as np\nimport seaborn as sns\nimport matplotlib.pyplot as plt\n%matplotlib inline\nfrom sklearn.model_selection import train_test_split \nfrom sklearn.linear_model import LinearRegression\nfrom sklearn import metrics\n<\/code><\/pre>\n\n\n\n
                    \ndata=pd.read_csv(\"Consumer.csv\")\ndata.head()\n<\/code><\/pre>\n\n\n\n
                    The above figure shows the top 5 rows of the data. We are actually trying to predict the Amount charged (dependent variable) based on the other two independent variables, Income and Household Size. We first check for our assumptions in our data set.<\/p>\n\n\n\n
                      \n
                    1. Check for Linearity<\/strong><\/li>\n<\/ol>\n\n\n\n
                      plt.figure(figsize=(14,5))\nplt.subplot(1,2,1)\nplt.scatter(data['AmountCharged'], data['Income'])\nplt.xlabel('AmountCharged')\nplt.ylabel('Income')\nplt.subplot(1,2,2)\nplt.scatter(data['AmountCharged'], data['HouseholdSize'])\nplt.xlabel('AmountCharged')\nplt.ylabel('HouseholdSize')\nplt.show()\n<\/code><\/pre>\n\n\n\n
                      We can see from the above graph, there exists a linear relationship between the Amount Charged and Income, Household Size. <\/p>\n\n\n\n
                      2. Check for Multicollinearity<\/strong><\/p>\n\n\n\n
                      sns.scatterplot(data['Income'],data['HouseholdSize'])\n<\/code><\/pre>\n\n\n\n
                      There exists no collinearity between Income and HouseholdSize from the above graph.<\/p>\n\n\n\n
                      We split our data to train and test in a ratio of 80:20, respectively, using the function train_test_split<\/strong><\/p>\n\n\n\n
                      X = pd.DataFrame(np.c_[data['Income'], data['HouseholdSize']], columns=['Income','HouseholdSize'])\ny=data['AmountCharged']\nX_train, X_test, y_train, y_test = train_test_split(X, y, test_size = 0.2, random_state=9)\n<\/code><\/pre>\n\n\n\n
                      3. Check for Homoscedasticity<\/strong><\/p>\n\n\n\n
                      First, we need to calculate residuals-<\/p>\n\n\n\n
                      resi=y_test-prediction\n<\/code><\/pre>\n\n\n\n
                      Polynomial Regression With scikit-learn<\/strong><\/h3>\n\n\n\n
                      For Polynomial regression, we will use the same data that we used for Simple Linear Regression. <\/p>\n\n\n\n
                      \"\"\/<\/figure>\n\n\n\n
                      \"\"\/<\/figure>\n\n\n\n
                      <\/p>\n\n\n\n
                      The graph shows that the relationship between horsepower and miles per gallon is not perfectly linear. It\u2019s a little bit curved. <\/p>\n\n\n\n
                      Graph for the Best fit line for Simple Linear Regression as per below:<\/p>\n\n\n\n
                      <\/p>\n\n\n\n
                      From the plot, we can infer that the best fit line is able to explain the effect of the independent variable, however, this does not apply to most of the data points. <\/p>\n\n\n\n
                      Let\u2019s try polynomial regression on the above dataset. Let\u2019s fit degree = 2 <\/p>\n\n\n\n
                      \"\"\/<\/figure>\n\n\n\n
                      Now, visualize the Polynomial Regression results<\/p>\n\n\n\n
                      \"\"\/<\/figure>\n\n\n\n
                      <\/p>\n\n\n\n
                      From the graph, the best fit line looks better than the Simple Linear Regression. <\/p>\n\n\n\n
                      Let\u2019s find out the model performance by calculating mean absolute Error, Mean squared error, and Root mean square.<\/p>\n\n\n\n
                      Simple Linear Regression Model Performance:<\/strong><\/p>\n\n\n\n
                      \"\"\/<\/figure>\n\n\n\n
                      Polynomial Regression (degree = 2) Model Performance:<\/strong><\/p>\n\n\n\n
                      \"\"\/<\/figure>\n\n\n\n
                      From the above results, we can see that Error-values are less in Polynomial regression but there is not much improvement. We can increase the polynomial degree and experiment with the model performance. <\/p>\n\n\n\n
                      Advanced Linear Regression with statsmodels<\/strong><\/h3>\n\n\n\n
                      There are many ways to perform regression in python. <\/p>\n\n\n\n
                        \n
                      1. scikit Learn <\/li>\n\n\n\n
                      2. statsmodels <\/li>\n<\/ol>\n\n\n\n
                        In the MLR in the python section explained above, we have performed MLR using the scikit learn library. Now, let\u2019s perform MLR using the statsmodels library.<\/p>\n\n\n\n
                        Import the below-required libraries<\/p>\n\n\n\n
                        \"\"\/<\/figure>\n\n\n\n
                        Now, perform Multiple Linear Regression using statsmodels<\/p>\n\n\n\n
                        \"\"\/<\/figure>\n\n\n\n
                        From the above results, R2<\/sup> and Adjusted R2<\/sup> are 0.708<\/span> and 0.704,<\/span> respectively. All the independent variables explain almost 71% of the variation in the dependent variables. The value of R2<\/sup> is the same as the result of the scikit learn library. <\/p>\n\n\n\n
                        By looking at the p-value for the independent variables, intercept, horsepower, and weight are important variables since the p-value is less than 0.05 (significance level). We can try to perform MLR by removing other variables which are not contributing to the model and selecting the best model.<\/p>\n\n\n\n
                        Now, let\u2019s check the model performance by calculating the RMSE value:<\/p>\n\n\n\n
                        \"\"\/<\/figure>\n\n\n\n
                        Linear Regression in R<\/strong><\/h2>\n\n\n\n
                        Contributed by:  By Mr. Abhay Poddar <\/em><\/strong><\/p>\n\n\n\n
                        To see an example of Linear Regression in R, we will choose the CARS, which is an inbuilt dataset in R. Typing CARS in the R Console can access the dataset. We can observe that the dataset has 50 observations and 2 variables, namely distance and speed. The objective here is to predict the distance traveled by a car when the speed of the car is known. Also, we need to establish a linear relationship between them with the help of an arithmetic equation. Before getting into modeling, it is always advisable to do an Exploratory Data Analysis, which helps us to understand the data and the variables.<\/p>\n\n\n\n
                        Exploratory Data Analysis<\/strong><\/h3>\n\n\n\n
                        This paper aims to build a Linear Regression Model that can help predict distance. The following are the basic visualizations that will help us understand more about the data and the variables:<\/p>\n\n\n\n
                          \n
                        1. Scatter Plot \u2013 To help establish whether there exists a linear relationship between distance and speed.<\/li>\n\n\n\n
                        2. Box Plot \u2013 To check whether there are any outliers in the dataset.<\/li>\n\n\n\n
                        3. Density Plot \u2013 To check the distribution of the variables; ideally, it should be normally distributed.<\/li>\n<\/ol>\n\n\n\n
                          Below are the steps to make these graphs in R.<\/p>\n\n\n\n
                          Scatter Plots to visualize Relationship<\/strong><\/h4>\n\n\n\n
                          A Scatter Diagram plots the pairs of numerical data with one variable on each axis, and helps establish the relationship between the independent and dependent variables.<\/p>\n\n\n\n
                          Steps in R<\/span><\/h4>\n\n\n\n
                          \"\"\/<\/figure>\n\n\n\n
                          If we carefully observe the scatter plot, we can see that the variables are correlated as they fall along the line\/curve. The higher the correlation, the nearer the points, will be to the line\/curve. <\/p>\n\n\n\n
                          <\/p>\n\n\n\n
                          As discussed earlier, the Scatter Plot shows a linear and positive relationship between Distance and Speed. Thus, it fulfills one of the assumptions of Linear Regression i.e., there should be a positive and linear relationship between dependent and independent variables.<\/p>\n\n\n\n
                          Check for Outliers using Boxplots.<\/strong><\/h4>\n\n\n\n
                          A boxplot is also called a box and whisker plot that is used in statistics to represent the five number summaries. It is used to check whether the distribution is skewed or whether there are any outliers in the dataset.<\/p>\n\n\n\n
                          Wikipedia defines \u2018Outliers\u2019 as an observation point that is distant from other observations in the dataset.<\/p>\n\n\n\n
                          Now, let's plot the Boxplot to check for outliers.
                          <\/p>\n\n\n\n
                          \"\"\/<\/figure>\n\n\n\n
                          <\/p>\n\n\n\n
                          After observing the Boxplots for both Speed and Distance, we can say that there are no outliers in Speed, and there seems to be a single outlier in Distance. Thus, there is no need for the treatment of outliers.<\/p>\n\n\n\n
                          Checking distribution of Data using Density Plots<\/strong><\/h4>\n\n\n\n
                          One of the key assumptions to performing Linear Regression is that the data should be normally distributed. This can be done with the help of Density Plots. A Density Plot helps us visualize the distribution of a numeric variable over a period of time.<\/p>\n\n\n\n
                          <\/p>\n\n\n\n
                          After looking at the Density Plots, we can conclude that the data set is more or less normally distributed.<\/p>\n\n\n\n
                          Linear Regression Modelling<\/strong><\/h3>\n\n\n\n
                          Now, let's get into the building of the Linear Regression Model. But before that, there is one check we need to perform, which is \u2018Correlation Computation\u2019. The Correlation Coefficients help us to check how strong is the relationship between the dependent and independent variables. The value of the Correlation Coefficient ranges from -1 to 1.<\/p>\n\n\n\n
                          A Correlation of 1 indicates a perfect positive relationship. It means if one variable's value increases, the other variable's value also increases.<\/p>\n\n\n\n
                          A Correlation of -1 indicates a perfect negative relationship. It means if the value of variable x increases, the value of variable y decreases.<\/p>\n\n\n\n
                          \"\"\/<\/figure>\n\n\n\n
                          A Correlation of 0 indicates there is no relationship between the variables.
                          <\/p>\n\n\n\n
                          The output of the above R Code is 0.8068949. It shows that the correlation between speed and distance is 0.8, which is close to 1, stating a positive and strong correlation.<\/p>\n\n\n\n
                          The linear regression model in R is built with the help of the lm() function.<\/p>\n\n\n\n
                          The formula uses two main parameters:<\/p>\n\n\n\n
                          Data \u2013 variable containing the dataset.<\/p>\n\n\n\n
                          \"\"\/<\/figure>\n\n\n\n
                          Formula \u2013 an object of the class formula.
                          <\/p>\n\n\n\n
                          The results show us the intercept and beta coefficient of the variable speed.<\/p>\n\n\n\n
                          From the output above,<\/p>\n\n\n\n
                          a) We can write the regression equation as distance = -17.579 + 3.932 (speed).<\/p>\n\n\n\n
                          Model Diagnostics<\/strong><\/h4>\n\n\n\n
                          Just building the model and using it for prediction is the job half done. Before using the model, we need to ensure that the model is statistically significant. This means:<\/p>\n\n\n\n
                            \n
                          1. To check if there is a statistically significant relationship between the dependent and independent variables.<\/li>\n\n\n\n
                          2. The model that we built fits the data very well.<\/li>\n<\/ol>\n\n\n\n
                            \"\"\/<\/figure>\n\n\n\n
                            We do this by a statistical summary of the model using the summary() function in R.<\/p>\n\n\n\n
                            <\/p>\n\n\n\n
                            The summary output shows the following:<\/p>\n\n\n\n
                              \n
                            1. Call \u2013 The function call used to compute the regression model.<\/li>\n\n\n\n
                            2. Residuals \u2013 Distribution of residuals, which generally has a mean of 0. Thus, the median should not be far from 0, and the minimum and maximum should be equal in absolute value.<\/li>\n\n\n\n
                            3. Coefficients \u2013 It shows the regression beta coefficients and their statistical significance.<\/li>\n\n\n\n
                            4. Residual stand effort (RSE), R \u2013 Square, and F \u2013Statistic \u2013 These are the metrics to check how well the model fits our data.<\/li>\n<\/ol>\n\n\n\n
                              Detecting t-statistics and P-Value<\/strong><\/h3>\n\n\n\n
                              T-Statistic and associated p-values are very important metrics while checking model fitment.<\/p>\n\n\n\n
                              The t-statistics tests whether there is a statistically significant relationship between the independent and dependent variables. This means whether the beta coefficient of the independent variable is significantly different from 0. So, the higher the t-value, the better.<\/p>\n\n\n\n
                              Whenever there is a p-value, there is always a null as well as an alternate hypothesis associated with it. The p-value helps us to test for the null hypothesis, i.e., the coefficients are equal to 0. A low p-value means we can reject the null hypothesis.<\/p>\n\n\n\n
                              The statistical hypotheses are as follows:<\/p>\n\n\n\n
                              Null Hypothesis (H0) \u2013 Coefficients are equal to zero.<\/p>\n\n\n\n
                              Alternate Hypothesis (H1) \u2013 Coefficients are not equal to zero.<\/p>\n\n\n\n
                              As discussed earlier, when the p-value < 0.05, we can safely reject the null hypothesis.<\/p>\n\n\n\n
                              In our case, since the p-value is less than 0.05, we can reject the null hypothesis and conclude that the model is highly significant. This means there is a significant association between the independent and dependent variables.<\/p>\n\n\n\n
                              R \u2013 Squared and Adjusted R - Squared<\/strong><\/h3>\n\n\n\n
                              R \u2013 Squared (R2) is a basic metric which tells us how much variance has been explained by the model. It ranges from 0 to 1. In Linear Regression, if we keep adding new variables, the value of R \u2013 Square will keep increasing irrespective of whether the variable is significant. This is where Adjusted R \u2013 Square comes to help. Adjusted R \u2013 Square helps us to calculate R \u2013 Square from only those variables whose addition to the model is significant. So, while performing Linear Regression, it is always preferable to look at Adjusted R \u2013 Square rather than just R \u2013 Square.<\/p>\n\n\n\n
                                \n
                              1. An Adjusted R \u2013 Square value close to 1 indicates that the regression model has explained a large proportion of variability.<\/li>\n\n\n\n
                              2. A number close to 0 indicates that the regression model did not explain too much variability.<\/li>\n<\/ol>\n\n\n\n
                                In our output, Adjusted R Square value is 0.6438, which is closer to 1, thus indicating that our model has been able to explain the variability.<\/p>\n\n\n\n
                                AIC and BIC<\/strong><\/h3>\n\n\n\n
                                \"\"\/<\/figure>\n\n\n\n
                                AIC and BIC are widely used metrics for model selection. AIC stands for Akaike Information Criterion, and BIC stands for Bayesian Information Criterion. These help us to check the goodness of fit for our model. For model comparison model with the lowest AIC and BIC is preferred.<\/p>\n\n\n\n
                                Which Regression Model is the best fit for the data?<\/strong><\/h3>\n\n\n\n
                                There are number of metrics that help us decide the best fit model for our data, but the most widely used are given below:<\/p>\n\n\n\n
                                Statistics<\/strong><\/td>Criterion<\/strong><\/td><\/tr>
                                R - Squared<\/td>Higher the better<\/td><\/tr>
                                Adjusted R - Squared<\/td>Higher the better<\/td><\/tr>
                                t-statistic<\/td>Higher the t-values lower the p-value<\/td><\/tr>
                                f-statistic<\/td>Higher the better<\/td><\/tr>
                                AIC<\/td>Lower the better<\/td><\/tr>
                                BIC<\/td>Lower the better<\/td><\/tr>
                                Mean Standard Error (MSE)<\/td>Lower the better<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
                                Predicting Linear Models<\/strong><\/h3>\n\n\n\n
                                Now we know how to build a Linear Regression Model In R using the full dataset. But this approach does not tell us how well the model will perform and fit new data.<\/p>\n\n\n\n
                                Thus, to solve this problem, the general practice in the industry is to split the data into the Train and Test datasets in the ratio of 80:20 (Train 80% and Test 20%). With the help of this method, we can now get the values for the test dataset and compare them with the values from the actual dataset.<\/p>\n\n\n\n
                                Splitting the Data<\/strong><\/h3>\n\n\n\n
                                \"\"\/<\/figure>\n\n\n\n
                                We do this with the help of the sample() function in R. <\/p>\n\n\n\n
                                \"\"\/<\/figure>\n\n\n\n
                                \"\"\/<\/figure>\n\n\n\n
                                Building the model on Train Data and Predict on Test Data<\/strong><\/h3>\n\n\n\n
                                \"\"\/<\/figure>\n\n\n\n
                                Model Diagnostics<\/strong><\/h4>\n\n\n\n
                                \"\"\/<\/figure>\n\n\n\n
                                If we look at the p-value, since it is less than 0.05, we can conclude that the model is significant. Also, if we compare the Adjusted R \u2013 Squared value with the original dataset, it is close to it, thus validating that the model is significant.<\/p>\n\n\n\n
                                K \u2013 Fold Cross-Validation<\/strong><\/h3>\n\n\n\n
                                Now, we have seen that the model performs well on the test dataset as well. But this does not guarantee that the model will be a good fit in the future as well. The reason is that there might be a case that a few data points in the dataset might not be representative of the whole population. Thus, we need to check the model performance as much as possible. One way to ensure this is to check whether the model performs well on train and test data chunks. This can be done with the help of K \u2013 Fold Cross-validation. <\/p>\n\n\n\n
                                The procedure of K \u2013 Fold Cross-validation is given below:<\/p>\n\n\n\n
                                  \n
                                1. The random shuffling of the dataset.<\/li>\n\n\n\n
                                2. Splitting of data into k folds\/sections\/groups.<\/li>\n\n\n\n
                                3. For each fold\/section\/group:<\/li>\n<\/ol>\n\n\n\n
                                    \n
                                  1. Make the fold\/section\/group the test data.<\/li>\n\n\n\n
                                  2. Take the rest data as train data.<\/li>\n\n\n\n
                                  3. Run the model on train data and evaluate the test data.<\/li>\n\n\n\n
                                  4. Keep the evaluation score and discard the model.<\/li>\n<\/ol>\n\n\n\n
                                    <\/p>\n\n\n\n
                                    \"\"\/<\/figure>\n\n\n\n
                                    \"\"\/<\/figure>\n\n\n\n
                                    After performing the K \u2013 Fold Cross-validation, we can observe that the R \u2013 Square value is close to the original data, as well, as MAE is 12%, which helps us conclude that model is a good fit.
                                    <\/p>\n\n\n\n
                                    Advantages of Using Linear Regression<\/strong><\/h2>\n\n\n\n
                                      \n
                                    1. The linear Regression method is very easy to use. If the relationship between the variables (independent and dependent) is known, we can easily implement the regression method accordingly (Linear Regression for linear relationship).<\/li>\n\n\n\n
                                    2. Linear Regression provides the significance level of each attribute contributing to the prediction of the dependent variable. With this data, we can choose between the variables which are highly contributing\/ important variables. <\/li>\n\n\n\n
                                    3. After performing linear regression, we get the best fit line, which is used in prediction, which we can use according to the business requirement.<\/li>\n<\/ol>\n\n\n\n
                                      Limitations of Linear Regression<\/strong><\/h3>\n\n\n\n
                                      The main limitation of linear regression is that its performance is not up to the mark in the case of a nonlinear relationship. Linear regression can be affected by the presence of outliers in the dataset. The presence of high correlation among the variables also leads to the poor performance of the linear regression model.<\/p>\n\n\n\n
                                      Linear Regression Examples<\/strong><\/h3>\n\n\n\n
                                        \n
                                      1. Linear Regression can be used for product sales prediction to optimize inventory management.<\/li>\n\n\n\n
                                      2. It can be used in the Insurance domain, for example, to predict the insurance premium based on various features.<\/li>\n\n\n\n
                                      3. Monitoring website click count on a daily basis using linear regression could help in optimizing the website efficiency etc.<\/li>\n\n\n\n
                                      4. Feature selection is one of the applications of Linear Regression. <\/li>\n<\/ol>\n\n\n\n
                                        Linear Regression - Learning the Model<\/b><\/h2>\n\n\n\n