{"id":13551,"date":"2020-03-23T15:48:09","date_gmt":"2020-03-23T10:18:09","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/logistic-regression-with-examples-in-python-and-r\/"},"modified":"2024-10-24T19:11:39","modified_gmt":"2024-10-24T13:41:39","slug":"logistic-regression-with-examples-in-python-and-r","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/logistic-regression-with-examples-in-python-and-r\/","title":{"rendered":"Logistic Regression With Examples in Python and R"},"content":{"rendered":"\n

In every algorithm of machine learning, there is an approach that is unique yet easily interpretable. Logistic regression is one such algorithm with an easy and unique approach. It is very often used in the credit and risk industry for its easy intuition on predicting the chances of default and risk cases. It is indeed quite a challenge to break down most of the algorithms due to their black-box nature and their hard to find parameters, but logistic regression outperforms all. So it is time to break down the entire algorithm and draw some inferences.<\/p>\n\n\n\n

We can say that logistic regression solely depends on maximum likelihood estimation. Now let us focus on this and after that, we will jump back to the core concepts including parameters. Learn more about the Logistic regression algorithm<\/a> here.<\/p>\n\n\n\n

Maximum likelihood estimation maximises the probability that classifies the event being 1 or 0 by estimating certain parameters. Every probability equation goes by the following.<\/em><\/p>\n\n\n\n

probability (event + non-event) = 1<\/strong><\/em><\/p>\n\n\n\n

Logistic regression is capable of finding out the probability only after transforming the dependent variable into a logit variable with respect to the independent variable or the features present in the data.<\/p>\n\n\n

\"\"<\/figure>\n\n\n\n

The reason being using log function is to increase the range of probability of classification of events. Let us explain how it works.<\/p>\n\n\n\n

Range of Probability = [0,1]<\/em><\/p>\n\n\n\n

Range of Odds = [0, + \u221e] as Odds = <\/em>(Probability)\/(1-Probability)<\/p>\n\n\n\n

Range of log(odds) = [- \u221e, + \u221e]<\/em><\/p>\n\n\n\n

Here as we can see how the range of [0,1] increased to [-,+] which is what we wanted.<\/p>\n\n\n\n

Now the entire mechanism behind the logistic regression has been covered, let us break further to the terminologies involved in it.<\/p>\n\n\n\n

\n
\n
\n
<\/div>\n Advance Data Science with MIT<\/span>\n <\/div>\n

\n MIT Data Science and Machine Learning Course<\/a>\n <\/p>\n

Unlock the power of data. Build hands-on data science and machine learning skills to drive innovation in your career.<\/p>\n

\n
\n
<\/div>\n Duration: 12 weeks<\/span>\n <\/div>\n
\n
<\/div>\n 4.62\/5 Rating<\/span>\n <\/div>\n <\/div>\n \n Discover the Program\n
<\/div>\n <\/a>\n <\/div>\n <\/div>\n\n\n\n

Information Value<\/strong><\/h2>\n\n\n\n

This parameter is considered even before building the model. This is useful in eliminating those variables which seem to be weak for prediction and is considered a weak contributor to the dependent variable. Let us understand the following equation.<\/p>\n\n\n

\"\"<\/figure>\n\n\n\n

Now this equation seems to be tricky at first, so let us go with an example and get a clear idea of how it works.<\/p>\n\n\n\n

The bins given above are the price range followed by the events and non-events. After we calculated the information value, we found a value of 0.0356 which is 3.5% and we can say it is a weak predictor as it contributes only 3.5% to the dependent variable.<\/p>\n\n\n\n

Let us see the predictive power of the variables based on their information value<\/p>\n\n\n\n

Information Value<\/strong><\/td>Predictive Power<\/strong><\/td><\/tr>
< 0.02<\/strong><\/td>Useless of prediction<\/td><\/tr>
0.02 - 0.1<\/strong><\/td>Weak Predictor<\/td><\/tr>
0.1 - 0.3<\/strong><\/td>Medium Predictor<\/td><\/tr>
0.3 \u2013 0.5<\/strong><\/td>Strong Predictor<\/td><\/tr>
> 0.5<\/strong><\/td>Suspicious or too good Predictor<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
Also Read: Linear Regression in Machine Learning<\/a><\/pre>\n\n\n\n
Akaike Information Criteria<\/strong><\/h2>\n\n\n\n
This parameter helps us to find the relatedness of our model by acting as a tradeoff between bias and variance.<\/p>\n\n\n\n
Key point - The lower the AIC the better is the model.<\/p>\n\n\n\n
AIC can be calculated by the following equation <\/p>\n\n\n\n
AIC = -2*ln(L) + 2*k<\/em><\/p>\n\n\n\n
L = Maximum value of Likelihood (log transformation applied for mathematical convenience)<\/em><\/p>\n\n\n\n
k = number of variables in the model<\/em><\/p>\n\n\n\n
Here the parameter k is used to penalise the overfitting issue in a model. <\/p>\n\n\n\n
ROC Curve<\/strong><\/h2>\n\n\n\n
ROC or Receiver Operating Characteristics curve is a graphical representation of the performance of a binary classification model.  It shows the variation between True positive and False positive rate at different threshold values.<\/p>\n\n\n\n
The ROC curve can give us a clear idea to set a threshold value to classify the label and also help in model optimisation.<\/p>\n\n\n\n
A low threshold value we will put most of the predicted observations under the positive category, even when some of them should be placed under the negative category. On the other hand, keeping the threshold at a very high level penalises the positive category, but the negative category will improve.<\/p>\n\n\n\n
For such case an optimum threshold value can give a better accuracy which can be found on ROC curve<\/p>\n\n\n\n
ROC curve will look as follow:<\/p>\n\n\n\n
To gain more information on sensitivity and specificity, we need to go through the following formula.<\/p>\n\n\n\n
Recall (Sensitivity or TPR) = <\/em>TP\/(TP+FN)<\/p>\n\n\n\n
Specificity (TNR) = <\/em>TN\/(FP+TN)<\/p>\n\n\n\n
Evaluation parameters<\/strong><\/h3>\n\n\n\n
Following are the evaluation parameters considered in logistic regression<\/p>\n\n\n\n
    \n
  1. Precision<\/li>\n\n\n\n
  2. Recall<\/li>\n\n\n\n
  3. F1-Score<\/li>\n\n\n\n
  4. ROC curve<\/li>\n<\/ol>\n\n\n\n
    All the above parameters are considered on positive and negative rates of the classes which are tabulated below<\/p>\n\n\n\n

    <\/td>    		Predicted: YES<\/strong><\/td>Predicted: NO<\/strong><\/td><\/tr>
    Actual: YES<\/strong><\/td>TP
    ( True Positive)<\/td>    		FN
    ( True Positive)<\/td><\/tr>
    Actual: NO<\/strong><\/td>FP
    ( True Positive)<\/td>    		TN
    ( True Positive)<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    (a) Precision:<\/strong> Precision is an evaluation measure which is the combination of relevant as well as retrieved items over the total number of retrieved results. It is basically used when the case of false-positive prediction is high.<\/p>\n\n\n\n
    Precision (TNR) = <\/em>TPTP+FP<\/p>\n\n\n\n
    b) Recall: <\/strong>Recall is a measure when False negative is considered.<\/p>\n\n\n\n
    Recall (Sensitivity or TPR) = <\/em>TP\/(TP+FN)<\/p>\n\n\n\n
    c) ROC Curve <\/strong>(Explained above)<\/p>\n\n\n\n
    d)F1-Score:<\/strong> F1-Score is an evaluation technique that maintains a balance between precision and recall.<\/p>\n\n\n\n
    F1 Score = <\/em>2 * (Precision*Recall)\/(Precision+Recall)<\/p>\n\n\n\n
    Now in some cases precision and recall can be a dangerous call. Let us understand that further.<\/p>\n\n\n\n
    Precision and recall are two of the evaluation techniques used in a machine learning<\/a> model which tells us how better a model can perform. But there are certain business domains where these two evaluation techniques can cause a business risk.<\/p>\n\n\n\n
    Medical Industry<\/strong><\/h3>\n\n\n\n
    Let us suppose we are analysing the records of patients suffering from cancer and we are trying to predict whether the upcoming patients will suffer from cancer or not. Now, the precision is considered when the false positive rates are high which means the patient is not suffering from cancer but the machine predicts as a cancer patient.<\/p>\n\n\n\n
    In case of recall, false negatives are high which means the patient is suffering from cancer and the model predicts the patient as cancer-free.<\/p>\n\n\n\n
    Cancer patients<\/strong><\/td>Predicted: Yes (Having cancer)<\/strong><\/td>Predicted: No (Not having cancer)<\/strong><\/td><\/tr>
    Actual: Yes (Having cancer)<\/strong><\/td>90<\/strong><\/td>10<\/strong><\/td><\/tr>
    Actual: No (Not having cancer)<\/strong><\/td>80<\/strong><\/td>20<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Accuracy of the model= (90+20)\/200=0.55<\/p>\n\n\n\n
    Precision of the model=90\/(90+80)=0.53<\/p>\n\n\n\n
    Recall of the model=90\/(90+10)=0.9<\/p>\n\n\n\n
    Now the model is 55% accurate but if we see the recall, then it is 90% accurate. But if we consider recall as accuracy then it will declare many patients as cancer-free although they are suffering from cancer.<\/p>\n\n\n\n
    Banking Sector<\/strong><\/h3>\n\n\n\n
    In the banking sector, let us assume building a model to check whether a customer is eligible for a loan or not. Let us again build a model on 200 customers and evaluate based on accuracy, precision and recall. Check out this Logistic regression on customer data<\/a><\/p>\n\n\n\n
     course.<\/p>\n\n\n\n
    Customer<\/strong><\/td>Predicted: Yes (Eligible for a loan)<\/strong><\/td>Predicted: No (Not eligible)<\/strong><\/td><\/tr>
    Actual: Yes (Eligible for a loan)<\/strong><\/td>80<\/strong><\/td>20<\/strong><\/td><\/tr>
    Actual: No (Not eligible)<\/strong><\/td>8<\/strong><\/td>92<\/strong><\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n
    Accuracy of the model= (80+92)\/200=0.86<\/p>\n\n\n\n
    Precision of the model=80\/(80+8)=0.9<\/p>\n\n\n\n
    Recall of the model=80\/(80+20)=0.8<\/p>\n\n\n\n
    Now the model is 86% accurate but if we consider the precision we see it is 90%. Now if we evaluate the model by precision, then there is a high chance that a person is not eligible for loan but the bank is sanctioning loan for them. In that case, the bank may incur huge losses.<\/p>\n\n\n\n
    Precision works fine when false-positive cases are high. On the other hand, recall works fine when false-negative cases are high. Many domains such as medical, educational or legal fields are delicate, where evaluating entities with such measures can cost more risk. In such cases, F1-Score performs better as it is the only measure that maintains a balance between precision and recall.<\/p>\n\n\n\n
    F1-Score is the harmonic mean of precision and recall. Multiplying the constant of 2 <\/em>scales the score to 1 <\/em>when both precision and recall are 1<\/em>.<\/p>\n\n\n\n
    In such cases, F1-score can be a good evaluation technique because it maintains a balance between precision and recall and can tell almost exactly whether a person is eligible for a loan or not.<\/p>\n\n\n\n
    The best evaluation technique is accuracy which can tell us the best prediction because it gives the ratio of true prediction to total actual values.<\/p>\n\n\n\n
    Accuracy = <\/em>TP+TNTP+FN+TN+FP<\/p>\n\n\n\n
    Merits of Logistic Regression<\/strong><\/h2>\n\n\n\n