{"id":14555,"date":"2020-05-18T08:39:11","date_gmt":"2020-05-18T03:09:11","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/bagging-boosting\/"},"modified":"2024-10-15T01:14:40","modified_gmt":"2024-10-14T19:44:40","slug":"bagging-boosting","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/bagging-boosting\/","title":{"rendered":"Understanding the Ensemble method Bagging and Boosting"},"content":{"rendered":"\n
    \n
  1. Bias and Variance\u00a0<\/a><\/li>\n\n\n\n
  2. Ensemble Methods<\/a><\/li>\n\n\n\n
  3. Bagging<\/a><\/li>\n\n\n\n
  4. Boosting<\/a><\/li>\n\n\n\n
  5. Bagging vs Boosting<\/a><\/li>\n\n\n\n
  6. Implementation<\/a><\/li>\n<\/ol>\n\n\n\n

    Bias and Variance <\/strong>
    <\/h2>\n\n\n\n

    For the same set of data, different algorithms behave differently. For example, if we want to predict the price of houses given for some dataset, some of the algorithms that can be used are Linear Regression<\/a> and Decision Tree Regressor<\/a>. Both of these algorithms will interpret the dataset in different ways and thus make different predictions. One of the key distinctions is how much bias and variance they produce.
    <\/p>\n\n\n\n

    There are 3 types of prediction error: bias, variance, and irreducible error. Irreducible error, also known as \"noise,\" can't be reduced by the choice of algorithm. The other two types of errors, however, can be reduced because they stem from your algorithm choice.<\/p>\n\n\n\n

    \u201c\u2026 bias is an error from erroneous assumptions in the learning algorithm. High bias can cause an algorithm to miss the relevant relations between features and target outputs (underfitting). \u201d
    <\/p>Wikipedia<\/cite><\/blockquote><\/figure>\n\n\n\n

    Bias is an assumption made by a model to make the target function easier to learn. Models with high bias are less flexible and are not fully able to learn from the training data.<\/p>\n\n\n\n

    In the given figure, we use a linear model, such as linear regression, to learn from the model. As we can see, the regression line fails to fit the majority of the data points and thus, this model has high bias and low learning power. Generally, models with low bias are preferred.<\/p>\n\n\n\n

     \u201c\u2026 variance is an error from sensitivity to small fluctuations in the training set. High variance can cause an algorithm to model the random noise in the training data, rather than the intended outputs (overfitting). \u201d 
    <\/p>Wikipedia<\/cite><\/blockquote><\/figure>\n\n\n\n

    Variance defines the deviation in prediction when switching from one dataset to another. In other words, it defines how much the predictions of a model will change from one dataset to another. It can also be defined as the amount that the estimate of the target function will change if different training data is used.<\/p>\n\n\n\n

    In the given figure, we can see a non-linear model such as SVR (Support Vector Regressor<\/a>) tries to generate a polynomial function that passes through all the data points. This may seem like the perfect model, but such models are not able to generalize the data well and perform poorly on data that has not been seen before. Ideally, we want a model with low variance.<\/p>\n\n\n\n

    But there seem to be tradeoffs between the bias and variance. This is known as a bias-variance tradeoff<\/a>. Hence when we decrease one, the other increases, and vice versa. <\/p>\n\n\n\n

    \n
    \n
    \n
    <\/div>\n Texas McCombs, UT Austin<\/span>\n <\/div>\n

    \n Post Graduate Program in Data Science with Generative AI: Applications to Business<\/a>\n <\/p>\n

    Learn how to turn data into strategy in this UT Data Science and Business Analytics Course \u2014 now with a focus on Generative AI. Gain practical experience through 7 hands-on projects over a 7-month duration.<\/p>\n

    \n
    \n
    <\/div>\n 7 Hands-on Projects<\/span>\n <\/div>\n
    \n
    <\/div>\n Duration: 7 months<\/span>\n <\/div>\n <\/div>\n \n Discover the Program\n
    <\/div>\n <\/a>\n <\/div>\n <\/div>\n\n\n\n

    Ensemble Methods<\/strong><\/h2>\n\n\n\n

    The general principle of an ensemble method in Machine Learning<\/a> to combine the predictions of several models. These are built with a given learning algorithm in order to improve robustness over a single model. Ensemble methods can be divided into two groups:
    <\/p>\n\n\n\n