{"id":19123,"date":"2020-08-24T19:41:28","date_gmt":"2020-08-24T14:11:28","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/"},"modified":"2024-09-02T15:30:56","modified_gmt":"2024-09-02T10:00:56","slug":"understanding-learning-rate-in-machine-learning","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/","title":{"rendered":"Understanding Learning Rate in Machine Learning"},"content":{"rendered":"\n

In supervised learning, to enable an algorithm\u2019s predictions to be as close to the actual values\/labels as possible, we employ two things: 1) A cost function and 2) A technique to minimize the cost function. There are popular forms of cost functions used for different tasks that the algorithms are expected to perform.  Also, a popular technique used to minimize the cost function is the gradient descent method. We will understand these concepts to understand the role of \u2018learning rate\u2019 in machine learning.<\/p>\n\n\n\n

Cost Function<\/strong><\/h2>\n\n\n\n

A cost function is a measure of the error in prediction committed by an algorithm. It indicates the difference between the predicted and the actual values for a given dataset. Closer the predicted value to the actual value, the smaller the difference and lower the value of the cost function. Lower the value of the cost function, the better the predictive capability of the model. An ideal value of the cost function is zero. Some of the popular cost functions used in machine learning for applications such as regression, classification, and density approximation are shown in table-1.<\/p>\n\n\n\n

To further the discussion, let\u2019s consider the cost function for regression, in which the objective is to learn a mapping function between the predictors<\/em> (independent variables) and target<\/em> (dependent variable). If we assume the relationship to be linear, the equation for the predicted value (yi) is shown below. <\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In equation-1, x<\/em>1<\/sub><\/em> and x<\/em>2<\/sub><\/em> are the two predictors and  \u03b2<\/em><\/strong>0<\/sub><\/em><\/strong>, \u03b2<\/em><\/strong>1<\/sub><\/em><\/strong>, \u03b2<\/em><\/strong>2<\/sub><\/em><\/strong> are the model parameters. The algorithm learns(estimates) the values of these parameters during training. <\/p>\n\n\n\n

In regression, the typical cost function (CF<\/em><\/strong>) used is the mean squared error (MSE) cost function. The form of the function is shown below.<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In the above equation, yi and yi are respectively the actual and predicted values. n<\/strong> is the number of records in the dataset. Replacing yi  in the above equation, the cost function can be re-written as shown below<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In the above equation, it is important to note that the values for yi, <\/strong>x1i <\/strong>and x2i <\/strong>come from the dataset and cannot be manipulated to minimize the cost function. Only the model parameters \u03b2<\/em><\/strong>0<\/sub><\/em><\/strong>, \u03b2<\/em><\/strong>1<\/sub><\/em><\/strong>, \u03b2<\/em><\/strong>2 <\/sub><\/em><\/strong>can be manipulated to minimize the cost function. For the given dataset, these parameters can be estimated using the gradient descent method such that the cost function value is minimum.<\/p>\n\n\n\n

Gradient Descent Method<\/strong><\/h2>\n\n\n\n

Gradient descent is the popular optimization algorithm used in machine learning to estimate the model parameters. During training a model, the value of each parameter is guessed or assigned random values initially. The cost function is calculated based on the initial values and the parameter estimates are improved over several steps such that the cost function assumes a minimum value eventually. This is shown in figure 1 below.<\/p>\n\n\n\n

The equation used to improve the estimate is shown below. It is important to note that when one of the parameter\u2019s estimates is being improved, the other parameters are held constant. In our example, while the estimate for \u03b2o<\/sub><\/em> is being improved, \u03b21<\/sub><\/em> and \u03b22<\/sub><\/em> are held constant. <\/p>\n\n\n\n

Learning rate<\/strong><\/h2>\n\n\n\n

In machine learning, we deal with two types of parameters; 1) machine learnable parameters and 2) hyper-parameters. The Machine learnable parameters are the one which the algorithms learn\/estimate on their own during the training for a given dataset. In equation-3,  \u03b2<\/em>0<\/sub><\/em>, \u03b2<\/em>1<\/sub><\/em> and \u03b2<\/em>2<\/sub><\/em> are the machine learnable parameters. The Hyper-parameters are the one which the machine learning engineers or data scientists will assign specific values to, to control the way the algorithms learn and also to tune the performance of the model. Learning rate, generally represented by the symbol \u2018\u03b1\u2019, shown in equation-4, is a hyper-parameter used to control the rate at which an algorithm updates the parameter estimates or learns the values of the parameters.  <\/p>\n\n\n\n

Get hands on practice, take Machine Learning Quiz<\/a>.<\/p>\n\n\n\n

Effect of different values for learning rate<\/strong><\/h2>\n\n\n\n

Learning rate is used to scale the magnitude of parameter updates during gradient descent. The choice of the value for learning rate can impact two things: 1) how fast the algorithm learns and 2) whether the cost function is minimized or not. Figure 2 shows the variation in cost function with a number of iterations\/epochs for different learning rates. <\/p>\n\n\n\n

It can be seen that for an optimal value of the learning rate, the cost function value is minimized in a few iterations (smaller time). This is represented by the blue line in the figure. If the learning rate used is lower than the optimal value, the number of iterations\/epochs required to minimize the cost function is high (takes longer time). This is represented by the green line in the figure. If the learning rate is high, the cost function could saturate at a value higher than the minimum value. This is represented by the red line in the figure. If the learning rate selected is very high, the cost function could continue to increase with iterations\/epochs. An optimal learning rate is not easy to find for a given problem. Though getting the right learning is always a challenge, there are some well-researched methods documented to figure out optimal learning rates. Some of these techniques are discussed in the following sections. In all these techniques the fundamental idea is to vary the learning rate dynamically instead of using a constant learning rate.<\/p>\n\n\n\n

Decaying Learning rate<\/strong><\/h2>\n\n\n\n

In the decaying learning rate approach, it decreases with increase in epochs\/iterations. The formula used is shown below:<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In the above equation, o is the initial learning rate, is the decay rate and is the learning rate at a given Epoch number. Figure 3 shows the learning rate decay with the epoch number for different initial learning rates and decay rates.<\/p>\n\n\n\n

Scheduled Drop Learning rate<\/strong><\/h2>\n\n\n\n

Unlike the decay method, where the learning rate drops monotonously, in the drop method, the learning rate is dropped by a predetermined proportion at a predetermined frequency. The formula used to calculate for a given epoch is shown in the below equation:<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In the above equation, o is the initial learning rate, \u2018n\u2019 <\/em>is the epoch\/iteration number, \u2018D<\/em>\u2019 is a hyper-parameter which specifies by how much the learning rate has to drop, and \u03c1 is another hyper-parameter which specifies the epoch-based frequency of dropping the learning rate. Figure 4 shows the variation with epochs for different values of \u2018D\u2019<\/em> and \u2018\u03c1\u2019<\/em>.<\/p>\n\n\n\n

The limitation with both the decay approach and the drop approach is that they do not evaluate if decreasing the learning rate is required or not. In both the methods, the learning rate decreases irrespective of difficulty involved in minimizing the cost function.<\/p>\n\n\n\n

Adaptive Learning rate<\/strong><\/h2>\n\n\n\n

In this approach, the learning rate increases or decreases based on the gradient value of the cost function. For higher gradient value, the learning rate will be smaller and for lower gradient value, the learning rate will be larger. Hence, the learning decelerates and accelerates respectively at steeper and shallower parts of the cost function curve. The formula used in this approach is shown in the below equation.<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In the above equation, <\/strong>o is the initial learning rate <\/strong>and \u2018s<\/em>n<\/sub><\/em>\u2019<\/em> is the momentum factor, which is calculated using below equation. \u2018n\u2019 <\/em>is the epoch\/iteration number<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In the above equation, <\/strong>\ud835\udefe is a hyperparameter whose value is typically between 0.7 and 0.9. Note that in equation-7, momentum factor Sn<\/sub><\/em> is an exponentially weighted average of gradients. So not only the value of the current gradient is considered, but also the values of gradients from the previous epochs are considered to calculate the momentum factor. Figure 5 shows the idea behind the gradient adapted learning rate. When the cost function curve is steep, the gradient is large, and the momentum factor \u2018Sn\u2019 is larger. Hence the learning rate is smaller. When the cost function curve is shallow, the gradient is small and the momentum factor \u2018Sn\u2019 is also small. The learning rate is larger. <\/p>\n\n\n\n

The gradient adapted learning rate approach eliminates the limitation in the decay and the drop approaches by considering the gradient of the cost function to increase or decrease the learning rate. This approach is widely used in training deep neural nets with stochastic gradient descent.<\/p>\n\n\n\n

Cycling Learning Rate<\/strong><\/h2>\n\n\n\n

In this approach, the learning rate varies between a base rate and a maximum rate cyclically. Figure 6 shows the idea behind this approach. The figure shows that the learning rate varies in a triangular form between the maximum and the base rates at a fixed frequency. <\/p>\n\n\n\n

It is reported [1]<\/sup> that other forms such as sinusoidal or parabolic too yield similar results. The frequency of variation can be adjusted by setting the value of \u2018step size\u2019.  The formula used in this approach is shown below.<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

In the above equation, E is the learning rates for a given epoch, E<\/em> is the epoch number, max and base are respectively the maximum and the base learning rates. S<\/em> is the step size. Note that the above equation valid when E>S<\/em>. for E\u2264S below equation can be used<\/p>\n\n\n

\"learning<\/figure>\n\n\n\n

An important step in making this approach work is identifying the right base and the maximum learning rates. The process to identify these, referred to as \u2018LR range test\u2019 in [1] is; train the model for a few epochs allowing the learning rate to vary linearly starting from a small number. Capture the accuracy of the model for different learning rates and plot. The plot could look like the image shown in figure 7. <\/p>\n\n\n\n

From the plot identify two learning rate values; 1) the value at which the accuracy starts to increase and 2) the value at which the accuracy begins to fluctuate or to decrease. The first point corresponds to the base learning rate, and the second point corresponds to the maximum learning rate. Figure 8 shows the comparison of model accuracy achieved with different learning rate approaches on CIFER-10 image dataset in a convolutional neural network (CNN). In the figure, \u2018Original learning rate\u2019 corresponds to a fixed learning rate, \u2018Exponential\u2019 corresponds to an exponential learning rate decay approach and \u2018CLR\u2019 corresponds to the cyclic learning rate approach. While the constant learning and the exponential approach seem to have taken close to 70000 iterations to achieve an accuracy of 81%, the cyclic learning rate seems to have taken close to 25000 iterations indicating close to 2.5 times quicker convergence. <\/p>\n\n\n\n

In this article, the role of learning rate as a hyper-parameter is discussed, highlighting the role it plays in the time taken to train a model and the prediction accuracy achieved by the model. Some of the important aspects to remember are; using variable learning rate instead of a constant learning rate could help achieve higher accuracy in smaller training time. One of these or a combination of these techniques, especially the adaptive learning rate and the cyclic learning rate can be used in practical applications to train algorithms using the gradient descent method.<\/p>\n\n\n\n

Contributed by: Arun K<\/a> <\/a><\/strong><\/p>\n\n\n\n

References:<\/p>\n\n\n\n

[1] Cyclical Learning Rates for Training Neural Networks<\/a>
[2] Neural Network: Tricks of the Trade<\/a>
[3] understand-the-dynamics-of-learning-rate-on-deep-learning-neural-networks<\/a><\/p>\n\n\n\n


<\/p>\n\n\n\n

Find Machine Learning Course in Top Indian Cities<\/b><\/p> \n Chennai<\/a> | \n Bangalore<\/a> | \n Hyderabad<\/a> | \n Pune<\/a> | \n Mumbai<\/a> | \n Delhi NCR<\/a><\/div>\n\n\n\n

<\/p>\n\n\n\n

Our Machine Learning Courses<\/h2>\n\n\n\n

Explore our Machine Learning and AI courses, designed for comprehensive learning and skill development.<\/p>\n\n\n\n

Program Name<\/strong><\/th>Duration<\/strong><\/th><\/tr>
MIT No code AI and Machine Learning Course<\/a><\/th>12 Weeks<\/th><\/tr>
MIT Data Science and Machine Learning Course<\/a><\/th>12 Weeks<\/th><\/tr>
Data Science and Machine Learning Course<\/a><\/th>12 Weeks<\/th><\/tr><\/thead><\/table><\/figure>\n","protected":false},"excerpt":{"rendered":"

In supervised learning, to enable an algorithm\u2019s predictions to be as close to the actual values\/labels as possible, we employ two things: 1) A cost function and 2) A technique to minimize the cost function. There are popular forms of cost functions used for different tasks that the algorithms are expected to perform.  Also, a […]<\/p>\n","protected":false},"author":41,"featured_media":19153,"comment_status":"open","ping_status":"open","sticky":false,"template":"","format":"standard","meta":{"_acf_changed":false,"_uag_custom_page_level_css":"","site-sidebar-layout":"default","site-content-layout":"","ast-site-content-layout":"default","site-content-style":"default","site-sidebar-style":"default","ast-global-header-display":"","ast-banner-title-visibility":"","ast-main-header-display":"","ast-hfb-above-header-display":"","ast-hfb-below-header-display":"","ast-hfb-mobile-header-display":"","site-post-title":"","ast-breadcrumbs-content":"","ast-featured-img":"","footer-sml-layout":"","ast-disable-related-posts":"","theme-transparent-header-meta":"","adv-header-id-meta":"","stick-header-meta":"","header-above-stick-meta":"","header-main-stick-meta":"","header-below-stick-meta":"","astra-migrate-meta-layouts":"set","ast-page-background-enabled":"default","ast-page-background-meta":{"desktop":{"background-color":"var(--ast-global-color-4)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"ast-content-background-meta":{"desktop":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"tablet":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""},"mobile":{"background-color":"var(--ast-global-color-5)","background-image":"","background-repeat":"repeat","background-position":"center center","background-size":"auto","background-attachment":"scroll","background-type":"","background-media":"","overlay-type":"","overlay-color":"","overlay-opacity":"","overlay-gradient":""}},"footnotes":""},"categories":[2],"tags":[],"content_type":[],"class_list":["post-19123","post","type-post","status-publish","format-standard","has-post-thumbnail","hentry","category-artificial-intelligence"],"acf":[],"yoast_head":"\nUnderstanding Learning Rate in Machine Learning<\/title>\n<meta name=\"description\" content=\"Understanding Learning Rate in Machine Learning: In this blog, we'll understand more about the term learning rate.\" \/>\n<meta name=\"robots\" content=\"index, follow, max-snippet:-1, max-image-preview:large, max-video-preview:-1\" \/>\n<link rel=\"canonical\" href=\"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/\" \/>\n<meta property=\"og:locale\" content=\"en_US\" \/>\n<meta property=\"og:type\" content=\"article\" \/>\n<meta property=\"og:title\" content=\"Understanding Learning Rate in Machine Learning\" \/>\n<meta property=\"og:description\" content=\"Understanding Learning Rate in Machine Learning: In this blog, we'll understand more about the term learning rate.\" \/>\n<meta property=\"og:url\" content=\"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/\" \/>\n<meta property=\"og:site_name\" content=\"Great Learning Blog: Free Resources what Matters to shape your Career!\" \/>\n<meta property=\"article:publisher\" content=\"https:\/\/www.facebook.com\/GreatLearningOfficial\/\" \/>\n<meta property=\"article:published_time\" content=\"2020-08-24T14:11:28+00:00\" \/>\n<meta property=\"article:modified_time\" content=\"2024-09-02T10:00:56+00:00\" \/>\n<meta property=\"og:image\" content=\"http:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg\" \/>\n\t<meta property=\"og:image:width\" content=\"1000\" \/>\n\t<meta property=\"og:image:height\" content=\"568\" \/>\n\t<meta property=\"og:image:type\" content=\"image\/jpeg\" \/>\n<meta name=\"author\" content=\"Great Learning Editorial Team\" \/>\n<meta name=\"twitter:card\" content=\"summary_large_image\" \/>\n<meta name=\"twitter:creator\" content=\"@https:\/\/twitter.com\/Great_Learning\" \/>\n<meta name=\"twitter:site\" content=\"@Great_Learning\" \/>\n<meta name=\"twitter:label1\" content=\"Written by\" \/>\n\t<meta name=\"twitter:data1\" content=\"Great Learning Editorial Team\" \/>\n\t<meta name=\"twitter:label2\" content=\"Est. reading time\" \/>\n\t<meta name=\"twitter:data2\" content=\"11 minutes\" \/>\n<script type=\"application\/ld+json\" class=\"yoast-schema-graph\">{\"@context\":\"https:\\\/\\\/schema.org\",\"@graph\":[{\"@type\":\"Article\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#article\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/\"},\"author\":{\"name\":\"Great Learning Editorial Team\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\"},\"headline\":\"Understanding Learning Rate in Machine Learning\",\"datePublished\":\"2020-08-24T14:11:28+00:00\",\"dateModified\":\"2024-09-02T10:00:56+00:00\",\"mainEntityOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/\"},\"wordCount\":1886,\"commentCount\":0,\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/shutterstock_1096541144.jpg\",\"articleSection\":[\"AI and Machine Learning\"],\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"CommentAction\",\"name\":\"Comment\",\"target\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#respond\"]}]},{\"@type\":\"WebPage\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/\",\"name\":\"Understanding Learning Rate in Machine Learning\",\"isPartOf\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\"},\"primaryImageOfPage\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#primaryimage\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#primaryimage\"},\"thumbnailUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/shutterstock_1096541144.jpg\",\"datePublished\":\"2020-08-24T14:11:28+00:00\",\"dateModified\":\"2024-09-02T10:00:56+00:00\",\"description\":\"Understanding Learning Rate in Machine Learning: In this blog, we'll understand more about the term learning rate.\",\"breadcrumb\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#breadcrumb\"},\"inLanguage\":\"en-US\",\"potentialAction\":[{\"@type\":\"ReadAction\",\"target\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/\"]}]},{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#primaryimage\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/shutterstock_1096541144.jpg\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2020\\\/08\\\/shutterstock_1096541144.jpg\",\"width\":1000,\"height\":568,\"caption\":\"learning rate\"},{\"@type\":\"BreadcrumbList\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/understanding-learning-rate-in-machine-learning\\\/#breadcrumb\",\"itemListElement\":[{\"@type\":\"ListItem\",\"position\":1,\"name\":\"Blog\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\"},{\"@type\":\"ListItem\",\"position\":2,\"name\":\"AI and Machine Learning\",\"item\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/artificial-intelligence\\\/\"},{\"@type\":\"ListItem\",\"position\":3,\"name\":\"Understanding Learning Rate in Machine Learning\"}]},{\"@type\":\"WebSite\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#website\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"name\":\"Great Learning Blog\",\"description\":\"Learn, Upskill & Career Development Guide and Resources\",\"publisher\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\"},\"alternateName\":\"Great Learning\",\"potentialAction\":[{\"@type\":\"SearchAction\",\"target\":{\"@type\":\"EntryPoint\",\"urlTemplate\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/?s={search_term_string}\"},\"query-input\":{\"@type\":\"PropertyValueSpecification\",\"valueRequired\":true,\"valueName\":\"search_term_string\"}}],\"inLanguage\":\"en-US\"},{\"@type\":\"Organization\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#organization\",\"name\":\"Great Learning\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/\",\"logo\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/06\\\/GL-Logo.jpg\",\"width\":900,\"height\":900,\"caption\":\"Great Learning\"},\"image\":{\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/logo\\\/image\\\/\"},\"sameAs\":[\"https:\\\/\\\/www.facebook.com\\\/GreatLearningOfficial\\\/\",\"https:\\\/\\\/x.com\\\/Great_Learning\",\"https:\\\/\\\/www.instagram.com\\\/greatlearningofficial\\\/\",\"https:\\\/\\\/www.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/in.pinterest.com\\\/greatlearning12\\\/\",\"https:\\\/\\\/www.youtube.com\\\/user\\\/beaconelearning\\\/\"],\"description\":\"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.\",\"email\":\"info@mygreatlearning.com\",\"legalName\":\"Great Learning Education Services Pvt. Ltd\",\"foundingDate\":\"2013-11-29\",\"numberOfEmployees\":{\"@type\":\"QuantitativeValue\",\"minValue\":\"1001\",\"maxValue\":\"5000\"}},{\"@type\":\"Person\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/#\\\/schema\\\/person\\\/6f993d1be4c584a335951e836f2656ad\",\"name\":\"Great Learning Editorial Team\",\"image\":{\"@type\":\"ImageObject\",\"inLanguage\":\"en-US\",\"@id\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"contentUrl\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/wp-content\\\/uploads\\\/2022\\\/02\\\/unnamed.webp\",\"caption\":\"Great Learning Editorial Team\"},\"description\":\"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.\",\"sameAs\":[\"https:\\\/\\\/www.mygreatlearning.com\\\/\",\"https:\\\/\\\/in.linkedin.com\\\/school\\\/great-learning\\\/\",\"https:\\\/\\\/x.com\\\/https:\\\/\\\/twitter.com\\\/Great_Learning\",\"https:\\\/\\\/www.youtube.com\\\/channel\\\/UCObs0kLIrDjX2LLSybqNaEA\"],\"award\":[\"Best EdTech Company of the Year 2024\",\"Education Economictimes Outstanding Education\\\/Edtech Solution Provider of the Year 2024\",\"Leading E-learning Platform 2024\"],\"url\":\"https:\\\/\\\/www.mygreatlearning.com\\\/blog\\\/author\\\/greatlearning\\\/\"}]}<\/script>\n<!-- \/ Yoast SEO Premium plugin. -->","yoast_head_json":{"title":"Understanding Learning Rate in Machine Learning","description":"Understanding Learning Rate in Machine Learning: In this blog, we'll understand more about the term learning rate.","robots":{"index":"index","follow":"follow","max-snippet":"max-snippet:-1","max-image-preview":"max-image-preview:large","max-video-preview":"max-video-preview:-1"},"canonical":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/","og_locale":"en_US","og_type":"article","og_title":"Understanding Learning Rate in Machine Learning","og_description":"Understanding Learning Rate in Machine Learning: In this blog, we'll understand more about the term learning rate.","og_url":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/","og_site_name":"Great Learning Blog: Free Resources what Matters to shape your Career!","article_publisher":"https:\/\/www.facebook.com\/GreatLearningOfficial\/","article_published_time":"2020-08-24T14:11:28+00:00","article_modified_time":"2024-09-02T10:00:56+00:00","og_image":[{"width":1000,"height":568,"url":"http:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg","type":"image\/jpeg"}],"author":"Great Learning Editorial Team","twitter_card":"summary_large_image","twitter_creator":"@https:\/\/twitter.com\/Great_Learning","twitter_site":"@Great_Learning","twitter_misc":{"Written by":"Great Learning Editorial Team","Est. reading time":"11 minutes"},"schema":{"@context":"https:\/\/schema.org","@graph":[{"@type":"Article","@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#article","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/"},"author":{"name":"Great Learning Editorial Team","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad"},"headline":"Understanding Learning Rate in Machine Learning","datePublished":"2020-08-24T14:11:28+00:00","dateModified":"2024-09-02T10:00:56+00:00","mainEntityOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/"},"wordCount":1886,"commentCount":0,"publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg","articleSection":["AI and Machine Learning"],"inLanguage":"en-US","potentialAction":[{"@type":"CommentAction","name":"Comment","target":["https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#respond"]}]},{"@type":"WebPage","@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/","url":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/","name":"Understanding Learning Rate in Machine Learning","isPartOf":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#website"},"primaryImageOfPage":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#primaryimage"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#primaryimage"},"thumbnailUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg","datePublished":"2020-08-24T14:11:28+00:00","dateModified":"2024-09-02T10:00:56+00:00","description":"Understanding Learning Rate in Machine Learning: In this blog, we'll understand more about the term learning rate.","breadcrumb":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#breadcrumb"},"inLanguage":"en-US","potentialAction":[{"@type":"ReadAction","target":["https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/"]}]},{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#primaryimage","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg","width":1000,"height":568,"caption":"learning rate"},{"@type":"BreadcrumbList","@id":"https:\/\/www.mygreatlearning.com\/blog\/understanding-learning-rate-in-machine-learning\/#breadcrumb","itemListElement":[{"@type":"ListItem","position":1,"name":"Blog","item":"https:\/\/www.mygreatlearning.com\/blog\/"},{"@type":"ListItem","position":2,"name":"AI and Machine Learning","item":"https:\/\/www.mygreatlearning.com\/blog\/artificial-intelligence\/"},{"@type":"ListItem","position":3,"name":"Understanding Learning Rate in Machine Learning"}]},{"@type":"WebSite","@id":"https:\/\/www.mygreatlearning.com\/blog\/#website","url":"https:\/\/www.mygreatlearning.com\/blog\/","name":"Great Learning Blog","description":"Learn, Upskill & Career Development Guide and Resources","publisher":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization"},"alternateName":"Great Learning","potentialAction":[{"@type":"SearchAction","target":{"@type":"EntryPoint","urlTemplate":"https:\/\/www.mygreatlearning.com\/blog\/?s={search_term_string}"},"query-input":{"@type":"PropertyValueSpecification","valueRequired":true,"valueName":"search_term_string"}}],"inLanguage":"en-US"},{"@type":"Organization","@id":"https:\/\/www.mygreatlearning.com\/blog\/#organization","name":"Great Learning","url":"https:\/\/www.mygreatlearning.com\/blog\/","logo":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/06\/GL-Logo.jpg","width":900,"height":900,"caption":"Great Learning"},"image":{"@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/logo\/image\/"},"sameAs":["https:\/\/www.facebook.com\/GreatLearningOfficial\/","https:\/\/x.com\/Great_Learning","https:\/\/www.instagram.com\/greatlearningofficial\/","https:\/\/www.linkedin.com\/school\/great-learning\/","https:\/\/in.pinterest.com\/greatlearning12\/","https:\/\/www.youtube.com\/user\/beaconelearning\/"],"description":"Great Learning is a leading global ed-tech company for professional training and higher education. It offers comprehensive, industry-relevant, hands-on learning programs across various business, technology, and interdisciplinary domains driving the digital economy. These programs are developed and offered in collaboration with the world's foremost academic institutions.","email":"info@mygreatlearning.com","legalName":"Great Learning Education Services Pvt. Ltd","foundingDate":"2013-11-29","numberOfEmployees":{"@type":"QuantitativeValue","minValue":"1001","maxValue":"5000"}},{"@type":"Person","@id":"https:\/\/www.mygreatlearning.com\/blog\/#\/schema\/person\/6f993d1be4c584a335951e836f2656ad","name":"Great Learning Editorial Team","image":{"@type":"ImageObject","inLanguage":"en-US","@id":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","url":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","contentUrl":"https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2022\/02\/unnamed.webp","caption":"Great Learning Editorial Team"},"description":"The Great Learning Editorial Staff includes a dynamic team of subject matter experts, instructors, and education professionals who combine their deep industry knowledge with innovative teaching methods. Their mission is to provide learners with the skills and insights needed to excel in their careers, whether through upskilling, reskilling, or transitioning into new fields.","sameAs":["https:\/\/www.mygreatlearning.com\/","https:\/\/in.linkedin.com\/school\/great-learning\/","https:\/\/x.com\/https:\/\/twitter.com\/Great_Learning","https:\/\/www.youtube.com\/channel\/UCObs0kLIrDjX2LLSybqNaEA"],"award":["Best EdTech Company of the Year 2024","Education Economictimes Outstanding Education\/Edtech Solution Provider of the Year 2024","Leading E-learning Platform 2024"],"url":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"}]}},"uagb_featured_image_src":{"full":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg",1000,568,false],"thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144-150x150.jpg",150,150,true],"medium":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144-300x170.jpg",300,170,true],"medium_large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144-768x436.jpg",768,436,true],"large":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg",1000,568,false],"1536x1536":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg",1000,568,false],"2048x2048":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg",1000,568,false],"web-stories-poster-portrait":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg",640,364,false],"web-stories-publisher-logo":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg",96,55,false],"web-stories-thumbnail":["https:\/\/www.mygreatlearning.com\/blog\/wp-content\/uploads\/2020\/08\/shutterstock_1096541144.jpg",150,85,false]},"uagb_author_info":{"display_name":"Great Learning Editorial Team","author_link":"https:\/\/www.mygreatlearning.com\/blog\/author\/greatlearning\/"},"uagb_comment_info":0,"uagb_excerpt":"In supervised learning, to enable an algorithm\u2019s predictions to be as close to the actual values\/labels as possible, we employ two things: 1) A cost function and 2) A technique to minimize the cost function. There are popular forms of cost functions used for different tasks that the algorithms are expected to perform.  Also, a…","_links":{"self":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/19123","targetHints":{"allow":["GET"]}}],"collection":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts"}],"about":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/types\/post"}],"author":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/users\/41"}],"replies":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/comments?post=19123"}],"version-history":[{"count":22,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/19123\/revisions"}],"predecessor-version":[{"id":110663,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/posts\/19123\/revisions\/110663"}],"wp:featuredmedia":[{"embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media\/19153"}],"wp:attachment":[{"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/media?parent=19123"}],"wp:term":[{"taxonomy":"category","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/categories?post=19123"},{"taxonomy":"post_tag","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/tags?post=19123"},{"taxonomy":"content_type","embeddable":true,"href":"https:\/\/www.mygreatlearning.com\/blog\/wp-json\/wp\/v2\/content_type?post=19123"}],"curies":[{"name":"wp","href":"https:\/\/api.w.org\/{rel}","templated":true}]}}