{"id":32485,"date":"2021-04-26T18:07:37","date_gmt":"2021-04-26T12:37:37","guid":{"rendered":"https:\/\/www.mygreatlearning.com\/blog\/introduction-to-stochastic-gradient-descent\/"},"modified":"2024-09-02T15:28:27","modified_gmt":"2024-09-02T09:58:27","slug":"introduction-to-stochastic-gradient-descent","status":"publish","type":"post","link":"https:\/\/www.mygreatlearning.com\/blog\/introduction-to-stochastic-gradient-descent\/","title":{"rendered":"Introduction to Stochastic Gradient Descent"},"content":{"rendered":"\n

Stochastic: \u201cProcess involving a randomly determined sequence of observations, each of which is considered as a sample of one element from a probability distribution.\u201d  Or, in simple terms, \u201cRandom selection.\u201d<\/p>\n\n\n\n

Contributed by: Sarveshwaran<\/a><\/strong><\/em><\/p>\n\n\n\n

Points discussed:<\/strong><\/p>\n\n\n\n

    \n
  1. Optimization<\/li>\n\n\n\n
  2. Optimization Function<\/li>\n\n\n\n
  3. Loss Function<\/li>\n\n\n\n
  4. Derivative<\/li>\n\n\n\n
  5. Optimization Methods<\/li>\n\n\n\n
  6. Gradient Descent<\/li>\n\n\n\n
  7. Stochastic Gradient Descent<\/li>\n<\/ol>\n\n\n\n

    What is the need of Optimization?<\/strong><\/h2>\n\n\n\n

    Any algorithm has an objective of reducing the error, reduction in error is achieved by optimization techniques.<\/p>\n\n\n\n

    Error: Cross-Entropy Loss in Logistic Regression, Sum of Squared Loss in Linear Regression<\/p>\n\n\n\n

    What is Optimization function?<\/strong><\/h2>\n\n\n\n

    Optimization is a mathematical technique of either minimizing or maximizing some function f(x)<\/em><\/strong> by altering x. In many real-time scenarios, we will be minimizing f(x). Well, if the thought comes up, how do we maximize the f(x)? It is just possible by minimizing the -f(x). Any function f(x)<\/em><\/strong> that we want to either minimize or maximize we call the objective function or criterion.<\/strong><\/p>\n\n\n\n

    \n
    \n
    \n
    <\/div>\n Texas McCombs, UT Austin<\/span>\n <\/div>\n

    \n PG Program in AI & Machine Learning<\/a>\n <\/p>\n

    Master AI with hands-on projects, expert mentorship, and a prestigious certificate from UT Austin and Great Lakes Executive Learning.<\/p>\n

    \n
    \n
    <\/div>\n Duration: 12 months<\/span>\n <\/div>\n
    \n
    <\/div>\n Ratings: 4.72<\/span>\n <\/div>\n <\/div>\n \n Start Learning today\n
    <\/div>\n <\/a>\n <\/div>\n <\/div>\n\n\n\n

    What is a Loss function?<\/strong><\/h2>\n\n\n\n

    Whenever we minimize our objective function f(x)<\/em><\/strong> we call it a loss function. Loss function takes various names such as Cost function or error function. <\/em><\/strong><\/p>\n\n\n\n

    Derivative:<\/strong><\/h4>\n\n\n\n

    Let\u2019s take an example of a function y = f(x), where both x and y are real numbers. The derivative of this function is denoted as f\u2019(x) or as dx\/dy. The derivative f\u2019(x) gives the slope of f(x) at the point x. In other words, it directs us how a small change in the input will correspond to the change in output. The derivative is useful to minimize the loss because it tells us how to change x to reduce the error or to make a small improvement in y.<\/p>\n\n\n\n

    Optimization Methods:<\/strong><\/h2>\n\n\n
    \"\"<\/figure>\n\n\n\n

    1. Un constrained - Closed Form Solution:<\/strong><\/h3>\n\n\n\n

    Steepest descent convergence when every element of the gradient is zero (at least very close to zero). In some cases, we may be able to avoid running an iterative algorithm and just jump to the critical point by solving equation  \u0394xfx=0 for x.<\/p>\n\n\n

    \"\"<\/figure>\n\n\n\n

    Why is the Iterative method more robust than the closed form?<\/strong><\/h2>\n\n\n\n