Q Learning, a model-free reinforcement learning algorithm, aims to learn the quality of actions and telling an agent what action is to be taken under which circumstance. Through the course of this blog, we will learn more about Q Learning, and it's learning process with the help of an example. <\/p>\n\n\n\n
Contributed by: Rahul Purohit<\/a> <\/strong><\/em><\/p>\n\n\n\n Richard S. Sutton in his book \u201cReinforcement Learning \u2013 An Introduction\u201d considered as the Gold Standard, gives a very intuitive definition \u2013 \u201cReinforcement learning is learning what to do\u2014how to map situations to actions\u2014to maximize a numerical reward signal.\u201d The field of reinforcement learning (RL from now on) is not new. It was initiated as early as the 1960s (earlier referred to as \u201chedonistic\u201d learning system). Although it failed to gain popularity with Supervised Learning (SL), attracting a large group of researchers\u2019 interest. Only in the last decade or so, researchers have come to realize untapped potential RL possesses. DeepMind\u2019s AlphaGo, Alpha Zero, are some brilliant examples of the powers of RL, and it is just the beginning. <\/p>\n\n\n\n Also, this would be a good place to understand how RL is different from SL. SL put in simple words is learning from a set of examples provided by an external supervisor, each example being a description of a phenomenon (independent variables) with a result (dependent variable) associated with it. The objective of SL is to exploit the knowledge gained from training examples and use it to determine the result of unseen data. This works pretty well for most of the problems, except it fails in the case of Interactive problems (e.g. Games, robotic manoeuvres etc.) where gathering a set of examples that are representative and exhaustive is not feasible. This is where the RL systems come to rescue, RL systems can learn without a set of examples explicitly given by an external supervisor, and rather, the agent itself interacts with the environment and can figure out a combination of actions that leads to the desired outcome. <\/p>\n\n\n\n Also Read: What is Machine Learning?<\/a><\/p>\n\n\n\n \n MIT Data Science and Machine Learning Course<\/a>\n <\/p>\n Unlock the power of data. Build hands-on data science and machine learning skills to drive innovation in your career.<\/p>\n 1. Policy Based- <\/strong><\/p>\n\n\n\n In this learning approach, a policy i.e. a function mapping each state to the best action is optimized. Once we have a well-defined policy, the agent can determine the best action to take by giving the current state as an input to the policy. We can further divide the policies in two types- <\/p>\n\n\n\n S=(s) \u27a1 A= (a)<\/p>\n\n\n\n Policy \u27a1 p (A = a | S = s) <\/p>\n\n\n\n 2. Value Based- <\/strong><\/p>\n\n\n\n In value-based RL, the objective is to optimize a value function, a function (can be thought of as a simple Lookup table) which maps maximum future reward to a given state. The value of each state is the total amount of reward an RL agent can expect to receive until the fulfilment of goal.<\/p>\n\n\n\n Q Learning comes under Value-based learning algorithms. The objective is to optimize a value function suited to a given problem\/environment. The \u2018Q\u2019 stands for quality; it helps in finding the next action resulting in a state of the highest quality. This approach is rather simple and intuitive. It a very good place to start the RL journey. The values are stored in a table, called a Q Table. <\/p>\n\n\n\n Let us devise a simple 2D game environment of size 4 x 4 and understand how Q- Learning can be used to arrive at the best solution.<\/p>\n\n\n\n Goal:<\/strong> Guide the kid to the Park<\/p>\n\n\n\n Reward System:<\/strong> End of an Episode:<\/strong> Now let us see how a typical Q learning agent will play this game. First, let us create a Q- table where we will keep a track of all values associated with each state. The Q Table will have rows equal to the number of states in the problem i.e. 16 in our case, and the number of columns would be equal to the number of actions an agent can make which happens to be 4 (Up, Down, Left & Right).<\/p>\n\n\n\n Sample Q-Table for 4 x 4 2D game environment<\/em><\/p>\n\n\n\n There are two popular Learning approaches<\/strong><\/h2>\n\n\n\n
<\/p>\n\n\n\n\n
\n
Q Learning <\/strong><\/h2>\n\n\n\n
A. Get candy = +10 points
B. Encounter Dog = -50 points
C. Reach Park = +50 points <\/p>\n\n\n\n
A. Encounter Dog
B. Reach Park <\/p>\n\n\n\n ACTIONS <\/strong>
STATES <\/strong><\/td>UP<\/strong><\/td> DOWN <\/strong><\/td> LEFT<\/strong><\/td> RIGHT <\/strong><\/td><\/tr> 1 (START)<\/strong><\/td> 0<\/td> 0<\/td> 0<\/td> 0<\/td><\/tr> 2<\/strong><\/td> 0<\/td> 0<\/td> 0<\/td> 0<\/td><\/tr> \u2026\u2026<\/strong><\/td> 0<\/td> 0<\/td> 0<\/td> 0<\/td><\/tr> 16<\/strong><\/td> 0<\/td> 0<\/td> 0<\/td> 0<\/td><\/tr><\/tbody><\/table><\/figure>\n\n\n\n