A helpful website that explains how policy-based deep reinforcement learning works:
https://medium.com/deep-math-machine-learning-ai/ch-13-deep-reinforcement-learning-deep-q-learning-and-policy-gradients-towards-agi-a2a0b611617e

University of Michigan - Ann Arbor

Policy-based methods learn the policy function directly, without calculating a value function for each action. These methods use a learning signal derived from sampling
instantiations of policy parameters and the set of policies is developed
towards policies that achieve better returns.

An example of a policy-based algorithm is Policy Gradient Methods.

Policy-Based Methods for Deep Reinforcement Learning

In policy gradient methods, we directly learn the policy function $$\pi$$, which outputs a probability distribution over actions. The term $$\pi(s,a;\theta) \in [0,1]$$ represents the probability of taking action $$a$$ given state $$s$$ with parameters $$\theta$$. Neural networks can be used to find the policy function, taking the state as input and producing the probability distribution of actions.

The general process is:
- The agent takes in a state and computes the probability of each action.
- It samples an action based on this probability distribution and observes the next state and reward.
- This cycle repeats until the end of the episode (game) and the total reward is evaluated.
- The parameters $$\theta$$ in the network are updated using backpropagation and gradient ascent based on the rewards.

Through this process, the network allows the agent to play and explore, gradually increasing the probabilities of actions that lead to positive returns.

Policy Gradient Methods for Deep Reinforcement Learning

In value-based methods, we calculate the Q values for all possible actions in the action space, but in real world , the number of actions can be a lot or can’t be imaginable. For example, the Robot walking problem has continuous and high dimensional action space. 

In policy based methods, we directly learn our policy function π without worrying about a value function, which means we can choose actions without calculate the Q(S,A) values.

Learn Before

Related