Google

By omitting the term for past rewards that do not contribute to the gradient, we arrive at a simplified version of the policy gradient with a baseline. This formulation isolates the future rewards (the reward-to-go) from time step $$t$$ onwards: 
$$ \frac{\partial J(\theta)}{\partial \theta} = \frac{1}{|\mathcal{D}|} \sum_{\tau \in \mathcal{D}} \frac{\partial}{\partial \theta} \left[ \sum_{t=1}^{T} \log \pi_{\theta}(a_t|s_t) \left( \sum_{k=t}^{T} r_k - b \right) \right] $$ 
Removing the past rewards not only simplifies the calculation but can further reduce the variance of the gradient estimate.

Policy Gradient with Reward-to-Go and Baseline

An engineer is training two reinforcement learning agents, Agent X and Agent Y, on the same complex task. Both agents use a policy gradient approach, but with different update rules for actions taken within a trajectory. After running several training sessions, the engineer observes that Agent Y learns a successful policy much faster and more consistently than Agent X. The variance of the gradient updates for Agent Y is also significantly lower.

*   **Agent X's Update Rule:** For each action taken at time step `t`, the policy update is weighted by the sum of all rewards from the entire trajectory (from `t=1` to `T`).
*   **Agent Y's Update Rule:** For each action taken at time step `t`, the policy update is weighted by the sum of rewards from that time step onward (from `t` to `T`), minus an estimate of the average reward typically received from the current state.

Based on this information, identify and explain the two distinct principles incorporated into Agent Y's update rule that contribute to its superior performance and lower variance compared to Agent X.

Analysis of Policy Update Mechanisms

An engineer is training a reinforcement learning agent and observes that the training process is very unstable, with large fluctuations in performance between updates. The current training algorithm updates the policy for every action in an episode by multiplying the gradient of the log-probability of that action by the total, undiscounted reward for the entire episode. Propose and justify two distinct modifications to the reward-scaling term in this calculation to reduce the observed instability. For each modification, explain the principle that makes it effective.

Stabilizing Policy Gradient Training

An agent completes an episode with the following sequence of rewards: `r_1 = -1, r_2 = -1, r_3 = -1, r_4 = +10`. When updating the policy for the action taken at time step `t=2`, a baseline value of `b(s_2) = 5` is used. According to the policy gradient method that incorporates both reward-to-go and a baseline, what is the numerical value of the term that multiplies the gradient of the log-probability of the action at `t=2`?

Learn Before

Related