Google

A simple way to implement a penalty for policy divergence in trust region optimization is to calculate the difference between the log-probabilities of a trajectory, $$\tau$$. This penalty measures the change between the current policy, $$\pi_{\theta}$$, and a reference policy, $$\pi_{\theta_{\text{ref}}}$$, using the following formula: $$ \text{Penalty} = \log \pi_{\theta}(\tau) - \log \pi_{\theta_{\text{ref}}}(\tau) $$ This value provides a straightforward measure of how much the policy has deviated.

Log-Probability Difference as a Policy Divergence Penalty

In practical applications, the policy divergence penalty, which is based on the log-probability of a trajectory, can be simplified. This approximation involves calculating the penalty using only the policy probabilities while ignoring the influence of the environment's dynamics, leading to a more computationally tractable measure.

Approximation of the Policy Divergence Penalty

The penalty term in PPO for language models quantifies the divergence between the current policy $\text{Pr}_{\theta}$ and a reference policy $\text{Pr}_{\theta_{\text{ref}}}$. It is defined as the difference in the log-probabilities of generating the response $\mathbf{y}$ given the prompt $\mathbf{x}$: $$ \text{Penalty} = \log \text{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) - \log \text{Pr}_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) $$ For autoregressive language models, this can be decomposed exactly into a sum over the tokens in the sequence: $$ \text{Penalty} = \sum_{t=1}^{T} \log \text{Pr}_{\theta}(y_t|\mathbf{x}, \mathbf{y}_{<t}) - \sum_{t=1}^{T} \log \text{Pr}_{\theta_{\text{ref}}}(y_t|\mathbf{x}, \mathbf{y}_{<t}) $$

Policy Divergence Penalty for Language Models

An optimization algorithm is updating a policy. For a specific trajectory, τ, the log-probability under the current policy, `log π_θ(τ)`, is -2.5. The log-probability under the reference policy, `log π_θ_ref(τ)`, is -4.0. Calculate the penalty used to measure the divergence between these two policies based on the difference in their log-probabilities.

Calculating Policy Divergence Penalty

In a reinforcement learning optimization step, you are evaluating a potential update to your policy. You analyze a specific trajectory of actions and states, τ, to decide if the update is within an acceptable range. Using the log-probability difference as a penalty, you gather the following data. Based on this data, calculate the penalty and determine whether the optimization process would be encouraged or discouraged from making this policy update. Justify your reasoning.

Interpreting Policy Divergence

In a policy optimization process, a penalty is used to measure the change between a current policy, $\pi_{\theta}$, and a reference policy, $\pi_{\theta_{\text{ref}}}$. The penalty is calculated for a specific sequence of actions and states (a trajectory, $\tau$) using the formula: 

$$ \text{Penalty} = \log \pi_{\theta}(\tau) - \log \pi_{\theta_{\text{ref}}}(\tau) $$ 

If the calculated penalty for a particular trajectory is a large positive value, what is the most accurate interpretation?

Learn Before

Related