Google

The penalty term in PPO for language models quantifies the divergence between the current policy $\text{Pr}_{\theta}$ and a reference policy $\text{Pr}_{\theta_{\text{ref}}}$. It is defined as the difference in the log-probabilities of generating the response $\mathbf{y}$ given the prompt $\mathbf{x}$: $$ \text{Penalty} = \log \text{Pr}_{\theta}(\mathbf{y}|\mathbf{x}) - \log \text{Pr}_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) $$ For autoregressive language models, this can be decomposed exactly into a sum over the tokens in the sequence: $$ \text{Penalty} = \sum_{t=1}^{T} \log \text{Pr}_{\theta}(y_t|\mathbf{x}, \mathbf{y}_{<t}) - \sum_{t=1}^{T} \log \text{Pr}_{\theta_{\text{ref}}}(y_t|\mathbf{x}, \mathbf{y}_{<t}) $$

Policy Divergence Penalty for Language Models

Based on the provided case study, analyze the relationship between the observed model behavior and the consistently low policy divergence penalty. What is the most likely reason for the model's failure to generate novel, helpful responses despite high reward scores?

Diagnosing Training Issues with Policy Divergence

The overall objective function for training language models with Proximal Policy Optimization (PPO), denoted as $$U$$, combines the clipped surrogate objective with a policy divergence penalty. This composite objective is formulated as: $$ U(\mathbf{x}, \mathbf{y}; \theta) = U_{\text{ppo-clip}}(\mathbf{x}, \mathbf{y}; \theta) - \beta \text{Penalty} $$ In this equation, $$U_{\text{ppo-clip}}$$ represents the PPO clipped objective, while the $$\text{Penalty}$$ term quantifies the divergence from a reference policy. The hyperparameter $$\beta$$ serves as a coefficient to control the magnitude of this penalty.

Overall PPO Objective Function for Language Models

Explain the primary purpose of using a policy divergence penalty when fine-tuning a language model. In your explanation, describe how this penalty is calculated at the token level for an autoregressive model and what a consistently high positive penalty value suggests about the relationship between the current and reference policies.

Interpreting the Policy Divergence Penalty

An autoregressive language model is generating the two-token response 'Good day' given a prompt. The table below shows the per-token log-probabilities from the current policy being trained ($	ext{Pr}_{	heta}$) and a fixed reference policy ($	ext{Pr}_{	heta_{	ext{ref}}}$). The policy divergence penalty is calculated as the sum of the differences between the log-probabilities of the current and reference policies for each token.

| Token | $\log \text{Pr}_{\theta}(y_t|\dots)$ | $\log \text{Pr}_{\t

Learn Before

Related