Google

In reinforcement learning, particularly for language model alignment, an optimal policy $\pi^*(y|x)$ is defined to be proportional to the product of a reference policy $\pi_{\theta_{\text{ref}}}(y|x)$ and the exponential of a scaled reward function $r(x, y)$. The unnormalized probability for an output $\mathbf{y}$ given an input $\mathbf{x}$ is given by the expression: $$\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y}))$$ This formulation effectively re-weights the probabilities from the reference policy, increasing the likelihood of outputs that receive a higher reward. The parameter $\beta$ acts as a temperature, controlling the influence of the reward function on the final policy. The full policy $\pi^*$ is obtained by normalizing this expression over all possible outputs $\mathbf{y}$.

Optimal Policy as a Product of Reference Policy and Exponentiated Reward

After training with the described method, the team observes that the chatbot's responses, while scoring high on 'helpfulness', have become grammatically awkward and unnatural. Based on the formulation described, what is the most likely cause of this degradation in language quality?

Diagnosing Language Model Alignment Issues

A language model's policy is being shaped using a reward function, $r(\mathbf{x}, \mathbf{y})$. Instead of defining the policy as being directly proportional to only the exponentiated reward, it is instead defined as proportional to the product of a reference policy and the exponentiated reward: $\pi_{\text{ref}}(\mathbf{y}|\mathbf{x}) \exp(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y}))$. Explain the primary benefit of including the $\pi_{\text{ref}}(\mathbf{y}|\mathbf{x})$ term in this formulation.

Justifying the Reference Policy

A team is refining a text-generation model. The final probability of a generated text sequence is proportional to the product of its probability from an initial base model and an exponentiated reward score. The reward's influence is controlled by a scaling parameter, β, in the exponent, where a smaller β gives the reward more weight. The team observes that when they significantly decrease the value of β, the model's outputs become more repetitive and sometimes nonsensical, even though they achie

Learn Before

Related