PPO Clipped Objective for Language Models
In the context of training language models with PPO, the clipped surrogate objective, denoted as , is calculated by summing over the generated tokens. For each token in the response , the objective considers the ratio of probabilities between the current policy and a reference policy . This ratio is clipped to prevent large policy updates and then multiplied by the advantage function . The formula is:
0
1
Tags
Ch.4 Alignment - Foundations of Large Language Models
Foundations of Large Language Models
Foundations of Large Language Models Course
Computing Sciences
Related
Incorporating Policy Divergence Penalty into the Clipped Surrogate Objective
A reinforcement learning agent is being trained using a utility function that incorporates an upper-bound clip on the policy probability ratio, defined as
min(ratio, 1+ε), whereεis a small positive constant. Consider two distinct actions taken during an episode:- Action A: Has a large positive advantage, and its probability ratio is
2.0. - Action B: Has a large negative advantage, and its probability ratio is
0.1.
Assuming
ε = 0.2, how does this specific clipping mecha- Action A: Has a large positive advantage, and its probability ratio is
A utility function that modifies the policy probability ratio
r_tusing the operationmin(r_t, 1+ε)is primarily intended to mitigate training instability caused by actions that are discovered to be substantially worse than the reference policy's actions (i.e., actions with a large negative advantage).Stabilizing Policy Gradient Training
Learn After
Overall PPO Objective Function for Language Models
During the training of a language model, the policy is updated based on a clipped objective function. Consider a single token generation step where the ratio of the current policy's probability to the reference policy's probability for a specific token is very large (e.g., 3.0), and the estimated advantage for generating this token is highly positive. The clipping range is set to [0.8, 1.2]. How does the clipping mechanism influence the calculation of the objective for this specific token?
Policy Update Analysis with Negative Advantage
Analysis of Clipping Mechanism based on Advantage Sign