Google

In policy optimization frameworks like RLHF, the target policy `π_θ` that is being learned is defined as being equal to an optimal distribution `π*`. This optimal distribution is created by re-weighting a reference policy `π_{θ_ref}` according to a reward function `r(x, y)`. The complete relationship is expressed by the formula: $$ \pi_{\theta}(\mathbf{y}|\mathbf{x}) = \pi^{*}(\mathbf{y}|\mathbf{x}) = \frac{\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)}{Z(\mathbf{x})} $$ This equation establishes the ideal policy that the model, parameterized by `θ`, aims to learn, balancing adherence to the reference model with maximization of the reward.

Target Policy as a Reward-Weighted Distribution

The RLHF policy optimization objective can be reformulated by substituting the definition of the optimal reward-weighted policy, $$\pi^{*}(\mathbf{y}|\mathbf{x})$$. This transforms the objective into one that minimizes the Kullback-Leibler (KL) divergence between the learned policy, $$\pi_{\theta}(\cdot|\mathbf{x})$$, and the optimal policy. The substitution yields: $$ \tilde{\theta} = \arg \min_{\theta} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \mathbb{E}_{\mathbf{y} \sim \pi_{\theta}(\cdot|\mathbf{x})} \left[ \log \pi_{\theta}(\mathbf{y}|\mathbf{x}) - \log \pi^{*}(\mathbf{y}|\mathbf{x}) - \log Z(\mathbf{x}) \right] $$ Because the expectation of the difference in log-probabilities over $$\pi_{\theta}$$ corresponds to the definition of KL divergence, this simplifies to: $$ \tilde{\theta} = \arg \min_{\theta} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \left[ \text{KL}(\pi_{\theta}(\cdot|\mathbf{x}) || \pi^{*}(\cdot|\mathbf{x})) - \log Z(\mathbf{x}) \right] $$ This derivation shows that the objective is equivalent to minimizing the KL divergence from the optimal policy, offset by the normalization term $$\log Z(\mathbf{x})$$.

Derivation of the KL Divergence Objective for Policy Optimization

In reinforcement learning from human feedback, a target policy `π*` is often defined by re-weighting a reference policy `π_ref` based on a reward `r(x, y)`, as shown in the equation: $$ \pi^{*}(\mathbf{y}|\mathbf{x}) = \frac{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)}{Z(\mathbf{x})} $$ A researcher proposes a simplification by removing the reference policy term entirely, creating a new target: $$ \pi_{\text{new}}^{*}(\mathbf{y}|\mathbf{x}) = \frac{\exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)}{Z'(\mathbf{x})} $$ Evaluate this proposed simplification. Discuss one potential advantage and two significant disadvantages of using this new formulation to guide a language model's learning process.

Critique of a Modified Policy Formulation

A language model's behavior is being refined. The goal is to create a new target distribution, π*, by adjusting a reference distribution, π_ref, based on a reward score, r(x, y). The relationship is defined by the formula: $$ \pi^{*}(\mathbf{y}|\mathbf{x}) = \frac{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)}{Z(\mathbf{x})} $$ where Z(x) is a normalization term ensuring the probabilities sum to 1. Given the scenario below, calculate the new probability for the output 'Bonjour' under the target distribution π*. Assume β = 1.0 for this calculation.

Calculating a Target Policy Distribution

A language model's behavior is guided by a target probability distribution, π*, which is defined by re-weighting a reference distribution, π_ref, based on a reward score, r(x, y). The relationship is given by the formula: $$ \pi^{*}(\mathbf{y}|\mathbf{x}) = \frac{\pi_{\text{ref}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)}{Z(\mathbf{x})} $$ In this formula, β is a positive scalar parameter. Analyze the effect of significantly *increasing* the value of β. Wh

Learn Before

Related