Google

In the Direct Preference Optimization (DPO) training objective, the penalty term utilizes a reference policy, denoted as $$\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x})$$. This policy is typically a fixed, supervised fine-tuned version of the language model that serves as a stable baseline. The penalty term's function is to regularize the optimized policy, $$\pi_\theta$$, discouraging it from deviating significantly from this reference, which helps maintain response quality and training stability.

Reference Policy in DPO's Penalty Term

An AI research team is setting up a training process to align a large language model with human preferences. They are using an objective function that penalizes the model for deviating too far from a fixed, stable baseline policy. Two options are proposed for this baseline:

1. The original, pre-trained base model, before any instruction tuning.
2. A version of the model that has already undergone supervised fine-tuning (SFT) on high-quality conversational data.

Evaluate the potential consequences of choosing each option as the baseline. Which option is generally preferred, and why does this choice contribute to better training stability and final model quality?

Choosing a Baseline for Preference Alignment

Which of the two models—the original base pre-trained model or the fine-tuned dialogue model—should the team use as the fixed reference policy during the preference alignment phase? Justify your choice by explaining the role this reference policy plays in ensuring training stability and maintaining response quality.

Selecting a Baseline for Policy Optimization

Before explicitly deriving the Direct Preference Optimization (DPO) objective, the method conceptually assumes a foundational policy training objective where the quality of an output $$\mathbf{y}$$ given an input $$\mathbf{x}$$ is evaluated by a theoretical reward model $$r(\mathbf{x}, \mathbf{y})$$. The goal is to find optimal parameters $$\tilde{\theta}$$ by minimizing a loss term (the negative reward, $$-r(\mathbf{x}, \mathbf{y})$$) and a penalty term that regularizes the target policy $$\pi_\theta$$ against a reference policy $$\pi_{\theta_{\text{ref}}}$$. The assumed training objective is given by:

$$\tilde{\theta} = \arg \min_{\theta} \mathbb{E}_{\mathbf{x} \sim \mathcal{D}} \mathbb{E}_{\mathbf{y} \sim \pi_{\theta}(\cdot|\mathbf{x})} \big[ \underbrace{-r(\mathbf{x}, \mathbf{y})}_{\text{loss}} + \beta \underbrace{(\log \pi_{\theta}(\mathbf{y}|\mathbf{x}) - \log \pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}))}_{\text{penalty}} \big]$$

Conceptual Objective Function Assumed in DPO

An AI development team is refining a pre-trained language model using a dataset of human preferences, where each example consists of a prompt, a preferred response, and a rejected response. As training progresses, they notice that while the model is learning to generate responses that align with the preferences, its general language quality is deteriorating; it produces more repetitive and nonsensical text. What is the most probable cause of this issue related to the optimization objective's des

Learn Before

Related