Google

The practical application of Reinforcement Learning from Human Feedback (RLHF) follows a specific training order composed of three main stages. First, the models are initialized: the reward and value models often start from a pre-trained Large Language Model (LLM), while the reference model and target model (policy) are initialized from an instruction fine-tuned model. At this point, the reference model is fixed and will not be updated further. Second, human preference data is collected to train the reward model. Third, the value model and the policy are trained simultaneously using the optimized reward model. At each position in an output sequence, the value model is updated by minimizing the Mean Squared Error (MSE) of its value prediction, while the policy is updated by minimizing the Proximal Policy Optimization (PPO) loss.

Three-Stage Training Process of RLHF

The first step in the RLHF training process is to initialize the necessary models from existing ones. Typically, the reward and value models are initialized using a pre-trained Large Language Model. In contrast, the reference and target (policy) models are initialized from a model that has already undergone instruction fine-tuning. After this initialization, the reference model's parameters are fixed and are not updated further during the subsequent training stages.

Model Initialization Strategy in RLHF

Given the following scenario, identify which of the three training stages in the human feedback alignment process is the most likely source of the problem and explain your reasoning.

Diagnosing a Flawed Alignment Process

An AI development team is in the final stage of a three-part alignment process for their language model. They observe that the model's outputs are becoming increasingly nonsensical, even though the reward scores assigned during this final stage are consistently high. The team has already confirmed that the initial models were set up correctly and that the dataset of human preferences used in the second stage is high-quality. Based on this information, what is the most probable cause of the model

A team is developing a large language model aligned with human preferences using a reinforcement learning approach. Arrange the following key phases of their training pipeline into the correct chronological order.

Learn Before

Related