Google

The starting point for Reinforcement Learning from Human Feedback (RLHF) is an initial policy, which is an LLM that has already undergone pre-training and instruction fine-tuning. This model is considered the version that would be deployed to interact with users and respond to their requests, forming the baseline for further alignment.

Establishing the Initial Policy in RLHF

Based on the following scenario, analyze the primary issue with the company's starting point for their human feedback process and recommend a corrective action.

Diagnosing an Inefficient Alignment Process

In a process designed to align a language model with human preferences using feedback, what are the two essential training stages the model should have completed *before* the feedback process begins? Briefly explain the purpose of each stage in preparing the model to serve as a functional baseline.

Characteristics of the Starting Model for Alignment

A development team is preparing to use a human feedback-driven process to improve an AI's helpfulness and safety. They have two candidate models to use as their starting point: 

Model A: A raw, pre-trained model that is very good at predicting the next word in a sentence but has not been specifically trained to follow user commands.

Model B: A model that has been pre-trained and then further fine-tuned on a dataset of instructions and high-quality answers, making it proficient at following use

Learn Before

Related