Google

To evaluate a response, the reward model processes a sequence created by concatenating the original input prompt $$\mathbf{x}$$ with the generated output $$\mathbf{y}_k$$. This combined sequence is formally denoted as $$\mathrm{seq}_k = [\mathbf{x},\mathbf{y}_k]$$, which is then fed into the model from left to right to derive its representation.

Input Formulation for the RLHF Reward Model

To generate a single vector representing an entire prompt-response sequence in a reward model, the sequence is processed from left to right using forced decoding. Because language modeling restricts each position to only accessing its left context, the output from the top-most Transformer layer at the first position cannot encapsulate the full sequence. To resolve this, a special symbol, such as $$\langle \backslash s \rangle$$, is appended to the end of the sequence. The corresponding output vector from the Transformer layer stack at this final position is then used as the comprehensive representation of the entire sequence.

Sequence Representation for Reward Calculation in RLHF

A data scientist is preparing a single data instance to be evaluated by a reward model. The instance consists of a prompt, `x = 'Explain the water cycle in simple terms.'`, and a generated response, `y = 'Water evaporates from oceans, forms clouds, and then falls back as rain.'`. Describe the exact structure of the input sequence that should be fed into the reward model for this instance to ensure a meaningful evaluation.

Reward Model Input Preparation

A development team is building a model to score the quality of chatbot-generated answers. To simplify the process, they train their model using only the chatbot's answers as input, labeling each as 'high-quality' or 'low-quality'. After training, they observe that the model performs poorly. For example, it rates the answer 'I am not sure' as low-quality, even when the original user question was an unanswerable philosophical query. Analyze this situation and identify the fundamental flaw in the team's input formulation. Explain why this flaw leads to poor performance.

Debugging a Reward Model's Input Formulation

A team is developing a model to automatically assign a quality score to an AI-generated response. To do this, the model must be given some text as input. Which of the following best explains why the model should be given the original prompt concatenated with the AI's response, instead of just the AI's response alone?

Learn Before

Related