Google

Once a comprehensive vector representation of the concatenated input sequence is obtained from the Transformer layer stack, a final output layer, such as a linear transformation layer, is built directly on top of this representation. This layer translates the vector into a final scalar reward score, denoted by $$R(\mathrm{seq}_k)$$ or $$R(\mathbf{x},\mathbf{y}_k)$$, representing the evaluation for the given prompt $$\mathbf{x}$$ and output $$\mathbf{y}_k$$.

Final Reward Score Calculation in RLHF

When a reward model is implemented using a pre-trained Large Language Model (LLM), the scalar reward score $$r(\mathbf{x}, \mathbf{y})$$ is computed by applying a linear transformation to the representation at the final position. The formula is: $$r(\mathbf{x}, \mathbf{y}) = \mathbf{h}_{\mathrm{last}} \mathbf{W}_r$$, where $$\mathbf{h}_{\mathrm{last}}$$ is a $$d$$-dimensional hidden state vector from the top-most Transformer layer corresponding to the last token of the concatenated prompt and response sequence, and $$\mathbf{W}_r$$ is a $$d \times 1$$ linear mapping matrix.

Reward Score Formula for LLM-based Reward Models

When applying an RLHF reward model, the reward signal is typically sparse. A reward score is generated only at the final position of the output sequence `y = y1...yn`. For all intermediate positions `t < n`, the model assigns a default value, such as 0. The actual, meaningful reward is calculated only upon completion of the entire sequence.

End-of-Sequence Reward Assignment in RLHF

Based on the function of a reward-scoring system, identify the fundamental design flaw in the final stage of this model and explain what the output *should* be and why.

Troubleshooting a Reward Model's Output

In a reward model, a network first produces a high-dimensional vector that represents the combined meaning of a prompt and its response. This vector is then passed through a final output layer to produce the reward score. Analyze the fundamental difference between the information encoded in the high-dimensional vector and the information represented by the final single numerical score. What is the purpose of this transformation?

From Representation to Reward

The formula $$r(\mathbf{x}, y) = \mathbf{h}_{\text{last}} \mathbf{W}_r$$ defines a reward function where the reward $$r$$ for a given prompt $$\mathbf{x}$$ and generated output $$y$$ is calculated as a linear function of the final hidden state, $$\mathbf{h}_{\text{last}}$$, of the language model that produced $$y$$. Here, $$\mathbf{h}_{\text{last}}$$ is the vector representation of the last token in the output sequence, and $$\mathbf{W}_r$$ is a learned weight matrix or vector that transforms this hidden state into a scalar reward value.

Reward Function as a Linear Transformation of the Last Hidden State

In a system designed to evaluate the quality of generated text, a complex neural network first processes a prompt and its corresponding response, ultimately producing a high-dimensional vector that captures the nuanced meaning and relationship between them. What is the essential final step required to convert this complex vector into a practical, usable evaluation, and what is the nature of its output?

Learn Before

Related