Google

To generate a single vector representing an entire prompt-response sequence in a reward model, the sequence is processed from left to right using forced decoding. Because language modeling restricts each position to only accessing its left context, the output from the top-most Transformer layer at the first position cannot encapsulate the full sequence. To resolve this, a special symbol, such as $$\langle \backslash s \rangle$$, is appended to the end of the sequence. The corresponding output vector from the Transformer layer stack at this final position is then used as the comprehensive representation of the entire sequence.

Sequence Representation for Reward Calculation in RLHF

Once a comprehensive vector representation of the concatenated input sequence is obtained from the Transformer layer stack, a final output layer, such as a linear transformation layer, is built directly on top of this representation. This layer translates the vector into a final scalar reward score, denoted by $$R(\mathrm{seq}_k)$$ or $$R(\mathbf{x},\mathbf{y}_k)$$, representing the evaluation for the given prompt $$\mathbf{x}$$ and output $$\mathbf{y}_k$$.

Final Reward Score Calculation in RLHF

Analyze the following implementation choice for a reward model. Explain the primary flaw in this approach given the typical architecture of the underlying model, and describe the standard, more effective method.

Reward Model Implementation Analysis

A text-processing model operates sequentially from left to right, where the representation for any given word is calculated based only on the words that came before it. To create a single summary vector for an entire sentence, a colleague suggests using the model's output vector corresponding to the very first word. Analyze this suggestion and explain its primary flaw.

Critique of a Sequence Representation Method

A team is building a system to evaluate text sequences. They use a model that processes text one token at a time from left to right, where the output for any given token is influenced only by the tokens that came before it. To obtain a single vector that represents an entire input sequence for scoring, which of the following strategies is most appropriate for this type of model?

Learn Before

Related