Google

The process of calculating a reward score using a Transformer-based LLM is illustrated by a data flow. First, input prompt tokens ($$x_0, \dots, x_m$$) are concatenated with response tokens ($$y_1, \dots, y_n$$), followed by a special end-of-sequence token like `⟨EOS⟩`. This combined sequence is fed into a Transformer Decoder (LLM), which outputs a hidden state representation for each token position ($$h_{x0}, \dots, h_{\mathrm{last}}$$). The final hidden state, $$h_{\mathrm{last}}$$, corresponding to the `⟨EOS⟩` token, is selected to represent the entire sequence. This vector is then transformed by a linear mapping layer with weights $$W_r$$ to produce a single scalar value, which serves as the reward score.

Diagram of Reward Score Calculation using an LLM

You are tasked with designing a system that uses a language model to generate a single numerical score representing the quality of a given text response to a prompt. Arrange the following steps into the correct logical sequence for this process.

Based on the standard architecture for this type of system, what specific component is the engineer most likely missing in their implementation that would convert the final hidden state vector into a single scalar score, and what is its function?

Reward Model Implementation Debugging

An engineer is building a system to generate a single quality score for a model's text response based on an initial prompt. Their proposed process is as follows:

1. Concatenate the prompt tokens and the response tokens into a single sequence.
2. Feed this combined sequence into a language model to get a final-layer hidden state vector for every token.
3. Average all of these hidden state vectors to create a single representative vector.
4. Pass this single vector through a linear layer to produ

Learn Before

Related