Google

The cumulative reward score for an entire output token sequence, represented as $$r(\mathbf{x}, \mathbf{y})$$, is determined by calculating the sum of the individual reward scores from all its segmented parts. The formal equation for this aggregation is:
$$r(\mathbf{x}, \mathbf{y}) = \sum_{k=1}^{n_s} r(\mathbf{x}, \mathbf{y}, \bar{\mathbf{y}}_k)$$
In this formula, $$n_s$$ stands for the total number of segments the sequence is divided into, and $$r(\mathbf{x}, \mathbf{y}, \bar{\mathbf{y}}_k)$$ denotes the computed reward for the $$k$$-th segment. This total score is typically used to update and train the policy model as usual.

Total Reward as Sum of Segment-Based Scores

The total reward score, $r(\mathbf{x}, \mathbf{y})$, which is aggregated from the scores of individual segments of a generated output, is utilized as the primary reward signal in the standard training process for the policy model.

Application of Segment-Based Total Reward in Policy Training

An AI training team is evaluating a language model's response, which has been divided into three segments. Based on the provided segment scores, calculate the total reward for the entire response and explain why a response containing a significantly flawed segment can still achieve a positive overall score.

Analyzing Reward Model Behavior

A language model's output is divided into four segments. A reward model assigns scores to each segment. The scores for the first three segments are +1.2, -0.5, and +0.8. If the total reward for the entire output is calculated to be 2.0, what is the score for the fourth segment? Explain your calculation.

Calculating a Missing Segment Score

A language model generates a three-segment response to a user's prompt. A separate reward model evaluates each segment, considering the full context of the prompt and the complete response, and assigns the following scores: Segment 1: 0.8, Segment 2: -0.3, Segment 3: 0.5. According to the principle of aggregating segment-based scores, what is the total reward for the entire generated response?

Learn Before

Related