Google

The combined reward, $$r_{\mathrm{combine}}$$, is calculated by taking a weighted average of the outputs from $$K$$ different reward models. Each individual reward model's output, $$r_k(\mathbf{x}, \mathbf{y})$$, is multiplied by a weight $$w_k$$. These products are summed up over all $$K$$ models, and the result is normalized by dividing by $$K$$. The formula is expressed as:

$$r_{\mathrm{combine}} = \frac{1}{K} \sum_{k=1}^{K} w_k \cdot r_k(\mathbf{x}, \mathbf{y})$$

Combined Reward Formula

The aggregated reward score, which is calculated by combining the outputs from multiple reward models, is used as the primary feedback signal to guide and supervise the training of a policy model.

Using Combined Reward for Policy Supervision

Based on the scenario provided, how should the development team adjust the weights assigned to their 'Helpfulness' and 'Safety' reward models to encourage the chatbot to produce safer responses? Explain the reasoning behind your proposed adjustment.

Adjusting Chatbot Behavior via Reward Model Weighting

In the formula for a combined reward score, $r_{\text{combine}} = \frac{1}{K} \sum_{k=1}^{K} w_k \cdot r_k(\mathbf{x}, \mathbf{y})$, explain the distinct purpose of the weight term ($w_k$) compared to the normalization factor ($\frac{1}{K}$). Why is it important to have both in the calculation?

Component Analysis of the Combined Reward Formula

An AI alignment team is evaluating a language model's response using three distinct reward models: Helpfulness, Harmlessness, and Conciseness. For a specific response, the models provide the following scores and are assigned the following weights:

- Helpfulness: Score = 8.0, Weight = 2.0
- Harmlessness: Score = 9.0, Weight = 3.0
- Conciseness: Score = 6.0, Weight = 1.0

Using the weighted average formula for combining rewards, what is the final aggregated reward score for this response? (Assume

Learn Before

Related