Google

In the context of RLHF, the Bradley-Terry model is adapted to formally express the probability that a given model output, $y_a$, is preferred over another, $y_b$. This application of the model provides a mathematical framework for quantifying human preferences.

Modeling Preference Probability with the Bradley-Terry Model in RLHF

Based on the principles of using a probabilistic model for pairwise comparisons to quantify human preferences, what can you infer about the *difference* in the underlying quality scores between Snippet A and Snippet B, compared to the difference between Snippet C and Snippet D? Explain your reasoning.

Interpreting Preference Data for AI Training

In a system designed to improve a language model based on human feedback, evaluators are consistently asked to choose the better of two generated responses for a given prompt. Explain why a probabilistic model originally designed for pairwise comparisons is a suitable mathematical framework for quantifying these human preferences.

Justifying the Choice of a Preference Model

The Bradley-Terry model can be used to express the probability of one item, $$\mathbf{y}_a$$, being preferred over another, $$\mathbf{y}_b$$, given a context $$\mathbf{x}$$. The model starts by defining this probability as the ratio of the exponentiated reward score of the preferred item to the sum of the exponentiated scores of both items. This formulation can be algebraically simplified to the sigmoid function of the difference between the two reward scores. The derivation proceeds as follows:

$$ \Pr(\mathbf{y}_a \succ \mathbf{y}_b | \mathbf{x}) = \frac{e^{r(\mathbf{x}, \mathbf{y}_a)}}{e^{r(\mathbf{x}, \mathbf{y}_a)} + e^{r(\mathbf{x}, \mathbf{y}_b)}} = \frac{e^{r(\mathbf{x}, \mathbf{y}_a) - r(\mathbf{x}, \mathbf{y}_b)}}{e^{r(\mathbf{x}, \mathbf{y}_a) - r(\mathbf{x}, \mathbf{y}_b)} + 1} = \mathrm{Sigmoid}(r(\mathbf{x}, \mathbf{y}_a) - r(\mathbf{x}, \mathbf{y}_b)) $$

This derivation shows how a model based on exponentiated scores is equivalent to modeling the preference probability using the sigmoid of the score difference.

Derivation of the Bradley-Terry Preference Formula

A team is training a language model using human feedback. For a given prompt, the model generates two distinct responses, Response A and Response B. A human evaluator indicates a preference for Response A over Response B. To learn from this feedback, the system uses a probabilistic model designed for pairwise comparisons to quantify this preference. Which statement best analyzes how this model represents the human's choice?

Learn Before

Related