Google

The probability that a response $$\mathbf{y}_a$$ is preferred over another response $$\mathbf{y}_b$$ given an input $$\mathbf{x}$$ is modeled using a learned reward function $$r(\mathbf{x}, \mathbf{y})$$. This is achieved by applying the sigmoid function to the difference between the reward scores of the two responses, as specified by the Bradley-Terry model. The formula is: $$\text{Pr}(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x}) = \text{Sigmoid}(r(\mathbf{x}, \mathbf{y}_a) - r(\mathbf{x}, \mathbf{y}_b))$$. This is a foundational component for training reward models in RLHF.

Modeling Pairwise Preference Probability with a Reward Function

The listwise loss, formulated by accumulating pairwise comparisons, is defined as the negative expected log-probability over all distinct pairs within a ranked list. The formula is:

$$ \mathcal{L}_{\mathrm{list}} = -\mathbb{E}_{\substack{(\mathbf{x},Y) \sim \mathcal{D}_r}} \Big[ \frac{1}{N(N-1)} \sum_{\substack{\mathbf{y}_a \in Y, \mathbf{y}_b \in Y \\ \mathbf{y}_a \ne \mathbf{y}_b}} \log \mathrm{Pr}(\mathbf{y}_a \succ \mathbf{y}_b | \mathbf{x}) \Big] $$

Here, $$ \mathcal{L}_{\mathrm{list}} $$ denotes the listwise loss. The expectation $$ \mathbb{E} $$ is taken over samples $$ (\mathbf{x},Y) $$ from the preference dataset $$ \mathcal{D}_r $$, where $$ Y $$ is a ranked list of $$ N $$ outputs for a given prompt $$ \mathbf{x} $$. The summation aggregates the log conditional probability of the preference for every ordered pair of distinct outputs $$ (\mathbf{y}_a, \mathbf{y}_b) $$ in the list $$ Y $$. The term $$ \frac{1}{N(N-1)} $$ normalizes the sum over the total number of possible ordered pairs.

Listwise Loss Formula from Accumulated Pairwise Comparisons

The theoretical reward model loss, defined as an expectation, is practically implemented as an empirical loss by averaging over the collected preference dataset $$\mathcal{D}_r$$. This is based on the assumption that the data points are sampled uniformly. The formula for this empirical loss is: $$\mathcal{L}_r(\phi) = -\frac{1}{|\mathcal{D}_r|} \sum_{(\mathbf{x},\mathbf{y}_a,\mathbf{y}_b)\in\mathcal{D}_r} \log \mathrm{Pr}_{\phi}(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x})$$. Here, $$|\mathcal{D}_r|$$ represents the total number of preference pairs in the dataset.

Empirical Reward Model Loss Formula

The pair-wise ranking loss function for a reward model with parameters $$\omega$$ can be formulated by summing over samples from the preference dataset $$\mathcal{D}_r$$. The expected loss incorporates the probability $$\Pr(\mathbf{x})$$ of drawing an input, and the conditional probability $$\Pr(\mathbf{y}_{k_1} \succ \mathbf{y}_{k_2} | \mathbf{x})$$ of drawing the preferred output pair. Assuming a uniform distribution over $$K$$ model inputs involved in sampling, the formula simplifies to:

$$\begin{aligned} \mathrm{Loss}_{\omega}(\mathcal{D}_r) &= -\sum \Pr(\mathbf{x}) \cdot \Pr(\mathbf{y}_{k_1} \succ \mathbf{y}_{k_2} | \mathbf{x}) \cdot \log(\mathrm{Sigmoid}(R_{\omega}(\mathbf{x}, \mathbf{y}_{k_1}) - R_{\omega}(\mathbf{x}, \mathbf{y}_{k_2}))) \\ &= -\frac{1}{K} \sum \Pr(\mathbf{y}_{k_1} \succ \mathbf{y}_{k_2} | \mathbf{x}) \cdot \log(\mathrm{Sigmoid}(R_{\omega}(\mathbf{x}, \mathbf{y}_{k_1}) - R_{\omega}(\mathbf{x}, \mathbf{y}_{k_2}))) \end{aligned}$$

Empirical Formulation of Pair-wise Ranking Loss

An AI development team is refining a language model using a reward function, `r(prompt, response)`, which assigns a quality score to a given response. The probability that a preferred response (`y_a`) is chosen over another response (`y_b`) is determined by the formula: `Pr(y_a ≻ y_b | prompt) = Sigmoid(r(prompt, y_a) - r(prompt, y_b))`, where `Sigmoid(z) = 1 / (1 + e^-z)`. Given the scenario below, calculate the probability that Response A is preferred over Response B. Show your calculation and provide the final probability rounded to two decimal places.

Preference Probability Calculation

A system models the probability that a response `y_a` is preferred over `y_b` for a given input `x` using the formula: `P(y_a > y_b | x) = Sigmoid(r(x, y_a) - r(x, y_b))`, where `r(x, y)` is a learned quality score. If a new reward function `r'(x, y)` is created by adding the same constant value `C` to all original scores (i.e., `r'(x, y) = r(x, y) + C`), how does this change affect the calculated preference probabilities? Explain your reasoning.

Invariance of Preference Probability

A system learns a function, `r(input, response)`, that assigns a numerical score indicating the quality of a `response` for a given `input`. The probability that response `Y_a` is preferred over response `Y_b` is then calculated using the formula: `Probability = Sigmoid(r(input, Y_a) - r(input, Y_b))`, where `Sigmoid(z) = 1 / (1 + e^-z)`. Given the following scenarios for a single input, which one presents a logical inconsistency between the assigned scores and the resulting preference probabili

Learn Before

Related