Google

A straightforward technique for modeling a listwise preference ordering is to formulate a loss function by aggregating the pairwise comparison losses. This involves calculating and summing the loss for every possible pair of outputs within the ranked list provided by human annotators.

Listwise Loss from Accumulated Pairwise Comparisons

The listwise loss, formulated by accumulating pairwise comparisons, is defined as the negative expected log-probability over all distinct pairs within a ranked list. The formula is:

$$ \mathcal{L}_{\mathrm{list}} = -\mathbb{E}_{\substack{(\mathbf{x},Y) \sim \mathcal{D}_r}} \Big[ \frac{1}{N(N-1)} \sum_{\substack{\mathbf{y}_a \in Y, \mathbf{y}_b \in Y \\ \mathbf{y}_a \ne \mathbf{y}_b}} \log \mathrm{Pr}(\mathbf{y}_a \succ \mathbf{y}_b | \mathbf{x}) \Big] $$

Here, $$ \mathcal{L}_{\mathrm{list}} $$ denotes the listwise loss. The expectation $$ \mathbb{E} $$ is taken over samples $$ (\mathbf{x},Y) $$ from the preference dataset $$ \mathcal{D}_r $$, where $$ Y $$ is a ranked list of $$ N $$ outputs for a given prompt $$ \mathbf{x} $$. The summation aggregates the log conditional probability of the preference for every ordered pair of distinct outputs $$ (\mathbf{y}_a, \mathbf{y}_b) $$ in the list $$ Y $$. The term $$ \frac{1}{N(N-1)} $$ normalizes the sum over the total number of possible ordered pairs.

Listwise Loss Formula from Accumulated Pairwise Comparisons

A common method for training a preference model is to take a ranked list of responses and break it down into all its constituent pairwise preferences. For example, the ranking A > B > C is decomposed into three preferences: A is preferred over B, A is preferred over C, and B is preferred over C. Following this method, if a human annotator provides a ranked list of 5 distinct responses, how many individual pairwise preferences will be extracted from this single list for the loss calculation?

Decomposing a Ranked List into Pairwise Preferences

A preference model is being trained using a loss function that is calculated by summing the losses of all individual pairwise comparisons derived from a ranked list. Based on the human preference and the two model predictions below, which model would incur a higher training loss for this specific example? Justify your reasoning by explaining how the number of incorrect pairwise comparisons contributes to the total loss.

Evaluating Preference Model Performance with Listwise Loss

A human annotator is given four model-generated responses (A, B, C, D) to a prompt and ranks them in order of preference from best to worst as: C > A > D > B. To train a preference model, a loss function is calculated by summing the individual losses for every pairwise comparison implied by this ranking. Which of the following sets represents all the pairwise preferences that would be used in this loss calculation?

Learn Before

Related