Google

The formula $$\text{Pr}(\mathbf{y}_{k_1} \succ \mathbf{y}_{k_2} | \mathbf{x})$$ represents the conditional probability that an outcome $$\mathbf{y}_{k_1}$$ is preferred to, or ranked higher than, another outcome $$\mathbf{y}_{k_2}$$, given a specific input context $$\mathbf{x}$$. The symbol $$\succ$$ denotes preference or a 'greater than' relationship in this context. This type of expression is fundamental in preference learning and ranking models, where the goal is to learn a function that can predict the relative order of items based on input features.

Conditional Probability of Pairwise Preference

The probability that one output, $$\mathbf{y}_a$$, is preferred over another, $$\mathbf{y}_b$$, given an input $$\mathbf{x}$$, can be defined using the Bradley-Terry model. This approach uses a reward function, $$r(\mathbf{x}, \mathbf{y})$$, to assign a score to each output. The preference probability is calculated as the sigmoid of the difference between their scores:
$$ \Pr(\mathbf{y}_a \succ \mathbf{y}_b | \mathbf{x}) = \mathrm{Sigmoid}(r(\mathbf{x}, \mathbf{y}_a) - r(\mathbf{x}, \mathbf{y}_b)) $$
This model maps the reward difference to a valid probability.

Bradley-Terry Model for Pairwise Preference Probability

Based on the scenario described below, use the notation `Pr(y_k1 ≻ y_k2 | x)` to write the specific conditional probability that the training process is trying to model. Clearly define what `x`, `y_k1`, `y_k2`, and the `≻` symbol represent in this context.

Ranking Chatbot Responses

The probability that a response $$\mathbf{y}_a$$ is preferred over another response $$\mathbf{y}_b$$ given an input $$\mathbf{x}$$ is modeled using a learned reward function $$r(\mathbf{x}, \mathbf{y})$$. This is achieved by applying the sigmoid function to the difference between the reward scores of the two responses, as specified by the Bradley-Terry model. The formula is: $$\text{Pr}(\mathbf{y}_a \succ \mathbf{y}_b|\mathbf{x}) = \text{Sigmoid}(r(\mathbf{x}, \mathbf{y}_a) - r(\mathbf{x}, \mathbf{y}_b))$$. This is a foundational component for training reward models in RLHF.

Modeling Pairwise Preference Probability with a Reward Function

A research team is developing a model to summarize news articles. For a given article, the model generates two different summaries. The team uses human feedback to determine which summary is better. The probability that one summary is preferred over another is represented by the expression:  $\text{Pr}(\mathbf{y}_A \succ \mathbf{y}_B | \mathbf{x})$. 

In the context of this specific scenario, explain what each of the following components represents: 
1. $\mathbf{x}$ 
2. $\mathbf{y}_A$ 
3. $\mathbf{y}_B$ 
4. The $\succ$ symbol

Interpreting Preference Probability Notation

A user provides the prompt, denoted as 'x', 'Translate the phrase "hello world" into French.' to a language model. The model generates two responses: Response A ('y_A'), which is 'Bonjour le monde', and Response B ('y_B'), which is 'Salut monde'. A human evaluator indicates that Response A is a better translation than Response B. Which of the following expressions correctly represents the probability of this specific preference, given the user's prompt?

Learn Before

Related