Google

To train a reward model in RLHF, preference data collected as a full ranking (listwise) must often be converted into a pairwise format. For instance, a single ranked list like `y1 ≻ y4 ≻ y2 ≻ y3` can be broken down into multiple pairwise comparisons, such as `(y1, y4)`, `(y1, y2)`, `(y1, y3)`, `(y4, y2)`, etc., where the first element is always preferred over the second. This process generates a dataset of `(prompt, preferred_response, rejected_response)` tuples, which is the standard input format for training the reward model using a pairwise ranking objective.

Converting Listwise Rankings to Pairwise Preferences for Reward Model Training

An annotation project collects human feedback by asking evaluators to rank 5 machine-generated responses to a given prompt, from best to worst. For training a model, each of these ranked lists is converted into a set of pairwise preferences, where every higher-ranked response is paired with every lower-ranked response. How many unique pairwise preference tuples are generated from a single ranked list of 5 responses? Explain your reasoning.

Calculating Pairwise Preference Dataset Size

Your task is to convert the single ranked list provided in the case study into a complete set of pairwise preference tuples. Each tuple should be in the format `(preferred_response, rejected_response)`. List all the valid tuples that can be generated from this ranking.

Generating a Pairwise Preference Dataset

A human evaluator has ranked four machine-generated responses to a prompt in order of preference, from best to worst, as follows: `Response D ≻ Response B ≻ Response A ≻ Response C`. To create a training dataset, this single ranked list is converted into a set of pairs, where the first element of each pair is preferred over the second. Which of the following pairs would be an *invalid* entry in the resulting dataset?

Learn Before

Related