Google

A practical instance of listwise ranking in Reinforcement Learning from Human Feedback (RLHF) involves human experts ordering multiple model-generated outputs for a single prompt. For example, if a dataset sample contains a set of four generated outputs, denoted as $$\{\mathbf{y}_1, \mathbf{y}_2, \mathbf{y}_3, \mathbf{y}_4\}$$, an expert might order them from most preferred to least preferred. One possible ranking could be $$\mathbf{y}_2 \succ \mathbf{y}_3 \succ \mathbf{y}_1 \succ \mathbf{y}_4$$, which indicates that $$\mathbf{y}_2$$ is the best response, followed sequentially by $$\mathbf{y}_3$$, $$\mathbf{y}_1$$, and finally $$\mathbf{y}_4$$.

Example of Listwise Ranking in RLHF

A human evaluator was asked to rank three different responses (Response A, Response B, Response C) generated by a language model for the same prompt. Match each formal preference notation with the correct description of the evaluator's ranking.

A language model generates three responses to the prompt 'Explain gravity to a 5-year-old.'

- **Response A:** 'Gravity is a fundamental interaction which causes mutual attraction between all things with mass or energy.'
- **Response B:** 'Imagine the Earth is a giant bowling ball on a trampoline. Things like you and me are like little marbles that roll towards it. That's kind of like gravity!'
- **Response C:** 'Gravity is the force that keeps you on the ground so you don't float away into space. It's what makes an apple fall from a tree.'

An evaluator provides the following ranking, where '≻' signifies 'is preferred over':
Response B ≻ Response C ≻ Response A

Based on this ranking, what primary characteristic was the evaluator likely prioritizing? Briefly justify your answer by referencing the content of the responses.

Interpreting Evaluator Preferences

A team is refining a language model. For a single user prompt, the model generates four distinct responses: Response 1, Response 2, Response 3, and Response 4. A human evaluator is tasked with ordering these responses from best to worst. The evaluator concludes that Response 3 is the most helpful. Response 1 is the second-best, followed by Response 4. Response 2 is deemed the least helpful. Using the notation where '≻' signifies 'is preferred over,' which option correctly represents the evaluato

Learn Before

Related