Google

In outcome-based approaches to LLM fine-tuning, supervision is applied exclusively when the end result is verified. The model is optimized to maximize some form of reward, such as $${}r(\mathbf{x},\mathbf{y})$$, based on the final outcome. This represents a standard methodology for learning from human feedback, where evaluation focuses on the complete input-output sequence rather than the intermediate steps.

Outcome-based Approaches for LLM Fine-Tuning

Reward models are often used to evaluate an entire sequence, providing feedback based solely on the final outcome. While this outcome-based approach is effective for tasks where correctness is easily verifiable, such as solving a mathematical expression, it proves insufficient for problems that demand complex reasoning. For these tasks, merely knowing if the final answer is right or wrong does not help the model learn the intermediate steps or logical process required to arrive at the correct solution, much like a student who only sees the final answer to a difficult problem cannot identify their mistakes in the reasoning process.

Limitations of Outcome-Based Rewards for Entire Sequences

A legal tech company is developing a system to summarize lengthy contracts. They are considering two different methods for fine-tuning their language model. Analyze both methods described in the case study and determine which one represents an outcome-based approach. Justify your answer by explaining how the chosen method's supervision and reward structure align with the principles of this approach.

Identifying Fine-Tuning Methodologies

A research team is fine-tuning a language model to summarize news articles. The model is trained to first extract key sentences from the article and then generate a summary based on them. The fine-tuning process provides a reward signal based on two criteria: (1) the factual accuracy of the final summary compared to a human-written one, and (2) whether the intermediate sentences it extracted match a predefined list of 'golden' key sentences. Based on the principles of fine-tuning, explain why this approach is *not* a purely outcome-based method.

Analyzing Fine-Tuning Methodologies

A team is fine-tuning a language model to act as a programming assistant that writes code. For each programming problem, the model generates a block of code. The fine-tuning process involves running the generated code against a set of predefined tests. If the code passes all the tests, the model receives a high reward. If it fails any test, it receives a low reward. The structure, style, or efficiency of the code itself is not directly evaluated for the reward signal. Which principle of model fi

Learn Before

Related