Google

Reward models can be designed to evaluate entire input-output sequences, a method that is highly effective for tasks where correctness can be verified by simply checking the final result. This approach focuses on the outcome rather than the process used to achieve it.

Outcome-Based Reward Models

A practical application of an outcome-based reward model is in evaluating mathematical calculations. In this scenario, the model provides positive feedback for a correct final answer and negative feedback for an incorrect one, without assessing the intermediate steps of the calculation.

Example of an Outcome-Based Reward Model in Mathematics

For tasks that require complex reasoning, reward models that only evaluate the correctness of the final output are insufficient for effective learning. This is because such outcome-based feedback does not provide information about errors made during the reasoning process, thus failing to guide the model on how to improve its problem-solving steps.

Insufficiency of Outcome-Based Rewards for Complex Reasoning

Evaluate the suitability of this proposed outcome-based reward model for *each* of the two capabilities. Justify your evaluation by explaining why the approach is or is not appropriate for each task.

Evaluating Reward Model Suitability

A developer is training a language model to generate short, engaging, and original marketing slogans. They decide to use a reward system that gives a high score to slogans that human raters find creative and a low score to those they find uninspired. This system does not analyze the intermediate steps the model took to generate the slogan. Explain why this focus on the final output is a particularly effective strategy for this specific training goal.

Reward Model Suitability for a Creative Task

A company is training a language model to act as an automated assistant for processing loan applications. The model must follow a specific, legally-mandated, multi-step procedure to ensure fairness and compliance (e.g., checking credit history, verifying income, providing specific disclosures). The company decides to train the model using a system that provides a large positive reward only if the final loan decision (approve/deny) is correct based on the applicant's overall profile. What is the 

Learn Before

Related