Google

Process Reward Models (PRMs) differ from Outcome Reward Models (ORMs) in the granularity of their feedback. PRMs offer a detailed, fine-grained supervisory signal by evaluating every intermediate step of a reasoning process, whereas ORMs provide a coarser signal by assessing only the final outcome.

Comparison of Process and Outcome Reward Models

Analyze the provided case study. Which type of supervisory signal—one that evaluates only the final outcome or one that evaluates each intermediate step of the reasoning process—would be more effective at identifying and correcting the model's flawed reasoning? Explain why the chosen approach would succeed and the other would likely fail in this specific scenario.

Diagnosing Flawed Reasoning in Language Models

A team is training a language model to act as a programming assistant that generates code. They observe that the model sometimes produces functionally correct code (the outcome is right) but uses inefficient, non-standard, or difficult-to-maintain methods (the process is poor). Which of the following feedback strategies would be most effective at specifically improving the quality of the reasoning process, rather than just the correctness of the final output?

When evaluating an AI's responses to a math problem, an outcome-based approach treats any response with the correct final result as entirely correct, ignoring any flaws in the intermediate reasoning. In contrast, a process-based approach assesses the correctness of each individual step, allowing it to identify and account for mistakes made during the reasoning process even if the final answer is coincidentally correct. This detailed step-level evaluation is essential for effectively guiding the model's logic through reward modeling.

Learn Before

Related