Google

When verifiers are used to score a model's reasoning, the evaluation can be categorized into two main approaches: outcome-based, which assesses the entire reasoning path, and process-based, which evaluates each individual step.

Classification of Verification Approaches

Outcome-based verification is an approach where a verifier is used to assign a score to the entire reasoning path generated for a problem. The evaluation focuses on the overall result or the complete chain of logic as a single unit.

Outcome-Based Verification

Process-based verification is an approach where a verifier assesses each individual step within a reasoning path. This method allows for a more granular evaluation by scoring the correctness of each part of the reasoning process separately.

Process-Based Verification

Verification approaches can be distinguished by the stage at which evaluation occurs. Solution-level verification methods, including self-refinement, assess and select from a set of fully generated candidate solutions. In contrast, step-level verification integrates the evaluation process directly into the reasoning path's construction, enabling real-time guidance or pruning at intermediate steps before a complete solution is formed.

Comparison of Solution-Level and Step-Level Verification

A team is building a system to verify the solutions to multi-step mathematical problems generated by a language model. Their verifier works by taking a complete, final solution, plugging the final answer back into the original problem statement, and checking if it holds true. The verifier does not inspect the intermediate calculations, only the final result. Which classification best describes this verification approach?

Analyze the two proposed verification methods in the case study below. For each method, identify whether it is an outcome-based or a process-based approach and explain one key advantage of using that method in this specific educational context.

Choosing a Verification Strategy for an AI Tutor

A team is developing verifiers to score the reasoning of a language model in different tasks. Match each verifier's description to the classification that best describes its approach.

Learn Before

Related