Google

An alternative method for evaluating a reasoning path involves using the log-probabilities generated by the step-level classification model. Instead of simply counting the number of steps deemed 'correct', this approach aggregates the log-probabilities to form the total reward for the entire path, providing a more nuanced score.

Log-Probability-Based Reward for Reasoning Paths

The reward for a reasoning path is calculated by summing the log-probabilities of each step being classified as 'correct' by the reward model. This method provides a more granular score than simply counting correct steps. The formula is:

$$r(\mathbf{x}, \mathbf{y}) = \sum_{k=1}^{n_s} \log \Pr(\text{correct}|\mathbf{x}, \bar{\mathbf{y}}_{\le k})$$

where:
- $$r(\mathbf{x}, \mathbf{y})$$ is the total reward for the reasoning path $$\mathbf{y}$$ given the input $$\mathbf{x}$$.
- $$n_s$$ is the total number of steps in the path.
- $$\log \Pr(\text{correct}|\mathbf{x}, \bar{\mathbf{y}}_{\le k})$$ is the log-probability of the 'correct' label for step $$k$$, as generated by the reward model, given the input and the reasoning path up to that step.

The reward score $$r(\mathbf{x},\mathbf{y})$$ can then be used to train the policy in RLHF.

Formula for Log-Probability-Based Reward for Reasoning Paths

You are evaluating two reasoning paths generated by an AI for the same problem. A scoring system calculates the total reward for a path by summing the log-probabilities of each step being 'correct', as determined by an automated checker. Based on the step-level probabilities provided in the case study, which path would this scoring system prefer? Justify your answer by explaining how this scoring method treats paths with varying levels of certainty across their steps.

Evaluating AI Reasoning Strategies

Imagine a system that evaluates multi-step reasoning by summing the log-probabilities of each step being 'correct'. Consider two reasoning paths, Path X and Path Y, that both solve a problem and have the same number of steps. In Path X, every step is evaluated with a moderately high probability of being correct (e.g., 80% for each step). In Path Y, most steps are evaluated with near-certainty of being correct (e.g., 99%), but one step is evaluated with a very low probability of being correct (e.g., 10%). Which path is likely to receive a higher total score from this evaluation system, and why? Explain your reasoning based on the properties of logarithms.

Nuanced Evaluation of Reasoning Paths

Consider two methods for scoring a multi-step reasoning process generated by an AI. Both methods use an underlying model that, for each step, outputs a probability that the step is 'correct'.

*   **Method A:** Assigns a score of +1 to each step where the probability of being 'correct' is greater than 0.5. The total score is the sum of these step scores.
*   **Method B:** Calculates the total score by summing the logarithm of the 'correct' probability for every step in the process.

Now, analyze

Learn Before

Related