Google

To transform a function that is not a probability distribution into one, it can be treated as an unnormalized probability. For instance, a term like `exp(r(x, y))` within an objective function can be converted into a valid, normalized probability distribution by dividing it by a normalization factor. This factor is calculated by summing or integrating the unnormalized function over its entire domain, ensuring the resulting distribution sums to one.

Normalizing a Function to Create a Probability Distribution

The normalization factor, often denoted as $Z(\mathbf{x})$, is a crucial component for converting an unnormalized, reward-weighted function into a valid probability distribution. It is calculated by summing or integrating the product of a reference policy, $\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x})$, and an exponentiated, scaled reward, $\exp(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y}))$, over the entire domain of possible outputs $\mathbf{y}$. The formula is: $$ Z(\mathbf{x}) = \sum_{\mathbf{y}} \pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right) $$ By dividing the unnormalized function by this factor, the resulting distribution is guaranteed to sum to one.

Normalization Factor for a Reward-Weighted Policy

A language model has evaluated three possible next words in a sequence and assigned them the following unnormalized scores: 'excellent' (score: 4.5), 'good' (score: 3.0), and 'okay' (score: 1.5). Explain the process required to convert these scores into a valid probability distribution and state why this conversion is necessary for the model to make a probabilistic choice.

From Model Scores to Probabilities

A machine learning model is designed to predict which of three categories (X, Y, Z) an input belongs to. Instead of probabilities, the model outputs raw, unnormalized scores. For a specific input, the model produces the following scores: Score(X) = 8, Score(Y) = 10, Score(Z) = 2. To use these outputs in a downstream process that requires a valid probability distribution, these scores must be transformed. Based on this scenario, what is the calculated probability for category Y after the correct transformation is applied?

Converting Model Scores to Probabilities

A function assigns the following unnormalized scores to three possible discrete outcomes: `score(A) = 12`, `score(B) = 7`, and `score(C) = 1`. To transform these scores into a valid probability distribution `P(outcome)`, each score must be divided by a normalization factor calculated from the sum of all scores. What is the resulting probability for outcome B, `P(B)`?

Learn Before

Related