Google

The formula $$\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp\left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)$$ represents a method for adjusting a probability distribution from a reference model, denoted by $$\pi_{\theta_{\text{ref}}}$$. The term $$\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x})$$ is the base probability of generating output $$\mathbf{y}$$ from input $$\mathbf{x}$$ according to the reference model parameterized by $$\theta_{\text{ref}}$$. This probability is then scaled by the exponential of a reward function $$r(\mathbf{x}, \mathbf{y})$$, which is itself scaled by an inverse temperature parameter, $$\frac{1}{\beta}$$. The temperature $$\beta$$ controls the extent to which the reward influences the final probability, with smaller values of $$\beta$$ amplifying the effect of the reward.

Re-weighting a Reference Probability Distribution with a Scaled Reward

Based on the case study below, analyze the engineer's problem. For each application (Medical Q&A and Marketing Slogans), determine whether a high or low value for the parameter β would be more appropriate and explain your reasoning by referencing how it impacts the overall formula.

Tuning a Generative Model for Different Tasks

An AI text generation model adjusts its output probabilities using the formula: Final_Prob = Ref_Prob * exp((1/β) * Reward), where 'Ref_Prob' is the initial probability from a base model, 'Reward' is a score for a specific quality (e.g., factual accuracy), and 'β' is a positive temperature parameter. A developer decreases the value of 'β' and observes that the model's outputs now adhere much more strictly to the rewarded quality, but have become less diverse and creative. Explain the mathematical reason for this change in the model's behavior.

Effect of Temperature Scaling on a Reward-Modified Distribution

A reward-weighted probability distribution, denoted as $$\pi^{*}(\mathbf{y}|\mathbf{x})$$, is a new distribution created by modifying a reference distribution, $$\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x})$$, based on a reward signal, $$r(\mathbf{x}, \mathbf{y})$$. The reference distribution's probability for each output $$\mathbf{y}$$ is scaled by an exponential factor of the reward, controlled by a temperature or inverse scaling parameter, $$\beta$$. The entire expression is then normalized by the partition function $$Z(\mathbf{x})$$ to ensure it sums to one and is a valid probability distribution. The formula is:

$$ \pi^{*}(\mathbf{y}|\mathbf{x}) = \frac{\pi_{\theta_{\text{ref}}}(\mathbf{y}|\mathbf{x}) \exp \left(\frac{1}{\beta}r(\mathbf{x}, \mathbf{y})\right)}{Z(\mathbf{x})} $$

Reward-Weighted Probability Distribution

An AI text generation system adjusts the likelihood of different outputs using the formula: New_Likelihood = Base_Likelihood * exp((1/β) * Reward). In this formula, 'Base_Likelihood' is the initial probability from a reference model, 'Reward' is a score for the output's quality, and 'β' is a positive 'temperature' parameter. A team wants to use this system to generate a diverse set of creative, high-quality story endings. They are comparing two settings for the temperature parameter: β = 0.5 and

Learn Before

Related