Google

In the context of minimum Bayes risk, a risk function, denoted as $$R(\mathbf{y},\mathbf{y}_r)$$, is defined for every pair of candidate outputs. This function quantifies the penalty or cost associated with replacing a chosen output $$\mathbf{y}$$ with an alternative reference output $$\mathbf{y}_r$$.

Risk Function in Minimum Bayes Risk Decoding

In Minimum Bayes Risk (MBR) decoding, the risk associated with selecting a particular output `y` from a set of candidates `Ω` is defined as the expected cost over all possible reference outputs `y_r`. This is calculated by summing the product of the risk function `R(y, y_r)` and the probability of the reference output `Pr(y_r|x)` for every candidate in `Ω`. The formula is:
`$\text{Risk}(y) = \mathbb{E}_{y_r \sim \Pr(y_r|\mathbf{x})} R(y, y_r) = \sum_{y_r \in \Omega} R(y, y_r) \cdot \Pr(y_r|\mathbf{x})$`

Risk of an Output in Minimum Bayes Risk Decoding

In a system that generates text, a simple risk function is proposed: `R(y, yr) = 0` if the candidate output `y` is exactly identical to the reference output `yr`, and `R(y, yr) = 1` otherwise. Analyze the primary limitation of using this function for tasks where semantic similarity is more important than exact string matching.

Evaluating a Simple Risk Function

A team is developing a translation system for legal documents where precision is paramount. Given the proposed risk function in the case study below, critically evaluate its suitability for this task. Identify its primary limitation and describe a key characteristic that a more appropriate risk function should possess.

Evaluating a Risk Function for Legal Translation

A language model is generating a summary for a document. The ideal, human-written reference summary is `yr`: "The study found a significant link between diet and health."

The model considers two candidate summaries:
- `y1`: "The study found a significant link between diet and health."
- `y2`: "Research showed a connection between food and wellness."

Given the purpose of a risk function, `R(candidate, reference)`, which of the following statements best describes how it would operate in this sce

Learn Before

Related