Google

A sequence-level loss is a type of loss function that computes the total error over an entire data sequence. Instead of evaluating each element of the sequence independently, it aggregates the individual losses from each step—often through summation or averaging—to provide a single measure of the model's performance on the whole sequence.

Sequence-Level Loss

Given the predictions for each time step $\hat{y}^{(t)}$, together with the ground truth labels $y^{(t)}$, we can calculate the loss on step t
$$J^{(t)} = \mathcal{L} (\hat{y}^{(t)}, y^{(t)})$$
To get the overall loss, we only need to average these:
$$J = \frac{1}{T} \sum_{t=1}^T J^{(t)} $$

Loss Function for RNN

When evaluating a model on a specific training instance, the loss function is calculated solely over the target sub-sequence, $$\mathbf{y}_{\mathrm{sample}}$$, instead of the full sequence. For a model defined by parameters $$\hat{\theta}^+$$, this loss is expressed as the negative log-likelihood of the probability of generating the output sub-sequence, given the input sub-sequence $$\mathbf{x}_{\mathrm{sample}}$$. The formula is:

$$\mathcal{L}_{\hat{\theta}^+}(\mathrm{sample}) = -\log \mathrm{Pr}_{\hat{\theta}^+}(\mathbf{y}_{\mathrm{sample}}|\mathbf{x}_{\mathrm{sample}})$$

Sample-wise Negative Log-Likelihood Loss for a Sub-sequence

A frequently used loss function in knowledge distillation is the sequence-level loss, which often takes the form of cross-entropy. This loss measures the dissimilarity between the teacher model's output distribution, $\text{Pr}^t(\mathbf{y}|\mathbf{c}, \mathbf{z})$, and the student model's distribution, $\text{Pr}_{\theta}^s(\mathbf{y}|\mathbf{c}', \mathbf{z})$. The total loss is calculated by summing the log probability of the student's predictions over all possible output sequences $\mathbf{y}$, weighted by the teacher's probability for each sequence. The formula is expressed as: $$ \text{Loss} = \sum_{\mathbf{y}} \text{Pr}^t(\mathbf{y}|\mathbf{c}, \mathbf{z}) \log \text{Pr}_{\theta}^s(\mathbf{y}|\mathbf{c}', \mathbf{z}) $$

Cross-Entropy Loss for Knowledge Distillation

The total loss for a given sequence of $$m$$ tokens $$(x_0, \dots, x_m)$$ is computed by summing the individual losses over each position from $$i=0$$ to $$m-1$$. At each position $$i$$, a loss function $$\mathcal{L}$$ measures the discrepancy between the model's predicted probability distribution for the next token ($$\mathbf{p}_{i+1}^{\theta}$$) and the actual ground-truth distribution ($$\mathbf{p}_{i+1}^{\mathrm{gold}}$$). This is expressed generally as:

$$ \mathrm{Loss}_{\theta}(x_0, \dots, x_m) = \sum_{i=0}^{m-1} \mathcal{L}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\mathrm{gold}}) $$

In natural language processing, this loss function $$\mathcal{L}$$ is typically the log-scale cross-entropy loss, leading to the specific formula:

$$ \mathrm{Loss}_{\theta}(x_0, \dots, x_m) = \sum_{i=0}^{m-1} \mathrm{LogCrossEntropy}(\mathbf{p}_{i+1}^{\theta}, \mathbf{p}_{i+1}^{\mathrm{gold}}) $$

Total Loss Calculation for a Token Sequence

A model is tasked with predicting a sequence of four numbers. For a specific training example, the model's error (loss) for each number in the sequence is calculated as follows: [0.8, 0.2, 0.5, 0.5]. If the overall performance on this sequence is measured by taking the *average* of the individual errors, what is the final sequence-level loss for this example?

Calculating Average Sequence-Level Loss

A team is training a model to translate sentences. They are debating two methods for calculating the error and updating the model for each training sentence:

**Method 1:** For a given sentence, the model generates the full translation. The errors for each individual word in the generated translation are then summed up. This single, total error value is used to adjust the model's parameters.

**Method 2:** For a given sentence, the model generates the first word of the translation. The error for this single word is calculated, and the model's parameters are immediately adjusted. Then, it generates the second word, calculates the error, and adjusts the parameters again. This process repeats for every word in the translation.

Analyze these two methods. Which method is a more appropriate application of calculating error for a complete data sequence, and why is it generally more effective for training models on sequential tasks like translation?

Evaluating Training Strategies for a Translation Model

A language model is being trained to generate the four-word sentence 'The quick brown fox'. The model generates one word at a time, and the error (loss) is calculated at each step:

- Loss for 'The' = 0.1
- Loss for 'quick' = 0.3
- Loss for 'brown' = 0.2
- Loss for 'fox' = 0.4

To update the model's parameters, the training process computes a single, overall loss value for the entire sentence. Which statement best analyzes this method of calculating the overall loss?

Learn Before

Related