Google

In Masked Language Modeling, the random selection of tokens for masking can result in the masking of consecutive tokens within a sequence [Joshi et al., 2020]. This means that two or more adjacent words may be replaced by `[MASK]` symbols in a single training instance.

Consecutive Token Masking in MLM

An example of a denoising autoencoder task involving consecutive token masking is training a model to reconstruct a full sentence from a corrupted input. The model receives an input where adjacent tokens have been replaced by `[MASK]`, and its objective is to generate the original, complete sequence. For instance, the input `[CLS] The puppies are [MASK] outside [MASK] [MASK] .` would be used to train the model to produce the target output: `⟨s⟩ The puppies are frolicking outside the house .`.

Example of Denoising Task with Consecutive Token Masking

An alternative approach to handling consecutive masked tokens is to treat them as a single span. Following the methodology of Raffel et al. (2020), a unique sentinel token, such as `[X]`, `[Y]`, or `[Z]`, replaces one or more adjacent masked tokens in the corrupted input sequence, effectively creating placeholder slots. The model's training task is then to fill these slots with the correct original tokens using the surrounding context. A significant advantage of consolidating multiple masks into a single placeholder is that the sequences used during training become shorter, leading to more computationally efficient training.

Representing Masked Spans with Sentinel Tokens

A language model is being trained with the following example:

**Input:** `The cat [MASK] [MASK] on the mat.`
**Target:** `The cat sat lazily on the mat.`

Explain why predicting the second `[MASK]` token (which corresponds to 'lazily') is a more difficult task for the model in this specific scenario than if the input had been `The cat sat [MASK] on the mat.`

Analyzing a Masked Language Modeling Task

Based on the training process described in the case study, what is the most likely reason for the model's poor performance on inputs with multiple adjacent `[MASK]` tokens? Explain the connection between the training data and the observed performance gap.

Analyzing Model Performance Discrepancy

A language model is pre-trained using a masked language modeling objective. During one training step, it sees the input: `The cat sat on the [MASK] [MASK].` Explain why predicting the two masked tokens in this scenario is a more challenging task for the model than predicting two separate, non-adjacent masked tokens in a longer sentence (e.g., `The [MASK] cat sat on the warm [MASK].`).

Analyzing the Challenge of Consecutive Masking

A language model is being trained to predict masked words in a text. Consider two different masking strategies: 

Strategy 1: 15% of the words in a sentence are masked individually at random positions. 
Example: `The quick [MASK] fox jumps [MASK] the lazy dog.`

Strategy 2: A contiguous span of several words is masked. 
Example: `The quick [MASK] [MASK] [MASK] the lazy dog.`

How does using Strategy 2 (masking a contiguous span) primarily alter the learning challenge for the model compared to St

Learn Before

Related