Google

In Masked Language Modeling (MLM), the masking process is formally defined for an input sequence $$\mathbf{x} = x_0 \dots x_m$$. A set of positions, $$\mathcal{A}(\mathbf{x}) = \{i_1, \dots, i_u\}$$, is selected to be masked. This procedure generates a modified token sequence, $$\bar{\mathbf{x}}$$, in which the token at each selected position in $$\mathcal{A}(\mathbf{x})$$ is replaced with a special $$[\mathrm{MASK}]$$ symbol.

Formal Definition of the Masking Process in Masked Language Modeling (MLM)

To illustrate Masked Language Modeling, consider the sentence 'The early bird catches the worm'. A training instance is created by masking one or more tokens. For example, masking a single token might yield 'The early bird $$[\mathrm{MASK}]$$ the worm', where the model predicts 'catches'. Alternatively, masking multiple tokens, such as 'early' and 'worm' (e.g., at indices $${}i_1 = 2$$ and $${}i_2 = 6$$), yields the corrupted sequence 'The $$[\mathrm{MASK}]$$ bird catches the $$[\mathrm{MASK}]$$'. The model must then correctly predict the masked words based on context.

Example of Masked Language Modeling with Single and Multiple Masks

A developer implements a text corruption process for a language model. Instead of selecting positions randomly from the input sequence, their algorithm always replaces the final token of every sequence with a special `[MASK]` symbol. Based on the formal definition of the masking process, identify the key component that is incorrectly implemented and explain why this deviation is problematic for the model's learning objective.

Critique of a Masking Implementation

In the formal definition of the masking process used in language models, several components are used to describe the transformation of an input sequence. Match each symbolic component with its correct description.

Given an original text sequence $$\mathbf{x}$$ and its corrupted version $$\bar{\mathbf{x}}$$, optimizing a model to predict $$\mathbf{x}$$ based on $$\bar{\mathbf{x}}$$ can be thought of as an autoencoding-like process. The fundamental training objective is to maximize the reconstruction probability $$\Pr(\mathbf{x}|\bar{\mathbf{x}})$$. However, because there is a simple position-wise alignment between the two sequences, an unmasked token in $$\bar{\mathbf{x}}$$ is the exact same as the token in $$\mathbf{x}$$ at the same position. Since there is no need to consider the prediction for these unmasked tokens, the training objective is simplified to only maximize the probabilities for the masked tokens.

Training Objective of Masked Language Modeling (MLM)

In a masked modeling approach, an input sequence `x` is transformed into a modified sequence `x̄` by replacing tokens at a randomly selected set of positions `A(x)` with a special `[MASK]` symbol. Given an input sequence `x = (T1, T2, T3, T4, T5, T6)` and a set of selected positions `A(x) = {2, 5}` (using 0-based indexing), what is the resulting modified sequence `x̄`?

Learn Before

Related