Google

A key limitation of the auto-encoding objective in Masked Language Modeling (MLM) is its failure to account for dependencies among the masked tokens. The model is trained to predict each masked token independently of the others. For example, if two tokens $$x_2$$ and $$x_6$$ in a sequence are masked, the prediction for the first masked token ($$x_2$$) is generated independently of the second masked token ($$x_6$$), even though $$x_6$$ should ideally be considered within the context of $$x_2$$.

Limitation of MLM: Ignoring Dependencies Between Masked Tokens

A language model is trained using an objective where it learns to predict words that have been hidden in a sentence. For example, given 'The cat sat on the [HIDDEN]', it learns to predict 'mat'. Now, consider the following behavior observed with this model:

Diagnosing a Language Model's Predictive Behavior

Consider a language model trained with a self-supervised objective where it learns to fill in randomly blanked-out words in a text. If the model encounters the sentence 'The artist picked up a [MASK] and a [MASK] to begin the painting,' it will try to predict the two missing words. However, it will predict each missing word without considering what the other missing word might be. Analyze the potential problem this creates for generating a coherent and logical sentence completion.

Consequences of Independent Predictions in Language Models

Permuted Language Modeling (PLM) is a pre-training approach designed to resolve specific issues found in Masked Language Modeling, such as the mismatch between training and inference and the independence assumption among masked tokens. While it is a sequential prediction task, the actual order of tokens in the original text remains completely unchanged. Instead, the model is trained to predict the tokens sequentially according to an arbitrarily determined, permuted order.

Permuted Language Modeling (PLM)

A language model pre-trained with a standard masked language modeling objective is given the input sentence: 'The capital of the United Kingdom is [MASK] [MASK].' Which statement best describes how the model will predict the two masked tokens?

Learn Before

Related