Google

To illustrate Masked Language Modeling, consider the sentence 'The early bird catches the worm'. A training instance is created by masking one or more tokens. For example, masking a single token might yield 'The early bird $$[\mathrm{MASK}]$$ the worm', where the model predicts 'catches'. Alternatively, masking multiple tokens, such as 'early' and 'worm' (e.g., at indices $${}i_1 = 2$$ and $${}i_2 = 6$$), yields the corrupted sequence 'The $$[\mathrm{MASK}]$$ bird catches the $$[\mathrm{MASK}]$$'. The model must then correctly predict the masked words based on context.

Example of Masked Language Modeling with Single and Multiple Masks

You are preparing training data for a language model using the sentence: 'The powerful engine roared to life on the racetrack.'

1. Rewrite the sentence, replacing exactly one word with a `[MASK]` token to create a training example.
2. Rewrite the sentence again, this time replacing two different words with `[MASK]` tokens to create a second, more challenging training example.
3. For both of your created examples, briefly describe what the model is expected to predict.

Applying Masking Strategies for Language Models

A language model is being trained using the sentence: 'The quick brown fox jumps over the lazy dog.' For one training instance, the model is presented with the following modified input: 'The quick [MASK] fox jumps over the [MASK] dog.' Based on this example, what is the model's primary task, and what specific words is it attempting to predict?

Applying Masked Language Modeling

A language model is being trained on the sentence: 'The chef carefully seasoned the delicious soup with a pinch of salt.' Consider two different training examples created from this sentence:

1. `... seasoned the delicious soup with a [MASK] of salt.`
2. `The [MASK] carefully seasoned the delicious [MASK] with a pinch of salt.`

Which statement best analyzes the difference in the learning objective between these two examples?

Learn Before

Related