Google

In the specific case of Masked Language Modeling where 100% of the input tokens are masked, the training objective becomes equivalent to sequence generation. Consequently, the decoder is trained to operate as a language model, responsible for generating the entire original text.

Training the Decoder as a Language Model in 100% Masking Scenarios

A research team is developing a system to generate short, creative story paragraphs from scratch, with no initial text provided other than a signal to start generating. They have access to a powerful pre-trained model that was exclusively trained using an objective where 100% of the input text tokens were consistently replaced with a special `[MASK]` token, and the model's goal was to reconstruct the original text. Based on its training method, evaluate the suitability of this pre-trained model for the team's creative story generation task. Justify your reasoning by explaining the core capability the model likely developed during its training.

Model Suitability for a Generation Task

A model is being trained on a text corpus. In one training configuration, 15% of the words in each sentence are randomly replaced with a special `[MASK]` token, and the model must predict the original words. In a second configuration, 100% of the words are replaced with `[MASK]` tokens. Analyze how the fundamental task the model is learning to perform differs between these two configurations.

Shift in Training Objective with 100% Masking

Consider a text-infilling model that is typically trained by masking about 15% of the words in a sentence and having the model predict them based on the surrounding unmasked words. If this training process is modified to mask 100% of the words in every input sentence, what is the most significant change in the fundamental skill the model is being trained to perform?

Learn Before

Related