Google

The pre-training pipeline for BERT's Masked Language Modeling (MLM) is a multi-step process. It begins with an input sequence, from which 15% of tokens are randomly selected. These chosen tokens are then altered: 80% are masked, 10% are replaced by random tokens, and 10% are left unchanged. This modified sequence is converted into embeddings and processed by a Transformer Encoder to produce contextualized hidden states. Finally, the model is trained using these hidden states to predict the original values of the altered tokens.

BERT's Masked Language Modeling Pre-training Pipeline

A language model is pre-trained using a masked language modeling objective. Arrange the following stages of its data processing and training pipeline in the correct chronological order.

Analyze the following implementation of a token modification strategy for pre-training a language model. Identify the primary conceptual flaw in this approach and explain the negative consequence it would have on the model's subsequent use.

Analyzing a Pre-training Pipeline Implementation

An input sequence of 200 tokens is processed during a model's self-supervised pre-training. The procedure first selects 15% of the tokens for modification. Of this selected group, 80% are replaced with a special mask symbol, 10% are replaced with a different, random token, and the final 10% are left as they are. Given this process, which statement accurately describes the state of the 200-token sequence after this modification step?

Learn Before

Related