Google

Self-supervised pre-training is a paradigm where a model learns from vast quantities of unlabeled data by creating its own supervisory signals directly from the input. For example, it might learn by predicting masked or corrupted parts of the data. Following this pre-training phase, the model is adapted for specific downstream applications, which can be accomplished through methods like supervised fine-tuning on labeled datasets or via prompting to enable zero-shot or few-shot learning.

Self-supervised Pre-training

Based on the provided scenario, propose a specific self-supervised pre-training task that could be used on the large, unlabeled database to create a powerful base model. Justify your proposal by explaining (1) why the task is considered 'self-supervised' and (2) how this pre-training step would improve performance on the final clause classification task.

Pre-training Strategy for a Specialized Domain

You are tasked with pre-training a language model on a massive, unlabeled dataset of computer code. Beyond the common approach of predicting randomly masked parts of the code, propose one distinct self-supervised objective that would be particularly well-suited for this dataset. Briefly justify why your proposed objective would help the model learn meaningful patterns specific to code.

Designing a Self-Supervised Task for Code

The development of a unified text-to-text system typically involves a two-stage pipeline. First, an encoder-decoder model is trained via self-supervision to acquire a broad, general-purpose understanding of language. Subsequently, this model undergoes fine-tuning for specific downstream applications using targeted training data that has been formatted into a text-to-text structure.

Training Process for Text-to-Text Models

A team is training a language model on a massive, unlabeled corpus of text from the internet. Their training objective is to randomly mask 15% of the words in each input sentence and require the model to predict the original masked words. Which of the following statements best analyzes why this specific training method is considered 'self-supervised'?

Learn Before

Related