Google

Approach 1: Train the model on an extremely large and diverse dataset, incorporating text from the web, academic articles, books, and code, using a general-purpose learning objective.
Approach 2: Train the model on a smaller, but very high-quality, curated dataset focused on a few key domains (e.g., customer service

The initial examination of the pre-training paradigm focuses on the fundamental ideas for addressing its two main challenges: optimizing a model for broad generalizability and adapting it effectively to specific downstream applications.

Scope of Foundational Concepts in Pre-training and Adaptation

Analyze the two distinct phases of the model development process described in the case study. For each phase, identify its primary objective and explain how that objective addresses one of the two fundamental challenges of building and using large-scale pre-trained models.

Analyzing a Model Development Lifecycle

A team is developing a large language model. During the initial training phase on a massive, diverse dataset, they focus on an objective that encourages the model to predict the next word in a sentence. Later, for a specific customer, they further train the model on a small, curated dataset of legal documents to improve its ability to answer legal questions. Explain how these two distinct training stages address the two fundamental challenges of the pre-training paradigm.

Balancing Generalization and Specialization

A research lab is developing a new foundation model with a limited computational budget. They are considering two primary approaches for the initial training phase:

*   **Approach 1:** Train the model on an extremely large and diverse dataset, incorporating text from the web, academic articles, books, and code, using a general-purpose learning objective.
*   **Approach 2:** Train the model on a smaller, but very high-quality, curated dataset focused on a few key domains (e.g., customer service 

Learn Before

Related