Google

A method for curating training datasets for Large Language Models involves using a smaller, auxiliary model to evaluate each data sequence. This small model calculates metrics such as likelihood or cross-entropy for the sequences. These metrics then serve as a basis for data selection: sequences that are poorly aligned with the small model's learned distribution (e.g., having low likelihood or high cross-entropy) can be filtered out, while sequences that are well-aligned (e.g., having high likelihood or low cross-entropy) can be prioritized. This technique helps focus the training of the larger model on more relevant or higher-quality data.

Data Selection and Filtering using Small Models

Critically evaluate the team's proposed data filtering strategy. Is this an effective approach for achieving their specific goal? Justify your conclusion by analyzing the relationship between the small model's training data and the desired characteristics of the final dataset.

Evaluating a Data Curation Strategy for a Specialized Model

A team is using a small, pre-trained language model to filter a large, diverse dataset for training a much larger model. They decide to keep only the data points for which the small model shows the lowest cross-entropy. Describe one significant potential risk or drawback of this data curation strategy.

Limitations of Small Model Data Filtering

A research team is building a large language model specialized in generating high-quality Python code. They have a massive dataset containing a mix of Python code, natural language text, and code from other programming languages. To curate this dataset, they use a smaller, pre-trained model that is already proficient in Python. Which of the following data filtering strategies would be most effective for their goal?

Learn Before

Related