Google

When using a weak model for data selection, the likelihood or cross-entropy of a sequence, as computed by the weak model, serves as a quantitative filtering criterion. Sequences that are less aligned with the weak model's learned distribution, indicated by low likelihood or high cross-entropy, may be excluded from the training set. Conversely, sequences that align well, indicated by high likelihood or low cross-entropy, can be prioritized to focus the training of a large model on more relevant or high-quality data.

Likelihood and Cross-Entropy as Data Filtering Criteria

Based on the scenario provided, which data sample would you recommend excluding from the final training set for the large, specialized model? Justify your decision based on the provided scores and the underlying principle of this filtering technique.

Data Filtering for a Specialized Language Model

A data curation team uses a small language model to pre-process a large text corpus. The model assigns a cross-entropy score to each document. They find two documents with the following scores:

*   Document A: Cross-entropy = 1.8
*   Document B: Cross-entropy = 9.5

Based on the goal of creating a high-quality, coherent training set, which document is more likely to be included, and why? Explain the relationship between the cross-entropy score and how well a document aligns with the small model's learned patterns.

Interpreting Cross-Entropy for Data Curation

A team is refining a large, general-purpose text corpus to train a specialized language model. They use a smaller, pre-existing model to calculate the cross-entropy for each document. Their goal is to create a high-quality, coherent, and well-structured training set. Which of the following filtering strategies should they implement and why?

Learn Before

Related