Google

Hoffmann et al. (2022) established a precise empirical equation for the Chinchilla scaling law to compute the test loss ($$\mathcal{L}$$) based on the model size ($$N$$) and the dataset size ($$D$$). The formulation is expressed as:

$$ \mathcal{L}(N,D) = \underbrace{\frac{406.4}{N^{0.34}}}_{\text{model scaling}} + \underbrace{\frac{410.7}{D^{0.28}}}_{\text{dataset scaling}} + \underbrace{1.69}_{\text{irreducible error}} $$

This relationship divides the overall loss into three distinct components: a model scaling term, a dataset scaling term, and a baseline irreducible error of $${}1.69$$.

Chinchilla Scaling Law Formula

Given the scenario below, determine which action would result in a greater reduction in the expected test loss and justify your reasoning by analyzing the components of the provided formula.

Optimizing Training Resources

Consider the empirical formula for the Chinchilla scaling law: $$ \mathcal{L}(N,D) = \frac{406.4}{N^{0.34}} + \frac{410.7}{D^{0.28}} + 1.69 $$ If a research team had access to a dataset of infinite size (i.e., as D approaches infinity), what would be the theoretical minimum loss for a model of a fixed size N? Explain how you arrived at your answer by analyzing the components of the formula.

Theoretical Loss Limit with Infinite Data

A research team is using the following empirical formula to guide their training strategy for a large language model, where `L` is the test loss, `N` is the model size, and `D` is the dataset size: $$ \mathcal{L}(N,D) = \frac{406.4}{N^{0.34}} + \frac{410.7}{D^{0.28}} + 1.69 $$ To achieve the most substantial reduction in test loss, which of the following strategies is predicted by this formula to be more effective?

Learn Before

Related