Google

To account for multiple factors simultaneously, the loss of a Large Language Model can be modeled as a function of both the number of model parameters, $$N$$, and the size of the training dataset, $$D$$. This relationship is captured by a combined scaling law developed by Rosenfeld et al., which incorporates an irreducible error term, $$\epsilon_{\infty}$$, resulting in the formula: $$\mathcal{L}(N,D) = aN^b + cD^d + \epsilon_{\infty}$$. In this equation, the terms $$aN^b$$ and $$cD^d$$ represent the independent contributions of model size and dataset size to the overall loss, following a power law.

Combined Power Law for LLM Loss with Model and Dataset Size

The Chinchilla scaling law provides a framework for predicting language model performance. According to this principle, the test loss per token is derived by adding a constant irreducible error term to two separate inverse proportional functions: one based on the model size ($$N$$) and the other based on the training dataset size ($$D$$).

Chinchilla Scaling Law

A research team is training a large language model and observes that after weeks of training, adding 50% more high-quality data to their training set results in a negligible decrease in the model's final loss. Based on the principle that loss (L) is a function of model size (N) and dataset size (D) as described by the relationship `L(N,D) = aN^b + cD^d + ε_∞`, propose two distinct and plausible explanations for this phenomenon. For each explanation, identify the specific term(s) in the equation that would be the primary cause.

Diagnosing LLM Training Plateaus

Based on the diagnostic analysis, which of the two strategies (A or B) should the lab choose to achieve the greatest improvement in model performance? Justify your answer by explaining how the chosen strategy addresses the identified bottleneck within the combined power law framework.

Optimizing LLM Training Strategy

A research team is working to improve a large language model and is using the combined power law, `L(N,D) = aN^b + cD^d + ε_∞`, to guide their efforts. Their analysis shows that the term `aN^b`, which depends on the model's parameter count (N), is currently the largest contributor to the total loss. The term `cD^d`, which depends on the dataset size (D), is comparatively small. To achieve the most significant reduction in loss with their limited resources, what should the team prioritize?

Learn Before

Related