Google

In theory, making a neural network larger and giving it more training examples can improve performance on many tasks. In real projects, however, very large models take much longer to train, and the pool of additional data can run out.

Why Increasing Model Size and Data Eventually Stops Helping

What is the main practical limit on repeatedly making a neural network much larger?

True or False: With enough model capacity and enough labeled data, a learning system can in principle achieve strong results on many tasks.

Training a very large model is _____, so increasing model size has practical limits.

Order the steps that explain why model and data scaling have practical limits.

Question:
Write a short essay explaining why the idea that a model could keep getting better forever by adding more capacity and more data does not translate into ordinary machine learning projects.

Sample answer:
If model capacity and training examples could grow without any limit, many problems would likely become easier and better performance would seem attainable. Real projects do not have that freedom. Very large models take much longer to train and require far more computing power, and datasets are not unlimited either. At some point, there is no more useful data to collect. Because of these constraints, engineers must improve systems within practical limits instead of depending on endless growth.

Key points:
- Unlimited capacity and unlimited data create an appealing theoretical picture
- Large models are expensive and slow to train, which creates computing constraints
- Data availability eventually runs out, so scaling cannot continue forever
- Real-world practice is limited by these constraints rather than the idealized picture

Rubric:
Full credit requires stating the idealized claim, identifying both real-world limits (computing cost and limited data supply), and clearly explaining why those limits separate theory from practice.

Why Infinite Scaling Is an Idea, Not a Plan

Case context:
A fintech company is building a large neural network to detect fraudulent card transactions. The team believes that if the model is still underperforming, the safest fix is to keep increasing model size and collecting more labeled transactions without stopping.

Question:
What practical limits should the team expect if it tries to keep scaling this approach, and how should those limits change its planning?

Sample answer:
In theory, a larger model trained on more data can continue to improve results. In practice, the team will run into two major limits. First, training ever-larger networks takes much longer and requires more compute, storage, and engineering effort. Second, there is only a finite amount of new labeled fraud data available, and each additional example can be costly to obtain. Because of these limits, the team should not assume that indefinite scaling is a realistic plan; it should budget for computation and accept that data growth will eventually slow or stop.

Key points:
- Notes that bigger models and more data can help in theory
- Identifies increasing compute time and cost as a constraint
- Identifies limited availability of additional labeled data
- Explains that these constraints make indefinite scaling impractical

Rubric:
Full credit requires identifying both the compute/time limit and the finite-data limit, and connecting them to the feasibility of the team's scaling plan.

Practical limits when scaling a fraud model

Question:
In one to three sentences, name two practical bottlenecks that make it unrealistic to keep increasing a neural network's size and its training data forever.

Sample answer:
One bottleneck is that very large models take much longer to train, so compute and time become limiting factors. The other is that additional training data may eventually be unavailable, too expensive, or too difficult to collect.

Key points:
- Slow, expensive training of larger models
- Limited availability or high cost of additional data

Rubric:
Full credit requires identifying both the compute/time bottleneck and the data-availability bottleneck.

Learn Before

Related