Google

A bigger model can raise variance and make overfitting more likely. However, when the training procedure includes effective regularization, increasing model size often does not harm test performance much. In that case, the biggest practical cost of using the larger model is usually extra computation and memory, not a large drop in generalization.

Regularization often lets a larger model remain usable

What Usually Happens When a Well-Regularized Model Gets Bigger?

True or False: A very large model can still memorize quirks in the training set if regularization is absent or too weak.

Match each training setup to its typical effect on variance and overfitting.

Put the model-size and regularization reasoning in order.

Question:
Explain why a good regularization method changes the usual concern about making a model larger, and identify the main tradeoff that still remains.

Sample answer:
Without regularization, increasing model size can lower bias but also increase variance and make overfitting more likely. With a well-chosen regularization method, such as L2 regularization or dropout tuned on a development set, larger models often do not become more prone to overfitting in practice. In many cases, performance stays about the same or gets better. As a result, the usual size-versus-overfitting concern is greatly reduced. The main tradeoff that remains is computational cost: larger models require more memory, training time, and inference time, so resources become the key reason not to keep increasing size.

Key points:
- Larger models can reduce bias but raise overfitting risk when unregularized
- Good regularization weakens that overfitting concern
- L2 and dropout are examples of regularization that can be tuned on a dev set
- Performance often stays steady or improves as size grows under regularization
- Compute and runtime become the main limiting factors

Rubric:
Full credit for explaining that regularization removes most of the overfitting penalty of larger models, noting that performance may stay the same or improve, and identifying computation as the remaining tradeoff. Partial credit for only part of this chain of reasoning.

How Regularization Changes the Model-Size Tradeoff

Case context:
A product team has a neural network for fraud detection. They use dropout and L2 regularization, and both were tuned on the validation set. The team wants to lower bias by moving from a medium-sized network to a much larger one. In an earlier project, a larger unregularized model fit the training data too closely, so some team members are worried that the same thing will happen again. They have access to a modest amount of GPU time, but not unlimited capacity.

Question:
Using the reasoning from the lesson, should the team mainly worry about overfitting, or about something else? What factor should drive the decision to increase model size?

Sample answer:
If regularization has already been chosen and tuned well, then overfitting is usually not the main reason to avoid a bigger model. A larger network can often be trained without increasing validation error, and it may even improve performance. The more important question is whether the team can afford the extra computation. They should compare the added training time, memory use, and deployment cost against the expected benefit from the larger model.

Key points:
- Regularization is already in place and tuned on validation data
- A larger model is not automatically a worse choice because of overfitting
- Validation performance can stay the same or improve with more capacity
- Compute, memory, and deployment cost are the main trade-offs

Rubric:
Full credit for stating that regularization reduces the overfitting concern and that computational cost is the key remaining issue. Partial credit for mentioning only one of those ideas.

Decide what matters most when expanding a well-regularized model.

Question:
In one to three sentences, explain what should be done if adding a new feature makes a model more variable.

Sample answer:
If the extra feature increases variance, the recommended response is to add regularization. Regularization usually offsets that extra variance, so the model can still improve overall.

Key points:
- Extra features can raise variance.
- Regularization is the recommended response.
- Regularization usually removes the added variance.

Rubric:
Full credit for saying that regularization should be used and that it usually cancels the variance increase. Partial credit for mentioning regularization without explaining the expected effect.

Learn Before

Related