Google

As the training set grows, development and test error usually move downward, but the error measured on the training set often moves upward. With a very small dataset, a model can often memorize the examples and achieve nearly perfect training performance. When the dataset becomes larger, more varied, or noisier, fitting every example exactly becomes harder, so training error tends to increase.

More Data Often Raises Training Error

What usually happens to training error as the training set becomes larger?

A very small training set can make a model seem impressively accurate on training data.

As the training set gets larger, training error usually _____.

Training Error and More Data

Match each training-data situation with its typical effect on error.

Order the reasoning that explains why training error can increase.

Question:
In a short explanation, describe why adding more training examples can make training performance look worse while improving development performance.

Sample answer:
With a very small training set, a model may be able to memorize the examples almost perfectly, so training error can be very low. As the training set grows, the model must fit more cases, including unusual, noisy, or ambiguous ones, so training error often increases. Even so, the larger and more varied training set usually helps the model learn patterns that generalize better, so dev error tends to fall.

Key points:
- Small training sets are easier to memorize.
- Training error often rises as the training set gets larger.
- Noisy, ambiguous, or hard examples make perfect fitting more difficult.
- Dev error often decreases because the model generalizes better.

Rubric:
A strong response explains both trends and links higher training error to reduced memorization and the greater difficulty of fitting a larger, more varied dataset, while also noting that dev performance typically improves.

Why can training error go up even as dev error goes down?

Case context:
A fruit-quality classifier has 0% training error on a tiny set of 3 examples. After the training set grows to 120 examples and then to 8,000 examples, the training error rises a little while the dev error falls. Some of the labels are uncertain or inconsistent.

Question:
What should the learner conclude about these changes?

Sample answer:
This pattern is expected. The 3-example set was easy to memorize, but fitting every example becomes harder as the training set gets larger. In addition, uncertain or inconsistent labels make perfect fitting more difficult. The rise in training error together with the drop in dev error shows that the model is learning a more useful pattern from the larger dataset.

Key points:
- Very low training error on a tiny set can come from memorization.
- Training error often increases as the training set becomes larger.
- Dev error should usually decrease as more training data is added.
- Uncertain or inconsistent labels make perfect training fit harder.

Rubric:
The response should explain both error trends, note that the smallest dataset could be memorized, and identify larger dataset size plus label inconsistency as reasons that perfect training fit becomes harder.

Explain the error trend after expanding a dataset

Question:
Answer in one to three sentences.

Sample answer:
With only two training cases, a sufficiently flexible model can simply fit both of them exactly. Because it has effectively memorized those examples, it can reach zero training error.

Key points:
- The training set is tiny.
- A flexible learner can memorize the examples.
- Perfect memorization can lead to 0% training error.

Rubric:
The answer should explain that such a small dataset is easy to memorize and that memorization can produce zero training error.

Learn Before

Related