Google

A model’s accuracy on the training set is usually at least as good as its accuracy on dev or test data. If it reaches only 82% on data it has already seen, it is not reasonable to expect 94% on unseen examples.

Training Performance Is a Baseline for Held-Out Performance

What usually has to happen before a model is expected to do well on validation and test data?

True or False: Performance on development and test data is usually lower than performance on the training set.

Training performance comes first

Interpreting Training and Held-Out Performance

Order the steps used to judge whether a better score on unseen data is believable.

Question:
In a concise analytical response, explain why an algorithm generally needs solid performance on the training set before high validation or test performance is realistic. Use a 78% and 92% example.

Sample answer:
Training performance reflects how well the model fits examples it has already seen, while validation or test performance measures how well it generalizes to new examples. In practice, performance on unseen data is usually lower than performance on the training set. So if a model reaches only 78% accuracy on training data, it would be unrealistic to expect 92% accuracy on new data. The model should first improve its training performance before a strong validation or test result is likely.

Key points:
- Training data are examples the model has already seen.
- Validation or test data are new examples the model has not seen.
- Performance on new data is usually lower than performance on training data.
- An 78% training result does not justify expecting 92% on unseen data.
- Better training performance should come first if strong generalization is expected.

Rubric:
A strong response distinguishes seen from unseen data, notes that validation or test performance is usually lower, correctly interprets the 78% and 92% example, and concludes that training performance must improve first.

Explain why training performance must come before strong validation or test performance.

Case context:
A team says its model gets 72% accuracy on the training examples it was fit on and 89% accuracy on the dev/test examples it has never seen.

Question:
Using only the principle described in the lesson, how should the team interpret this report, and what should it work on first?

Sample answer:
The report should be treated as inconsistent with the principle. When the same evaluation setup is used, performance on unseen dev/test examples is usually worse than performance on the training set, not better. The team should first improve training-set performance before expecting good dev/test results.

Key points:
- The numbers describe an unlikely relationship between training and dev/test performance.
- Training examples are the ones the model has already seen.
- Dev/test examples are examples the model has not seen before.
- Dev/test performance is typically lower than training performance.
- The first priority is to improve training-set performance.

Rubric:
Full credit requires noting that the report conflicts with the principle, distinguishing seen training data from unseen dev/test data, and stating that training-set performance should be improved first.

Diagnose a claim of 72% training accuracy and 89% dev/test accuracy.

Question:
Respond in one to three sentences by relating performance on data the model has already learned from to performance on new data.

Sample answer:
A model is judged first on the data it was trained with, because that shows whether it has learned the task at all. If it does not do well on familiar training examples, there is little reason to expect it to do well on new validation or test examples, which are usually harder.

Key points:
- Training examples are the data used to fit the model.
- Results on validation or test data are typically lower than training results.
- Strong learning on the training set should come before expecting good generalization.

Rubric:
The response should refer to training examples as the data already seen, note that validation or test results are usually worse, and explain that strong training performance should come first.

Learn Before

Related