Google

Informally, variance is the amount by which a model performs worse on dev or test data than it does on its training data. In plain terms, it describes the performance gap between the training set and a held-out set.

Variance, Informally

What does variance mean in this informal comparison?

A larger performance gap between training data and validation data is an informal sign of variance.

Variance compares test performance to the ____ performance.

Match each term to its role in the informal variance idea.

Order the steps for judging whether a model has high variance

Question:
In a concise analytical response, explain why training performance by itself or held-out performance by itself does not fully capture the informal meaning of variance.

Sample answer:
Informal variance refers to how much worse a model performs on a validation or test set than it does on the training set. That means the concept is based on a comparison between two performance levels. If you look at only one result, you do not see the performance gap that the term is meant to describe.

Key points:
- Training performance provides the baseline.
- Validation or test performance is compared against that baseline.
- Variance is the amount of degradation from training to held-out data.
- One performance number alone does not show the gap.

Rubric:
A strong response identifies both parts of the comparison, explains that variance is the difference between them, and states why either result alone is incomplete.

Why does the informal idea of variance require a comparison between training and held-out results?

Case context:
A team trains a spam filter that scores better on its training emails than on its validation emails. The team wants to understand what that gap means.

Question:
What should the team compare, and what does the size of the drop indicate?

Sample answer:
The team should compare the model's validation performance with its training performance. The amount by which validation performance is worse than training performance is informally called the model's variance.

Key points:
- Compare validation performance against training performance.
- Focus on how much worse the validation result is.
- Treat that drop as an informal measure of variance.

Rubric:
The response should name both training and validation performance, keep the comparison direction correct, and identify the performance drop on validation as informal variance.

Interpret a gap between training and validation performance.

Question:
In an informal comparison of training and validation or test performance, what does it mean when one side is described as worse, and which result is used as the baseline?

Sample answer:
It means the validation or test performance is poorer than the training performance. Training performance is the reference point used to judge how much the model has degraded.

Key points:
- The worse result is usually on the validation or test set.
- Training performance is the comparison baseline.

Rubric:
Credit answers that identify validation or test as worse than training and state that training is the reference used for the comparison.

What does “worse” mean in an informal variance comparison?

An informal variance score is defined as training-set error minus test-set error.

In this document-classification example, training error is 3% and development error is 14%, an 11-percentage-point generalization gap. If the training and development sets come from the same distribution, this large gap is evidence of high variance, or overfitting: the model performs much better on its training examples than on unseen examples. If the development set has a different distribution, these two error rates alone cannot distinguish variance from data mismatch; that diagnosis requires comparison with a training-dev set drawn from the training distribution.

Learn Before

Related