Google

Development-set labels should be corrected when the mislabeled fraction is large enough to interfere with fast algorithm comparison and make it hard to tell whether one candidate model is better than another.

When to Repair Noisy Development Labels

When to Repair Validation Labels

Choose the word that best completes the sentence.

Noisy labels can ____ the comparison itself.

Match each development-set concept to its role in deciding whether labels should be fixed.

Order the reasoning for deciding whether a noisy development set should be corrected.

Question:
A team uses a dev set to compare two image classifiers and choose the better one. Explain why simply discovering mislabeled dev examples is not enough to justify spending time correcting them.

Sample answer:
A dev set is meant to support fast model comparison and selection. Because of that, the key issue is not whether any mistakes exist in the labels, but whether those mistakes are large enough to distort the comparison between models. If the mislabeled examples do not affect the decision about which classifier is better, fixing them may not be worth the effort. If they do interfere with that judgment, then correcting them becomes important because it restores the dev set's usefulness for choosing between models.

Key points:
- The dev set is used for quick model evaluation and selection.
- The presence of label errors alone does not determine whether they should be fixed.
- The deciding factor is whether the errors change or weaken the comparison between models.
- Label correction is worthwhile when the errors interfere with choosing the better algorithm.

Rubric:
A strong response states the purpose of the dev set, explains that the effect of the mislabels matters more than their mere existence, and concludes that correction is justified when the errors affect model comparison.

When should mislabeled dev-set examples be fixed?

Case context:
A team is comparing Model A and Model B using a held-out validation set. They discover that some validation examples are labeled incorrectly, and those mistakes are making it hard to trust the comparison and slowing down the review process.

Question:
Based only on the decision rule in the lesson, what should the team do, and why?

Sample answer:
The team should correct the mislabeled validation examples. The labeling errors are interfering with both the speed and reliability of the A-versus-B comparison, which is the situation in which fixing the labels is justified.

Key points:
- The validation set is being used to choose between two models.
- Wrong labels are making the comparison unreliable.
- The evaluation process has become slower or less decisive.
- Under these conditions, the labels should be fixed.

Rubric:
The response should recommend fixing the labels and explicitly connect that choice to the fact that the model comparison is no longer fast or dependable.

Decide When Incorrect Validation Labels Should Be Fixed

Question:
In one to three sentences, explain the condition under which correcting mislabeled validation labels is worth the effort.

Sample answer:
It is worth correcting them when the label errors are large enough to make validation checks slow or unreliable. In particular, if the mistakes can change which model appears better, they should be fixed.

Key points:
- The errors must interfere with the validation set's purpose.
- That purpose includes fast evaluation and comparing models.

Rubric:
The answer must identify impaired evaluation or model comparison as the reason to correct the labels.

When is it worth fixing mislabeled validation examples?

When is cleaning dev-set labels worth the effort?

Every mislabeled example in the validation set must be corrected before you can use the set for model comparison.

Fixing validation labels one by one is always required.

When a model is still making many mistakes, a few incorrect labels in the development or test set may have little effect on the overall error picture. As the model gets better and its true error rate drops, those same mislabeled examples can account for a larger share of the observed errors. At that point, improving the label quality of the development set can become worth the effort.

Learn Before

Related