If the development set is easier than the test set because the examples come from different sources, improving development accuracy may fail to improve test performance.
0
1
Tags
Machine Learning
Deep Learning
Supervised Learning
Dive into Deep Learning @ D2L
Data Science
Machine Learning Strategy
Machine Learning Yearning @ DeepLearning.AI
Related
Focus on the Scoring Model When Most Errors Come from It
When Is a Dev-Set Error Attributed to Optimization?
True or False: If the best achievable development-set score is already no better than the current system's score, the optimization algorithm is the main problem.
A development-set example is counted as a mistake by the _____ metric when it lowers the score used to tune the model.
Which condition indicates that a wrong development-set example is an optimization problem rather than a scoring problem?
When the evaluation score prefers the wrong output, the dev-set issue is a _____ error.
Match each verification expression to its meaning.
Order the steps for deciding whether each development-set mistake points to the optimizer or the scoring rule.
What does Score_A(S*) ≤ Score_A(S_out) indicate in dev set analysis?
A troubleshooting method can start by analyzing one example and then be extended to every error in the development set.
Meaning of T* in an Optimization Check
Match each diagnosis label to its role in separating search problems from scoring problems.
Put the validation analysis steps in the correct order after reviewing example-level errors.
How can you sort validation mistakes into model and metric problems?
Using a Development-Set Check to Separate Search Problems from Metric Problems
When does a verification test point to a scoring problem?