Google

When a model performs poorly on the training set, a practical first step is to review about 100 training examples where it makes the largest mistakes. Group those cases into a few error categories to see which kinds of failures are most common.

Sample the Worst Training Cases to Find Error Patterns

What should you do first when a model is making many mistakes on the training set?

True or False: A useful way to inspect training mistakes is to group them into categories and count how often each type appears, similar to dev-set error analysis.

How many bad training examples should you inspect first?

Match Each Training-Review Idea to Its Description

Put the training-set review process in the correct order.

Question:
Explain why looking through roughly 100 training examples that a model handles badly can reveal the main kinds of training-set mistakes, and how this resembles error analysis on a development set.

Sample answer:
Reviewing about 100 low-performing training examples gives a direct view of what is going wrong, instead of only seeing a single overall loss or accuracy number. By sorting those cases into a few error categories, a practitioner can see which problems occur most often and which data issues deserve attention first. This is the same basic idea used in development-set error analysis: classify mistakes by type, count how often each type appears, and use those counts to decide what to fix.

Key points:
- Inspect roughly 100 examples the model gets wrong
- Look for recurring categories of training-set errors
- Count examples in each category to find the biggest problems
- The method is analogous to dev-set error analysis

Rubric:
Full credit requires mentioning: (1) reviewing about 100 examples, (2) identifying major categories of errors, and (3) the link to counting errors in a way similar to dev-set analysis.

Why inspecting a small set of bad training cases can reveal training-set problems

Case context:
A team has built an automatic speech recognition model, and it is making many mistakes even on the examples it was trained on, not just on new evaluation data.

Question:
What should the team do to figure out why the model is struggling on the training data, and what should they generate from that review?

Sample answer:
They should examine about 100 training examples that the model handles badly and inspect them by hand. As they review them, they should sort the mistakes into categories and count how often each category appears. The result should be a list of the main types of training-set errors, which helps the team decide what problem to address first.

Key points:
- Inspect roughly 100 badly handled training examples
- Group the mistakes into categories
- Count how often each category appears
- Use the summary to prioritize fixes, similar to development-set error analysis

Rubric:
Full credit for identifying the review of about 100 bad training examples, the categorization and counting of errors, and the comparison to development-set error analysis.

Diagnosing Poor Training Performance in a Speech Model

Question:
In one to three sentences, explain the recommended next step when a model is not fitting the training data well.

Sample answer:
Inspect about 100 examples the model handles badly, group the mistakes into a few categories, and count how often each category appears.

Key points:
- Review roughly 100 weak training examples
- Classify and tally the error types

Rubric:
Full credit for stating that about 100 bad examples should be reviewed and that the errors should be grouped and counted by category.

What should you do when training performance is poor?

What is the main purpose of reviewing training examples that the model handles poorly?

When reviewing poorly performing examples from a speech-to-text training set of volunteer recordings, an error-analysis worksheet can label each problematic clip by observable issues such as loud background noise, speech that is too fast, or a speaker being too far from the microphone. Tallying these labels shows which failure patterns recur most often.

Learn Before

Related