Google

When a model is part of a user-facing product, experience with the product can help estimate what level of error is acceptable for a strong user experience. For an important system one has worked on for years, experience can also help judge how much improvement is realistically achievable over the next quarter or year, and that judgment can guide the target error rate.

Using Experience to Set a Target Error Rate

What kind of reference is most useful for choosing a target error rate in a user-facing classifier?

A product team may care about the level of _____ the model must achieve to deliver a strong user experience.

Match each situation to the kind of intuition it uses when choosing a target error rate.

Order the steps for setting a target error rate in a hospital triage classifier.

Question:
In a few sentences, explain how a team’s sense of what users need for a great product experience can help set a desired error rate for a learning system.

Sample answer:
When a team knows what level of mistakes users will tolerate, that experience-based judgment can be translated into a desired error rate. For instance, if a photo-search feature can still feel smooth even when it occasionally returns the wrong result, the acceptable error rate may be higher than for a safety-critical system. If even a small number of errors would frustrate users, the target must be much stricter. In this way, the error rate is tied to the product experience the team wants to deliver.

Key points:
- A product serves users, so user experience matters.
- Intuition about acceptable performance gives a practical benchmark.
- That benchmark can be turned into a concrete desired error rate.
- The target should reflect what users need, not an arbitrary number.

Rubric:
Full credit for explaining that intuition about the user experience helps translate product needs into a concrete desired error rate.

How can product experience guide a target error rate?

Case context:
A company has maintained a medical transcription service for many years. The engineering team is now deciding what accuracy target to set for the next release cycle.

Question:
According to the underlying idea, how should the team choose the target, and what kind of judgment should guide it?

Sample answer:
The team should use its long history with this system to estimate what improvement is realistically attainable in the next cycle. Because they have seen the system’s behavior over many releases, they can make a practical judgment about likely progress, and that judgment should be used to choose the target accuracy.

Key points:
- Years of experience with one application can support realistic forecasting
- The team should estimate how much improvement is feasible in the coming cycle
- That estimate should shape the target they set
- This is about operational performance, not about user-facing product taste

Rubric:
Full credit for explaining that long-term experience with the system should be used to judge realistic near-term improvement, and that this judgment should determine the target accuracy.

Choosing a Quarterly Target from Years of Operational Experience

Question:
In one to three sentences, name the two kinds of intuition that can help a team choose a target error rate for a learning system.

Sample answer:
One source is intuition about the level of performance people need for the product to feel useful and satisfying. The other is intuition, developed from long work on the problem, about how much additional improvement is realistically achievable in the next few months or year.

Key points:
- Product or user-experience needs
- Realistic near-term improvement based on long-term experience

Rubric:
Full credit for identifying both sources: product-focused intuition and long-term improvement intuition.

Learn Before

Related