Google

An optimizing metric is the main measure you use to compare candidate systems after any minimum requirements on other metrics have already been set. For example, in a loan-approval model, you might require the false positive rate to stay below a fixed limit and then choose the model with the highest recall among the remaining candidates.

Primary Optimization Metric

What is the optimizing metric in this fraud-screening example?

True or False: Satisficing criteria are usually set before choosing the final optimizing metric.

Match each metric role to its meaning in a mobile app ranking task.

Order the steps for choosing the optimization target in a mobile app example.

Question:
A team is building an email spam filter. They want inference latency to stay under 50 milliseconds, and among all models that satisfy that limit, they want the highest possible precision. Explain why precision is the optimizing metric and latency is the satisficing metric.

Sample answer:
Latency is a satisficing metric because the model only needs to stay within an acceptable limit; once it is under 50 milliseconds, the requirement is met. Precision is the optimizing metric because there is no fixed cutoff that merely counts as acceptable. Instead, the team wants to improve precision as much as possible while still meeting the latency constraint. That makes precision the quantity being optimized and latency the constraint being satisfied.

Key points:
- Latency has a pass/fail threshold
- Precision is improved continuously rather than checked against a minimum cutoff
- The goal is to maximize precision while keeping latency under the limit
- This makes precision the optimizing metric and latency the satisficing metric

Rubric:
Full credit requires explaining that latency must only meet the time limit, precision is the quantity to maximize, and the optimization is done subject to the latency constraint. Partial credit for identifying the roles without clearly describing the threshold-versus-maximization difference.

Distinguish an Optimizing Metric from a Satisficing Metric

Case context:
A team is building a classifier that must return a prediction within 120 milliseconds on average. That speed requirement has already been decided. The team now has one other important measure of success: accuracy.

Question:
After the time limit has been fixed, how should the team treat accuracy, and why?

Sample answer:
The team should make accuracy the optimizing metric and try to maximize it, as long as the classifier still satisfies the 120-millisecond time requirement. The speed requirement acts as a threshold that must be met, while accuracy is the metric to improve among solutions that meet that threshold.

Key points:
- The 120-millisecond limit is a satisficing requirement.
- Accuracy is the remaining metric to optimize.
- Accuracy should be maximized, not just checked against a cutoff.
- The model must still meet the speed requirement while improving accuracy.

Rubric:
Full credit requires identifying accuracy as the optimizing metric after the time constraint is fixed, and explaining that accuracy should be maximized subject to meeting the runtime threshold. Partial credit is available for noting that accuracy matters without stating the subject-to relationship.

Choose the Metric to Maximize After Setting a Time Limit

Question:
In a few words, explain what an optimizing metric is and name the metric used for the classifier example.

Sample answer:
An optimizing metric is the measure you try to make as good as possible after the essential pass/fail requirements have been set. In the classifier example, the optimizing metric is accuracy, because the model is chosen to maximize accuracy once the speed requirement is satisfied.

Key points:
- It is the metric used for choosing among acceptable options
- It is considered after required constraints are set
- Accuracy is the example optimizing metric in the classifier case

Rubric:
Full credit for defining it as the metric to maximize after required constraints are met and for naming accuracy in the example. Partial credit for an incomplete but directionally correct definition.

What is an optimizing metric in model selection?

When Does a Metric Become the Optimizing Metric?

For a mobile classifier that must answer within 150 milliseconds, inference speed is the metric to maximize and accuracy only has to clear a minimum cutoff.

Learn Before

Related