Google

To estimate human-level error on mobile cat images, ask people to label the same dataset and then measure how often they make mistakes.

Estimating Human Error on Mobile Cat Images

Measuring Human Error on Phone Photos

Estimating Human Performance on a Domain-Specific Image Task

Human-level performance on a retail shelf-image set can be measured by asking people to _____ the image set and then calculating their error rate.

Estimating Human Error on an Image Set

Parts of an Evaluation on the Target Dataset

Question:
Explain how to establish a human performance benchmark for a specific subset of data, such as delivery-room X-ray images.

Sample answer:
To measure human performance for one exact data set, you must use examples drawn from that same data distribution rather than from a different one. For example, if the benchmark is for delivery-room X-ray images, you should collect X-rays from that setting and have people label those images directly. After that, calculate the labelers’ error rate on that specific set. That error rate is the human benchmark for the target distribution.

Key points:
- Use examples from the exact target data set.
- Ask humans to do the same task on that data.
- Compute the error rate on that target set.

Rubric:
A complete response should state that the benchmark must be measured on the target distribution itself, that humans should perform the labeling task on those examples, and that the resulting error rate is the benchmark.

How to Measure Human Performance on the Target Data Set

Case context:
Your team has measured human-level performance on clear daytime warehouse images at 1%. The system will actually be used on night-time phone photos that are dark, blurry, and sometimes partially cropped. Your current model makes 6% error on this deployment data.

Question:
What should your team do to determine a fair benchmark for the night-time application data?

Sample answer:
The team should sample real night-time phone photos from the target distribution and have humans label them. By computing the human error rate on those specific images, they can establish the benchmark that matches the deployment setting.

Key points:
- Use images from the night-time phone-photo distribution.
- Have humans label those target-domain images.
- Measure human error on that distribution to set the benchmark.

Rubric:
The response must say that humans should label the target-domain data specifically and that this is how the benchmark error rate should be measured.

Choosing a Benchmark for Night-Time Phone Photos

Question:
What must people do with a collection of smartphone bird photos before their performance can be evaluated?

Sample answer:
People must annotate the photos with the correct labels.

Key points:
- The images need human annotation.
- The labels should identify the correct category for each image.

Rubric:
Award full credit for stating that humans must annotate, mark, or label the images.

Learn Before

Related