Google

A random 70%/30% split can give misleading results if the data used for training and testing does not match the distribution the system will face after deployment.

Random Train/Test Splits Can Be Misleading When Deployment Data Differs

Why can a random 70/30 train/test split be misleading in deployed machine-learning systems?

True or False: A random 70/30 split always creates a test set that matches the deployment distribution.

Match each data situation to whether a random 70/30 split is a good choice.

Order the steps for deciding whether a random split is appropriate.

Question:
Write a short essay explaining why a 70%/30% random split can give a misleading evaluation when the data the model is trained on is not drawn from the same conditions where it will later be used.

Sample answer:
A random split divides only the examples that are already available, so it keeps the same mix of patterns, noise, and biases found in that pool of data. If the available examples come from one setting, but the model must work in a different setting, the test set will still reflect the first setting rather than the future one. For example, a model trained on product photos taken in a studio may be tested on a random holdout from those studio photos, even though the real job is to recognize items in dim, handheld snapshots from customers. In that case, the score on the holdout set can look good while saying little about how the model will behave in the customer setting. That is why the split is not a reliable check of real-world performance.

Key points:
- A random split mirrors the distribution of the data already collected
- The collected data can differ from the deployment environment
- Example: studio product photos versus handheld customer photos
- The holdout set may stay representative of the source pool, not the target use case
- The evaluation can therefore give an overly optimistic or otherwise misleading estimate of deployment performance

Rubric:
Full credit requires stating that a random split preserves the source-data distribution, identifying a concrete source-versus-target mismatch, and explaining that the resulting test score may not reflect performance in the environment where the model will actually be used.

Why a Random Split Can Mislead Under Distribution Shift

Case context:
A team is building a classifier that identifies product placements on store shelves. Their labeled training images mostly come from a public photo archive because those images are easy to download in large quantities. They randomly split those archive images 80%/20% into training and test sets.

Question:
What problem should the team identify with this split, and why?

Sample answer:
The split is likely misleading because both the training set and the test set come from the same archive-photo source, while the model will actually be used on photos taken by store employees on handheld devices. The archive images and the deployment images may differ in angle, lighting, clutter, and framing, so a strong score on the random split would not reliably predict real performance. The issue is a source mismatch between the evaluation data and the intended use environment.

Key points:
- Training and test images both come from the archive source
- Real use will involve handheld device photos from stores
- Random splitting does not address the source mismatch
- Test results may overstate real-world accuracy

Rubric:
Full credit identifies the mismatch between archive images and handheld store photos, and explains that the split is misleading because it does not reflect the deployment distribution.

Diagnose a training split for a retail shelf classifier.

Question:
In one to three sentences, explain when a random 80%/20% train/test split is a bad way to evaluate a system.

Sample answer:
A random split is a poor choice when the data the model is trained on does not match the data it will face after deployment. For example, a model trained on studio product photos may need to work on customer photos taken in very different lighting.

Key points:
- The problem is a mismatch between training and deployment data.
- Random splitting can hide that mismatch.
- The training and target distributions should be similar for the split to be reliable.

Rubric:
Full credit requires stating that the split fails when the training data distribution differs from the real-world use distribution; partial credit for mentioning a random split without naming the mismatch condition.

Learn Before

Related