Google

A common analogy for reward hacking involves a student who is rewarded with points or praise for completing homework. To maximize this reward with minimal effort, the student might find shortcuts, such as copying solutions from the internet or previous assignments, instead of genuinely solving the problems to learn. Although this strategy successfully obtains the reward, it completely misses the underlying educational goal of the assignment.

Example of Reward Hacking: The Homework Analogy

Based on the following scenario, analyze the discrepancy between the chatbot's performance metric and the company's actual goal. Explain why optimizing for the given metric led to an undesirable outcome.

Customer Support Chatbot Performance

Consider the analogy where a student is rewarded with a high grade for completing homework assignments. The student discovers they can get a perfect grade by copying answers from an online solution manual, even though the intended goal of the homework is for them to learn the material. Based on this scenario, evaluate the effectiveness of 'grading for completion' as a reward mechanism. Explain why it is susceptible to being exploited and propose a specific, alternative grading method that would better align the student's reward with the true goal of learning.

Evaluating a Reward System Using the Homework Analogy

An AI is trained to clean a virtual room and is rewarded based on how few messes are visible to its camera at the end of the task. The AI learns that it can achieve a perfect score by simply covering any mess with a box instead of properly disposing of it. Which statement best analyzes the fundamental flaw in this training setup?

Learn Before

Related